Amazon S3 plugin for Fluentd 🔗︎
Overview 🔗︎
s3 output plugin buffers event logs in local file and upload it to S3 periodically. This plugin splits files exactly by using the time of event logs (not the time when the logs are received). For example, a log ‘2011-01-02 message B’ is reached, and then another log ‘2011-01-03 message B’ is reached in this order, the former one is stored in “20110102.gz” file, and latter one in “20110103.gz” file.
Example: S3 Output Deployment
Example output configurations 🔗︎
spec:
s3:
aws_key_id:
valueFrom:
secretKeyRef:
name: logging-s3
key: awsAccessKeyId
aws_sec_key:
valueFrom:
secretKeyRef:
name: logging-s3
key: awsSecretAccessKey
s3_bucket: logging-amazon-s3
s3_region: eu-central-1
path: logs/${tag}/%Y/%m/%d/
buffer:
timekey: 10m
timekey_wait: 30s
timekey_use_utc: true
Configuration 🔗︎
Output Config 🔗︎
aws_key_id (*secret.Secret, optional) {#output config-aws_key_id} 🔗︎
AWS access key id Secret
Default: -
aws_sec_key (*secret.Secret, optional) {#output config-aws_sec_key} 🔗︎
AWS secret key. Secret
Default: -
check_apikey_on_start (string, optional) {#output config-check_apikey_on_start} 🔗︎
Check AWS key on start
Default: -
grant_read (string, optional) {#output config-grant_read} 🔗︎
Allows grantee to read the object data and its metadata
Default: -
overwrite (string, optional) {#output config-overwrite} 🔗︎
Overwrite already existing path
Default: -
path (string, optional) {#output config-path} 🔗︎
Path prefix of the files on S3
Default: -
grant_write_acp (string, optional) {#output config-grant_write_acp} 🔗︎
Allows grantee to write the ACL for the applicable object
Default: -
check_bucket (string, optional) {#output config-check_bucket} 🔗︎
Check bucket if exists or not
Default: -
sse_customer_key (string, optional) {#output config-sse_customer_key} 🔗︎
Specifies the customer-provided encryption key for Amazon S3 to use in encrypting data
Default: -
sse_customer_key_md5 (string, optional) {#output config-sse_customer_key_md5} 🔗︎
Specifies the 128-bit MD5 digest of the encryption key according to RFC 1321
Default: -
compute_checksums (string, optional) {#output config-compute_checksums} 🔗︎
AWS SDK uses MD5 for API request/response by default
Default: -
warn_for_delay (string, optional) {#output config-warn_for_delay} 🔗︎
Given a threshold to treat events as delay, output warning logs if delayed events were put into s3
Default: -
use_bundled_cert (string, optional) {#output config-use_bundled_cert} 🔗︎
Use aws-sdk-ruby bundled cert
Default: -
s3_endpoint (string, optional) {#output config-s3_endpoint} 🔗︎
Custom S3 endpoint (like minio)
Default: -
ssekms_key_id (string, optional) {#output config-ssekms_key_id} 🔗︎
Specifies the AWS KMS key ID to use for object encryption
Default: -
s3_metadata (string, optional) {#output config-s3_metadata} 🔗︎
Arbitrary S3 metadata headers to set for the object
Default: -
force_path_style (string, optional) {#output config-force_path_style} 🔗︎
If true, the bucket name is always left in the request URI and never moved to the host as a sub-domain
Default: -
auto_create_bucket (string, optional) {#output config-auto_create_bucket} 🔗︎
Create S3 bucket if it does not exists
Default: -
index_format (string, optional) {#output config-index_format} 🔗︎
sprintf format for %{index}
Default: -
signature_version (string, optional) {#output config-signature_version} 🔗︎
Signature version for API Request (s3,v4)
Default: -
enable_transfer_acceleration (string, optional) {#output config-enable_transfer_acceleration} 🔗︎
If true, S3 Transfer Acceleration will be enabled for uploads. IMPORTANT: You must first enable this feature on your destination S3 bucket
Default: -
ssl_verify_peer (string, optional) {#output config-ssl_verify_peer} 🔗︎
If false, the certificate of endpoint will not be verified
Default: -
proxy_uri (string, optional) {#output config-proxy_uri} 🔗︎
URI of proxy environment
Default: -
grant_read_acp (string, optional) {#output config-grant_read_acp} 🔗︎
Allows grantee to read the object ACL
Default: -
check_object (string, optional) {#output config-check_object} 🔗︎
Check object before creation
Default: -
sse_customer_algorithm (string, optional) {#output config-sse_customer_algorithm} 🔗︎
Specifies the algorithm to use to when encrypting the object
Default: -
use_server_side_encryption (string, optional) {#output config-use_server_side_encryption} 🔗︎
The Server-side encryption algorithm used when storing this object in S3 (AES256, aws:kms)
Default: -
s3_region (string, optional) {#output config-s3_region} 🔗︎
S3 region name
Default: -
acl (string, optional) {#output config-acl} 🔗︎
Permission for the object in S3
Default: -
grant_full_control (string, optional) {#output config-grant_full_control} 🔗︎
Allows grantee READ, READ_ACP, and WRITE_ACP permissions on the object
Default: -
hex_random_length (string, optional) {#output config-hex_random_length} 🔗︎
The length of %{hex_random} placeholder(4-16)
Default: -
s3_object_key_format (string, optional) {#output config-s3_object_key_format} 🔗︎
The format of S3 object keys (default: %{path}%{time_slice}%{uuid_hash}%{index}.%{file_extension})
Default: %{path}%{time_slice}%{uuid_hash}%{index}.%{file_extension}
s3_bucket (string, required) {#output config-s3_bucket} 🔗︎
S3 bucket name
Default: -
store_as (string, optional) {#output config-store_as} 🔗︎
Archive format on S3
Default: -
storage_class (string, optional) {#output config-storage_class} 🔗︎
The type of storage to use for the object, for example STANDARD, REDUCED_REDUNDANCY, STANDARD_IA, ONEZONE_IA, INTELLIGENT_TIERING, GLACIER, DEEP_ARCHIVE, OUTPOSTS, GLACIER_IR For a complete list of possible values, see the Amazon S3 API reference.
Default: -
aws_iam_retries (string, optional) {#output config-aws_iam_retries} 🔗︎
The number of attempts to load instance profile credentials from the EC2 metadata service using IAM role
Default: -
buffer (*Buffer, optional) {#output config-buffer} 🔗︎
Default: -
slow_flush_log_threshold (string, optional) {#output config-slow_flush_log_threshold} 🔗︎
The threshold for chunk flush performance check. Parameter type is float, not time, default: 20.0 (seconds) If chunk flush takes longer time than this threshold, fluentd logs warning message and increases metric fluentd_output_status_slow_flush_count.
Default: -
format (*Format, optional) {#output config-format} 🔗︎
Default: -
assume_role_credentials (*S3AssumeRoleCredentials, optional) {#output config-assume_role_credentials} 🔗︎
Default: -
instance_profile_credentials (*S3InstanceProfileCredentials, optional) {#output config-instance_profile_credentials} 🔗︎
Default: -
shared_credentials (*S3SharedCredentials, optional) {#output config-shared_credentials} 🔗︎
Default: -
compress (*Compress, optional) {#output config-compress} 🔗︎
Parquet compressor
Default: -
oneeye_format (bool, optional) {#output config-oneeye_format} 🔗︎
One-eye format trigger
Default: false
clustername (string, optional) {#output config-clustername} 🔗︎
Custom cluster name
Default: one-eye
Assume Role Credentials 🔗︎
assume_role_credentials
role_arn (string, required) {#assume role credentials-role_arn} 🔗︎
The Amazon Resource Name (ARN) of the role to assume
Default: -
role_session_name (string, required) {#assume role credentials-role_session_name} 🔗︎
An identifier for the assumed role session
Default: -
policy (string, optional) {#assume role credentials-policy} 🔗︎
An IAM policy in JSON format
Default: -
duration_seconds (string, optional) {#assume role credentials-duration_seconds} 🔗︎
The duration, in seconds, of the role session (900-3600)
Default: -
external_id (string, optional) {#assume role credentials-external_id} 🔗︎
A unique identifier that is used by third parties when assuming roles in their customers’ accounts.
Default: -
Instance Profile Credentials 🔗︎
instance_profile_credentials
ip_address (string, optional) {#instance profile credentials-ip_address} 🔗︎
IP address
Default: 169.254.169.254
port (string, optional) {#instance profile credentials-port} 🔗︎
Port number
Default: 80
http_open_timeout (string, optional) {#instance profile credentials-http_open_timeout} 🔗︎
Number of seconds to wait for the connection to open
Default: -
http_read_timeout (string, optional) {#instance profile credentials-http_read_timeout} 🔗︎
Number of seconds to wait for one block to be read
Default: -
retries (string, optional) {#instance profile credentials-retries} 🔗︎
Number of times to retry when retrieving credentials
Default: -
Shared Credentials 🔗︎
shared_credentials
profile_name (string, optional) {#shared credentials-profile_name} 🔗︎
Profile name. Default to ‘default’ or ENV[‘AWS_PROFILE’]
Default: -
path (string, optional) {#shared credentials-path} 🔗︎
Path to the shared file.
Default: $HOME/.aws/credentials
Parquet compressor 🔗︎
parquet compressor
parquet_compression_codec (string, optional) {#parquet compressor-parquet_compression_codec} 🔗︎
Parquet compression codec. (uncompressed, snappy, gzip, lzo, brotli, lz4, zstd)
Default: snappy
parquet_page_size (string, optional) {#parquet compressor-parquet_page_size} 🔗︎
Parquet file page size.
Default: 8192 bytes
parquet_row_group_size (string, optional) {#parquet compressor-parquet_row_group_size} 🔗︎
Parquet file row group size.
Default: 128 MB
record_type (string, optional) {#parquet compressor-record_type} 🔗︎
Record data format type. (avro csv jsonl msgpack tsv msgpack json)
Default: msgpack
schema_type (string, optional) {#parquet compressor-schema_type} 🔗︎
Schema type. (avro, bigquery)
Default: avro
schema_file (string, optional) {#parquet compressor-schema_file} 🔗︎
Path to schema file.
Default: -
