Amazon S3 plugin for Fluentd 🔗︎

Overview 🔗︎

s3 output plugin buffers event logs in local file and upload it to S3 periodically. This plugin splits files exactly by using the time of event logs (not the time when the logs are received). For example, a log ‘2011-01-02 message B’ is reached, and then another log ‘2011-01-03 message B’ is reached in this order, the former one is stored in “20110102.gz” file, and latter one in “20110103.gz” file.

Example: S3 Output Deployment

Example output configurations 🔗︎

spec:
 s3:
   aws_key_id:
     valueFrom:
       secretKeyRef:
         name: logging-s3
         key: awsAccessKeyId
   aws_sec_key:
     valueFrom:
       secretKeyRef:
         name: logging-s3
         key: awsSecretAccessKey
   s3_bucket: logging-amazon-s3
   s3_region: eu-central-1
   path: logs/${tag}/%Y/%m/%d/
   buffer:
     timekey: 10m
     timekey_wait: 30s
     timekey_use_utc: true

Configuration 🔗︎

Output Config 🔗︎

aws_key_id (*secret.Secret, optional) 🔗︎

AWS access key id Secret

Default: -

aws_sec_key (*secret.Secret, optional) 🔗︎

AWS secret key. Secret

Default: -

check_apikey_on_start (string, optional) 🔗︎

Check AWS key on start

Default: -

grant_read (string, optional) 🔗︎

Allows grantee to read the object data and its metadata

Default: -

overwrite (string, optional) 🔗︎

Overwrite already existing path

Default: -

path (string, optional) 🔗︎

Path prefix of the files on S3

Default: -

grant_write_acp (string, optional) 🔗︎

Allows grantee to write the ACL for the applicable object

Default: -

check_bucket (string, optional) 🔗︎

Check bucket if exists or not

Default: -

sse_customer_key (string, optional) 🔗︎

Specifies the customer-provided encryption key for Amazon S3 to use in encrypting data

Default: -

sse_customer_key_md5 (string, optional) 🔗︎

Specifies the 128-bit MD5 digest of the encryption key according to RFC 1321

Default: -

compute_checksums (string, optional) 🔗︎

AWS SDK uses MD5 for API request/response by default

Default: -

warn_for_delay (string, optional) 🔗︎

Given a threshold to treat events as delay, output warning logs if delayed events were put into s3

Default: -

use_bundled_cert (string, optional) 🔗︎

Use aws-sdk-ruby bundled cert

Default: -

s3_endpoint (string, optional) 🔗︎

Custom S3 endpoint (like minio)

Default: -

ssekms_key_id (string, optional) 🔗︎

Specifies the AWS KMS key ID to use for object encryption

Default: -

s3_metadata (string, optional) 🔗︎

Arbitrary S3 metadata headers to set for the object

Default: -

force_path_style (string, optional) 🔗︎

If true, the bucket name is always left in the request URI and never moved to the host as a sub-domain

Default: -

auto_create_bucket (string, optional) 🔗︎

Create S3 bucket if it does not exists

Default: -

index_format (string, optional) 🔗︎

sprintf format for %{index}

Default: -

signature_version (string, optional) 🔗︎

Signature version for API Request (s3,v4)

Default: -

enable_transfer_acceleration (string, optional) 🔗︎

If true, S3 Transfer Acceleration will be enabled for uploads. IMPORTANT: You must first enable this feature on your destination S3 bucket

Default: -

ssl_verify_peer (string, optional) 🔗︎

If false, the certificate of endpoint will not be verified

Default: -

proxy_uri (string, optional) 🔗︎

URI of proxy environment

Default: -

grant_read_acp (string, optional) 🔗︎

Allows grantee to read the object ACL

Default: -

check_object (string, optional) 🔗︎

Check object before creation

Default: -

sse_customer_algorithm (string, optional) 🔗︎

Specifies the algorithm to use to when encrypting the object

Default: -

use_server_side_encryption (string, optional) 🔗︎

The Server-side encryption algorithm used when storing this object in S3 (AES256, aws:kms)

Default: -

s3_region (string, optional) 🔗︎

S3 region name

Default: -

acl (string, optional) 🔗︎

Permission for the object in S3

Default: -

grant_full_control (string, optional) 🔗︎

Allows grantee READ, READ_ACP, and WRITE_ACP permissions on the object

Default: -

hex_random_length (string, optional) 🔗︎

The length of %{hex_random} placeholder(4-16)

Default: -

s3_object_key_format (string, optional) 🔗︎

The format of S3 object keys (default: %{path}%{time_slice}%{uuid_hash}%{index}.%{file_extension})

Default: %{path}%{time_slice}%{uuid_hash}%{index}.%{file_extension}

s3_bucket (string, required) 🔗︎

S3 bucket name

Default: -

store_as (string, optional) 🔗︎

Archive format on S3

Default: -

storage_class (string, optional) 🔗︎

The type of storage to use for the object, for example STANDARD, REDUCED_REDUNDANCY, STANDARD_IA, ONEZONE_IA, INTELLIGENT_TIERING, GLACIER, DEEP_ARCHIVE, OUTPOSTS, GLACIER_IR For a complete list of possible values, see the Amazon S3 API reference.

Default: -

aws_iam_retries (string, optional) 🔗︎

The number of attempts to load instance profile credentials from the EC2 metadata service using IAM role

Default: -

buffer (*Buffer, optional) 🔗︎

Buffer

Default: -

format (*Format, optional) 🔗︎

Format

Default: -

assume_role_credentials (*S3AssumeRoleCredentials, optional) 🔗︎

Assume Role Credentials

Default: -

instance_profile_credentials (*S3InstanceProfileCredentials, optional) 🔗︎

Instance Profile Credentials

Default: -

shared_credentials (*S3SharedCredentials, optional) 🔗︎

Shared Credentials

Default: -

compress (*Compress, optional) 🔗︎

Parquet compressor

Default: -

oneeye_format (bool, optional) 🔗︎

One-eye format trigger

Default: false

clustername (string, optional) 🔗︎

Custom cluster name

Default: one-eye

Assume Role Credentials 🔗︎

assume_role_credentials

role_arn (string, required) {#assume role credentials-role_arn} 🔗︎

The Amazon Resource Name (ARN) of the role to assume

Default: -

role_session_name (string, required) {#assume role credentials-role_session_name} 🔗︎

An identifier for the assumed role session

Default: -

policy (string, optional) {#assume role credentials-policy} 🔗︎

An IAM policy in JSON format

Default: -

duration_seconds (string, optional) {#assume role credentials-duration_seconds} 🔗︎

The duration, in seconds, of the role session (900-3600)

Default: -

external_id (string, optional) {#assume role credentials-external_id} 🔗︎

A unique identifier that is used by third parties when assuming roles in their customers’ accounts.

Default: -

Instance Profile Credentials 🔗︎

instance_profile_credentials

ip_address (string, optional) {#instance profile credentials-ip_address} 🔗︎

IP address

Default: 169.254.169.254

port (string, optional) {#instance profile credentials-port} 🔗︎

Port number

Default: 80

http_open_timeout (string, optional) {#instance profile credentials-http_open_timeout} 🔗︎

Number of seconds to wait for the connection to open

Default: -

http_read_timeout (string, optional) {#instance profile credentials-http_read_timeout} 🔗︎

Number of seconds to wait for one block to be read

Default: -

retries (string, optional) {#instance profile credentials-retries} 🔗︎

Number of times to retry when retrieving credentials

Default: -

Shared Credentials 🔗︎

shared_credentials

profile_name (string, optional) {#shared credentials-profile_name} 🔗︎

Profile name. Default to ‘default’ or ENV[‘AWS_PROFILE’]

Default: -

path (string, optional) {#shared credentials-path} 🔗︎

Path to the shared file.

Default: $HOME/.aws/credentials

Parquet compressor 🔗︎

parquet compressor

parquet_compression_codec (string, optional) {#parquet compressor-parquet_compression_codec} 🔗︎

Parquet compression codec. (uncompressed, snappy, gzip, lzo, brotli, lz4, zstd)

Default: snappy

parquet_page_size (string, optional) {#parquet compressor-parquet_page_size} 🔗︎

Parquet file page size.

Default: 8192 bytes

parquet_row_group_size (string, optional) {#parquet compressor-parquet_row_group_size} 🔗︎

Parquet file row group size.

Default: 128 MB

record_type (string, optional) {#parquet compressor-record_type} 🔗︎

Record data format type. (avro csv jsonl msgpack tsv msgpack json)

Default: msgpack

schema_type (string, optional) {#parquet compressor-schema_type} 🔗︎

Schema type. (avro, bigquery)

Default: avro

schema_file (string, optional) {#parquet compressor-schema_file} 🔗︎

Path to schema file.

Default: -