Aliyun OSS plugin for Fluentd 🔗︎

Overview 🔗︎

Fluent OSS output plugin buffers event logs in local files and uploads them to OSS periodically in background threads.

This plugin splits events by using the timestamp of event logs. For example, a log ‘2019-04-09 message Hello’ is reached, and then another log ‘2019-04-10 message World’ is reached in this order, the former is stored in “20190409.gz” file, and latter in “20190410.gz” file.

Fluent OSS input plugin reads data from OSS periodically.

This plugin uses MNS on the same region of the OSS bucket. We must setup MNS and OSS event notification before using this plugin.

This document shows how to setup MNS and OSS event notification.

This plugin will poll events from MNS queue and extract object keys from these events, and then will read those objects from OSS. More info at https://github.com/aliyun/fluent-plugin-oss

Configuration 🔗︎

Output Config 🔗︎

Variable Name Type Required Default Description
endpoint string Yes - OSS endpoint to connect to'
bucket string Yes - Your bucket name
access_key_id *secret.Secret Yes - Your access key id
Secret
aaccess_key_secret *secret.Secret Yes - Your access secret key
Secret
path string No fluent/logs Path prefix of the files on OSS
upload_crc_enable bool No true Upload crc enabled
download_crc_enable bool No true Download crc enabled
open_timeout int No 10 Timeout for open connections
read_timeout int No 120 Timeout for read response
oss_sdk_log_dir string No /var/log/td-agent OSS SDK log directory
key_format string No %{path}/%{time_slice}%{index}%{thread_id}.%{file_extension} The format of OSS object keys
store_as string No gzip Archive format on OSS: gzip, json, text, lzo, lzma2
auto_create_bucket bool No false desc ‘Create OSS bucket if it does not exists
overwrite bool No false Overwrite already existing path
check_bucket bool No true Check bucket if exists or not
check_object bool No true Check object before creation
hex_random_length int No 4 The length of %{hex_random} placeholder(4-16)
index_format string No %d sprintf format for %{index}
warn_for_delay string No - Given a threshold to treat events as delay, output warning logs if delayed events were put into OSS
format *Format No - Format
buffer *Buffer No - Buffer