Kafka output plugin for Fluentd 🔗︎
Overview 🔗︎
More info at https://github.com/fluent/fluent-plugin-kafka
Example Deployment: Transport Nginx Access Logs into Kafka with Logging Operator
Example output configurations 🔗︎
spec:
kafka:
brokers: kafka-headless.kafka.svc.cluster.local:29092
default_topic: topic
sasl_over_ssl: false
format:
type: json
buffer:
tags: topic
timekey: 1m
timekey_wait: 30s
timekey_use_utc: true
Configuration 🔗︎
Kafka 🔗︎
Send your logs to Kafka 🔗︎
| Variable Name | Type | Required | Default | Description |
|---|---|---|---|---|
| brokers | string | Yes | - | The list of all seed brokers, with their host and port information. |
| topic_key | string | No | “topic” | Topic Key |
| partition_key | string | No | “partition” | Partition |
| partition_key_key | string | No | “partition_key” | Partition Key |
| message_key_key | string | No | “message_key” | Message Key |
| default_topic | string | No | nil | The name of default topic . |
| default_partition_key | string | No | nil | The name of default partition key . |
| default_message_key | string | No | nil | The name of default message key . |
| exclude_topic_key | bool | No | false | Exclude Topic key |
| exclude_partion_key | bool | No | false | Exclude Partition key |
| get_kafka_client_log | bool | No | false | Get Kafka Client log |
| headers | map[string]string | No | {} | Headers |
| headers_from_record | map[string]string | No | {} | Headers from Record |
| use_default_for_unknown_topic | bool | No | false | Use default for unknown topics |
| idempotent | bool | No | false | Idempotent |
| sasl_over_ssl | bool | Yes | true | SASL over SSL |
| username | *secret.Secret | No | - | Username when using PLAIN/SCRAM SASL authentication |
| password | *secret.Secret | No | - | Password when using PLAIN/SCRAM SASL authentication |
| scram_mechanism | string | No | - | If set, use SCRAM authentication with specified mechanism. When unset, default to PLAIN authentication |
| max_send_retries | int | No | 1 | Number of times to retry sending of messages to a leader |
| required_acks | int | No | -1 | The number of acks required per request . |
| ack_timeout | int | No | nil => Uses default of ruby-kafka library | How long the producer waits for acks. The unit is seconds |
| compression_codec | string | No | nil | The codec the producer uses to compress messages . The available options are gzip and snappy. |
| ssl_ca_certs_from_system | *bool | No | false | System’s CA cert store |
| ssl_ca_cert | *secret.Secret | No | - | CA certificate |
| ssl_client_cert | *secret.Secret | No | - | Client certificate |
| ssl_client_cert_chain | *secret.Secret | No | - | Client certificate chain |
| ssl_client_cert_key | *secret.Secret | No | - | Client certificate key |
| ssl_verify_hostname | *bool | No | - | Verify certificate hostname |
| format | *Format | Yes | - | Format |
| buffer | *Buffer | No | - | Buffer |
