Parser Filter 🔗︎
Overview 🔗︎
Parses a string field in event records and mutates its event record with the parsed result.
Configuration 🔗︎
ParserConfig 🔗︎
key_name (string, optional) 🔗︎
Specify field name in the record to parse. If you leave empty the Container Runtime default will be used.
Default: -
reserve_time (bool, optional) 🔗︎
Keep original event time in parsed result.
Default: -
reserve_data (bool, optional) 🔗︎
Keep original key-value pair in parsed result.
Default: -
remove_key_name_field (bool, optional) 🔗︎
Remove key_name field when parsing is succeeded
Default: -
replace_invalid_sequence (bool, optional) 🔗︎
If true, invalid string is replaced with safe characters and re-parse it.
Default: -
inject_key_prefix (string, optional) 🔗︎
Store parsed values with specified key name prefix.
Default: -
hash_value_field (string, optional) 🔗︎
Store parsed values as a hash value in a field.
Default: -
emit_invalid_record_to_error (*bool, optional) 🔗︎
Emit invalid record to @ERROR label. Invalid cases are: key not exist, format is not matched, unexpected error
Default: -
parse (ParseSection, optional) 🔗︎
Default: -
parsers ([]ParseSection, optional) 🔗︎
Deprecated, use parse instead
Default: -
Parse Section 🔗︎
type (string, optional) {#parse section-type} 🔗︎
Parse type: apache2, apache_error, nginx, syslog, csv, tsv, ltsv, json, multiline, none, logfmt, grok, multiline_grok
Default: -
expression (string, optional) {#parse section-expression} 🔗︎
Regexp expression to evaluate
Default: -
time_key (string, optional) {#parse section-time_key} 🔗︎
Specify time field for event time. If the event doesn’t have this field, current time is used.
Default: -
keys (string, optional) {#parse section-keys} 🔗︎
Names for fields on each line. (seperated by coma)
Default: -
null_value_pattern (string, optional) {#parse section-null_value_pattern} 🔗︎
Specify null value pattern.
Default: -
null_empty_string (bool, optional) {#parse section-null_empty_string} 🔗︎
If true, empty string field is replaced with nil
Default: -
estimate_current_event (bool, optional) {#parse section-estimate_current_event} 🔗︎
If true, use Fluent::EventTime.now(current time) as a timestamp when time_key is specified.
Default: -
keep_time_key (bool, optional) {#parse section-keep_time_key} 🔗︎
If true, keep time field in the record.
Default: -
types (string, optional) {#parse section-types} 🔗︎
Types casting the fields to proper types example: field1:type, field2:type
Default: -
time_format (string, optional) {#parse section-time_format} 🔗︎
Process value using specified format. This is available only when time_type is string
Default: -
time_type (string, optional) {#parse section-time_type} 🔗︎
Parse/format value according to this type available values: float, unixtime, string
Default: string
local_time (bool, optional) {#parse section-local_time} 🔗︎
Ff true, use local time. Otherwise, UTC is used. This is exclusive with utc.
Default: true
utc (bool, optional) {#parse section-utc} 🔗︎
If true, use UTC. Otherwise, local time is used. This is exclusive with localtime
Default: false
timezone (string, optional) {#parse section-timezone} 🔗︎
Use specified timezone. one can parse/format the time value in the specified timezone.
Default: nil
format (string, optional) {#parse section-format} 🔗︎
Only available when using type: multi_format
Default: -
format_firstline (string, optional) {#parse section-format_firstline} 🔗︎
Only available when using type: multi_format
Default: -
delimiter (string, optional) {#parse section-delimiter} 🔗︎
Only available when using type: ltsv
Default: “\t”
delimiter_pattern (string, optional) {#parse section-delimiter_pattern} 🔗︎
Only available when using type: ltsv
Default: -
label_delimiter (string, optional) {#parse section-label_delimiter} 🔗︎
Only available when using type: ltsv
Default: “:”
multiline ([]string, optional) {#parse section-multiline} 🔗︎
The multiline parser plugin parses multiline logs.
Default: -
patterns ([]SingleParseSection, optional) {#parse section-patterns} 🔗︎
Only available when using type: multi_format Parse Section
Default: -
grok_pattern (string, optional) {#parse section-grok_pattern} 🔗︎
Only available when using type: grok, multiline_grok. The pattern of grok. You cannot specify multiple grok pattern with this.
Default: -
custom_pattern_path (*secret.Secret, optional) {#parse section-custom_pattern_path} 🔗︎
Only available when using type: grok, multiline_grok. File that includes custom grok patterns.
Default: -
grok_failure_key (string, optional) {#parse section-grok_failure_key} 🔗︎
Only available when using type: grok, multiline_grok. The key has grok failure reason.
Default: -
grok_name_key (string, optional) {#parse section-grok_name_key} 🔗︎
Only available when using type: grok, multiline_grok. The key name to store grok section’s name.
Default: -
multiline_start_regexp (string, optional) {#parse section-multiline_start_regexp} 🔗︎
Only available when using type: multiline_grok The regexp to match beginning of multiline.
Default: -
grok_patterns ([]GrokSection, optional) {#parse section-grok_patterns} 🔗︎
Only available when using type: grok, multiline_grok. Grok Section Specify grok pattern series set.
Default: -
Parse Section (single) 🔗︎
type (string, optional) {#parse section (single)-type} 🔗︎
Parse type: apache2, apache_error, nginx, syslog, csv, tsv, ltsv, json, multiline, none, logfmt, grok, multiline_grok
Default: -
expression (string, optional) {#parse section (single)-expression} 🔗︎
Regexp expression to evaluate
Default: -
time_key (string, optional) {#parse section (single)-time_key} 🔗︎
Specify time field for event time. If the event doesn’t have this field, current time is used.
Default: -
null_value_pattern (string, optional) {#parse section (single)-null_value_pattern} 🔗︎
Specify null value pattern.
Default: -
null_empty_string (bool, optional) {#parse section (single)-null_empty_string} 🔗︎
If true, empty string field is replaced with nil
Default: -
estimate_current_event (bool, optional) {#parse section (single)-estimate_current_event} 🔗︎
If true, use Fluent::EventTime.now(current time) as a timestamp when time_key is specified.
Default: -
keep_time_key (bool, optional) {#parse section (single)-keep_time_key} 🔗︎
If true, keep time field in the record.
Default: -
types (string, optional) {#parse section (single)-types} 🔗︎
Types casting the fields to proper types example: field1:type, field2:type
Default: -
time_format (string, optional) {#parse section (single)-time_format} 🔗︎
Process value using specified format. This is available only when time_type is string
Default: -
time_type (string, optional) {#parse section (single)-time_type} 🔗︎
Parse/format value according to this type available values: float, unixtime, string
Default: string
local_time (bool, optional) {#parse section (single)-local_time} 🔗︎
Ff true, use local time. Otherwise, UTC is used. This is exclusive with utc.
Default: true
utc (bool, optional) {#parse section (single)-utc} 🔗︎
If true, use UTC. Otherwise, local time is used. This is exclusive with localtime
Default: false
timezone (string, optional) {#parse section (single)-timezone} 🔗︎
Use specified timezone. one can parse/format the time value in the specified timezone.
Default: nil
format (string, optional) {#parse section (single)-format} 🔗︎
Only available when using type: multi_format
Default: -
grok_pattern (string, optional) {#parse section (single)-grok_pattern} 🔗︎
Only available when using format: grok, multiline_grok. The pattern of grok. You cannot specify multiple grok pattern with this.
Default: -
custom_pattern_path (*secret.Secret, optional) {#parse section (single)-custom_pattern_path} 🔗︎
Only available when using format: grok, multiline_grok. File that includes custom grok patterns.
Default: -
grok_failure_key (string, optional) {#parse section (single)-grok_failure_key} 🔗︎
Only available when using format: grok, multiline_grok. The key has grok failure reason.
Default: -
grok_name_key (string, optional) {#parse section (single)-grok_name_key} 🔗︎
Only available when using format: grok, multiline_grok. The key name to store grok section’s name.
Default: -
multiline_start_regexp (string, optional) {#parse section (single)-multiline_start_regexp} 🔗︎
Only available when using format: multiline_grok The regexp to match beginning of multiline.
Default: -
grok_patterns ([]GrokSection, optional) {#parse section (single)-grok_patterns} 🔗︎
Only available when using format: grok, multiline_grok. Grok Section Specify grok pattern series set.
Default: -
Grok Section 🔗︎
name (string, optional) {#grok section-name} 🔗︎
The name of grok section.
Default: -
pattern (string, required) {#grok section-pattern} 🔗︎
The pattern of grok.
Default: -
keep_time_key (bool, optional) {#grok section-keep_time_key} 🔗︎
If true, keep time field in the record.
Default: -
time_key (string, optional) {#grok section-time_key} 🔗︎
Specify time field for event time. If the event doesn’t have this field, current time is used.
Default: time
time_format (string, optional) {#grok section-time_format} 🔗︎
Process value using specified format. This is available only when time_type is string.
Default: -
timezone (string, optional) {#grok section-timezone} 🔗︎
Use specified timezone. one can parse/format the time value in the specified timezone.
Default: -
Example Parser filter configurations 🔗︎
apiVersion: logging.banzaicloud.io/v1beta1
kind: Flow
metadata:
name: demo-flow
spec:
filters:
- parser:
remove_key_name_field: true
reserve_data: true
parse:
type: multi_format
patterns:
- format: nginx
- format: regexp
expression: /foo/
- format: none
selectors: {}
localOutputRefs:
- demo-output
Fluentd Config Result 🔗︎
<filter **>
@type parser
@id test_parser
key_name message
remove_key_name_field true
reserve_data true
<parse>
@type multi_format
<pattern>
format nginx
</pattern>
<pattern>
expression /foo/
format regexp
</pattern>
<pattern>
format none
</pattern>
</parse>
</filter>
