Parser Filter 🔗︎

Overview 🔗︎

Parses” string field in event records and mutates its

Configuration 🔗︎

ParserConfig 🔗︎

Variable Name Type Required Default Description
key_name string No - Specify field name in the record to parse. If you leave empty the Container Runtime default will be used.
reserve_time bool No - Keep original event time in parsed result.
reserve_data bool No - Keep original key-value pair in parsed result.
remove_key_name_field bool No - Remove key_name field when parsing is succeeded
replace_invalid_sequence bool No - If true, invalid string is replaced with safe characters and re-parse it.
inject_key_prefix string No - Store parsed values with specified key name prefix.
hash_value_field string No - Store parsed values as a hash value in a field.
emit_invalid_record_to_error *bool No - Emit invalid record to @ERROR label. Invalid cases are: key not exist, format is not matched, unexpected error
parse ParseSection No - Parse Section
parsers []ParseSection No - Deprecated, use parse instead

Parse Section 🔗︎

Variable Name Type Required Default Description
type string No - Parse type: apache2, apache_error, nginx, syslog, csv, tsv, ltsv, json, multiline, none, logfmt
expression string No - Regexp expression to evaluate
time_key string No - Specify time field for event time. If the event doesn’t have this field, current time is used.
null_value_pattern string No - Specify null value pattern.
null_empty_string bool No - If true, empty string field is replaced with nil
estimate_current_event bool No - If true, use Fluent::EventTime.now(current time) as a timestamp when time_key is specified.
keep_time_key bool No - If true, keep time field in the record.
types string No - Types casting the fields to proper types example: field1:type, field2:type
time_format string No - Process value using specified format. This is available only when time_type is string
time_type string No string Parse/format value according to this type available values: float, unixtime, string
local_time bool No true Ff true, use local time. Otherwise, UTC is used. This is exclusive with utc.
utc bool No false If true, use UTC. Otherwise, local time is used. This is exclusive with localtime
timezone string No nil Use specified timezone. one can parse/format the time value in the specified timezone.
format string No - Only available when using type: multi_format
format_firstline string No - Only available when using type: multi_format
delimiter string No “\t” Only available when using type: ltsv
delimiter_pattern string No - Only available when using type: ltsv
label_delimiter string No “:” Only available when using type: ltsv
multiline []string No - The multiline parser plugin parses multiline logs.
patterns []SingleParseSection No - Only available when using type: multi_format
Parse Section

Parse Section (single) 🔗︎

Variable Name Type Required Default Description
type string No - Parse type: apache2, apache_error, nginx, syslog, csv, tsv, ltsv, json, multiline, none, logfmt
expression string No - Regexp expression to evaluate
time_key string No - Specify time field for event time. If the event doesn’t have this field, current time is used.
null_value_pattern string No - Specify null value pattern.
null_empty_string bool No - If true, empty string field is replaced with nil
estimate_current_event bool No - If true, use Fluent::EventTime.now(current time) as a timestamp when time_key is specified.
keep_time_key bool No - If true, keep time field in the record.
types string No - Types casting the fields to proper types example: field1:type, field2:type
time_format string No - Process value using specified format. This is available only when time_type is string
time_type string No string Parse/format value according to this type available values: float, unixtime, string
local_time bool No true Ff true, use local time. Otherwise, UTC is used. This is exclusive with utc.
utc bool No false If true, use UTC. Otherwise, local time is used. This is exclusive with localtime
timezone string No nil Use specified timezone. one can parse/format the time value in the specified timezone.
format string No - Only available when using type: multi_format

Example Parser filter configurations 🔗︎ 

apiVersion: logging.banzaicloud.io/v1beta1
kind: Flow
metadata:
 name: demo-flow
spec:
 filters:
   - parser:
       remove_key_name_field: true
       reserve_data: true
       parse:
         type: multi_format
         patterns:
         - format: nginx
         - format: regexp
           expression: /foo/
         - format: none
 selectors: {}
 localOutputRefs:
   - demo-output

Fluentd Config Result 🔗︎ 

<filter **>
 @type parser
 @id test_parser
 key_name message
 remove_key_name_field true
 reserve_data true
 <parse>
   @type multi_format
   <pattern>
     format nginx
   </pattern>
   <pattern>
     expression /foo/
     format regexp
   </pattern>
   <pattern>
     format none
   </pattern>
 </parse>
</filter>
Edit this page on GitHub