Grok Parser Transform

The Vector grok_parser transform accepts log events and allows you to parse a log field value with Grok.

Configuration

vector.toml

[transforms.my_transform_id]
  # REQUIRED - General
  type = "grok_parser" # example, must be: "grok_parser"
  inputs = ["my-source-id"] # example
  pattern = "%{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:level} %{GREEDYDATA:message}" # example
  
  # OPTIONAL - General
  drop_field = true # default
  field = "message" # default
  
  # OPTIONAL - Types
  [transforms.my_transform_id.types]
    status = "int"

Options

4 items

common onlyrequired only

commonbooloptional

drop_field

If true will drop the specified field after parsing.

Default: true

View examples

commonstringoptional

field

The log field to execute the pattern against. Must be a string value.

Default: "message"

View examples

commonstringrequired

pattern

The Grok pattern

No default

View examples

commontableoptional

types

Key/Value pairs representing mapped log field types.

commonstringenumrequired

[field-name]

A definition of log field type conversions. They key is the log field name and the value is the type. strptime specifiers are supported for the timestamp type.

No default

Enum, must be one of: "bool" "float" "int" "string" "timestamp"

View examples

How It Works

Available Patterns

Vector uses the Rust grok library. All patterns listed here are supported. It is recommended to use maintained patterns when possible since they can be improved over time by the community.

Debugging

We recommend the Grok debugger for Grok testing.

Environment Variables

Environment variables are supported through all of Vector's configuration. Simply add ${MY_ENV_VAR} in your Vector configuration file and the variable will be replaced before being evaluated.

You can learn more in the Environment Variables section.

Performance

Grok is approximately 50% slower than the regex_parser transform. We plan to add a performance test for this in the future. While this is still plenty fast for most use cases we recommend using the regex_parser transform if you are experiencing performance issues.