Split Transform

The Vector split transform accepts log events and allows you to split a field's value on a given separator and zip the tokens into ordered field names.

Configuration

vector.toml

[transforms.my_transform_id]
  # REQUIRED - General
  type = "split" # example, must be: "split"
  inputs = ["my-source-id"] # example
  field_names = ["timestamp", "level", "message"] # example
  
  # OPTIONAL - General
  drop_field = true # default
  field = "message" # default
  separator = "," # default
  
  # OPTIONAL - Types
  [transforms.my_transform_id.types]
    status = "int"

Options

5 items

common onlyrequired only

commonbooloptional

drop_field

If true the field will be dropped after parsing.

Default: true

View examples

commonstringoptional

field

The field to apply the split on.

Default: "message"

View examples

common[string]required

field_names

The field names assigned to the resulting tokens, in order.

No default

View examples

common[string]optional

separator

The separator to split the field on. If no separator is given, it will split on whitespace.

Default: "whitespace"

View examples

commontableoptional

types

Key/Value pairs representing mapped log field types.

commonstringenumrequired

[field-name]

A definition of log field type conversions. They key is the log field name and the value is the type. strptime specifiers are supported for the timestamp type.

No default

Enum, must be one of: "bool" "float" "int" "string" "timestamp"

View examples

Output

Given the following log line:

{
  "message": "5.86.210.12,zieme4647,19/06/2019:17:20:49 -0400,GET /embrace/supply-chains/dynamic/vertical,201,20574"
}

And the following configuration:

[transforms.<transform-id>]
type = "split"
field = "message"
fields = ["remote_addr", "user_id", "timestamp", "message", "status", "bytes"]
  [transforms.<transform-id>.types]
    status = "int"
    bytes = "int"

A log event will be output with the following structure:

{
  // ... existing fields
  "remote_addr": "5.86.210.12",
  "user_id": "zieme4647",
  "timestamp": "19/06/2019:17:20:49 -0400",
  "message": "GET /embrace/supply-chains/dynamic/vertical",
  "status": 201,
  "bytes": 20574
}

A few things to note about the output:

1. The message field was overwritten.
2. The status and bytes fields are integers because of type coercion.

How It Works

Environment Variables

Environment variables are supported through all of Vector's configuration. Simply add ${MY_ENV_VAR} in your Vector configuration file and the variable will be replaced before being evaluated.

You can learn more in the Environment Variables section.