GCP PubSub Sink

The Vector gcp_pubsub sink batches log events to Google Cloud Platform's Pubsub service via the REST Interface.

Configuration

• Common
• Advanced

vector.toml

[sinks.my_sink_id]
  type = "gcp_pubsub" # example, must be: "gcp_pubsub"
  inputs = ["my-source-id"] # example
  project = "vector-123456" # example
  topic = "this-is-a-topic" # example

Options

15 items

common onlyrequired only

stringoptional

api_key

A Google Cloud API key used to authenticate access the pubsub project and topic. Either this or credentials_path must be set.

intbytesoptional

batch_size

The maximum size of a batch before it is flushed.

Default: 10485760

View examples

intsecondsoptional

batch_timeout

The maximum age of a batch before it is flushed.

Default: 1

View examples

tableoptional

buffer

Configures the sink specific buffer.

intbytesoptional

max_size

The maximum size of the buffer on the disk.

No default

Only relevant when: type = "disk"

View examples

inteventsoptional

num_items

The maximum number of events allowed in the buffer.

Default: 500

Only relevant when: type = "memory"

View examples

stringenumoptional

type

The buffer's type / location. disk buffers are persistent and will be retained between restarts.

Default: "memory"

Enum, must be one of: "memory" "disk"

View examples

stringenumoptional

when_full

The behavior when the buffer becomes full.

Default: "block"

Enum, must be one of: "block" "drop_newest"

View examples

stringoptional

credentials_path

The filename for a Google Cloud service account credentials JSON file used to authenticate access to the pubsub project and topic. Either this or api_key must be set.

booloptional

healthcheck

Enables/disables the sink healthcheck upon start.

Default: true

View examples

commonstringrequired

project

The project name to which to publish logs.

No default

View examples

intoptional

request_in_flight_limit

The maximum number of in-flight requests allowed at any given time.

Default: 5

View examples

intsecondsoptional

request_rate_limit_duration_secs

The window used for the request_rate_limit_num option

Default: 1

View examples

intoptional

request_rate_limit_num

The maximum number of requests allowed within the request_rate_limit_duration_secs window.

Default: 100

View examples

intoptional

request_retry_attempts

The maximum number of retries to make for failed requests.

Default: 9223372036854776000

View examples

intsecondsoptional

request_retry_backoff_secs

The amount of time to wait before attempting a failed request again.

Default: 1

View examples

intsecondsoptional

request_timeout_secs

The maximum time a request can take before being aborted. It is highly recommended that you do not lower value below the service's internal timeout, as this could create orphaned requests, pile on retries, and result in deuplicate data downstream.

Default: 60

View examples

tableoptional

tls

Configures the TLS options for connections from this sink.

stringoptional

ca_path

Absolute path to an additional CA certificate file, in DER or PEM format (X.509).

No default

View examples

stringoptional

crt_path

Absolute path to a certificate file used to identify this connection, in DER or PEM format (X.509) or PKCS#12. If this is set and is not a PKCS#12 archive, key_path must also be set.

No default

View examples

stringoptional

key_pass

Pass phrase used to unlock the encrypted key file. This has no effect unless key_pass above is set.

No default

View examples

stringoptional

key_path

Absolute path to a certificate key file used to identify this connection, in DER or PEM format (PKCS#8). If this is set, crt_path must also be set.

No default

View examples

booloptional

verify_certificate

If true (the default), Vector will validate the TLS certificate of the remote host. Do NOT set this to false unless you understand the risks of not verifying the remote certificate.

Default: true

View examples

booloptional

verify_hostname

If true (the default), Vector will validate the configured remote host name against the remote host's TLS certificate. Do NOT set this to false unless you understand the risks of not verifying the remote hostname.

Default: true

View examples

commonstringrequired

topic

The topic within the project to which to publish logs.

No default

View examples