Writes events to Amazon S3 as batched objects.

S3🔗

Writes events to Amazon S3 as batched objects.

Configuration🔗

Field	Type	Required	Default	Description
`inputs`	`[string]`	yes	—	Upstream component names that feed events into this sink.
`error_policy`	`ErrorPolicy`	no	`drop_on_error`	Policy for handling errors during event processing.
`encoding`	`EncoderConfigEnum (optional)`	no	—	Optional encoder configuration for output events.
`buffer`	`BufferConfig`	no	—	Buffer configuration for event batching.
`auth`	`AwsConfig (optional)`	no	—	AWS authentication and region settings.
`region`	`string (optional)`	no	—	AWS region to operate in.
`endpoint`	`string (optional)`	no	—	Custom AWS endpoint URL (for LocalStack, MinIO, etc.).
`access_key_id`	`string (optional)`	no	—	AWS access key ID for static credentials.
`secret_access_key`	`string (optional)`	no	—	AWS secret access key for static credentials. (sensitive)
`session_token`	`string (optional)`	no	—	AWS session token for temporary credentials. (sensitive)
`assume_role`	`string (optional)`	no	—	IAM role ARN to assume.
`session_name`	`string (optional)`	no	—	Session name when assuming a role.
`external_id`	`string (optional)`	no	—	External ID for cross-account role assumption.
`profile`	`string (optional)`	no	—	AWS profile name from shared credentials file.
`credentials_file`	`string (optional)`	no	—	Path to an AWS credentials file.
`imds`	`ImdsConfig (optional)`	no	—	Instance Metadata Service (IMDS) configuration.
`connect_timeout_seconds`	`unsigned integer (optional)`	no	`1`	Timeout in seconds for IMDS connection.
`max_attempts`	`unsigned integer (optional)`	no	`3`	Maximum number of IMDS retry attempts.
`read_timeout_seconds`	`unsigned integer (optional)`	no	`1`	Timeout in seconds for IMDS read operations.
`load_timeout_secs`	`unsigned integer (optional)`	no	`10`	Timeout in seconds for loading AWS credentials.
`bucket`	`string`	yes	—	S3 bucket name.
`key_prefix`	`string`	yes	—	Key prefix for S3 objects.
`compression`	`string (optional)`	no	—	Compression algorithm for S3 objects.
`encoding`	`string (optional)`	no	—	Encoding format for the S3 object (e.g. 'json' or 'parquet').
`batch`	`BatchConfig`	no	—	Batching configuration for S3 writes.
`max_size`	`unsigned integer (optional)`	no	—	Maximum number of events in a batch before flushing.
`timeout_secs`	`unsigned integer (optional)`	no	—	Maximum time in seconds before flushing a batch.

Telemetry🔗

This component emits the following standard telemetry metrics and events.

Metrics🔗

Name	Type	Description
`component_sent_events_total`	Counter	Total number of Kinetic events successfully sent by this sink.
`component_sent_event_bytes_total`	Counter	Total estimated byte size of events sent.
`component_sent_network_bytes_total`	Counter	Total number of raw network bytes sent.
`component_errors_total`	Counter	Total number of errors encountered during processing.

Events🔗

Name	Description
`ComponentError`	Emitted when an error occurs during component execution.
`ComponentEventsDropped`	Emitted when events are intentionally or unintentionally dropped.

Example🔗

sinks:
  my_s3:
    type: s3
    inputs: ["my_transform"]
    auth:
      region: us-west-2
      endpoint: http://localhost:4566
      access_key_id: AKIAIOSFODNN7EXAMPLE
      secret_access_key: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
      assume_role: arn:aws:iam::123456789012:role/kinetic-ingest
      session_name: kinetic-session
      profile: default
      credentials_file: /home/kinetic/.aws/credentials
      imds:
        connect_timeout_seconds: 5
        max_attempts: 5
        read_timeout_seconds: 5
    bucket: my-data-lake
    key_prefix: logs/kinetic/
    compression: gzip
    encoding: parquet
    batch:
      max_size: 1000
      timeout_secs: 30

S3 Stable

S3🔗

Configuration🔗

Telemetry🔗

Metrics🔗

Events🔗

Example🔗