Writes events to Google Cloud Storage as batched objects.
Gcp Cloud Storage🔗
Writes events to Google Cloud Storage as batched objects.
Configuration🔗
| Field | Type | Required | Default | Description |
|---|
inputs | [string] | yes | — | Upstream component names that feed events into this sink. |
error_policy | ErrorPolicy | no | drop_on_error | Policy for handling errors during event processing. |
encoding | EncoderConfigEnum (optional) | no | — | Optional encoder configuration for output events. |
buffer | BufferConfig | no | — | Buffer configuration for event batching. |
auth | GcpConfig (optional) | no | — | GCP authentication and project settings. |
project_id | string (optional) | yes | — | GCP Project ID. |
credentials_file | string (optional) | no | — | Path to a Service Account JSON key file. If not provided, Application Default Credentials (ADC) will be used. |
credentials_json | string (optional) | no | — | Raw Service Account JSON key content. If not provided, credentials_file or ADC will be used. (sensitive) |
bucket | string | yes | — | GCS bucket name. |
key_prefix | string | yes | — | Key prefix for GCS objects. |
compression | string (optional) | no | — | Compression algorithm for GCS objects. |
encoding | string (optional) | no | — | Encoding format for the GCS object (e.g. 'json' or 'parquet'). |
batch | aws_common::config::BatchConfig | no | — | Batching configuration for GCS writes. |
Telemetry🔗
This component emits the following standard telemetry metrics and events.
Metrics🔗
| Name | Type | Description |
|---|
component_sent_events_total | Counter | Total number of Kinetic events successfully sent by this sink. |
component_sent_event_bytes_total | Counter | Total estimated byte size of events sent. |
component_sent_network_bytes_total | Counter | Total number of raw network bytes sent. |
component_errors_total | Counter | Total number of errors encountered during processing. |
Events🔗
| Name | Description |
|---|
ComponentError | Emitted when an error occurs during component execution. |
ComponentEventsDropped | Emitted when events are intentionally or unintentionally dropped. |
Example🔗
sinks:
my_gcp_cloud_storage:
type: gcp_cloud_storage
inputs: ["my_transform"]
auth:
project_id: my-gcp-project
credentials_file: /secrets/gcp-sa.json
bucket: my-data-lake
key_prefix: logs/kinetic/
compression: gzip
encoding: parquet