Reads events from Google Cloud Storage objects. Supports both list-based polling and Pub/Sub notifications.

Gcp Cloud Storage🔗

Reads events from Google Cloud Storage objects. Supports both list-based polling and Pub/Sub notifications.

Configuration🔗

Field	Type	Required	Default	Description
`decoding`	`DecoderConfigEnum (optional)`	no	—	Optional decoder configuration for input events.

Source mode: list for polling or event_stream for Pub/Sub notifications.

Mode	Description
`list`	List mode: periodically polls for new objects.
`event_stream`	Event stream mode: consumes GCS notifications from Pub/Sub.

Field	Type	Required	Default	Description
`auth`	`GcpConfig (optional)`	no	—	[Authentication] GCP authentication and project settings.
`project_id`	`string (optional)`	yes	—	GCP Project ID.
`credentials_file`	`string (optional)`	no	—	Path to a Service Account JSON key file. If not provided, Application Default Credentials (ADC) will be used.
`credentials_json`	`string (optional)`	no	—	Raw Service Account JSON key content. If not provided, credentials_file or ADC will be used. (sensitive)
`bucket`	`string`	yes	—	GCS bucket to list objects from.
`prefix`	`string (optional)`	no	—	Key prefix filter.
`interval_secs`	`unsigned integer`	no	`60`	Polling interval in seconds.
`delete_after_read`	`boolean`	no	`false`	Whether to delete objects from GCS after they have been successfully processed.

Example:

mode:
mode: list
  bucket: my-gcs-bucket
  interval_secs: 60

Field	Type	Required	Default	Description
`pubsub`	`PubSubConfig`	yes	—	[Event Stream] Pub/Sub configuration for consuming GCS event notifications.
`auth`	`GcpConfig (optional)`	no	—	GCP authentication and project settings.
`project_id`	`string (optional)`	yes	—	GCP Project ID.
`credentials_file`	`string (optional)`	no	—	Path to a Service Account JSON key file. If not provided, Application Default Credentials (ADC) will be used.
`credentials_json`	`string (optional)`	no	—	Raw Service Account JSON key content. If not provided, credentials_file or ADC will be used. (sensitive)
`topic`	`string`	yes	—	Pub/Sub topic name.
`subscription`	`string (optional)`	no	—	Pub/Sub subscription name (required for sources).
`dead_letter_topic`	`string (optional)`	no	—	Dead Letter Topic name.
`enable_message_ordering`	`boolean`	no	—	Whether to enable message ordering.
`ordering_key_field`	`string (optional)`	no	—	Field name to use as the ordering key.
`bucket`	`string`	yes	—	GCS bucket to read objects from.
`include_metadata_updates`	`boolean`	no	`false`	Whether to listen for OBJECT_METADATA_UPDATE events.

Example:

mode:
mode: event_stream
  bucket: my-gcs-bucket
  pubsub:
    topic: my-topic
    subscription: my-sub

This component emits the following standard telemetry metrics and events.

Name	Type	Description
`component_received_events_total`	Counter	Total number of Kinetic events created by this source.
`component_received_event_bytes_total`	Counter	Total estimated byte size of events created.
`component_received_network_bytes_total`	Counter	Total number of raw network bytes received.
`component_errors_total`	Counter	Total number of errors encountered during processing.

Name	Description
`ComponentError`	Emitted when an error occurs during component execution.
`ComponentEventsDropped`	Emitted when events are intentionally or unintentionally dropped.

sources:
  my_gcp_cloud_storage:
    type: gcp_cloud_storage
    mode: list