Reads events from one or more Apache Kafka topics using a consumer group.
Kafka🔗
Reads events from one or more Apache Kafka topics using a consumer group.
Configuration🔗
| Field | Type | Required | Default | Description |
decoding | DecoderConfigEnum (optional) | no | — | Optional decoder configuration for input events. |
client | KafkaClientConfig | no | — | Shared Kafka client connection settings. |
bootstrap_servers | string | yes | — | Comma-separated list of Kafka broker addresses. |
client_id | string (optional) | no | — | Optional Kafka client identifier. |
properties | std::collections::HashMap<String,String> | no | — | Additional librdkafka configuration properties. |
group_id | string | yes | — | Kafka consumer group identifier. |
topics | [string] | yes | — | List of Kafka topics to consume from. |
auto_offset_reset | string | no | earliest | Where to begin consuming when no committed offset exists. |
Telemetry🔗
This component emits the following standard telemetry metrics and events.
Metrics🔗
| Name | Type | Description |
component_received_events_total | Counter | Total number of Kinetic events created by this source. |
component_received_event_bytes_total | Counter | Total estimated byte size of events created. |
component_received_network_bytes_total | Counter | Total number of raw network bytes received. |
component_errors_total | Counter | Total number of errors encountered during processing. |
Events🔗
| Name | Description |
ComponentError | Emitted when an error occurs during component execution. |
ComponentEventsDropped | Emitted when events are intentionally or unintentionally dropped. |
Example🔗
sources:
my_kafka:
type: kafka
client:
bootstrap_servers: localhost:9092
client_id: kinetic-worker-1
properties: {"compression.type": "lz4"}
group_id: kinetic-ingest
topics: ["events", "logs"]
auto_offset_reset: latest