Reads events from one or more Apache Kafka topics using a consumer group.

Kafka🔗

Reads events from one or more Apache Kafka topics using a consumer group.

Configuration🔗

FieldTypeRequiredDefaultDescription
decodingDecoderConfigEnum (optional)noOptional decoder configuration for input events.
clientKafkaClientConfignoShared Kafka client connection settings.
  bootstrap_serversstringyesComma-separated list of Kafka broker addresses.
  client_idstring (optional)noOptional Kafka client identifier.
  propertiesstd::collections::HashMap<String,String>noAdditional librdkafka configuration properties.
group_idstringyesKafka consumer group identifier.
topics[string]yesList of Kafka topics to consume from.
auto_offset_resetstringnoearliestWhere to begin consuming when no committed offset exists.

Telemetry🔗

This component emits the following standard telemetry metrics and events.

Metrics🔗

NameTypeDescription
component_received_events_totalCounterTotal number of Kinetic events created by this source.
component_received_event_bytes_totalCounterTotal estimated byte size of events created.
component_received_network_bytes_totalCounterTotal number of raw network bytes received.
component_errors_totalCounterTotal number of errors encountered during processing.

Events🔗

NameDescription
ComponentErrorEmitted when an error occurs during component execution.
ComponentEventsDroppedEmitted when events are intentionally or unintentionally dropped.

Example🔗

sources:
  my_kafka:
    type: kafka
    client:
      bootstrap_servers: localhost:9092
      client_id: kinetic-worker-1
      properties: {"compression.type": "lz4"}
    group_id: kinetic-ingest
    topics: ["events", "logs"]
    auto_offset_reset: latest