Decodes JSON and NDJSON input into events.

JSON Decoder

Decodes JSON and newline-delimited JSON (NDJSON) input into Arrow RecordBatches.

Configuration

Field	Type	Required	Default	Description
`codec`	`string`	yes	—	Must be `"json"`
`batch_size`	`integer`	no	`1024`	Number of rows per batch
`max_size`	`integer`	no	`10485760`	Maximum message size in bytes (10MB default)

Example

sources:
  my_kafka:
    type: kafka
    bootstrap_servers: localhost:9092
    group_id: my-group
    topics: [events]
    decoding:
      codec: json
      batch_size: 1024

Input Formats

NDJSON

One JSON object per line:

{"timestamp": "2024-01-15T10:30:00Z", "level": "info", "message": "Event 1"}
{"timestamp": "2024-01-15T10:30:01Z", "level": "warn", "message": "Event 2"}

JSON Array

Array of objects:

[
  {"timestamp": "2024-01-15T10:30:00Z", "level": "info", "message": "Event 1"},
  {"timestamp": "2024-01-15T10:30:01Z", "level": "warn", "message": "Event 2"}
]

When to Use

Log ingestion - Application logs, access logs
Webhook receivers - HTTP endpoints receiving JSON
Message queues - Kafka, SQS with JSON payloads