Codecs

Kinetic supports multiple serialization formats for encoding and decoding events. Sources use decoders to parse incoming data, while sinks use encoders to serialize outgoing data.

Available Codecs

Encoders (for Sinks)

Codec	Description	Sink Usage
JSON	JSON and NDJSON output	`encoding: { codec: json }`
Arrow IPC	Apache Arrow IPC format	`encoding: { codec: arrow }`
Avro	Apache Avro binary format	`encoding: { codec: avro }`
Parquet	Apache Parquet columnar format	`encoding: { codec: parquet }`
Protobuf	Protocol Buffers	`encoding: { codec: protobuf }`
OCSF	Open Cybersecurity Schema Framework	`encoding: { codec: ocsf }`

Decoders (for Sources)

Codec	Description	Source Usage
JSON	JSON and NDJSON input	`decoding: { codec: json }`
Arrow IPC	Apache Arrow IPC format	`decoding: { codec: arrow }`
Avro	Apache Avro binary format	`decoding: { codec: avro }`
Parquet	Apache Parquet columnar format	`decoding: { codec: parquet }`
Protobuf	Protocol Buffers	`decoding: { codec: protobuf }`
OCSF	Open Cybersecurity Schema Framework	`decoding: { codec: ocsf }`

Configuration Pattern

Both encoders and decoders use a tagged enum pattern with a codec field:

# Source with decoder
sources:
  my_kafka:
    type: kafka
    bootstrap_servers: localhost:9092
    group_id: my-group
    topics: [events]
    decoding:
      codec: json
      batch_size: 1024

# Sink with encoder
sinks:
  my_s3:
    type: s3
    bucket: my-bucket
    prefix: logs/
    encoding:
      codec: parquet
      compression: zstd

Codec Selection Guide

JSON: Best for human-readable formats, debugging, and interoperability with web services
Arrow IPC: Efficient for internal Kinetic-to-Kinetic communication
Avro: Compact binary format with schema evolution support
Parquet: Optimized for analytics workloads and columnar storage (S3, data lakes)
Protobuf: High-performance binary format for gRPC and microservices
OCSF: Standardized security event format for SIEM integration

Ocsf

Configuration for the OCSF Encoder.