Configuration Reference🔗

Kinetic uses a structured YAML configuration to define data pipelines.

Root Configuration🔗

The root configuration consists of three main sections: sources, transforms, and sinks.

sources:
  my_source:
    type: kafka
    # ... source configuration ...

transforms:
  my_transform:
    type: filter
    inputs: [my_source]
    # ... transform configuration ...

sinks:
  my_sink:
    type: s3
    inputs: [my_transform]
    # ... sink configuration ...

Error Policies🔗

All transforms and sinks support an error_policy to define how failures are handled.

Policy	Description
drop_on_error	(Default) Silently drop the failing event.
reroute_on_error	Send the event to the configured DLQ.
halt_on_error	Stop the pipeline processing.

Component Types🔗

Detailed configuration for each component type can be found in the respective sections:

• Sources
• Transforms
• Sinks