Kafka

Module for consuming data in real time from kafka.

ConsumerMetrics dataclass

Metrics for consumed data.

ExecutionMetrics dataclass

Metrics for the execution of a dag.

KafkaDriver

Control the execution of a dag, using data from kafka.

__init__(dag, runtime_source_topics, runtime_sink_topics, consumer_manager, producer_manager)

Do not use directly, use create instead.

KafkaMessageDeserializer

Bases: Protocol[T]

Interface for converting incoming kafka messages to custom data.

__call__(messages)

Convert batch of messages to data.

KafkaMessageSerializer

Bases: Protocol[T]

Interface for converting custom data to outgoing kafka messages.

__call__(value)

Convert batch of custom data to KafkaProducerMessage.

KafkaProducerMessage dataclass

Data for a message that needs to be produced.

OffsetPolicy

Bases: Enum

Type defining kafka historic replay policy.

ProducerMetrics dataclass

Metrics for kafka produced data.

SourceTopic dataclass

Bases: Generic[T]

Configuration of a source topic.

Do not use the constructor directly use from_xxx instead.