Crate laminar_connectors 

LaminarDB connector framework.

Modules

changelog

Sink-agnostic changelog collapse for upsert sinks (Z-set / CDC → key-unique _op batch). Pulled in by upsert-capable sink features (e.g. delta-lake). Sink-agnostic changelog collapse for upsert sinks. A commit epoch can carry many changelog events for one merge key (every aggregate value change is a retract + insert), which an upsert writer rejects as a cardinality violation. collapse_changelog folds an epoch down to one row per key.

checkpoint

Connector checkpoint types. Source checkpoints: the minimal state needed to resume a connector where it left off after a restart (Kafka offsets, CDC LSN/GTID, etc.).

config

Connector configuration types.

connector

Core connector traits (SourceConnector, SinkConnector). Connector traits — async SourceConnector / SinkConnector.

error

Connector error types.

files

AutoLoader-style file source and sink connectors. File source and sink connectors.

generator

Synthetic data generator source — no external infrastructure.

kafka

Kafka source and sink connectors. Kafka source and sink connectors.

lakehouse

Lakehouse connectors (Delta Lake, Iceberg). Lakehouse connectors (Delta Lake, Apache Iceberg).

mongodb

MongoDB CDC source and sink connectors. MongoDB CDC source and sink connectors.

nats

NATS core and JetStream source and sink connectors. NATS source and sink — core (non-durable, at-most-once) or jetstream (default; replayable, durable at-least-once with optional bounded Nats-Msg-Id broker deduplication).

otel

OpenTelemetry OTLP/gRPC source connector. OpenTelemetry (OTLP/gRPC) source connector.

postgres

PostgreSQL connector-specific configuration and implementations. PostgreSQL connector-specific configuration and implementations.

prom

Shared Prometheus registry/counter helpers used by per-connector metric structs. Shared Prometheus counter/gauge construction for per-connector metric structs. Replaces the per-file let local; let reg = … boilerplate and reg_c! macros that used to be duplicated in eleven places.

reference

Finite startup snapshot sources for reference tables. Startup snapshot sources for reference tables.

registry

Connector registry with factory pattern. Registry of connector factories, keyed by connector type string.

retry

Reconnect/backoff helper shared across source connectors. Retry/backoff helper shared by connectors.

schema

Schema inference, resolution, and evolution framework. Schema evolution, format codecs, and connector schema traits.

security

Secret classification and durable connector-identity sanitization. Shared secret classification for catalog and checkpoint metadata.

serde

Record serialization and deserialization framework. Record (de)serialization: bytes ↔ Arrow RecordBatch. Format implementations in json, csv, raw, debezium.

storage

Cloud storage infrastructure (credential resolution, validation, secret masking). Cloud storage infrastructure for lakehouse connectors.

testing

Testing utilities (mock connectors, helpers). Testing utilities for connector implementations.

websocket

WebSocket source and sink connectors. WebSocket client source plus client/server sinks.