Expand description
LaminarDB connector framework.
Modulesยง
- cdc
- Change Data Capture connectors for databases.
Change Data Capture (CDC) source connectors.
PostgreSQL uses logical replication (
pgoutput); MySQL uses the binary log with GTID support. - checkpoint
- Connector checkpoint types. Source checkpoints: the minimal state needed to resume a connector where it left off after a restart (Kafka offsets, CDC LSN/GTID, etc.).
- config
- Connector configuration types.
- connector
- Core connector traits (
SourceConnector,SinkConnector). Connector traits โ asyncSourceConnector/SinkConnector. - error
- Connector error types.
- files
- AutoLoader-style file source and sink connectors. File source and sink connectors.
- health
- Connector health status types.
- kafka
- Kafka source and sink connectors. Kafka source and sink connectors.
- lakehouse
- Lakehouse connectors (Delta Lake, Iceberg). Lakehouse connectors (Delta Lake, Apache Iceberg).
- lookup
- Lookup table support for enrichment joins. Lookup tables for enrichment joins.
- metrics
- Connector metrics types.
- mongodb
- MongoDB CDC source and sink connectors.
MongoDBCDC source and sink connectors. - nats
- NATS core and JetStream source and sink connectors.
NATS source and sink โ
core(non-durable, at-most-once) orjetstream(default; replayable; at-least-once, or exactly-once withNats-Msg-Iddedup). - otel
- OpenTelemetry OTLP/gRPC source connector. OpenTelemetry (OTLP/gRPC) source connector.
- postgres
- PostgreSQL sink connector.
PostgreSQLsink connector. - reference
- Reference table source trait and refresh modes. Reference table source trait and refresh modes.
- registry
- Connector registry with factory pattern. Registry of connector factories, keyed by connector type string.
- schema
- Schema inference, resolution, and evolution framework. Schema evolution, format codecs, and connector schema traits.
- serde
- Record serialization and deserialization framework.
Record (de)serialization: bytes โ Arrow
RecordBatch. Format implementations injson,csv,raw,debezium. - storage
- Cloud storage infrastructure (credential resolution, validation, secret masking). Cloud storage infrastructure for lakehouse connectors.
- testing
- Testing utilities (mock connectors, helpers). Testing utilities for connector implementations.
- websocket
- WebSocket source and sink connectors. WebSocket source/sink connectors. Four modes: source-client (connect to a WS server), source-server (listen for clients), sink-server (fan out results to subscribers), sink-client (push to an external server).