Module lookup 

Lookup table support for enrichment joins.

External Lookup Tables for Enrichment Joins

This module provides the TableLoader trait for loading data from external reference tables (dimension tables) to enrich streaming events.

Architecture

┌─────────────────────────────────────────────────────────────┐
│                    LookupJoinOperator                       │
│  (laminar-core)                                             │
│  ┌─────────┐    ┌─────────┐    ┌─────────────────────────┐ │
│  │ Event   │───▶│  Cache  │───▶│  Output (enriched)      │ │
│  │ Stream  │    │ (State) │    │                         │ │
│  └─────────┘    └────┬────┘    └─────────────────────────┘ │
│                      │ miss                                 │
│                      ▼                                      │
│               ┌─────────────┐                               │
│               │ TableLoader │  (trait, implemented here)    │
│               └──────┬──────┘                               │
└──────────────────────┼──────────────────────────────────────┘
                       │
                       ▼
            ┌──────────────────────┐
            │   External Systems   │
            │ (Redis, PostgreSQL,  │
            │  HTTP APIs, etc.)    │
            └──────────────────────┘

Implementations

• InMemoryTableLoader - For testing and static reference data
• Redis, PostgreSQL, HTTP loaders

Re-exports

pub use postgres_source::PostgresLookupSource;

pub use postgres_source::PostgresLookupSourceConfig;

pub use parquet_source::ParquetLookupSource;

pub use parquet_source::ParquetLookupSourceConfig;

Modules

cdc_adapter

CDC-to-reference-table adapter for using CDC sources as lookup tables. CDC-to-reference-table adapter.

parquet_source

Parquet file lookup source for static/slowly-changing dimension tables. Parquet file lookup source.

postgres_source

PostgreSQL lookup source with connection pooling and predicate pushdown. PostgreSQL lookup source implementation.

Structs

NoOpTableLoader

A no-op table loader that always returns NotFound.

Enums

LookupError

Errors that can occur during table lookup operations.

LookupResult

Result of a lookup operation.

Traits

TableLoader

Trait for loading data from external reference tables.