Module files 

AutoLoader-style file source and sink connectors. AutoLoader-style file source and sink connectors.

Watches local directories or cloud storage for new files, infers and evolves schemas automatically, and checkpoints exactly which files have been processed.

Formats

• CSV (via CsvDecoder)
• JSON Lines (via JsonDecoder)
• Plain text (via TextLineDecoder)
• Apache Parquet (via ParquetDecoder)

Example DDL

CREATE SOURCE logs (ts BIGINT, level VARCHAR, message VARCHAR)
WITH (
    connector = 'files',
    path = '/data/logs/*.csv',
    format = 'csv'
);

Re-exports

pub use config::FileFormat;

pub use config::FileSinkConfig;

pub use config::FileSourceConfig;

pub use config::SinkMode;

pub use manifest::FileEntry;

pub use manifest::FileIngestionManifest;

pub use sink::FileSink;

pub use source::FileSource;

pub use text_decoder::TextLineDecoder;

Modules

config

Configuration types for the file source and sink connectors.

discovery

File discovery engine (Ring 2).

manifest

File ingestion manifest with bounded retention.

sink

File sink connector implementing SinkConnector.

source

File source connector implementing SourceConnector.

text_decoder

Plain text line decoder implementing FormatDecoder.

Functions

register_file_sink

Registers the file sink connector in the registry.

register_file_source

Registers the file source connector in the registry.