Struct FileSourceConfig 

pub struct FileSourceConfig {

    pub path: String,
    pub format: Option<FileFormat>,
    pub poll_interval: Duration,
    pub stabilisation_delay: Duration,
    pub max_files_per_poll: usize,
    pub include_metadata: bool,
    pub allow_overwrites: bool,
    pub manifest_retention_count: usize,
    pub manifest_retention_age_days: u64,
    pub max_file_bytes: usize,
    pub glob_pattern: Option<String>,
    pub csv_delimiter: u8,
    pub csv_has_header: bool,
}

Parsed configuration for super::source::FileSource.

Fields

path: String

Directory path, glob pattern, or cloud URL.

format: Option<FileFormat>

Data format (csv, json, text, parquet). None = auto-detect.

poll_interval: Duration

Discovery polling interval.

stabilisation_delay: Duration

Wait after last modify event before considering a file complete.

max_files_per_poll: usize

Maximum files to process per poll_batch call.

include_metadata: bool

Whether to append a _metadata struct column.

allow_overwrites: bool

Whether to allow re-ingesting files whose size has changed.

manifest_retention_count: usize

Maximum active entries in the file ingestion manifest.

manifest_retention_age_days: u64

Maximum age (days) for manifest entries before eviction.

max_file_bytes: usize

Safety limit for reading a single file (bytes). Primarily for Parquet.

glob_pattern: Option<String>

Additional glob pattern to filter discovered file names.

csv_delimiter: u8

CSV-specific: field delimiter.

csv_has_header: bool

CSV-specific: whether the first row is a header.

Implementations

impl FileSourceConfig

pub fn from_connector_config( config: &ConnectorConfig, ) -> Result<Self, ConnectorError>

Parse from connector properties.

Errors

Returns ConnectorError::ConfigurationError if required options are missing or values cannot be parsed.

Trait Implementations

impl Clone for FileSourceConfig

fn clone(&self) -> FileSourceConfig

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for FileSourceConfig

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.