Struct RateLimitedProvider 

pub struct RateLimitedProvider { /* private fields */ }

An InferenceProvider that paces calls to a steady rate.

Implementations

impl RateLimitedProvider

pub fn new( inner: Arc<dyn InferenceProvider>, requests_per_second: NonZeroU32, ) -> Self

Wrap inner, limiting to requests_per_second (burst up to the same).

Trait Implementations

impl InferenceProvider for RateLimitedProvider

fn infer_batch<'life0, 'async_trait>( &'life0 self, request: InferenceRequest, ) -> Pin<Box<dyn Future<Output = Result<InferenceResponse, ProviderError>> + Send + 'async_trait>>

where Self: 'async_trait, 'life0: 'async_trait,

Run one batch of inputs through the model.

fn name(&self) -> &'static str

Stable backend-kind identity for logging and the laminar.ai_calls log (e.g. anthropic, openai, local). Constant per implementor.

fn intrinsic_labels(&self, model: &str) -> Option<Vec<String>>

Classifier labels intrinsic to a model, discovered from its own metadata. Returns None for backends that have none (remote providers, embedding models). A local classifier returns its config.json id2label once the model is on disk — the seam that lets a lazily downloaded classifier score without the labels having been known at startup. The default is None.