Expand description
Client-side token-bucket rate limiting for remote providers. One cell is acquired per input row before dispatch; the wait happens on Ring 1 only.
Structsยง
- Rate
Limited Provider - An
InferenceProviderthat paces calls to a steady rate.