Provider-Level Timeouts Now Available

Vercel's AI Gateway has introduced provider-level custom timeouts, allowing developers to define how long to wait for responses from individual AI providers before automatically falling back to alternatives. This feature enables faster failover strategies and better control over inference latency in production applications.

How It Works

The timeout mechanism is configured via the providerTimeouts parameter in providerOptions.gateway. When a provider doesn't respond within the specified millisecond threshold, AI Gateway cancels the request and attempts the next available provider in the configured sequence.

Key implementation details:

• Configure timeouts per provider in milliseconds
• Works with BYOK (Bring Your Own Key) credentials in beta
• Integrates with the order parameter to control both provider sequence and failover timing
• Can be combined with multi-provider setups for optimized latency

Usage Examples

For simple scenarios, set a single timeout:

providerOptions: {
  gateway: {
    providerTimeouts: {
      byok: { openai: 15000 }, // 15 seconds
    },
  },
}

For advanced multi-provider failover, configure different timeouts per provider:

providerOptions: {
  gateway: {
    order: ['anthropic', 'bedrock', 'vertex'],
    providerTimeouts: {
      byok: {
        anthropic: 10000,  // 10 seconds
        bedrock: 15000,    // 15 seconds
      },
    },
  },
}

Important Considerations

Some providers don't support stream cancellation, meaning you may still incur charges for timed-out requests depending on your provider. Developers should review provider-specific billing policies before implementing aggressive timeout strategies.

For detailed configuration options and best practices, see the custom provider timeouts documentation.