vllm-sr/auto API

Overview

Any OpenAI SDK or tool that lets you set a base URL works: the OpenAI Python and Node SDKs, LangChain, LlamaIndex, agent frameworks and coding assistants.

Authentication

Send your key as a Bearer token. Create a key in your browser; it looks like sk-pt- followed by 43 letters and digits.

Authorization: Bearer $VLLM_SR_API_KEY

X-API-Key: <key> is accepted too. Keep keys on servers: never put them in client-side code, apps or public repositories. If a key leaks, revoke it on the Keys page.

Chat completions

POST /v1/chat/completions takes the OpenAI request body. The parameters you need most:

curl https://mom.vllm-sr.ai/v1/chat/completions \
  -H "Authorization: Bearer $VLLM_SR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "vllm-sr/auto",
    "messages": [
      {"role": "system", "content": "You are a concise assistant."},
      {"role": "user", "content": "What does a KV cache store?"}
    ],
    "max_tokens": 300
  }'

The answer is a standard chat.completion object with choices and usage. Its model field names the backend that answered; the routing headers below name the model in the pool.

Streaming

With "stream": true the answer arrives as server-sent events: one data: {...} line per chunk, ending with data: [DONE]. Reasoning models stream their thinking in delta.reasoning before the answer in delta.content. The stream passes through untouched.

import os
from openai import OpenAI

client = OpenAI(base_url="https://mom.vllm-sr.ai/v1", api_key=os.environ["VLLM_SR_API_KEY"])
stream = client.chat.completions.create(
    model="vllm-sr/auto",
    messages=[{"role": "user", "content": "Explain speculative decoding in five sentences."}],
    stream=True,
    stream_options={"include_usage": True},
)
for chunk in stream:
    if chunk.choices and chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="", flush=True)
    if chunk.usage:
        print("\n", chunk.usage)

Tools

Declare functions in tools; the model answers with tool_calls, you run them and send the results back as tool messages. Agent loops run with up to 16 requests at once per key.

tools = [{"type": "function", "function": {
    "name": "get_weather",
    "description": "Current weather for a city",
    "parameters": {"type": "object", "properties": {"city": {"type": "string"}}, "required": ["city"]},
}}]
resp = client.chat.completions.create(model="vllm-sr/auto", tools=tools,
    messages=[{"role": "user", "content": "Do I need an umbrella in San Jose today?"}])
print(resp.choices[0].message.tool_calls)

Routing headers

Every answer says how it was routed:

In Python, client.chat.completions.with_raw_response.create(...) gives you the headers.

Models

GET /v1/models lists vllm-sr/auto, the one model name you call. The pool behind it can change; your code does not.

Limits

Each key has its own budget, sized against the measured capacity of the pool so that one agent can run freely. The live values:

• Daily quotas reset at 00:00 UTC. Tokens count prompt and output, charged when a request finishes.
• When the whole service is busy, a request waits up to 20 seconds for a free slot before it is answered 429 server_overloaded.
• Every 429 carries Retry-After in seconds. Back off and retry; do not open more keys to get around a limit.

Errors

Errors use the OpenAI shape: {"error": {"message", "type", "code"}}.

Keys API

The site uses these endpoints; you can call them too. Key-authenticated endpoints take the key in X-API-Key.

Data and logging

Requests and responses are logged and kept to operate, secure, debug, analyse and improve the service, including the routing, and to train and evaluate models. No request header turns logging off. Do not send secrets, credentials or sensitive personal data. IP addresses are kept for up to 30 days. The privacy notice has the details, and the terms of use apply.