Gateway Inference

Gateway inference lets your code call Islo-managed models without configuring a model provider key. Use the islo package to create a short-lived session token, then pass that token to an OpenAI-compatible or Anthropic-compatible SDK.

The islo package is used here only for token management. Inference requests are sent through SDKs that accept a compatible base URL and API key in code.

The gateway chooses the upstream provider from the requested model. No provider header or provider-specific route is required.

Setup

uv add islo
export ISLO_API_KEY="your-islo-api-key"

OpenAI SDK

Use the OpenAI-compatible base URL:

https://gateway.islo.dev/inference/openai/v1

For Python:

uv add openai

import os
from islo.custom.auth import SyncTokenProvider
from openai import OpenAI
session_token = SyncTokenProvider(
    "https://api.islo.dev",
    os.environ["ISLO_API_KEY"],
)()
client = OpenAI(
    api_key=session_token,
    base_url="https://gateway.islo.dev/inference/openai/v1",
)
response = client.chat.completions.create(
    model="kimi-k2.7-code",
    messages=[
        {"role": "user", "content": "Say hello from Islo gateway inference."},
    ],
    max_tokens=128,
)
print(response)

Call the OpenAI Responses API with the same client:

response = client.responses.create(
    model="kimi-k2.7-code",
    input="Say hello from Islo gateway inference.",
    max_output_tokens=128,
)
print(response)

Anthropic SDK

Use the Anthropic-compatible base URL:

https://gateway.islo.dev/inference/anthropic

For Python:

uv add anthropic

import os
from anthropic import Anthropic
from islo.custom.auth import SyncTokenProvider
session_token = SyncTokenProvider(
    "https://api.islo.dev",
    os.environ["ISLO_API_KEY"],
)()
client = Anthropic(
    api_key=session_token,
    base_url="https://gateway.islo.dev/inference/anthropic",
)
message = client.messages.create(
    model="kimi-k2.7-code",
    max_tokens=128,
    messages=[
        {"role": "user", "content": "Say hello from Islo gateway inference."},
    ],
)
print(message)

For long-running Anthropic clients, create a fresh session token before the current token expires.

Claude Agent SDK

Claude Agent SDK reads Anthropic-compatible connection settings from the process environment. Set those values from a freshly created Islo session token before constructing the client:

uv add claude-agent-sdk

import os
from claude_agent_sdk import ClaudeAgentOptions, ClaudeSDKClient
from islo.custom.auth import SyncTokenProvider
session_token = SyncTokenProvider(
    "https://api.islo.dev",
    os.environ["ISLO_API_KEY"],
)()
os.environ["ANTHROPIC_BASE_URL"] = "https://gateway.islo.dev/inference/anthropic"
os.environ["ANTHROPIC_API_KEY"] = session_token
options = ClaudeAgentOptions(
    system_prompt="You are a helpful assistant.",
    model="kimi-k2.7-code",
    max_turns=20,
)
async with ClaudeSDKClient(options=options) as client:
    await client.query("Say hello from Islo gateway inference.")
    async for message in client.receive_response():
        if hasattr(message, "content"):
            for block in message.content:
                if hasattr(block, "text"):
                    print(block.text, end="", flush=True)

For long-running agents, create a fresh session token before starting a new client.

OpenAI Agents SDK

Use the OpenAI Agents SDK with an AsyncOpenAI client configured for the gateway:

uv add openai-agents openai

import asyncio
import os
from agents import Agent, OpenAIChatCompletionsModel, Runner, set_tracing_disabled
from islo.custom.auth import SyncTokenProvider
from openai import AsyncOpenAI
session_token = SyncTokenProvider(
    "https://api.islo.dev",
    os.environ["ISLO_API_KEY"],
)()
openai_client = AsyncOpenAI(
    api_key=session_token,
    base_url="https://gateway.islo.dev/inference/openai/v1",
)
set_tracing_disabled(disabled=True)
async def main() -> None:
    agent = Agent(
        name="Assistant",
        instructions="You are a helpful assistant.",
        model=OpenAIChatCompletionsModel(
            model="kimi-k2.7-code",
            openai_client=openai_client,
        ),
    )
    result = await Runner.run(agent, "Say hello from Islo gateway inference.")
    print(result.final_output)
asyncio.run(main())

LangChain

Use LangChain’s OpenAI chat model with the gateway base URL:

uv add langchain-openai

import os
from islo.custom.auth import SyncTokenProvider
from langchain_openai import ChatOpenAI
session_token = SyncTokenProvider(
    "https://api.islo.dev",
    os.environ["ISLO_API_KEY"],
)()
chat = ChatOpenAI(
    api_key=session_token,
    base_url="https://gateway.islo.dev/inference/openai/v1",
    model="kimi-k2.7-code",
)
response = chat.invoke("Say hello from Islo gateway inference.")
print(response.content)

Instructor

Use Instructor with an OpenAI client configured for the gateway:

uv add instructor openai pydantic

import os
import instructor
from islo.custom.auth import SyncTokenProvider
from openai import OpenAI
from pydantic import BaseModel
class Greeting(BaseModel):
    message: str
session_token = SyncTokenProvider(
    "https://api.islo.dev",
    os.environ["ISLO_API_KEY"],
)()
openai_client = OpenAI(
    api_key=session_token,
    base_url="https://gateway.islo.dev/inference/openai/v1",
)
client = instructor.from_openai(openai_client)
greeting = client.chat.completions.create(
    model="kimi-k2.7-code",
    response_model=Greeting,
    messages=[
        {"role": "user", "content": "Say hello from Islo gateway inference."},
    ],
)
print(greeting.message)