> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.islo.dev/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.islo.dev/_mcp/server.

# Gateway inference

> Use Islo-managed models through OpenAI-compatible and Anthropic-compatible SDKs.

In a factory, Agents use their configured model and Machine. This advanced guide covers direct gateway inference from your own code when you need a model call outside a Line. See [Agents](/concepts/agents) for factory work.

Gateway inference lets your code call Islo-managed models without configuring a model provider key. Use the `islo` package to create a short-lived session token, then pass that token to an OpenAI-compatible or Anthropic-compatible SDK.

The `islo` package is used here only for token management. Inference requests are sent through SDKs that accept a compatible base URL and API key in code.

The gateway chooses the upstream provider from the requested model. No provider header or provider-specific route is required.

## Setup

```bash
uv add islo
export ISLO_API_KEY="your-islo-api-key"
```

## OpenAI SDK

Use the OpenAI-compatible base URL:

```text
https://gateway.islo.dev/inference/openai/v1
```

For Python:

```bash
uv add openai
```

```python
import os

from islo.custom.auth import SyncTokenProvider
from openai import OpenAI

session_token = SyncTokenProvider(
    "https://api.islo.dev",
    os.environ["ISLO_API_KEY"],
)()

client = OpenAI(
    api_key=session_token,
    base_url="https://gateway.islo.dev/inference/openai/v1",
)

response = client.chat.completions.create(
    model="kimi-k3",
    messages=[
        {"role": "user", "content": "Say hello from Islo gateway inference."},
    ],
    max_tokens=128,
)
print(response)
```

Call the OpenAI Responses API with the same client:

```python
response = client.responses.create(
    model="kimi-k3",
    input="Say hello from Islo gateway inference.",
    max_output_tokens=128,
)
print(response)
```

## Anthropic SDK

Use the Anthropic-compatible base URL:

```text
https://gateway.islo.dev/inference/anthropic
```

For Python:

```bash
uv add anthropic
```

```python
import os

from anthropic import Anthropic
from islo.custom.auth import SyncTokenProvider

session_token = SyncTokenProvider(
    "https://api.islo.dev",
    os.environ["ISLO_API_KEY"],
)()

client = Anthropic(
    api_key=session_token,
    base_url="https://gateway.islo.dev/inference/anthropic",
)

message = client.messages.create(
    model="kimi-k3",
    max_tokens=128,
    messages=[
        {"role": "user", "content": "Say hello from Islo gateway inference."},
    ],
)
print(message)
```

For long-running Anthropic clients, create a fresh session token before the current token expires.

## Claude Agent SDK

Claude Agent SDK reads Anthropic-compatible connection settings from the process environment. Set those values from a freshly created Islo session token before constructing the client:

```bash
uv add claude-agent-sdk
```

```python
import os

from claude_agent_sdk import ClaudeAgentOptions, ClaudeSDKClient
from islo.custom.auth import SyncTokenProvider

session_token = SyncTokenProvider(
    "https://api.islo.dev",
    os.environ["ISLO_API_KEY"],
)()

os.environ["ANTHROPIC_BASE_URL"] = "https://gateway.islo.dev/inference/anthropic"
os.environ["ANTHROPIC_API_KEY"] = session_token

options = ClaudeAgentOptions(
    system_prompt="You are a helpful assistant.",
    model="kimi-k3",
    max_turns=20,
)

async with ClaudeSDKClient(options=options) as client:
    await client.query("Say hello from Islo gateway inference.")

    async for message in client.receive_response():
        if hasattr(message, "content"):
            for block in message.content:
                if hasattr(block, "text"):
                    print(block.text, end="", flush=True)
```

For long-running agents, create a fresh session token before starting a new client.

## OpenAI Agents SDK

Use the OpenAI Agents SDK with an `AsyncOpenAI` client configured for the gateway:

```bash
uv add openai-agents openai
```

```python
import asyncio
import os

from agents import Agent, OpenAIChatCompletionsModel, Runner, set_tracing_disabled
from islo.custom.auth import SyncTokenProvider
from openai import AsyncOpenAI

session_token = SyncTokenProvider(
    "https://api.islo.dev",
    os.environ["ISLO_API_KEY"],
)()

openai_client = AsyncOpenAI(
    api_key=session_token,
    base_url="https://gateway.islo.dev/inference/openai/v1",
)

set_tracing_disabled(disabled=True)

async def main() -> None:
    agent = Agent(
        name="Assistant",
        instructions="You are a helpful assistant.",
        model=OpenAIChatCompletionsModel(
            model="kimi-k3",
            openai_client=openai_client,
        ),
    )

    result = await Runner.run(agent, "Say hello from Islo gateway inference.")
    print(result.final_output)


asyncio.run(main())
```

## LangChain

Use LangChain's OpenAI chat model with the gateway base URL:

```bash
uv add langchain-openai
```

```python
import os

from islo.custom.auth import SyncTokenProvider
from langchain_openai import ChatOpenAI

session_token = SyncTokenProvider(
    "https://api.islo.dev",
    os.environ["ISLO_API_KEY"],
)()

chat = ChatOpenAI(
    api_key=session_token,
    base_url="https://gateway.islo.dev/inference/openai/v1",
    model="kimi-k3",
)

response = chat.invoke("Say hello from Islo gateway inference.")
print(response.content)
```

## Instructor

Use Instructor with an OpenAI client configured for the gateway:

```bash
uv add instructor openai pydantic
```

```python
import os

import instructor
from islo.custom.auth import SyncTokenProvider
from openai import OpenAI
from pydantic import BaseModel


class Greeting(BaseModel):
    message: str


session_token = SyncTokenProvider(
    "https://api.islo.dev",
    os.environ["ISLO_API_KEY"],
)()

openai_client = OpenAI(
    api_key=session_token,
    base_url="https://gateway.islo.dev/inference/openai/v1",
)
client = instructor.from_openai(openai_client)

greeting = client.chat.completions.create(
    model="kimi-k3",
    response_model=Greeting,
    messages=[
        {"role": "user", "content": "Say hello from Islo gateway inference."},
    ],
)
print(greeting.message)
```