LangChain

LangChain is a framework for building LLM-powered applications using composable chains and agents. Because the Portkey AI Gateway is OpenAI-compatible, you can use it as the LLM backend in any LangChain application by setting the base_url. This gives your LangChain app automatic retries, fallbacks, load balancing, caching, and guardrails — without changing your chain logic.

Installation

Start the gateway

Local (npx)
Portkey Cloud

npx @portkey-ai/gateway

The gateway listens at http://localhost:8787/v1.

Install dependencies

pip install langchain langchain-openai portkey-ai

Basic setup

Set base_url on ChatOpenAI to point at the gateway.

from langchain_openai import ChatOpenAI

llm = ChatOpenAI(
    api_key="sk-***",          # your OpenAI API key
    base_url="http://localhost:8787/v1",
    model="gpt-4o-mini",
)

response = llm.invoke("What is the Portkey AI Gateway?")
print(response.content)

Adding routing configs via headers

Pass gateway configs and Portkey metadata through default_headers (when constructing the client) or extra_headers (per-request).

from langchain_openai import ChatOpenAI
import json

config = {
    "retry": {"attempts": 3},
    "cache": {"mode": "simple"}
}

llm = ChatOpenAI(
    api_key="sk-***",
    base_url="http://localhost:8787/v1",
    model="gpt-4o-mini",
    default_headers={
        "x-portkey-config": json.dumps(config)
    }
)

from langchain_openai import ChatOpenAI
from portkey_ai import createHeaders, PORTKEY_GATEWAY_URL

llm = ChatOpenAI(
    api_key="dummy",  # not used — Portkey handles auth
    base_url=PORTKEY_GATEWAY_URL,
    model="gpt-4o-mini",
    default_headers=createHeaders(
        api_key="YOUR_PORTKEY_API_KEY",
        virtual_key="YOUR_VIRTUAL_KEY"
    )
)

response = llm.invoke("Explain LLM routing in one sentence.")
print(response.content)

createHeaders from portkey-ai is a convenience function that produces the correct x-portkey-* header dict. You can also construct headers manually.

Using in a chain

Drop the configured llm into any LangChain chain or pipeline as usual.

from langchain_openai import ChatOpenAI
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.output_parsers import StrOutputParser
from portkey_ai import createHeaders, PORTKEY_GATEWAY_URL

llm = ChatOpenAI(
    api_key="dummy",
    base_url=PORTKEY_GATEWAY_URL,
    model="gpt-4o-mini",
    default_headers=createHeaders(
        api_key="YOUR_PORTKEY_API_KEY",
        virtual_key="YOUR_VIRTUAL_KEY"
    )
)

prompt = ChatPromptTemplate.from_template(
    "Summarize the following text in one sentence: {text}"
)

chain = prompt | llm | StrOutputParser()

result = chain.invoke({"text": "The AI Gateway routes requests to multiple LLM providers with reliability features."})
print(result)

Real-world use case: RAG pipeline with fallback

A retrieval-augmented generation pipeline that falls back to a secondary model if the primary is unavailable.

from langchain_openai import ChatOpenAI
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.output_parsers import StrOutputParser
import json

fallback_config = {
    "strategy": {"mode": "fallback"},
    "targets": [
        {
            "provider": "openai",
            "api_key": "sk-***",
            "override_params": {"model": "gpt-4o-mini"}
        },
        {
            "provider": "anthropic",
            "api_key": "sk-ant-***",
            "override_params": {"model": "claude-3-haiku-20240307"}
        }
    ]
}

llm = ChatOpenAI(
    api_key="dummy",
    base_url="http://localhost:8787/v1",
    model="gpt-4o-mini",
    default_headers={
        "x-portkey-config": json.dumps(fallback_config)
    }
)

prompt = ChatPromptTemplate.from_template(
    "Answer the question using only the context below.\n\nContext: {context}\n\nQuestion: {question}"
)

rag_chain = prompt | llm | StrOutputParser()

result = rag_chain.invoke({
    "context": "Portkey is an AI gateway that routes requests to 250+ LLMs.",
    "question": "What does Portkey do?"
})
print(result)

All requests made through LangChain are logged and traceable in the Portkey dashboard when using the hosted gateway. Use trace_id in your headers to group related requests.

Get Started

Deployment

Core Concepts

Guardrails

MCP Gateway

Integrations

Plugin Development

Installation

Basic setup

Adding routing configs via headers

Using in a chain

Real-world use case: RAG pipeline with fallback

​Installation

​Basic setup

​Adding routing configs via headers

​Using in a chain

​Real-world use case: RAG pipeline with fallback

Installation

Basic setup

Adding routing configs via headers

Using in a chain

Real-world use case: RAG pipeline with fallback