> ## Documentation Index
> Fetch the complete documentation index at: https://portkey-docs-mintlify-vertex-ai-context-caching-docs-39214.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Universal API

> Portkey's Universal API provides a consistent interface to integrate a wide range of modalities (text, vision, audio) and LLMs (hosted OR local) into your apps.

<Info>
  This feature is available on all Portkey plans.
</Info>

So, instead of maintaining separate integrations for different multimodal LLMs, you can interact with models from OpenAI, Anthropic, Meta, Cohere, Mistral, and many more (100+ models, 15+ providers) - all using a common, unified API signature.

## Portkey Follows OpenAI Spec

Portkey API is powered by its [battle-tested open-source AI Gateway](https://github.com/portkey-ai/gateway), which converts all incoming requests to the OpenAI signature and returns OpenAI-compliant responses.

## Switching Providers is a Breeze

<Tabs>
  <Tab title="Node">
    ```JS theme={null}
    import Portkey from 'portkey-ai';

    // Calling OpenAI
    const portkey = new Portkey({
      provider: "openai",
      Authorization: "Bearer sk-xxxxx"
    })

    const response = await portkey.chat.completions.create({
      messages: [{ role: 'user', content: 'Hello' }],
      model: 'gpt-4',
    });

    // Swithing to Anthropic
    const portkey = new Portkey({
      provider: "anthropic",
      Authorization: "Bearer sk-ant-xxxxx"
    })

    const response = await portkey.chat.completions.create({
      messages: [{ role: 'user', content: 'Hello' }],
      model: 'claude-3-opus-20240229',
    });
    ```
  </Tab>

  <Tab title="Python">
    ```py theme={null}
    from portkey_ai import Portkey

    # Calling OpenAI
    portkey = Portkey(
      provider = "openai",
      Authorization = "sk-xxxxx"
    )

    response = portkey.chat.completions.create(
      messages = [{ "role": 'user', "content": 'Hello' }],
      model = 'gpt-4'
    )

    # Switching to Anthropic
    portkey = Portkey(
      provider = "anthropic",
      Authorization = "sk-ant-xxxxx"
    )

    response = portkey.chat.completions.create(
      messages = [{ "role": 'user', "content": 'Hello' }],
      model = 'claude-3-opus-20240229'
    )
    ```
  </Tab>
</Tabs>

## Integrating Local or Private Models

Portkey can also route to and observe your locally or privately hosted LLMs, as long as the model is compliant with one of the 15+ providers supported by Portkey and the URL is exposed publicly.

Simply specify the `custom_host` parameter along with the `provider` name, and Portkey will handle the communication with your local model.

<Tabs>
  <Tab title="NodeJS">
    ```js theme={null}
    import Portkey from 'portkey-ai';

    const portkey = new Portkey({
        apiKey: "PORTKEY_API_KEY",
        provider: "mistral-ai",
        customHost: "http://MODEL_URL/v1/" // Point Portkey to where the model is hosted
    })

    async function main(){
        const response = await portkey.chat.completions.create({
            messages: [{ role: 'user', content: '1729' }],
            model: 'mixtral-8x22b',
        });
        console.log(response)
    }

    main()
    ```
  </Tab>

  <Tab title="Python">
    ```py theme={null}
    from portkey_ai import Portkey

    portkey = Portkey(
        api_key="PORTKEY_API_KEY",
        provider="mistral-ai",
        custom_host="http://MODEL_URL/v1/" # Point Portkey to where the model is hosted
    )

    chat = portkey.chat.completions.create(
        messages = [{ "role": 'user', "content": 'Say this is a test' }],
        model="mixtral-8x22b"
    )

    print(chat)
    ```
  </Tab>

  <Tab title="cURL">
    ```sh theme={null}
    curl https://api.portkey.ai/v1/chat/completions \
      -H "Content-Type: application/json" \
      -H "x-portkey-api-key: $PORTKEY_API_KEY" \
      -H "x-portkey-provider: mistral-ai" \
      -H "x-portkey-custom-host: http://MODEL_URL/v1/" \
      -d '{
        "model": "mixtral-8x22b",
        "messages": [{ "role": "user", "content": "Say this is a test" }]
      }'
    ```
  </Tab>
</Tabs>

<Info>
  **Note:**

  When using `custom_host`, include the version identifier (e.g., `/v1`) in the URL. Portkey will append the actual endpoint path (`/chat/completions`, `/completions`, or `/embeddings`) automatically. (For Ollama models, this works differently. [Check here](/integrations/llms/ollama))
</Info>

## Powerful Routing and Fallback Strategies

With Portkey you can implement sophisticated routing and fallback strategies. Route requests to different providers based on various criteria, loadbalance them, set up retries or fallbacks to alternative models in case of failures or resource constraints.

Here's an example config where we set up a fallback from OpenAI to a locally hosted Llama3 on Ollama:

```py theme={null}
config = {
	"strategy": { "mode": "loadbalance" },
	"targets": [
		{
			"provider": "openai",
			"api_key": "xxx",
			"weight": 1,
			"override_params": { "model": "gpt-3.5-turbo" }
		},
		{
			"provider": "mistral-ai",
			"custom_host": "http://MODEL_URL/v1/",
			"weight": 1,
			"override_params": { "model": "mixtral-8x22b" }
		}
	]
}

from portkey_ai import Portkey

portkey = Portkey(
    api_key="PORTKEY_API_KEY",
    config=config
)
```

## Using the  Anthropic's /messages Route

Access models on [Anthropic](/integrations/llms/anthropic), [Bedrock](/integrations/llms/bedrock) and [Vertex AI](/integrations/llms/vertex-ai) through Anthropic's native`/messages` endpoint using Portkey's SDK or Anthropic's SDK.

<Note>
  This route only works with Claude models on Anthropic, Bedrock and Vertex AI. For other models, use the standard OpenAI compliant endpoint.
</Note>

<Tabs>
  <Tab title="cURL">
    ```sh theme={null}
    curl --location 'https://api.portkey.ai/v1/messages' \
    --header 'x-portkey-provider: @your-provider-slug' \
    --header 'Content-Type: application/json' \
    --header 'x-portkey-api-key: YOUR_PORTKEY_API_KEY' \
    --data '{
        "model": "your-model-name",
        "max_tokens": 250,
        "messages": [
            {
                "role": "user",
                "content": "Hello, Claude"
            }
        ]
    }'
    ```
  </Tab>

  <Tab title="Python SDK">
    ```python theme={null}
        Coming Soon!
    ```
  </Tab>

  <Tab title="NodeJS SDK">
    ```javascript theme={null}
        Coming Soon!
    ```
  </Tab>

  <Tab title="Anthropic Python SDK">
    ```python Anthropic Python SDK theme={null}
      import anthropic

      client = anthropic.Anthropic(
          api_key="dummy", # we will use portkey's provider slug
          default_headers={"x-portkey-api-key": "YOUR_PORTKEY_API_KEY"},
          base_url="https://api.portkey.ai"
      )
      message = client.messages.create(
          model="@your-provider-slug/your-model-name",
          max_tokens=250,
          messages=[
              {"role": "user", "content": "Hello, Claude"}
          ],
      )
      print(message.content)
    ```
  </Tab>

  <Tab title="Anthropic TypeScript SDK">
    ```typescript Anthropic TS SDK theme={null}
       import Anthropic from '@anthropic-ai/sdk';

       const anthropic = new Anthropic({
         apiKey: 'dummy', // we will use portkey's provider slug
         baseURL: "https://api.portkey.ai",
         defaultHeaders: { "x-portkey-api-key": "YOUR_PORTKEY_API_KEY" }
       });

       const msg = await anthropic.messages.create({
         model: "@your-provider-slug/your-model-name",
         max_tokens: 1024,
         messages: [{ role: "user", content: "Hello, Claude" }],
       });
       console.log(msg);
    ```
  </Tab>
</Tabs>

## Multimodality

Portkey integrates with multimodal models through the same unified API and supports vision, audio, image generation, and more capabilities across providers.

[Multimodal Capabilities](/product/ai-gateway/multimodal-capabilities)

## Supported Endpoints

Portkey's Universal API supports a comprehensive range of endpoints across all major AI capabilities. Each endpoint follows the OpenAI specification while working seamlessly with 15+ providers.

<Info>
  Not all providers support every endpoint. Check our [provider compatibility matrix](/api-reference/inference-api/supported-providers) to see which endpoints are available for each provider.
</Info>

### Core Endpoints

* **[Chat Completions](/api-reference/inference-api/chat)** - An OpenAI compatible unified endpoint for generating responses with support for streaming, function calling, and multi-modal inputs across 50+ providers like OpenAI, Anthropic, Vertex AI, Bedrock, and more
* **[Responses](/api-reference/inference-api/responses)** - An OpenAI compatible unified endpoint using the Responses API format. Works with **all 70+ providers**.
* **[Messages API](/integrations/llms/anthropic#using-messages-route-to-call-anthropics-api)** - An Anthropic-compatible unified endpoint for generating responses with support for streaming, function calling, and multi-modal inputs across providers like Anthropic, Vertex AI, Bedrock, and more.
* **[Images](/api-reference/inference-api/images)** - Generate, edit, and create variations of images using models like DALL-E, Stable Diffusion, and others
* **[Audio](/api-reference/inference-api/audio)** - Convert speech to text (transcription) and text to speech across multiple languages

### Advanced Capabilities

* **[Fine-tuning](/product/ai-gateway/fine-tuning)** - Customize models on your specific datasets
* **[Batch Processing](/product/ai-gateway/batches)** - Process large volumes of requests efficiently
* **[Files](/product/ai-gateway/files)** - Upload and manage files for fine-tuning and batch operations
* **[Moderations](/api-reference/inference-api/moderations)** - Check content for safety and compliance

### Additional Endpoints

* **[Gateway to other APIs](/api-reference/inference-api/gateway-for-other-apis)** - Gateway to other APIs
* **[Assistants API](/api-reference/inference-api/assistants-api/assistants/create-assistant)** - OpenAI assistants with persistent threads and file handling
* **[Completions](/api-reference/inference-api/completions)** - Legacy text completion endpoint for backward compatibility