OpenAI Access Gateway

Defang makes it easy to deploy on your favourite cloud's managed LLM service with our OpenAI Access Gateway. This service sits between your application and the cloud service and acts as a compatibility layer. It handles incoming OpenAI requests, translates those requests to the appropriate cloud-native API, handles the native response, and re-constructs an OpenAI-compatible response.

See our tutorial which describes how to configure the OpenAI Access Gateway for your application.

Docker Model Provider Services

As of Docker Compose v2.35 and Docker Desktop v4.41, Compose introduces a new service type called provider that allows you to declare platform capabilities required by your application. For AI models, you use the model type to declare model dependencies. This will expose an OpenAI compatible API for your service. Check the Docker Model Runner documentation for more details.

services:
  chat:
    build: .
    depends_on:
      - ai_runner

  ai_runner:
    provider:
      type: model
      options:
        model: ai/mistral
    x-defang-llm: true

Under the hood, when you use the model provider, Defang will deploy the OpenAI Access Gateway in a private network. This allows you to use the same code for both local development and cloud deployment.

The x-defang-llm extension is used to configure the appropriate roles and permissions for your service. See the Managed Language Models page for more details.

Model Mapping

Defang supports model mapping through the openai-access-gateway on AWS and GCP. This takes a model with a Docker naming convention (e.g. ai/llama3.3) and maps it to the closest matching model name on the target platform. If no such match can be found, it can fallback onto a known existing model (e.g. ai/mistral).

This can be configured through the following environment variables:

• USE_MODEL_MAPPING (default to true) - configures whether or not model mapping should be enabled.
• FALLBACK_MODEL (no default) - configure a model which will be used if model mapping fails to find a target model.

Current Support

Provider	Managed Language Models
Playground	✅
AWS Bedrock	✅
DigitalOcean GenAI	❌
GCP Vertex AI	✅