Deploying your OpenAI Application to AWS Bedrock or GCP Vertex AI

Let's assume you have an app that uses an OpenAI client library and you want to deploy it to the cloud, either on AWS Bedrock or GCP Vertex AI.

This tutorial shows you how Defang makes it easy.

Suppose you start with a compose file like this:

services:
  app:
    build:
      context: .
    ports:
      - 3000:3000
    environment:
      OPENAI_API_KEY:
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:3000/"]

---

Add an LLM Service to Your Compose File

You need to add a new service that acts as a proxy between your app and the backend LLM provider (Bedrock or Vertex).

Add Defang's openai-access-gateway service:

+  llm:
+    image: defangio/openai-access-gateway
+    x-defang-llm: true
+    ports:
+      - target: 80
+        published: 80
+        mode: host
+    environment:
+      - OPENAI_API_KEY
+      - GCP_PROJECT_ID # if using GCP Vertex AI
+      - GCP_REGION # if using GCP Vertex AI, AWS_REGION not necessary for Bedrock

Notes:

• The container image is based on aws-samples/bedrock-access-gateway, with enhancements.
• x-defang-llm: true signals to Defang that this service should be configured to use target platform AI services.
• New environment variables:
  • GCP_PROJECT_ID and GCP_REGION are needed if using Vertex AI. (e.g. GCP_PROJECT_ID = my-project-456789 and GCP_REGION = us-central1)

tip

OpenAI Key

You no longer need your original OpenAI API Key.
We recommend generating a random secret for authentication with the gateway:

defang config set OPENAI_API_KEY --random

---

Redirect Application Traffic

Modify your app service to send API calls to the openai-access-gateway:

 services:
   app:
     ports:
       - 3000:3000
     environment:
       OPENAI_API_KEY:
+      OPENAI_BASE_URL: "http://llm/api/v1"
     healthcheck:
       test: ["CMD", "curl", "-f", "http://localhost:3000/"]

Now, all OpenAI traffic will be routed through your gateway service and onto AWS Bedrock or GCP Vertex.

---

Selecting a Model

You should configure your application to specify the model you want to use.

 services:
   app:
     ports:
       - 3000:3000
     environment:
       OPENAI_API_KEY:
       OPENAI_BASE_URL: "http://llm/api/v1"
+      MODEL: "anthropic.claude-3-sonnet-20240229-v1:0" # for Bedrock
+      # MODEL: "google/gemini-2.5-pro-preview-03-25" # for Vertex AI
     healthcheck:
       test: ["CMD", "curl", "-f", "http://localhost:3000/"]

Choose the correct MODEL depending on which cloud provider you are using.

info

Choosing the Right Model

• For AWS Bedrock, use a Bedrock model ID (e.g., anthropic.claude-3-sonnet-20240229-v1:0) See available Bedrock models.
• For GCP Vertex AI, use a full model path (e.g., google/gemini-2.5-pro-preview-03-25) See available Vertex models

Complete Example Compose File

services:
  app:
    build:
      context: .
    ports:
      - 3000:3000
    environment:
      OPENAI_API_KEY:
      OPENAI_BASE_URL: "http://llm/api/v1"
      MODEL: "anthropic.claude-3-sonnet-20240229-v1:0" # or your Vertex AI model path
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:3000/"]

  llm:
    image: defangio/openai-access-gateway
    x-defang-llm: true
    ports:
      - target: 80
        published: 80
        mode: host
    environment:
      - OPENAI_API_KEY
      - GCP_PROJECT_ID     # required if using Vertex AI
      - GCP_REGION         # required if using Vertex AI

---

Environment Variable Matrix

Variable	AWS Bedrock	GCP Vertex AI
GCP_PROJECT_ID	(not used)	Required
GCP_REGION	(not used)	Required
MODEL	Bedrock model ID	Vertex model path

---

You now have a single app that can:

• Talk to AWS Bedrock or GCP Vertex AI
• Use the same OpenAI-compatible client code
• Easily switch cloud providers by changing a few environment variables