Skip to main content

Managed Language Models

Each cloud provider offers their own managed Large Language Model services. AWS offers Bedrock, GCP offers Vertex AI, Azure offers Microsoft Foundry, and Digital Ocean offers their GenAI platform. Defang makes it easy to leverage these services in your projects.

Availability

Managed LLM services are available on all tiers (Starter, Pro, Enterprise) when deploying via BYOC.

Current Support​

ProviderManaged Language Models
AWS Bedrock✅
Microsoft Foundry✅
DigitalOcean GenAI❌
GCP Vertex AI✅

Usage​

In order to leverage cloud-native managed language models from your Defang services, all you need to do is add the x-defang-llm extension to the service config and Defang will configure the approprate roles and permissions for you.

Example​

info

If you are planning to use Google Vertex AI, you must enable model access for each model you intend to use. This is not necessary on AWS Bedrock.

Assume you have a web service like the following, which uses the cloud native SDK, for example:

 services:
     app:
         build:
             context: .
+        x-defang-llm: true

Deploying OpenAI-compatible apps​

If you already have an OpenAI-compatible application, Defang makes it easy to deploy on your favourite cloud's managed LLM service. See our OpenAI Access Gateway.

tip

Defang has a Managed LLM sample that uses the OpenAI Access Gateway, and a Managed LLM with Docker Model Provider sample that uses a Docker Model Provider.