Skip to main content

Managed Language Models

Each cloud provider offers their own managed Large Language Model services. AWS offers Bedrock, GCP offers Vertex AI, and Digital Ocean offers their GenAI platform. Defang makes it easy to leverage these services in your projects.

Current Support​

ProviderManaged Language Models
Playground✅
AWS Bedrock✅
DigitalOcean GenAI❌
GCP Vertex AI✅

Usage​

In order to leverage cloud-native managed language models from your Defang services, all you need to do is add the x-defang-llm extension to the service config and Defang will configure the approprate roles and permissions for you.

Example​

info

Ensure you have enabled model access for the model you intend to use:

Assume you have a web service like the following, which uses the cloud native SDK, for example:

 services:
     app:
         build:
             context: .
+        x-defang-llm: true

Deploying OpenAI-compatible apps​

If you already have an OpenAI-compatible application, Defang makes it easy to deploy on your favourite cloud's managed LLM service. See our OpenAI Access Gateway.

tip

Defang has a Managed LLM sample that uses the OpenAI Access Gateway, and a Managed LLM with Docker Model Provider sample that uses a Docker Model Provider.

Managed LLM on Playground​

If you are using the Managed LLM feature on Defang Playground, please note that your MODEL (model ID) will be limited to a default model chosen by Defang. To access a full range of models, consider using Defang BYOC.