Managed Language Models
Each cloud provider offers their own managed Large Language Model services. AWS offers Bedrock, GCP offers Vertex AI, and Digital Ocean offers their GenAI platform. Defang makes it easy to leverage these services in your projects.
Current Support
Provider | Managed Language Models |
---|---|
Playground | ✅ |
AWS Bedrock | ✅ |
DigitalOcean GenAI | ❌ |
GCP Vertex AI | ✅ |
Usage
In order to leverage cloud-native managed language models from your Defang services, all you need to do is add the x-defang-llm
extension to the service config and Defang will configure the approprate roles and permissions for you.
Example
Ensure you have enabled model access for the model you intend to use:
Assume you have a web service like the following, which uses the cloud native SDK, for example:
services:
app:
build:
context: .
+ x-defang-llm: true
Deploying OpenAI-compatible apps
If you already have an OpenAI-compatible application, Defang makes it easy to deploy on your favourite cloud's managed LLM service. See our OpenAI Access Gateway.
Defang has a Managed LLM sample that uses the OpenAI Access Gateway, and a Managed LLM with Docker Model Provider sample that uses a Docker Model Provider.
Managed LLM on Playground
If you are using the Managed LLM feature on Defang Playground, please note that your MODEL
(model ID) will be limited to a default model chosen by Defang.
To access a full range of models, consider using Defang BYOC.