Skip to main content

Managed Language Models

Each cloud provider offers their own managed Large Language Model services. AWS offers Bedrock, GCP offers Vertex, and Digital Ocean offers their GenAI platform. Defang makes it easy to leverage these services in your projects.

Usage​

In order to leverage cloud-native managed language models from your Defang services, all you need to do is add the x-defang-llm extension to the service config and Defang will configure the approprate roles and permissions for you.

Example​

Assume you have a web service like the following, which uses the cloud native SDK, for example:

 services:
     app:
         build:
             context: .
+        x-defang-llm: true

Deploying OpenAI-compatible apps​

If you already have an OpenAI-compatible application, Defang makes it easy to deploy on your favourite cloud's managed LLM service. See our OpenAI Access Gateway

Current Support​

ProviderManaged Language Models
Playground❌
AWS Bedrock✅
DigitalOcean GenAI❌
GCP Vertex❌