Privatemode proxy

The Privatemode proxy exposes a local HTTP endpoint that connects applications in any programming language and tools such as coding agents to the Privatemode API. It:

1. Verifies the Privatemode deployment at api.privatemode.ai, where encrypted prompts are processed by the GenAI. The verification process is described in the attestation section.
2. Transparently encrypts user prompts and decrypts responses from the Privatemode API.

The Privatemode proxy is published as a Docker image on GitHub.

Running the container

The following command starts the Privatemode proxy and exposes it on localhost port 8080:

docker run -p 127.0.0.1:8080:8080 ghcr.io/edgelesssys/privatemode/privatemode-proxy:latest

Omit 127.0.0.1: if you want to expose the port on all network interfaces.

CLI flags

To see all available CLI option flags, use:

docker run ghcr.io/edgelesssys/privatemode/privatemode-proxy:latest --help

Options

      --api-endpoint string                    The endpoint for the Privatemode API (default "api.privatemode.ai:443")
      --api-key string                         The API key for the Privatemode API. Accepts either a direct literal or a file path prefixed with '@'. If no key is set, the proxy will not authenticate with the API.
      --api-key-fallback string                When to use the configured API key for forwarded requests: never, always, or if-placeholder (only when the API key in the request is 'placeholder'). The default will change to never in a future release. (default "always")
      --cors-allowed-origins strings           Allowed browser origins (comma-separated). Empty disables CORS; '*' allows any origin.
      --disable-prompt-cache                   Use a new random salt per request instead of sharing a cache across the proxy. Request cache_salt can still be used to achieve shared caches. (default true)
      --dump-requests                          If set, the proxy dumps request and response logs to the '/requests' sub‑directory of the workspace. Leaving this flag unset disables request and response dumping.
  -h, --help                                   help for privatemode-proxy
      --listen-address string                  The IP address or hostname on which the proxy listens for incoming API requests. (default "0.0.0.0")
      --log-format string                      set logging format (json or text) (default "text")
  -l, --log-level string                       set logging level (debug, info, warn, error, or a number) (default "info")
      --manifest-path string                   The path for the manifest file. If not provided, the manifest will be read from the remote source.
      --nvidia-ocsp-allow-unknown              Whether it should be tolerated if the NVIDIA OCSP service cannot be reached. (default true)
      --nvidia-ocsp-revoked-grace-period int   The grace period (in hours) for which to accept NVIDIA attestation certificates that are revoked according to the OCSP service. Supplying a value of 0 disables the grace period, meaning that revoked certificates are rejected immediately. (default 48)
      --port string                            The port on which the proxy listens for incoming API requests. (default "8080")
      --prompt-cache-salt string               The salt used to separate prompt caches. Can either be a literal, @file, or random (one salt per proxy startup). Requires --disable-prompt-cache=false. Request cache_salt takes precedence. Literal and file salts must be valid UTF-8, at least 32 bytes and at most 128 characters, without '@', '/', '\', or NUL. (default "random")
      --tls-cert-path string                   The path to the TLS certificate. If not provided, the server will start without TLS.
      --tls-key-path string                    The path to the TLS key. If not provided, the server will start without TLS.
      --workspace string                       The path into which the binary writes files. This includes the manifest log data in the 'manifests' subdirectory. (default ".")

CORS

CORS is disabled by default. Enable browser access with --cors-allowed-origins=https://client.example, a comma-separated list of exact origins, or --cors-allowed-origins='*' for any origin. The proxy handles preflight requests automatically and allows browsers to cache successful preflight responses for up to 24 hours.

Extract a static binary

If you want to run the proxy as a binary, you can extract it from the container image. Depending on your architecture (arm64 or amd64), insert the <arch> variable below to obtain a static Linux binary like this:

containerID=$(docker create --platform linux/<arch> ghcr.io/edgelesssys/privatemode/privatemode-proxy:latest)
docker cp -L "${containerID}":/bin/privatemode-proxy ./privatemode-proxy
docker rm "${containerID}"

Outbound network traffic

When running the Privatemode proxy in an environment with restricted firewall settings, you might need to allow the following domains and ports:

• cdn.confidential.cloud:443: for fetching the latest manifest
• api.privatemode.ai:443: for communication with the Privatemode backend

Setting up TLS

If the Privatemode proxy runs on another machine and is accessed over a network, configure TLS to encrypt the traffic between the application and the proxy.

To enable TLS, provide a certificate and a private key to the container. Mount the directory containing these files into the container so the proxy can access them.

Use the following flags to specify the paths inside the container:

• --tls-cert-path: Path to the TLS certificate file.
• --tls-key-path: Path to the private key file.

For example, if your certificates are in the current directory on your host, you can mount it to /certs inside the container:

docker run \
  -p 8080:8080 \
  -v "$(pwd):/certs" \
  ghcr.io/edgelesssys/privatemode/privatemode-proxy:latest \
  --tls-cert-path /certs/server.crt \
  --tls-key-path /certs/server.key

By providing these, the Privatemode proxy will serve traffic from and to your application client via HTTPS, ensuring secure communication. If these flags aren't set, the Privatemode proxy will fall back to serving traffic over HTTP.

TLS inspection / MitM appliances

If your network uses a TLS inspection or Man-in-the-Middle (MitM) appliance, the Privatemode proxy might fail to verify the appliance-issued certificates for outbound connections to cdn.confidential.cloud or api.privatemode.ai.

In this case, mount a CA bundle into the container and point SSL_CERT_FILE to it.

Docker

Create or obtain a PEM bundle, then mount it into the container:

docker run \
  -p 8080:8080 \
  -v /path/to/ca-bundle.crt:/etc/ssl/certs/custom-ca-bundle.crt:ro \
  -e SSL_CERT_FILE=/etc/ssl/certs/custom-ca-bundle.crt \
  ghcr.io/edgelesssys/privatemode/privatemode-proxy:latest

To verify that the configuration works, restart the container and check its logs. Startup should include Coordinator manifest fetched successfully, and errors like x509: certificate signed by unknown authority should no longer appear.

API key

The Privatemode API requires authentication with the API key you received when you signed up. Providing it to the proxy with the --api-key flag lets the proxy handle authentication.

Use --api-key-fallback to control when the configured key replaces request authentication:

• never: Forward request authentication unchanged. The configured key is still used for attestation.
• always (default): Use the configured key for every forwarded request, replacing any client-provided API key.
• if-placeholder: Use the configured key only when the API key in the request is placeholder (the Authorization header is exactly Bearer placeholder). Missing headers and other API keys are forwarded unchanged.

Without a configured --api-key, all modes forward request authentication unchanged.

Using multiple API keys

If you need to support multiple API keys (e.g., for different departments or cost centers), you have two options:

1. Pass the API key in the request: Start the proxy without the --api-key flag, or use --api-key-fallback=never to reserve the configured key for attestation. The proxy will forward the Authorization header from your client's request to the Privatemode API. This allows each client to provide their own API key.

   from openai import OpenAI
   import os
   
   client = OpenAI(
       # Pass the API key here
       # A placeholder suffices when the proxy runs with --api-key.
       api_key=os.environ.get("PRIVATEMODE_API_KEY", "placeholder"),
       # Point to your local proxy instance
       base_url="http://localhost:8080/v1",
   )
   
   completion = client.chat.completions.create(
       model="gpt-oss-120b", messages=[{"role": "user", "content": "Hello Privatemode!"}]
   )

2. Run multiple proxy instances: Deploy a separate proxy instance for each API key, configuring each with a different --api-key and listening on a different port.

Prompt caching

Privatemode supports prompt caching to reduce response latency when the first part of a prompt can be reused across requests. This is especially relevant for requests with long shared context or long conversation history. For the threat model and cache-isolation design, see Prompt cache security.

Modes of operation

By default, the proxy generates a new random salt for each request without its own cache_salt, preventing cache sharing for those requests.

Set --disable-prompt-cache=false to share a prompt cache across clients of the proxy instance using one random salt per startup.

Additionally, set --prompt-cache-salt to configure the shared salt. This requires --disable-prompt-cache=false:

• A literal salt: A fixed salt shared across proxy instances and restarts.
• @file: Read a fixed salt from a file, trimming surrounding whitespace.
• random: Generate one random salt for the lifetime of the proxy, sharing a cache until restart.

Literal and file salts must be valid UTF-8, contain at least 32 bytes and at most 128 Unicode characters, and exclude @, /, \, and NUL.

Clients can always override the salt using cache_salt in each chat completion request, including when --disable-prompt-cache is set.

Security

Cache sharing is disabled by default. When enabling sharing, all clients using the same salt share a cache. For isolation between clients, use separate request salts or keep --disable-prompt-cache enabled.

The cache is stored in the Privatemode AI worker in GPU memory and encrypted CPU memory. Caches with different salts are isolated from each other.

Custom cache salts should be kept private and randomly generated. Prefer at least 256 bits of entropy, using a salt file or per-request salt. You can generate a secure salt file with openssl rand -hex 32 > cache-salt and use --disable-prompt-cache=false --prompt-cache-salt=@cache-salt.

NVIDIA OCSP

You can set the policy for handling responses of the NVIDIA OCSP using the following flags:

• nvidia-ocsp-allow-unknown: Whether the "unknown" OCSP status (i.e., OCSP is unreachable or doesn't provide information about this certificate) should be tolerated. (Default: true)
• nvidia-ocsp-revoked-grace-period: How long "revoked" OCSP responses should be accepted for after the revocation time, in hours. A value of 0 means that "revoked" OCSP responses aren't accepted at all. (Default: 48)

For a more detailed explanation of the policy, see the documentation on certificate revocation.

Proxy updates

It’s possible that an update to the Privatemode API introduces a new manifest that's incompatible with your current version of the Privatemode proxy. In such cases, you may encounter issues where the updated manifest can't be processed by the Privatemode proxy. This is known as an "unmarshaling" error.

When this happens, please update the Privatemode proxy (the Docker image) to the latest version.

In the future, we will provide documentation on how to implement automatic updates, which will help mitigate these types of issues.

HTTP proxy

You can run the Privatemode proxy behind an HTTP proxy that supports HTTP CONNECT. Set the https_proxy environment variable like this:

docker run -p 127.0.0.1:8080:8080 -e https_proxy=<proxy-address> ghcr.io/edgelesssys/privatemode/privatemode-proxy:latest

Manifest management

Whenever the Privatemode proxy verifies the Privatemode deployment, it relies on a manifest to determine whether the services should be trusted. The manifest contains fingerprints of expected configurations and states of trusted endpoints. If they differ from the actual configurations and states, the services aren't to be trusted.

By default, the manifest is managed automatically. Manual control requires extra work each time an update to the Privatemode API is rolled out.

Automatically

By default, the Privatemode proxy fetches a manifest from a file service managed by Edgeless Systems (you can get it here). Whenever validation of the Privatemode deployment fails, the Privatemode proxy fetches the latest manifest from the file service and retries validation. This allows the proxy to continue working without manual intervention, even if the deployment changes.

To ensure auditability of the enforced manifests over time, changes to the manifest are logged to the local file system. These logs serve as a transparency log, recording which manifest was used at what point in time to verify the Privatemode deployment.

The proxy writes a file called log.txt. For each manifest that's enforced by the proxy, log.txt contains a new line with the timestamp at which enforcement began and the filename of the manifest that was enforced.

log.txt and the corresponding manifests are stored in a folder manifests. You can use the CLI flag --workspace to control where the folder manifests is stored.

Mounting the workspace to the Docker host preserves this transparency log when the container is removed:

docker run -p 127.0.0.1:8080:8080 -v proxy-logs:/app/privatemode-proxy ghcr.io/edgelesssys/privatemode/privatemode-proxy:latest --workspace /app/privatemode-proxy

Manually

You can generate a manifest manually and provide the file path to the Privatemode proxy via its --manifest-path CLI flag.

Helm chart

You can use the privatemode-proxy Helm chart for easy deployment to Kubernetes.

Prerequisites

• Kubernetes 1.16+
• Helm 3+
• (Optional) Persistent Volume for workspace
• (Optional) ConfigMap for manifest file
• (Optional) TLS secret for certificates

Installation

You can install the chart with the following commands:

helm repo add edgeless https://helm.edgeless.systems/stable
helm repo update

helm install privatemode-proxy edgeless/privatemode-proxy

Configuration

API key

Store the API key in a Kubernetes secret. For example:

kubectl create secret generic privatemode-api-key --from-literal=apiKey=your-api-key

Set config.apiKey.fallback to never, always, or if-placeholder to control when the configured key replaces request authentication. The default empty value omits the flag and uses the proxy's default. See API key for the modes' behavior.

For example, to use the configured key only when the API key in the request is placeholder:

config:
  apiKey:
    fallback: 'if-placeholder'

CORS

Set config.corsAllowedOrigins to a list of exact browser origins, or ['*'] to allow any origin. The default empty list disables CORS.

config:
  corsAllowedOrigins:
    - 'https://client.example'

See CORS for preflight behavior.

Shared prompt cache

Cache sharing is disabled by default. Set config.sharedPromptCache.enabled=true to share a cache across clients of the proxy instance. Client-provided salts always take precedence.

To share prompt caches across proxy instances, store a random salt of at least 32 bytes in a Kubernetes Secret:

openssl rand -base64 32 | kubectl create secret generic privatemode-proxy-prompt-cache-salt --from-file=promptCacheSalt=/dev/stdin

Then, configure these values for your chart:

config:
  sharedPromptCache:
    enabled: true
    promptCacheSalt:
      enabled: true
      secretName: 'privatemode-proxy-prompt-cache-salt'

Persistent volume

To persist the application’s data beyond the lifetime of the current deployment, you can configure a Persistent Volume. The data includes the transparency log and manifests that allow you to audit the enforced manifests over time.

First, create a PersistentVolumeClaim:

kubectl apply -f - <<EOF
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: privatemode-proxy-pvc
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
EOF

Then, configure these values for your chart:

config:
  workspace:
    enabled: true
    volumeClaimName: "privatemode-proxy-pvc"

TLS configuration

To enable TLS for communication between your application and the Privatemode proxy, provide the TLS certificate and key through a Kubernetes secret:

You can use cert-manager to manage the TLS secret. Or you can create it manually as follows:

kubectl create secret tls privatemode-proxy-tls \
  --cert=<path-to-cert> --key=<path-to-key>

Then, configure these values for your chart:

config:
  tls:
    enabled: true
    secretName: "privatemode-proxy-tls"

TLS inspection / MitM appliance CA bundle

If your cluster reaches Privatemode through a TLS inspection appliance, create a secret that contains a PEM CA bundle:

kubectl create secret generic privatemode-proxy-custom-ca \
  --from-file=ca-bundle.crt=/path/to/ca-bundle.crt

Then, configure these values for your chart:

config:
  customCA:
    enabled: true
    secretName: "privatemode-proxy-custom-ca"
    fileName: "ca-bundle.crt"

If the bundle is already available as a ConfigMap, for example one managed by cert-manager's trust-manager, set configMapName instead. It takes precedence over secretName. Set fileName to the ConfigMap key that holds the bundle:

config:
  customCA:
    enabled: true
    configMapName: "corporate-ca-bundle"
    fileName: "ca-bundle.crt"

When enabled, the chart mounts the bundle and sets SSL_CERT_FILE for the container. The bundle is read at startup, so restart the proxy after updating it:

kubectl rollout restart deployment/privatemode-proxy-<release-name>

HTTP proxy configuration

You can run the Privatemode proxy behind an HTTP proxy that supports HTTP CONNECT. To this end, set the https_proxy environment variable for your chart:

config:
  extraEnv:
    - name: https_proxy
      value: <proxy-address>

Manifest file configuration

While manually managing manifests isn't recommended (see Manifest management), you can pass in the manifest via a ConfigMap:

Create the ConfigMap from your manifest file:

kubectl create configmap privatemode-proxy-config --from-file=manifest.json=/path/to/your/manifest.json

Then, configure these values for your chart:

config:
  manifest:
    enabled: true
    configMapName: "privatemode-proxy-config"
    fileName: "manifest.json"
    mountPath: "/etc/config/manifest.json"

Accessing the proxy

Once the deployment is complete, you can configure your application to access the API through the Privatemode proxy service’s domain.

By default, the proxy can be accessed at the following URL:

http://privatemode-proxy-privatemode-proxy.default.svc.cluster.local:8080/v1

This URL is constructed as follows:

http://privatemode-proxy-{helm-release}.{namespace}.svc.cluster.local:{port}/v1

• {helm-release}: The name of your Helm release.
• {namespace}: The Kubernetes namespace where the proxy is deployed.
• {port}: The port configured for the proxy service (default is 8080).

If you configured a custom DNS entry in your cluster, adjust the URL accordingly. Replace the default service domain with your custom domain, ensuring that your application can correctly resolve and communicate with the Privatemode proxy service.

Uninstallation

You can uninstall the chart as follows:

helm uninstall privatemode-proxy