Custom image registry

Cloud Self-Managed Community (OSS)Self-Managed Enterprise

You can optionally configure Airbyte to pull Docker images from a custom image registry rather than Airbyte's public Docker repository. In this case, Airbyte pulls both platform images (e.g. server, webapp, workload-launcher, etc.) and connector images (e.g. Postgres Source, S3 Destination, etc.) from the configured registry.

Implementing Airbyte this way has several advantages.

• Security: Private custom image registries keep images in your network, reducing the risk of external threats.
• Access control: You have more control over who can access and modify images.
• Compliance: By keeping images in a controlled environment, it's easier to prove compliance with regulatory requirements for data storage and handling.

Custom Docker connectors in your workspace that specify an image using a fully qualified domain name (for example, example.com/airbyte/your-custom-source) ignore your configured custom image registry and pull images from the domain specified by that connector.

Before you start

Set up your custom image registry. The examples in this article use GitHub, but you have many options. Here are some popular ones:

Cloud provider	Service name	Documentation
Google Cloud	Artifact Registry	Quickstart
AWS	Amazon ECR	Getting started with Amazon ECR
Azure	Container Registry	Quickstart
DockerHub	Repositories	DockerHub Quickstart
GitHub	Container Registry	Working with the Container registry
Self hosted	Open-source Docker Registry	Deploy a registry server

Custom connectors using fully qualified domain names

Custom internal connectors that specify an image using a fully qualified domain name (for example, example.com/airbyte/source-postgres) ignore your configured image registry and pull images from the domain specified by that connector.

Get a list of all Airbyte images

To get a list of Airbyte images for the latest version, use abctl.

abctl images manifest

You should see something like this:

airbyte/bootloader:1.3.1
airbyte/connector-builder-server:1.3.1
airbyte/connector-sidecar:1.3.1
airbyte/container-orchestrator:1.3.1
airbyte/cron:1.3.1
airbyte/db:1.3.1
airbyte/mc:latest
airbyte/server:1.3.1
airbyte/webapp:1.3.1
airbyte/worker:1.3.1
airbyte/workload-api-server:1.3.1
airbyte/workload-init-container:1.3.1
airbyte/workload-launcher:1.3.1
bitnami/kubectl:1.28.9
busybox:1.35
busybox:latest
curlimages/curl:8.1.1
minio/minio:RELEASE.2023-11-20T22-40-07Z
temporalio/auto-setup:1.23.0

Step 1: Customize Airbyte to use your image registry

To pull all platform and connector images from a custom image registry, add the following customization to Airbyte's values.yaml file, replacing the registry value with your own registry location.

values.yaml

global:
  image:
    registry: ghcr.io/NAMESPACE

If your registry requires authentication, you can create a Kubernetes secret and reference it in the Airbyte config:

1. Create a Kubernetes secret. In this example, you create a secret called regcred from a config file. That file contains authentication information for a private custom image registry. Learn more about Kubernetes secrets.

   kubectl create secret generic regcred \
   --from-file=.dockerconfigjson=<path/to/.docker/config.json> \
   --type=kubernetes.io/dockerconfigjson

2. Add the secret you created to your values.yaml file. In this example, you use your regcred secret to authenticate.

   values.yaml

   global:
     image:
       registry: ghcr.io/NAMESPACE
     imagePullSecrets:
       - name: regcred

Step 2: Tag and push Airbyte images

Tag and push Airbyte's images to your custom image registry.

In this example, you tag all platform images and push them all to GitHub.

abctl images manifest | xargs -L1 -I{} docker tag {} ghcr.io/NAMESPACE/{} && docker push ghcr.io/NAMESPACE/{}

You can also pull Airbyte's connector images from Docker, tag them, and push them to your custom image registry. You must do this prior to adding a source or destination.

In this example, you pull a connector from Docker, tag it, and push it to GitHub.

docker pull airbyte/destination-google-sheets:latest
docker tag airbyte/desination-google-sheets:latest ghcr.io/NAMESPACE/desination-google-sheets:latest
docker push ghcr.io/NAMESPACE/destination-google-sheets:latest    

Now, when you install Airbyte, images will come from the custom image registry you configured.