Airbyte 1.4 Release

Happy new year! Airbyte version 1.4.0 was released on January 17, 2025. We’re excited to share new improvements and changes to the Airbyte platform.

🚀 Platform Changes

Platform changes improve Airbyte for everyone with a self-managed instance.

Configure the schema refresh rate

At the start of a sync, Airbyte occasionally re-runs schema discovery to detect changes in your source, such as new fields or a change in column type. But running schema discovery has a performance cost, and may not always be necessary before every sync.

You may now use a new toggle, DISCOVER_REFRESH_WINDOW_MINUTES, to set the minimum number of minutes Airbyte will wait to refresh the schema for your sources. By setting a larger number, you run automatic schema detection less frequently, which can result in gains in sync performance. The default configuration in Airbyte Open Source is 1440, which refreshes schemas every 24 hours. The lowest interval you can set is 1, which refreshes schemas before every sync.

worker:
  env_vars:
    DISCOVER_REFRESH_WINDOW_MINUTES: 1440 # Airbyte automatically refreshes schemas no more than once per day (1440 minutes).

Set this to 0 to disable automatic schema refreshes. All schema refreshes will need to be done manually via the Airbyte UI or API.

worker:
  env_vars:
    DISCOVER_REFRESH_WINDOW_MINUTES: 0 # Airbyte does not automatically detect schema changes.

Connectors support custom image registries

Connectors can now use custom image registries rather than Airbyte’s public Docker registry. If you configure Airbyte to use a custom image registry, it now automatically uses that registry for connector images as well as platform images. Previously, only platform images supported this. In this example, we set Airbyte’s values.yaml file to pull all images from GitHub.

global:
  image:
    registry: ghcr.io/NAMESPACE

You must ensure copies of platform and connector images are available in your custom image registry. Learn how to set up custom image registries.

Custom Docker connectors in your workspace that specify an image using a fully qualified domain name (for example, example.com/airbyte/your-custom-source) ignore your configured custom image registry and pull images from the domain specified by that connector.

🐛 Bug fixes

• Reduced resource consumption: Instances of Airbyte running multiple connections at a time now consume fewer resources, thanks to optimizations to the airbyte-worker pod.

• Canceled syncs do not rerun until the next scheduled sync: If you previously cancelled an in-progress sync, and the next sync as configured in the connection frequency was behind schedule, Airbyte would immediately start a new sync. In practice, many users had to choose to ‘Cancel Sync’ twice in a row to stop moving data. Airbyte now automatically waits until the next scheduled sync to move data.

• Reduce rate limit errors from the airbyte-cron service: We fixed an issue reported by the community that caused excessive rate limit errors on the airbyte-cron pod when users scheduled connections using the Airbyte CRON capability.

• Autorecovery for hanging connections: Airbyte now packages a service to detect connections blocked by the unlikely event that a sync becomes stuck, and remains in a perpetual ‘in-progress’ state without moving data. This heartbeat service will detect syncs that are hanging, and automatically create a new job attempt.