# Pipeline breaks from outdated dependency images in Dataflow Canonical: https://www.pointfive.co/efficiency-hub/inefficiencies/pipeline-breaks-from-outdated-dependency-images-in-dataflow-fbd63 In restricted or isolated network environments, Dataflow workers often cannot reach the public internet to download runtime dependencies. By: Damian Ohienmhen Updated: 2026-09-28 [Cloud Efficiency Hub](https://www.pointfive.co/efficiency-hub) # Pipeline Breaks from Outdated Dependency Images in Dataflow The short version In restricted or isolated network environments, Dataflow workers often cannot reach the public internet to download runtime dependencies. [Damian Ohienmhen](https://www.linkedin.com/in/damian-ohienmhen/) Specialist, Systems Engineer @ McKesson GCP service [GCP Dataflow](https://www.pointfive.co/efficiency-hub/cloud-services/gcp-dataflow) Category [Compute](https://www.pointfive.co/efficiency-hub/service-category/compute) Reference CER-0251 Type Outdated Version ## Explanation Why the waste happens and who it affects. To operate securely, organizations build custom worker images that bundle required libraries. However, these images must be manually updated to keep dependencies current. As upstream packages evolve, outdated internal images can cause pipeline errors, execution delays, or total job failures. Each failure wastes worker runtime, increases troubleshooting time, and leads to rebuild cycles that inflate operational and compute costs. ## Billing model The pricing dimensions that drive this cost. Dataflow billing is based on worker instance time, storage, and additional data transfer. Pipeline interruptions or rebuilds caused by dependency issues increase both compute cost and engineering effort, leading to inefficiency even when direct spend isn't immediately visible. ## How to detect 4 checks to find it in your estate. - Review Dataflow job error logs for dependency or package resolution failures - Identify whether pipelines rely on custom worker images with static dependency sets - Assess the frequency of image rebuilds and whether delays correlate with dependency updates in public repositories - Evaluate whether repeated pipeline restarts or failed jobs are linked to dependency mismatches ## How to fix 4 ways to remove the waste. - Implement a scheduled process to rebuild and validate custom Dataflow images using the latest stable dependencies - Maintain version tracking of bundled packages to detect when key libraries are updated upstream - Automate dependency validation and image refresh through CI/CD workflows to minimize manual effort - Evaluate whether workers can retrieve vetted dependencies through an Artifact Registry remote repository (for example, a PyPI proxy reached through Private Google Access) or through Cloud NAT, instead of relying fully on manual image maintenance; VPC Service Controls restricts access to Google Cloud APIs and does not provide internet access ## Documentation Vendor references for pricing and configuration. - [Using Custom Containers with Dataflow cloud.google.com](https://cloud.google.com/dataflow/docs/guides/using-custom-containers) - [Create remote repositories docs.cloud.google.com](https://docs.cloud.google.com/artifact-registry/docs/repositories/remote-repo) ## Related inefficiencies [Browse the library](https://www.pointfive.co/efficiency-hub) - GCP Dataflow CER-0244 ### [Idle Dataflow Workers Running After Pipeline Failure](https://www.pointfive.co/efficiency-hub/inefficiencies/idle-dataflow-workers-running-after-pipeline-failure-f6b1a) When a Dataflow pipeline fails - often due to dependency issues, misconfigurations, or data format mismatches-its worker instances may remain active temporarily until the service terminates them. In some cases, misconfigured jobs, stuck... Compute - GCP Dataflow CER-0489 ### [Missed FlexRS for Delay-Tolerant Batch Dataflow Jobs](https://www.pointfive.co/efficiency-hub/inefficiencies/missed-flexrs-for-delay-tolerant-batch-dataflow-jobs) Flexible Resource Scheduling (FlexRS) is a Dataflow option for batch pipelines that trades start time for price. Dataflow queues the job and starts it within six hours of creation, and runs it on a mix of preemptible and regular VMs with... Compute - GCP Dataflow CER-0490 ### [Exactly-Once Streaming Mode Where At-Least-Once Is Sufficient in Dataflow](https://www.pointfive.co/efficiency-hub/inefficiencies/exactly-once-streaming-mode-where-at-least-once-is-sufficient-in-dataflow) Every Dataflow streaming job runs in exactly-once mode unless configured otherwise. Exactly-once guarantees that records are not dropped or duplicated, which requires Dataflow to deduplicate and checkpoint state along the pipeline. Google... Compute --- Source: the public page above. Product screenshots and illustrative interfaces are examples, not live customer data.