Explanation
Why the waste happens and who it affects.
In some cases, misconfigured jobs, stuck retries, or delayed monitoring can cause workers to continue running for extended periods. These idle workers consume vCPU, memory, and storage resources without performing useful work. The inefficiency is compounded in large or high-frequency batch environments where repeated failures can leave many orphaned workers running concurrently.
Billing model
The pricing dimensions that drive this cost.
Dataflow charges for the compute time of active workers, as well as associated resources such as persistent disks and networking. If pipeline failures prevent graceful shutdown or cleanup, these workers continue incurring compute charges even though no processing occurs.
How to detect
4 checks to find it in your estate.
- Review Dataflow job logs for failed or cancelled jobs that show prolonged worker activity afterward
- Assess cost and utilization metrics to identify worker instances continuing to accrue compute charges after pipeline termination
- Evaluate the frequency of job restarts or repeated retries that result in idle worker time
- Confirm whether monitoring and alerting mechanisms detect failed or stalled pipelines promptly
How to fix
4 ways to remove the waste.
- Implement automated job monitoring and alerting to detect pipeline failures and trigger termination of orphaned workers
- Use the max_workflow_runtime_walltime_seconds service option to stop jobs that run longer than expected, and use dead-letter queues so streaming pipelines, which retry failing work items indefinitely, do not stall (batch jobs fail after a bundle fails four times; retry counts are not configurable)
- Regularly review active Dataflow jobs and terminate those that are stuck, failed, or idle
- Use error handling and pipeline health checks to ensure worker cleanup occurs after job failures
Documentation
Vendor references for pricing and configuration.