Resource-Aware Job Orchestration in Enterprise Data Platforms

Abstract

Enterprise data platforms increasingly run batch and streaming jobs on shared infrastructure, creating resource contention between deadline-bound processing and low-latency event pipelines. This article presents a resourceaware orchestration approach that profiles workload behavior, estimates resource demand, classifies SLA sensitivity, and applies statetransition logic for mixed batch–stream execution. The proposed model adjusts scheduling decisions according to platform states such as streaming pressure, batch deadline risk, memory saturation, burst ingestion, sink bottleneck, idle recovery, and failure-prone execution. Simulated results show that orchestration states change across a 24-hour workload cycle, with batch deadline risk reaching 44% during overnight processing and streaming pressure reaching 41% during midday event bursts. Resource reallocation results show that streaming lag spikes shift streaming share from 34% to 52%, while batch deadline pressure increases batch share from 39% to 51%. These findings show that mixed workload orchestration should be driven by runtime state, resource pressure, and SLA risk rather than static scheduling priority.