Building a Task Orchestration Engine
Too many times I’ve seen workflows fail because tasks ran in the wrong order, database migrations kicking in before the schema check, services booting before the database was ready, or build steps firing out of sequence. These aren’t just deployment headaches; they’re symptoms of a more general problem: coordinating dependent tasks efficiently.
I wanted a clean, reusable way to declare dependencies once and let the runtime figure out the rest. Surprisingly, Rust didn’t have a general-purpose library for this. So I built one: when2task, a dependency-aware task executor for async Rust.
The Problem Space
Most systems have tasks that depend on other tasks. Some examples:
Build systems: Compile utils before core, compile core before app
Deployment scripts: Start database before API server, start auth service before web frontend
Data pipelines: Extract data, then validate, then transform, then load
The naive solution is to run everything sequentially:
Sequential (8 min)
A(2m) → B(3m) → C(1m) → D(2m)
But if Task A and B are independent, we're wasting time. The optimal approach requires coordination:
Optimal (6 min)
A(2m) ─┐
├─> C(1m) → D(2m)
B(3m) ─┘
Why Existing Solutions Don't Work
Other ecosystems solve this with build systems or orchestration frameworks, but in Rust you’d usually end up hand-rolling a DAG (Directed Acyclic Graph) executor. I wanted something different: lightweight, async-native, ergonomic, no config files, no boilerplate, just a clean way to declare dependencies and run tasks.
That meant four things:
Declare dependencies once, get optimal execution
Zero configuration
Work with async Rust code
Catch dependency cycles at compile time
The when2task Approach
The core insight is that task scheduling with dependencies is just topological sorting. You build a DAG of tasks, then execute them in topological order with maximum parallelization.
Basic Usage
use when2task::{ExecutionMode, Task, TaskExecutor};
use std::time::Duration;
fn work(
start_msg: &'static str,
secs: u64,
done_msg: &'static str,
ok: &'static str,
) -> impl Future<Output = Result<&'static str, &'static str>> {
async move {
println!("{start_msg}");
tokio::time::sleep(Duration::from_secs(secs)).await;
println!("{done_msg}");
Ok::<_, &str>(ok)
}
}
#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
let setup_db = Task::new_independent(work(
"Setting up database...",
2,
"Database ready",
"db_ready",
));
let download_assets = Task::new_independent(work(
"Downloading assets...",
3,
"Assets ready",
"assets_ready",
));
let start_server = Task::new(
work("Starting server...", 1, "Server ready", "server_ready"),
*setup_db.id(),
);
let run_tests = Task::new(
work("Running tests...", 2, "Tests complete", "tests_done"),
[*start_server.id(), *download_assets.id()],
);
let result = TaskExecutor::new(ExecutionMode::true_async())
.insert(setup_db)
.insert(download_assets)
.insert(start_server)
.insert(run_tests)
.execute()
.await?;
println!(
"Completed {}/{} tasks",
result.successful_tasks, result.total_tasks
);
Ok(())
}
Note that the example code might not be up to date with the dependency, refer the dependency’s README at https://crates.io/crates/when2task for latest code.
Execution Flow
The library automatically figures out the optimal execution plan:
Step 1: [setup_db, download_assets] run in parallel (3 seconds)
Step 2: [start_server] runs after setup_db (1 second)
Step 3: [run_tests] runs after both start_server and download_assets (2 seconds)
Total: 6 seconds instead of 8 sequential
Error Handling
The library distinguishes between setup errors and runtime errors:
match executor.execute().await {
Ok(result) => {
if result.has_failures() {
println!("Some tasks failed:");
for failed in result.failed_results() {
println!(" {} failed: {:?}", failed.task_id, failed.result);
}
}
}
Err(when2task::ExecutionError::BlueprintError(
when2task::BlueprintError::CircularDependency(cycle)
)) => {
eprintln!("Circular dependency detected: {:?}", cycle);
}
Err(e) => eprintln!("Execution failed: {}", e),
}
Design Decisions
Why Topological Sorting?
It's the proven algorithm for dependency resolution. Every build system uses some variant of this. The innovation isn't the algorithm - it's the ergonomic API and async integration.
Execution Modes
For now, the library supports two execution modes:
TaskExecutor::new(ExecutionMode::true_async())
TaskExecutor::new(ExecutionMode::pseudo_async(tokio::spawn))
Use spawned execution if you need task isolation or cancellation support.
Setup: O(V + E) to build the execution plan (V = tasks, E = dependencies)
Memory: O(V) for task storage, plus whatever your tasks allocate
Execution: Bounded by the critical path, not total task count
For typical workloads, the library overhead is negligible compared to actual task execution time.
When to Use This
Good fit:
Not a good fit:
PS
Building when2task was refreshing. It turned chaotic workflows into something orderly. For me, the fun wasn’t just solving the problem, but shaping the solution: the three-level design, the architecture, and the API all reflect the programming taste I’ve been refining over the years.
Sources code
The source code is on GitHub and the crate is on crates.io. MIT licensed, because coordination should be free.
Written by a developer who got tired of tasks running in the wrong order.
