Too many times I’ve seen workflows fail because tasks ran in the wrong order, database migrations kicking in before the schema check, services booting before the database was ready, or build steps firing out of sequence. These aren’t just deployment headaches; they’re symptoms of a more general problem: coordinating dependent tasks efficiently.

I wanted a clean, reusable way to declare dependencies once and let the runtime figure out the rest. Surprisingly, Rust didn’t have a general-purpose library for this. So I built one: when2task, a dependency-aware task executor for async Rust.

The Problem Space

Most systems have tasks that depend on other tasks. Some examples:

• Build systems: Compile utils before core, compile core before app

• Deployment scripts: Start database before API server, start auth service before web frontend

• Data pipelines: Extract data, then validate, then transform, then load

The naive solution is to run everything sequentially:

Sequential (8 min)
A(2m) → B(3m) → C(1m) → D(2m)

But if Task A and B are independent, we're wasting time. The optimal approach requires coordination:

Optimal (6 min)
 A(2m) ─┐
        ├─> C(1m) → D(2m)
 B(3m) ─┘

Why Existing Solutions Don't Work

Other ecosystems solve this with build systems or orchestration frameworks, but in Rust you’d usually end up hand-rolling a DAG (Directed Acyclic Graph) executor. I wanted something different: lightweight, async-native, ergonomic, no config files, no boilerplate, just a clean way to declare dependencies and run tasks.

That meant four things:

• Declare dependencies once, get optimal execution

• Zero configuration

• Work with async Rust code

• Catch dependency cycles at compile time

The when2task Approach

The core insight is that task scheduling with dependencies is just topological sorting. You build a DAG of tasks, then execute them in topological order with maximum parallelization.

Basic Usage

use when2task::{ExecutionMode, Task, TaskExecutor};
use std::time::Duration;

// Small helper: create the async work future once,
// reuse it for independent and dependent tasks.
fn work(
    start_msg: &'static str,
    secs: u64,
    done_msg: &'static str,
    ok: &'static str,
) -> impl Future<Output = Result<&'static str, &'static str>> {
    async move {
        println!("{start_msg}");
        tokio::time::sleep(Duration::from_secs(secs)).await;
        println!("{done_msg}");
        Ok::<_, &str>(ok)
    }
}

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    // Independent tasks
    let setup_db = Task::new_independent(work(
        "Setting up database...",
        2,
        "Database ready",
        "db_ready",
    ));

    let download_assets = Task::new_independent(work(
        "Downloading assets...",
        3,
        "Assets ready",
        "assets_ready",
    ));

    // Dependent tasks
    let start_server = Task::new(
        work("Starting server...", 1, "Server ready", "server_ready"),
        *setup_db.id(), // depends on DB only
    );

    let run_tests = Task::new(
        work("Running tests...", 2, "Tests complete", "tests_done"),
        [*start_server.id(), *download_assets.id()], // depends on both
    );

    let result = TaskExecutor::new(ExecutionMode::true_async())
        .insert(setup_db)
        .insert(download_assets)
        .insert(start_server)
        .insert(run_tests)
        .execute()
        .await?;

    println!(
        "Completed {}/{} tasks",
        result.successful_tasks, result.total_tasks
    );
    Ok(())
}

Note that the example code might not be up to date with the dependency, refer the dependency’s README at https://crates.io/crates/when2task for latest code.

Execution Flow

The library automatically figures out the optimal execution plan:

Step 1: [setup_db, download_assets] run in parallel (3 seconds)
Step 2: [start_server] runs after setup_db (1 second)
Step 3: [run_tests] runs after both start_server and download_assets (2 seconds)

Total: 6 seconds instead of 8 sequential

Error Handling

The library distinguishes between setup errors and runtime errors:

match executor.execute().await {
    Ok(result) => {
        if result.has_failures() {
            println!("Some tasks failed:");
            for failed in result.failed_results() {
                println!("  {} failed: {:?}", failed.task_id, failed.result);
            }
        }
    }
    Err(when2task::ExecutionError::BlueprintError(
        when2task::BlueprintError::CircularDependency(cycle)
    )) => {
        eprintln!("Circular dependency detected: {:?}", cycle);
        // This is caught at setup time, not runtime
    }
    Err(e) => eprintln!("Execution failed: {}", e),
}

Design Decisions

Why Topological Sorting?

It's the proven algorithm for dependency resolution. Every build system uses some variant of this. The innovation isn't the algorithm - it's the ergonomic API and async integration.

Execution Modes

For now, the library supports two execution modes:

// Direct execution - tasks run on the current runtime
TaskExecutor::new(ExecutionMode::true_async())

// Spawned execution - tasks run on separate spawned tasks
TaskExecutor::new(ExecutionMode::pseudo_async(tokio::spawn))

Use spawned execution if you need task isolation or cancellation support.

Performance Characteristics

• Setup: O(V + E) to build the execution plan (V = tasks, E = dependencies)

• Memory: O(V) for task storage, plus whatever your tasks allocate

• Execution: Bounded by the critical path, not total task count

For typical workloads, the library overhead is negligible compared to actual task execution time.

When to Use This

Good fit:

• You have tasks with dependencies that could run in parallel

• You want optimal execution without manual coordination

Not a good fit:

• Tasks have no dependencies (just use tokio::try_join!)

• You want dynamic dependency discovery at runtime

PS

Building when2task was refreshing. It turned chaotic workflows into something orderly. For me, the fun wasn’t just solving the problem, but shaping the solution: the three-level design, the architecture, and the API all reflect the programming taste I’ve been refining over the years.

Sources code

The source code is on GitHub and the crate is on crates.io. MIT licensed, because coordination should be free.

Written by a developer who got tired of tasks running in the wrong order.

What is JSON Lines (JSONL)?

Before diving into my library, let's quickly cover what JSONL actually is. JSON Lines is a text format where each line is a valid JSON object, separated by newline characters. It looks like this:

{"id": 1, "name": "Alice", "age": 30}
{"id": 2, "name": "Bob", "age": 25}
{"id": 3, "name": "Charlie", "age": 35}

Why JSONL is Awesome

1. Streamable: You can process one line at a time without loading the entire file into memory

2. Appendable: Easy to add new records to the end of a file

3. Fault-tolerant: One corrupted line doesn't break the entire dataset

4. Tool-friendly: Works great with unix tools like cat, head, tail, and grep

5. Language-agnostic: Supported across virtually every programming language

These advantages make JSONL perfect for log files, data pipelines, and any scenario where you're dealing with large volumes of structured data.

Enter async-jsonl: Built for Performance and Simplicity

While working on various Rust projects that needed to process large JSONL files, I kept running into the same issues:

• Existing solutions weren't async-first

• Memory usage would explode with large files

• Error handling was either too strict (one bad line kills everything) or too loose

• Type safety was often sacrificed for convenience

So I built async-jsonl to solve these problems. Here's what makes it special:

🚀 Async/Await from the Ground Up

Built on Tokio, async-jsonl is designed for high-performance async I/O. No blocking the event loop here!

use async_jsonl::Jsonl;
use futures::StreamExt;

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    let jsonl = Jsonl::from_path("data.jsonl").await?;

    let lines: Vec<_> = jsonl.collect().await;
    for line_result in lines {
        let line = line_result?;
        println!("Raw JSON: {}", line);
    }

    Ok(())
}

💾 Memory Efficient Streaming

Process files of any size without loading everything into memory. The streaming approach means you can handle gigabyte files with minimal RAM usage.

use async_jsonl::{Jsonl, JsonlDeserialize};
use futures::StreamExt;
use serde::Deserialize;

#[derive(Debug, Deserialize)]
struct LogEntry {
    timestamp: String,
    level: String,
    message: String,
}

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    let jsonl = Jsonl::from_path("huge_log_file.jsonl").await?;
    let mut stream = jsonl.deserialize::<LogEntry>();

    // Process one record at a time - O(1) memory usage!
    while let Some(entry_result) = stream.next().await {
        let entry = entry_result?;
        if entry.level == "ERROR" {
            println!("🚨 Error at {}: {}", entry.timestamp, entry.message);
        }
    }

    Ok(())
}

🔒 Type-Safe with Serde Integration

Full integration with serde means you get compile-time type checking and zero-cost deserialization.

use async_jsonl::{Jsonl, JsonlDeserialize};
use futures::StreamExt;
use serde::Deserialize;

#[derive(Debug, Deserialize)]
struct User {
    id: u64,
    name: String,
    email: String,
    #[serde(default)]
    is_verified: bool,
}

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    let jsonl = Jsonl::from_path("users.jsonl").await?;
    let users = jsonl.deserialize::<User>();

    let results: Vec<_> = users.collect().await;
    for user_result in results {
        let user = user_result?;
        println!("{:?}", user);
    }

    Ok(())
}

🛡️ Resilient Error Handling

One of my favorite features: the library continues processing even when individual lines fail to parse. Perfect for real-world data that's not always perfect.

use async_jsonl::{Jsonl, JsonlValueDeserialize};
use futures::StreamExt;

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    let jsonl = Jsonl::from_path("messy_data.jsonl").await?;
    let mut stream = jsonl.deserialize_values();

    let mut valid_records = 0;
    let mut error_count = 0;

    while let Some(result) = stream.next().await {
        match result {
            Ok(value) => {
                valid_records += 1;
                // Process valid JSON
                println!("Valid record: {}", value);
            }
            Err(e) => {
                error_count += 1;
                eprintln!("Skipping invalid line: {}", e);
                // Keep going!
            }
        }
    }

    println!("Processed {} valid records, {} errors", valid_records, error_count);
    Ok(())
}

📖 Flexible Input Sources

Whether you're reading from files, memory, or any AsyncRead source, async-jsonl has you covered.

use async_jsonl::Jsonl;
use std::io::Cursor;

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    let data = r#"{"id": 1, "name": "Alice"}
{"id": 2, "name": "Bob"}
{"id": 3, "name": "Charlie"}
"#;

    // Read from memory
    let reader = Cursor::new(data.as_bytes());
    let jsonl = Jsonl::new(reader);

    // Or read from a file
    let file_jsonl = Jsonl::from_path("data.jsonl").await?;

    Ok(())
}

🎯 Advanced Features for Power Users

The library also includes some advanced features I found myself needing repeatedly:

Line Counting Without Full Processing:

use async_jsonl::{Jsonl, JsonlReader};

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    let jsonl = Jsonl::from_path("massive_file.jsonl").await?;
    let count = jsonl.count().await?;

    println!("File contains {} records", count);
    Ok(())
}

The Technical Journey

Building this library taught me a lot about async Rust and streaming APIs. Some key decisions I made:

1. Tokio as the Foundation: Built on tokio::io for maximum compatibility with the async ecosystem

2. Stream-based Architecture: Using futures's Stream trait for composable, lazy processing

3. Zero-copy Where Possible: Minimizing allocations while maintaining safety

4. Comprehensive Error Types: Using anyhow for ergonomic error handling without sacrificing information

Real-World Performance

In my testing with large datasets (>1GB JSONL files), async-jsonl consistently shows:

• Memory usage: Constant regardless of file size

• Throughput: Competitive with the fastest JSONL parsers in any language

• CPU efficiency: Low overhead thanks to Rust's zero-cost abstractions

Try It Yourself!

The library is available on crates.io and the source is on GitHub. Here's how to get started:

[dependencies]
async-jsonl = "0.3.1"
tokio = { version = "1.0", features = ["full"] }
futures = "0.3"
serde = { version = "1.0", features = ["derive"] }
anyhow = "1.0"

I'd love to hear how you use it in your projects! Whether you're processing logs, handling data pipelines, or working with APIs, async-jsonl is designed to make your life easier.

What do you think? Have you worked with JSONL files before? What features would you find most useful in a JSONL processing library? Let me know in the comments below!

I am Sandipsinh Rathod, a 2nd year student at Nirma University pursuing Bachelor's degree in Computer Science.

What is Rust?

Rust is a fairly new programming language, first stable version was released in May 2015. It is rapidly gaining popularity and it's been most admired language for 8 years in a row.

How I feel about Rust

I have been practicing Rust since 2021, the learning curve for Rust is indeed quite steep but I can say that it's all worth it. Rust forces to "unlearn" terrible practices that you might have learn from other languages and it forces you to write some better code! Yes, Rust requires a lot of practice, which may lead you to lose friends; but hey you will definitely have Rust's compiler as your new friend!

The contribution and hackathon phase

I never focused on money and I just like exploring stuff.. Past year, in Oct. 2023, I was looking for some repository to contribute for hacktoberfest and I stumbled upon this awesome repository called Tailcall. It not only had a lot of issues open but also bounties on it! Which means that I can actually earn while contributing to the repo. Win-win situation right?

My first contribution to Tailcall was to bring HTTP support for reading configuration files (#609) where I made my first ~$150 in open-source contributions.

Then a few months passed by and I had been contributing to Tailcall all that time, and I got a big opportunity, where Tailcall hosted a mini hackathon to bring WASM compatibility to the project with MASSIVE $2000 bounty on it (#750). Bringing WASM compatibility to a CLI application was huge challenge, it took me 3 days to make my first working PR (Pull Request) which make Tailcall compatible with Cloudflare Workers (i.e. WASM compatible).

Of course, 3 days of work wasn't enough, the code worked but it was huge mess. Luckily, at that period of time, my university announced that my batch had to do mandatory internship in upcoming weeks, and of course, I applied to Tailcall and they happily accepted me as their intern for upcoming month. I started my internship at Tailcall under Tushar Mathur (Ex VPE @Dream11) and Amit Singh. In the internship, basically my task was to cleanup the code and grab the bounty :p

After a month of hard work and 20+ PRs, I was finally awarded the bounty on 3rd Feb, 2024.

What did I learn during internship?

Well, this is more important section in my opinion. Before any contributions, I was just a "programmer" who can just get the job done, but the code quality might not be great etc etc...

After the completion of internship, I can proudly say that I became a software engineer. The team at tailcall is quite supportive, especially Tushar, being a great leader, supported me throughout the duration of internship. I can say that Tushar played an important role in upgrading my skills.

I always like exploring stuff and I really wanted to explore GraphQL and GRPC for a long period of time, but I always procrastinated. I can say that bounties were huge motivation for me to explore them and now I can say that I have fairly good knowledge about both of them.

The story still continues

algora.io is a platform which allows people to put bounties on the github issues and allows contributors to claim 100% of the amount upon completion of the issue.

Tailcall uses the same platform. After taking some break I started contributing to the same repository again. Sticking to same repository helps in gaining deep understanding of the code-base. Now I am able to solve majority of the issues which are published at Tailcall. I never tried to contribute to any other repo, provided that financial gain isn't my first goal.

However by contributing to just to single repository, I am currently ranked #7 on algora.io leader board by making over $5000 in bounties, in 2 months.

Conclusion

I am not paid by any of the platforms which I mentioned in this blog, this blog is purely based on my personal experience in past few months.

I hope you learnt something new and interesting from this blog. Happy contributing!