As a developer working with large datasets, I've often found myself wrestling with JSON Lines (JSONL) files. Whether it's processing log files, handling data exports, or working with streaming APIs, JSONL has become ubiquitous in the data world. However, I noticed a gap in the Rust ecosystem for efficient, async-first JSONL processing. That's why I decided to create async-jsonl - and I'm excited to share what I've built!
What is JSON Lines (JSONL)?
Before diving into my library, let's quickly cover what JSONL actually is. JSON Lines is a text format where each line is a valid JSON object, separated by newline characters. It looks like this:
{"id": 1, "name": "Alice", "age": 30}
{"id": 2, "name": "Bob", "age": 25}
{"id": 3, "name": "Charlie", "age": 35}
Why JSONL is Awesome
Streamable: You can process one line at a time without loading the entire file into memory
Appendable: Easy to add new records to the end of a file
Fault-tolerant: One corrupted line doesn't break the entire dataset
Tool-friendly: Works great with unix tools like cat, head, tail, and grep
Language-agnostic: Supported across virtually every programming language
These advantages make JSONL perfect for log files, data pipelines, and any scenario where you're dealing with large volumes of structured data.
While working on various Rust projects that needed to process large JSONL files, I kept running into the same issues:
Existing solutions weren't async-first
Memory usage would explode with large files
Error handling was either too strict (one bad line kills everything) or too loose
Type safety was often sacrificed for convenience
So I built async-jsonl to solve these problems. Here's what makes it special:
🚀 Async/Await from the Ground Up
Built on Tokio, async-jsonl is designed for high-performance async I/O. No blocking the event loop here!
use async_jsonl::Jsonl;
use futures::StreamExt;
#[tokio::main]
async fn main() -> anyhow::Result<()> {
let jsonl = Jsonl::from_path("data.jsonl").await?;
let lines: Vec<_> = jsonl.collect().await;
for line_result in lines {
let line = line_result?;
println!("Raw JSON: {}", line);
}
Ok(())
}
💾 Memory Efficient Streaming
Process files of any size without loading everything into memory. The streaming approach means you can handle gigabyte files with minimal RAM usage.
use async_jsonl::{Jsonl, JsonlDeserialize};
use futures::StreamExt;
use serde::Deserialize;
#[derive(Debug, Deserialize)]
struct LogEntry {
timestamp: String,
level: String,
message: String,
}
#[tokio::main]
async fn main() -> anyhow::Result<()> {
let jsonl = Jsonl::from_path("huge_log_file.jsonl").await?;
let mut stream = jsonl.deserialize::<LogEntry>();
while let Some(entry_result) = stream.next().await {
let entry = entry_result?;
if entry.level == "ERROR" {
println!("🚨 Error at {}: {}", entry.timestamp, entry.message);
}
}
Ok(())
}
🔒 Type-Safe with Serde Integration
Full integration with serde means you get compile-time type checking and zero-cost deserialization.
use async_jsonl::{Jsonl, JsonlDeserialize};
use futures::StreamExt;
use serde::Deserialize;
#[derive(Debug, Deserialize)]
struct User {
id: u64,
name: String,
email: String,
#[serde(default)]
is_verified: bool,
}
#[tokio::main]
async fn main() -> anyhow::Result<()> {
let jsonl = Jsonl::from_path("users.jsonl").await?;
let users = jsonl.deserialize::<User>();
let results: Vec<_> = users.collect().await;
for user_result in results {
let user = user_result?;
println!("{:?}", user);
}
Ok(())
}
🛡️ Resilient Error Handling
One of my favorite features: the library continues processing even when individual lines fail to parse. Perfect for real-world data that's not always perfect.
use async_jsonl::{Jsonl, JsonlValueDeserialize};
use futures::StreamExt;
#[tokio::main]
async fn main() -> anyhow::Result<()> {
let jsonl = Jsonl::from_path("messy_data.jsonl").await?;
let mut stream = jsonl.deserialize_values();
let mut valid_records = 0;
let mut error_count = 0;
while let Some(result) = stream.next().await {
match result {
Ok(value) => {
valid_records += 1;
println!("Valid record: {}", value);
}
Err(e) => {
error_count += 1;
eprintln!("Skipping invalid line: {}", e);
}
}
}
println!("Processed {} valid records, {} errors", valid_records, error_count);
Ok(())
}
Whether you're reading from files, memory, or any AsyncRead source, async-jsonl has you covered.
use async_jsonl::Jsonl;
use std::io::Cursor;
#[tokio::main]
async fn main() -> anyhow::Result<()> {
let data = r#"{"id": 1, "name": "Alice"}
{"id": 2, "name": "Bob"}
{"id": 3, "name": "Charlie"}
"#;
let reader = Cursor::new(data.as_bytes());
let jsonl = Jsonl::new(reader);
let file_jsonl = Jsonl::from_path("data.jsonl").await?;
Ok(())
}
🎯 Advanced Features for Power Users
The library also includes some advanced features I found myself needing repeatedly:
Line Counting Without Full Processing:
use async_jsonl::{Jsonl, JsonlReader};
#[tokio::main]
async fn main() -> anyhow::Result<()> {
let jsonl = Jsonl::from_path("massive_file.jsonl").await?;
let count = jsonl.count().await?;
println!("File contains {} records", count);
Ok(())
}
The Technical Journey
Building this library taught me a lot about async Rust and streaming APIs. Some key decisions I made:
Tokio as the Foundation: Built on tokio::io for maximum compatibility with the async ecosystem
Stream-based Architecture: Using futures's Stream trait for composable, lazy processing
Zero-copy Where Possible: Minimizing allocations while maintaining safety
Comprehensive Error Types: Using anyhow for ergonomic error handling without sacrificing information
In my testing with large datasets (>1GB JSONL files), async-jsonl consistently shows:
Memory usage: Constant regardless of file size
Throughput: Competitive with the fastest JSONL parsers in any language
CPU efficiency: Low overhead thanks to Rust's zero-cost abstractions
Try It Yourself!
The library is available on crates.io and the source is on GitHub. Here's how to get started:
[dependencies]
async-jsonl = "0.3.1"
tokio = { version = "1.0", features = ["full"] }
futures = "0.3"
serde = { version = "1.0", features = ["derive"] }
anyhow = "1.0"
I'd love to hear how you use it in your projects! Whether you're processing logs, handling data pipelines, or working with APIs, async-jsonl is designed to make your life easier.
What do you think? Have you worked with JSONL files before? What features would you find most useful in a JSONL processing library? Let me know in the comments below!
