Working with Large JSON Files
2026-03-24#performance#large-files#javascript#streaming
The Challenge of Large JSON
When dealing with JSON files that exceed several megabytes, standard parsing approaches can quickly exhaust available memory. A 100MB JSON file can easily consume 500MB or more when parsed into memory structures.
Memory-Efficient Parsing Strategies
Streaming JSON Parsers
Instead of loading the entire file at once, use streaming parsers that process data incrementally:
import { JSONParser } from "@streamparser/json";
const parser = new JSONParser({
onValue: (value, key, parent, stack) => {
// Process each value as it's found
console.log(`Found key: ${key}, value:`, value);
}
});
fs.createReadStream("large-file.json").pipe(parser);
Chunked Reading
Read and parse file in chunks:
const CHUNK_SIZE = 64 * 1024; // 64KB chunks
async function* parseJSONChunks(filePath) {
const fileHandle = await fs.promises.open(filePath, "r");
let buffer = "";
try {
while (true) {
const { bytesRead, buffer: chunk } = await fileHandle.read({
length: CHUNK_SIZE,
position: fileHandle.position
});
if (bytesRead === 0) break;
buffer += chunk.toString();
const lines = buffer.split("\n");
buffer = lines.pop() || "";
for (const line of lines) {
if (line.trim()) {
yield JSON.parse(line);
}
}
}
} finally {
await fileHandle.close();
}
}
JSON Lines Format (JSONL)
Instead of one giant JSON array, use JSON Lines format where each line is a separate JSON object:
{"id": 1, "name": "Alice", "age": 25}
{"id": 2, "name": "Bob", "age": 30}
{"id": 3, "name": "Charlie", "age": 35}
This format allows you to:
- Process records one at a time
- Append new records without re-parsing the entire file
- Use standard line-based tools like
grep,awk, andsed
Converting Arrays to JSONL
async function jsonArrayToJsonl(inputFile, outputFile) {
const data = JSON.parse(await fs.promises.readFile(inputFile, "utf8"));
const writeStream = fs.createWriteStream(outputFile);
for (const item of data) {
writeStream.write(JSON.stringify(item) + "\n");
}
writeStream.end();
}
Incremental JSON Construction
Build large JSON objects incrementally using array buffers:
class IncrementalJSONBuilder {
constructor() {
this.chunks = [];
this.isObject = false;
}
startObject() {
this.chunks.push("{");
this.isObject = true;
return this;
}
addKeyValue(key, value) {
if (this.chunks.length > 1) {
this.chunks.push(",");
}
this.chunks.push(`"${key}":${JSON.stringify(value)}`);
return this;
}
endObject() {
this.chunks.push("}");
return this;
}
toString() {
return this.chunks.join("");
}
}
// Usage
const builder = new IncrementalJSONBuilder();
builder.startObject();
builder.addKeyValue("name", "Test");
builder.addKeyValue("data", [1, 2, 3]);
builder.endObject();
console.log(builder.toString());
Memory-Mapped Files
For read-heavy operations, memory-mapped files can be more efficient:
import mmap from "memory-map";
const buffer = mmap("large-file.json");
const json = JSON.parse(buffer.toString());
Practical Benchmarks
| File Size | Traditional Parse | Streaming | JSONL |
|---|---|---|---|
| 10 MB | ~100ms | ~150ms | ~50ms |
| 100 MB | ~1.5s | ~1.2s | ~400ms |
| 1 GB | Out of memory | ~15s | ~3s |
When to Use Each Approach
- Standard JSON parse: Files under 10MB
- Streaming parser: Very large files, network streams
- JSON Lines: Log files, data pipelines, appendable data
- Memory mapping: Random access patterns, read-heavy workloads
Pro Tips
- Pre-validate before parsing - Check file structure without full parse
- Compress JSON - Gzip can reduce file size by 80-90%
- Consider alternative formats - For numeric data, consider binary formats like Protocol Buffers
- Use pagination - Split large datasets into pages
Processing large JSON files doesn't have to be a headache. Choose the right strategy for your use case and your available resources.
Related articles
Understanding JSON Schema Validation
Learn how to use JSON Schema to validate your JSON data structure and ensure data integrity across your applications.
JSON Best Practices for Modern APIs
Design principles and patterns for building robust, developer-friendly APIs using JSON as the data exchange format.
Working with Large JSON Files - A Practical Guide
Techniques and tools for handling JSON files that exceed memory limits or browser constraints.
JSON vs XML - Choosing the Right Format for Your Use Case
A comprehensive comparison of JSON and XML to help you make informed format decisions.
JSON Tools Ecosystem - A Comprehensive Overview
Explore the best tools, libraries, and utilities for working with JSON across different platforms and use cases.
JSON Security Best Practices - Protecting Your Applications
Essential security measures for handling JSON data safely and preventing common vulnerabilities.