The Challenge of Large JSON

When dealing with JSON files that exceed several megabytes, standard parsing approaches can quickly exhaust available memory. A 100MB JSON file can easily consume 500MB or more when parsed into memory structures.

Memory-Efficient Parsing Strategies

Streaming JSON Parsers

Instead of loading the entire file at once, use streaming parsers that process data incrementally:

import { JSONParser } from "@streamparser/json";

const parser = new JSONParser({
  onValue: (value, key, parent, stack) => {
    // Process each value as it's found
    console.log(`Found key: ${key}, value:`, value);
  }
});

fs.createReadStream("large-file.json").pipe(parser);

Chunked Reading

Read and parse file in chunks:

const CHUNK_SIZE = 64 * 1024; // 64KB chunks

async function* parseJSONChunks(filePath) {
  const fileHandle = await fs.promises.open(filePath, "r");
  let buffer = "";

  try {
    while (true) {
      const { bytesRead, buffer: chunk } = await fileHandle.read({
        length: CHUNK_SIZE,
        position: fileHandle.position
      });

      if (bytesRead === 0) break;

      buffer += chunk.toString();
      const lines = buffer.split("\n");
      buffer = lines.pop() || "";

      for (const line of lines) {
        if (line.trim()) {
          yield JSON.parse(line);
        }
      }
    }
  } finally {
    await fileHandle.close();
  }
}

JSON Lines Format (JSONL)

Instead of one giant JSON array, use JSON Lines format where each line is a separate JSON object:

{"id": 1, "name": "Alice", "age": 25}
{"id": 2, "name": "Bob", "age": 30}
{"id": 3, "name": "Charlie", "age": 35}

This format allows you to:

• Process records one at a time
• Append new records without re-parsing the entire file
• Use standard line-based tools like grep, awk, and sed

Converting Arrays to JSONL

async function jsonArrayToJsonl(inputFile, outputFile) {
  const data = JSON.parse(await fs.promises.readFile(inputFile, "utf8"));

  const writeStream = fs.createWriteStream(outputFile);
  for (const item of data) {
    writeStream.write(JSON.stringify(item) + "\n");
  }
  writeStream.end();
}

Incremental JSON Construction

Build large JSON objects incrementally using array buffers:

class IncrementalJSONBuilder {
  constructor() {
    this.chunks = [];
    this.isObject = false;
  }

  startObject() {
    this.chunks.push("{");
    this.isObject = true;
    return this;
  }

  addKeyValue(key, value) {
    if (this.chunks.length > 1) {
      this.chunks.push(",");
    }
    this.chunks.push(`"${key}":${JSON.stringify(value)}`);
    return this;
  }

  endObject() {
    this.chunks.push("}");
    return this;
  }

  toString() {
    return this.chunks.join("");
  }
}

// Usage
const builder = new IncrementalJSONBuilder();
builder.startObject();
builder.addKeyValue("name", "Test");
builder.addKeyValue("data", [1, 2, 3]);
builder.endObject();
console.log(builder.toString());

Memory-Mapped Files

For read-heavy operations, memory-mapped files can be more efficient:

import mmap from "memory-map";

const buffer = mmap("large-file.json");
const json = JSON.parse(buffer.toString());

Practical Benchmarks

When to Use Each Approach

• Standard JSON parse: Files under 10MB
• Streaming parser: Very large files, network streams
• JSON Lines: Log files, data pipelines, appendable data
• Memory mapping: Random access patterns, read-heavy workloads

Pro Tips

1. Pre-validate before parsing - Check file structure without full parse
2. Compress JSON - Gzip can reduce file size by 80-90%
3. Consider alternative formats - For numeric data, consider binary formats like Protocol Buffers
4. Use pagination - Split large datasets into pages

Processing large JSON files doesn't have to be a headache. Choose the right strategy for your use case and your available resources.