Convert CSV into newline-delimited JSON
The CSV to NDJSON Converter turns each CSV row into a single JSON object on its own line. This newline-delimited format — also called JSON Lines or JSONL — is what BigQuery loads, the Elasticsearch bulk API, and most log shippers expect, because it can be read and written one record at a time without holding the whole file in memory.
How it works
The tool parses your CSV with a quote-aware parser, so commas, newlines, and
escaped quotes ("") inside quoted fields are handled correctly. The first row is
treated as the header and supplies the JSON keys. Each subsequent row becomes an
object whose keys are the headers and whose values come from the cells.
When type inference is enabled, each cell is checked against strict rules: a value
matching a JSON number pattern becomes a number, true and false become
booleans, and null becomes JSON null. Leading-zero strings like 007 and
grouped numbers like 1,000 deliberately stay strings so IDs and codes are not
mangled. The output is then joined with \n so every object sits on its own line.
Example
Given this CSV:
id,name,active
1,Ada Lovelace,true
2,"Hopper, Grace",falsethe converter produces:
{"id":1,"name":"Ada Lovelace","active":true}
{"id":2,"name":"Hopper, Grace","active":false}Notice the quoted comma inside "Hopper, Grace" is preserved as part of one field.
NDJSON versus regular JSON arrays
| Property | JSON array | NDJSON |
|---|---|---|
| Structure | [{...}, {...}] | {...}\n{...} |
| Streamable? | No — must buffer entire file | Yes — read one line at a time |
| Partial read? | No | Yes — read first N lines without touching the rest |
| Appendable? | No — requires rewriting the closing ] | Yes — append a new line |
| Memory for 1 M rows | Entire array in RAM | One object at a time |
BigQuery, Elasticsearch, Apache Kafka, Logstash, and most log aggregators prefer NDJSON specifically because of streamability. A 5 GB NDJSON file can be ingested line by line without ever needing 5 GB of free RAM; a 5 GB JSON array cannot.
Loading NDJSON into BigQuery
When using bq load, set the --source_format flag:
bq load --source_format=NEWLINE_DELIMITED_JSON \
mydataset.mytable \
gs://mybucket/data.ndjson \
schema.jsonIf your schema matches your CSV headers, BigQuery can often infer it — but providing an explicit schema avoids surprises with type coercion. Ensure the NDJSON you generate has type inference enabled so numeric columns arrive as integers or floats rather than strings.
Loading NDJSON into Elasticsearch
The Elasticsearch bulk API expects an alternating pattern:
{"index":{"_index":"myindex"}}
{"id":1,"name":"Ada Lovelace","active":true}
{"index":{"_index":"myindex"}}
{"id":2,"name":"Hopper, Grace","active":false}This tool emits only the document lines. To use the output with the bulk API,
either add the action lines with a text editor, a shell awk script, or a
purpose-built Elasticsearch import tool. For Elasticsearch, remember the bulk
API needs an action line between documents, which this tool does not add — it
emits the raw document stream.
Tips and notes
- Keep type inference on for numeric and boolean columns so the receiving system does not have to re-parse them from strings.
- NDJSON files should have one object per line with no trailing comma — this converter always produces compliant output.
- The file extension
.ndjsonand.jsonlare both common; either is accepted by BigQuery and Elasticsearch.