NumPy/Pandas JSON Sanitizer
Clean up messy JSON output from Python libraries like Pandas and NumPy.
Automatically detects and replaces non-standard values like NaN, Infinity, and -Infinity with valid JSON (null, 0, or string).
Why is my JSON invalid?
Python's json library is compliant with an old spec that allowed NaN and Infinity.
However, the standard JSON spec (RFC 8259) widely used by Web APIs and JavaScript does NOT support them.
How to fix in Pandas
🐼 Replace NaN
Replace NaN with None before dumping.
df = df.where(pd.notnull(df), None) # OR df.fillna(0, inplace=True)
🐍 Python Strict Mode
Force Python to throw error on NaN.
import json # Will raise ValueError if NaN exists json.dumps(data, allow_nan=False)
Advanced: Handling Large Datasets
If you are processing millions of rows, using df.fillna() can be memory intensive.
Here are optimized approaches used by high-performance data pipelines.
🚀 Use `orjson` (Fastest)
orjson natively handles Numpy types 10x faster.
import orjson
import numpy as np
data = {"score": np.nan}
# Serializes NaN to null automatically!
json_bytes = orjson.dumps(data)
# Output: b'{"score":null}'
🛠️ Use `simplejson`
Explicitly ignore NaN to force compliance.
import simplejson # ignore_nan=True -> outputs 0 or null? # Actually simplejson defaults to allow_nan=True. simplejson.dumps(data, ignore_nan=True) # Output: ValueError (It forces you to fix it!)
⚠️ The "Integer to Float" Trap
In Pandas, if an integer column has one single NaN, the entire column is cast to float.
This means IDs like `12345` become `12345.0` in your JSON.
Fix: Use `Int64` (capital 'I') nullable integer type in Pandas:
df['id'] = df['id'].astype('Int64').
🔢 Numpy TypeErrors
int64/float32 break `json.dumps`.
class NumpyEncoder(json.JSONEncoder):
def default(self, obj):
if isinstance(obj, np.integer):
return int(obj)
if isinstance(obj, np.floating):
return float(obj)
return super().default(obj)
🌐 JavaScript "Crash"
JS JSON.parse() destroys NaN.
❌ Uncaught SyntaxError
This tool fixes exactly this crash by converting to `null`.