Adventures with Big Data - oh 32 bit…why do we still have this around?! · Tech

One of my challenges with parsing large amounts of data, and that after all my excitement of getting my scripts working...the 32 bit version of Python has a memory limitation!

How did I stumble across this problem?

I ran my scripts...everything was going well, so I decided to leave it overnight. I woke up to a very short and stubborn response from Python:

Console output ending in a Python MemoryError after about 383 million lines processed

Big Data Memory Error

After some quick googling, I figured out what could be the cause - I had installed the 32bit version of Python instead of the 64bit version...doh!

After a quick uninstall (32bit) and install (64bit) I was able to run my scripts just fine:

Console output from the 64-bit run: per-file progress for output1015 to output102.json.gz at about 50,000 lines per second

Big Data Processing Speed (64 bit)

One interesting that I noticed was that I was parsing about 10,000 - 20,000 more lines per second when running in 64bit than when running 32bit.

Recommendation - if your OS supports it, make sure you install the 64bit version!