Adventures with Big Data - Part 1
Part 1
I've been volunteering some of my time with the COVID-19 Cyber Threat Coalition. It's a collection of volunteers that have banded together thwart malicious people (artificial intelligence...maybe?)…
Adventures with Big Data
While volunteering with the COVID-19 Cyber Threat Coalition (external site), I took on the task of working with the anonymised query data from Quad9 to give some visual context to our efforts. The plan: download the query data with Python, get it into a big data database, and then find ways to visualise it.
These posts follow that work as it happened, including the detours.
-
-
Adventures with Big Data - Part 1.5
Part 2
While I had ambitions to get all the data into the database, the challenge is trying to manage large files (larger than say 1GB compressed) because if you run into any problems while parsing the…
-
Adventures with Big Data - oh 32 bit…why do we still have this around?!
Part 3
One of my challenges with parsing large amounts of data, and that after all my excitement of getting my scripts working...the 32 bit version of Python has a memory limitation!