When your data doesn’t fit in memory: the basic techniques

Hi,

You’re writing software that processes data, and it works fine when you test it on a small sample file. But when you load the real data, your program crashes.

The problem is that you don’t have enough memory—if you have 16GB of RAM, you can’t load a 100GB file. At some point the operating system will run out of memory, fail to allocate, and there goes your program.

So what can you do? You could spin up a Big Data cluster—all you’ll need to do is:

• Get a cluster of computers.
• Spend a week on setup.
• In many cases, learn a completely new API and rewrite all your code.

This is a bit of exaggeration, to be fair, since you can spin up Big Data clusters in the cloud, but it can still be expensive and frustrating; luckily, in many cases it’s also unnecessary.

You need a solution that’s simple and easy: processing your data on a single computer, with minimal setup, and as much as possible using the same libraries you’re already using. And much of the time you can actually do that, using a set of techniques that are sometimes called “out-of-core computation”.

In this article I’ll cover:

• Why you need RAM at all.
• The easiest way to process data that doesn’t fit in memory: spending some money.
• The three basic software techniques for handling too much data: compression, chunking, and indexing.

Read the rest of the article...

—Itamar

Python code too slow? There are many ways to speed it up, from compilation to parallelism to process improvements. Learn more about the practices of performance and how they can help you speed up your code.