Read a 1 GB Log File with Python Generators: Day 2 - Generator Pipelines and a Live Log Watcher

Read a 1 GB Log File with Python Generators: Day 2 - Generator Pipelines and a Live Log Watcher

Connect small generators into a pipeline and watch a live log for errors with almost no memory.

Ardit Sulce

Oct 21

∙

Preview

Yesterday we read a 1 GB log file with one generator. Today we connect small generators into a pipeline, and point it at a log that a web server is still writing.

Here is today’s program watching a live log. A fake web server adds about 50 requests a second, and the watcher prints each error as it appears:

It ran for 45 seconds, saw 2,248 requests and 9 errors, and never used more than 0.03 MB of memory.

Projects in this week’s series:

Day 1: Yesterday, we read a 1 GB log file with a generator and measured how much memory it saves.

Day 2 (Today): We connect generators into a pipeline, and build a live watcher that prints new errors while the log is still being written.

View All Projects This Week

What you get today

• A pipeline of four small generators that read, count, parse and filter a log

• The first 5 errors of the 1 GB file in 0.002 seconds

• A live watcher that works like tail -f and prints errors as they happen

• A fake web server, so you can test the watcher without a real website

The key idea

Each step is a generator that takes the previous one as its input:

lines = read_lines("access.log")
requests = parse(lines)
errors = only_errors(requests)

for when, page, status in errors:
    print(when, status, page)

The first three lines do not read anything. They only connect the steps. The work starts when the for loop asks errors for a value. Then only_errors asks parse, parse asks read_lines, and one line travels through all the steps.

To watch a live log, we change only the first step. follow() reads to the end of the file and waits for new lines. The rest stays the same.

Because nothing runs ahead, the pipeline can also stop early. Here it asks yesterday’s 1 GB file for the first 5 server errors:

It read only 772 of the 5 million lines, so it finished in 0.002 seconds.

Worth knowing

• islice() from itertools is like slicing a list, but for a generator. islice(errors, 5) takes the first 5 values and then stops asking, so the whole pipeline stops too.

• A generator that never ends is normal. follow() has a while True loop, so the watcher runs until you press Ctrl+C.

• The watcher can catch a line that is only half written. follow() keeps the half and waits for the rest, so the parser only ever sees whole lines.

Run it

Save the code as logwatch.py next to yesterday’s access.log, then run:

python logwatch.py access.log
python logwatch.py --demo

The second command starts the live demo. It writes a small file called live.log and runs until you press Ctrl+C. There is nothing to install.

Try this

Change only_errors() so it also lets status 404 through, and run python logwatch.py access.log 20. One page fills the whole list. Which page is it, and who do you think keeps asking for it? Reply and tell me.

Solution

The full code is below. It is one file, and every step has a comment...

Watch with a 7-day free trial

A subscription gets you:

© 2026 Ardit Sulce
[email protected]