valthonis.net

Dug up an old post from an earlier incarnation of the blog today.  My logs say that folks found this at least a little bit useful.  This was originally posted on 28 April 2009:

I recently had need to parse a large (100 MB) CSV file with Python 2.x and was pleased with how speedy it was.  However, the test machine I was using was strapped for disk space and a 100 MB file was, shall we say, less than welcome.  Zipping the file compressed it to 5 MB, which was far more reasonable.

Parsing CSV in Python is straightforward… and so is reading file contents from a ZIP archive.  Unless you’re using Python 2.6 or higher, though, getting the file contents out of the ZIP archive in a way csv.reader() likes (as a file-like object) is not as simple as calling ZipFile.open().  All you get in Python 2.5 or earlier is ZipFile.read(), which returns the file contents as a string.  Since disk space was a premium, unarchiving the CSV even temporarily was out of the question.  Using the StringIO module, it is possible to take the string contents of the CSV within the ZIP archive and make a file-like object that csv.reader() will accept.

Assume you have a ZIP archive called test.csv.zip with a single file in it called test.csv.  Here’s the code:

import csv
import StringIO
import zipfile

dataFile = 'test.csv'
archive = '.'.join([dataFile, 'zip'])

filehandle = open(archive, 'rb')
zfile = zipfile.ZipFile(filehandle)
data = StringIO.StringIO(zfile.read(dataFile))
reader = csv.reader(data)

for row in reader:
    print row

zfile.close()
filehandle.close()