Approximate Line Count for Very Large Files

%expand% Approximate Line Count for Very Large Files %set(body,%html-clean({

When dealing with very large files, the unix tool "wc" can be extremely slow. The alternative, byte size, is often not what I want to look at, especially when trying to estimate the number of reads in a fastq file.

A good estimate (2 sig figs) is, 90% of the time, what I need.

alc is my "approximate" line count tool. It counts the number of lines in a file, just like wc, except it only "samples" the file in a series of segments. By seeking and reading 200K from a dozen places in the file, rather than reading the whole thing, I get a good representative sample, and an accurate-enough count.

UPDATE: The new version 1.2 uses a log-linear model to predict the line-count of compressed (gzipped) files. For files ending in ".gz", the output is always the predicted uncompressed line count, not the total line count. It's off by no more than 10% for the vast majority of files tested, and a huge improvement over the older version.

}))


Bracing against the wind
www.documentroot.com

View Archive

Free Blog Commenter & Counter

Memebot: Free Website Hosting

Americans for Approval Voting

2002 WTC Pics: Aug 12 Aug 16 Aug 30 Sep 11 Sep 23 Oct 24

Shop with Bitcoin At Amazon

Yvette's Weblog

All sites by me

RSS for this site

Lo! In the east against the blazing shadow of a rising sun, there come four men striding in unison, each step a journey, against the terrible winds.

Thursday, February 10, 2011

Approximate Line Count for Very Large Files
Posted 2:06 AM

When dealing with very large files, the unix tool "wc" can be extremely slow. The alternative, byte size, is often not what I want to look at, especially when trying to estimate the number of reads in a fastq file.

A good estimate (2 sig figs) is, 90% of the time, what I need.

alc is my "approximate" line count tool. It counts the number of lines in a file, just like wc, except it only "samples" the file in a series of segments. By seeking and reading 200K from a dozen places in the file, rather than reading the whole thing, I get a good representative sample, and an accurate-enough count.

UPDATE: The new version 1.2 uses a log-linear model to predict the line-count of compressed (gzipped) files. For files ending in ".gz", the output is always the predicted uncompressed line count, not the total line count. It's off by no more than 10% for the vast majority of files tested, and a huge improvement over the older version.

[View/Post Comments] [Digg] [Del.icio.us] [Stumble]

| Bloghop: | Blogarama | Technorati | Blogwise