Get block size bash
Posted 10:13 AM

Probe for block size. Useful when you've probably got no tools. Like when trying to optimize sqlite paging at run-time.

function block_size() {( 
   [ "$1" ] && cd $1; echo 1 > block_size.tmp;  
   ls --block-size 1 -s block_size.tmp | cut -f 1 -d ' '; 
   rm block_size.tmp; 
)}

[View/Post Comments] [Digg] [Del.icio.us] [Stumble]

Viral load
Posted 1:35 AM

Quintiles put the poster I made up on their site: Identifying viral load in tumors using nextgen sequencing. Thanks Kim!
[View/Post Comments] [Digg] [Del.icio.us] [Stumble]

Gene therapies I want to see developed
Posted 10:58 AM

Gene therapies I'd love to see developed/tested in my lifetime:

​1. daf-2 - Downregulation increases lifespan in c.elegans

Next steps:  RNAi and/or Crispr to cause daf-2 homolog (FOXO3) downregulation in mice + tests on resulting phenotype.
​
​2. PEPCK-C    - Beta-actin promoter linked to pepck gene shown to improve longevity, intelligence metabolism in mice.

Next steps:  Lentiviral transfection vector as a longevity vaccine (develop in mouse ageing model with human gene)
​
​3. m-CAT    -  Mitochondrial targeted catalase : upregulation shown to improve cancer resistance and longevity

Next steps:  Develop transfection vector as a cancer vaccine (develop in mouse ageing model with human gene)
​
4. HAS2 - Molecular weight and production of hyaluron decreases over time.   Organisms with large amounts of HMWHA live longer and resist cancer.

Next steps:  Test regular intravenous injection of HMWHA in rat models of cancer and aging.   Develop gene therapy intervention to increase HAS2 expression.

5. TERT - TERT+ mice live longer, but also produce more tumors.   HAS2, m-CAT and PEPCK-C all interfere with tumorgenesis.  The upregulation of TERT while simultaneously increasing m-CAT and HAS2 expression should have a strong synergistic effect.

Next steps:  Develop gene therapy intervention to increase TERT expression, matching prior experiments, but combine with other interventions.

[View/Post Comments] [Digg] [Del.icio.us] [Stumble]

Clustered Object Encoding 2.0
Posted 1:38 PM

Without going into detail about of "the cloud" and the exaggerations and mysticism that accompany it, here's some definitions that it took me a while to truly understand:

Object storage: Some manky, idiosyncratic api that allows developers to store "stuff". Avoids dealing with things like partial writes, seeks, memory maps, etc. Generally space wasteful by design. Not a place you'd want to put anything you care about. Yet, secretly, is probably the location of everythingyou care about. Your bank probably keeps your PDF statements in something like this.

Block storage: Like a horse, one can mount this kind of storage... but only for exclusive access. No shared, lockable access, no parallel reads. Essentially it's the virtual hard drive for your virtual machine from which you derive your virtual paycheck.

Clustered storage: Typically one of the above, but your data is now copied ad nauseum. Someone probably didn't stripe it. The things you expect from clustered storage, like striped files, full POSIX support, parallel writes, and erasure coding don't work in your version of the software, but every whitepaper you read refers to them.

POSIX, Erasure Coded, Clustered, Storage (PECCS): 1. This is what you actually want if you run your own data center. Used in a sentence: "Wow, that guy's got PECCS". 2. A mythical beast. 3. Maybe some company named "NextEMCApp" does it. Maybe some guy names Xavier in Spain got it to work once, but then he got hired by [SOME BIG COMPANY] and nobody heard from him again.

[View/Post Comments] [Digg] [Del.icio.us] [Stumble]

Variable Length, Zipf's Law and Density Functions
Posted 9:17 AM

When coding, variables should be named based on both frequency and the length of their context. Variables, structures and other constructs should have a name length that follows to Zipf's law as it applies to word length and frequency. Please glance at those references before continuing.

When these laws of linguistic use are ignored, readability is demonstrably diminished.

Consider a variable, the hated "x", used 100 times in a file. If "x" is used 100 times in 40 lines of mathematical manipulation, such as in a digest function, this would be acceptable... the high frequency of use strictly precludes a longer name. If, however, it was spread out across 10000 lines of code, a far longer name would be needed.

Of course, suppose that function was embedded in 10k lines of code. Does this change the freqency of x? No. One important rule is that, for the purposes of linguistic calculation, the frequency of any variables usage should be computed relative to the scope of it's use.

As a demonstration, cluttering a short for i=1 to 10 ... loop with for iterationIndex = 1 to 10, can cause code to be nigh unreadable.

Operation can often be obscured by foolish verbosity:

for iterationIndex = 1 to 10
    countArray[iterationIndex]=sizeArray[iterationIndex]*factorArray[iterationIndex] + computeDensity(histogram,iterationIndex)

Versus

for i = 1 to 10
    countArray[i]=sizeArray[i]*factorArray[i] + computeDensity(histogram,i)

Likewise stuffing common prefixes on sets of variables in order to lengthen them can hide their meaning in a limited scope:

for i = 1 to 10
    wordSuffixCountArray[i]=wordSuffixSizeArray[i]*wordSuffixFactorArray[i] + ComputeDensityForWordSuffix(currentWordSuffixHistogram,i)

It's clear that a minimum set of descriptive, distinct terms within the scope of use should, instead, be used.

Recently, software engineering pundits seem to take the stance that there is no such thing as a variable with too-long a name. This is false. Numerous linguistic studies on readability, word length and frequency, bear that out and so, of course, does common sense. Maximum clarity can be achieved in coding languages by using the same types of statistical distributions that occur in natural languages.

[View/Post Comments] [Digg] [Del.icio.us] [Stumble]

Capricious Job Scheduling
Posted 12:16 PM

Using a perfectly fair scheduler, I launch 5 processes with the following pattern:

 IIIIIIIIICCCCCCCCCCCCIIIIIIIIIIIIIICCCCCCCCCCCCC
 IIIIIIIIICCCCCCCCCCCCIIIIIIIIIIIIIICCCCCCCCCCCCC
 IIIIIIIIICCCCCCCCCCCCIIIIIIIIIIIIIICCCCCCCCCCCCC
 IIIIIIIIICCCCCCCCCCCCIIIIIIIIIIIIIICCCCCCCCCCCCC
 IIIIIIIIICCCCCCCCCCCCIIIIIIIIIIIIIICCCCCCCCCCCCC

Where 'I' is one second of I/O intensive operation, and 'C' is one second of CPU intensive operation. The result of a perfectly fair scheduler is that there will be periods of high I/O contention and high CPU contention, as the 5 processes are "fairly" switched between, and each keeps the pace with the other.

Using a more "capricious" scheduler, where processes may "get lucky" for a while, my processes will be staggered

 IIIIIIIIICCCCCCCCCCCCIIIIIIIIIIIIIICCCCCCCCCCCCC      lucky (goes first)
           IIIIIIIIICCCCCCCCCCCCIIIIIIIIIIIIIICCCCCCCCCCCCC unlucky (goes last)
          IIIIIIIIICCCCCCCCCCCCIIIIIIIIIIIIIICCCCCCCCCCCCC
   IIIIIIIIICCCCCCCCCCCCIIIIIIIIIIIIIICCCCCCCCCCCCC
  IIIIIIIIICCCCCCCCCCCCIIIIIIIIIIIIIICCCCCCCCCCCCC

It's clear that "everyone wins" when this method is chosen.

My argument for "capricious" scheduling is that a system cannot know, in advance whether something will be an I or a C. Programs can be profiled over time (modern o/s systems don't do this... but probably should), but most program execution paths vary to the point where profiling may be impractical. Accordingly, a scheduler should be, to some extent, random.

[View/Post Comments] [Digg] [Del.icio.us] [Stumble]

New verison of bowtie-1.0
Posted 9:35 AM

Since the Ben Langmead version has been languishing a bit, I forked the repo, and incorporated most of the pull requests (with some edits): https://github.com/earonesty/bowtie Support for compiling on Apple, stream i/o, and gzip support has been added.
[View/Post Comments] [Digg] [Del.icio.us] [Stumble]

Latest Printable Firearm Kit From DefDist
Posted 6:38 AM

Printable automatic weapons? DIY genome editing?

We've come to a point in our evolution as a species where it's not possible to limit the "things" people can obtain. What's more important is creating a world where people have no reason to do harm.

"Well there's no way to guarantee that". Sure. But we allow corporations to draft zoning laws so that people have to drive further to work and so that public transportation remains a failure in most cities. We also allow gross wealth inequity. We try to con mothers into abandoning breast feeding (yes, this still happens, I witnessed it a couple weeks ago), and when that doesn't work we make it as difficult as possible for families to spend time with their kids.

When the majority of the population is stuck in a perpetual state of inadequacy, when the society is structured around things like "employment" and not "enjoyment", it's not surprising that violence seems a means to an end.

A more radical transformation of society, where from birth a nation's citizens feel nurtured, loved and supported would probably go further to prevent violence than "gun laws" possibly could.

If only 10% of the people in America went to work, or if we worked only 10% of the time, we could, with the technology we have, produce all the food, shelter and goods needed for every citizen to live a decent middle class lifestyle. But the focus is not on creating awesome lifestyles. The focus is on more expensive stuff for the few stakeholders that got in early in the pyramid scheme that is capitalism.

We continue to blame the weapon, and not the society that produces it's wielder.

You download printable firearms here: http://kickass.to/defdist-defcad-mega-pack-4-5-otacon-zipped-t7677520.html

[View/Post Comments] [Digg] [Del.icio.us] [Stumble]

bitk.in, mojolicious and liteapi
Posted 1:09 AM

I wanted to build a site that demonstrated how easy it was to take bitcoin and litecoin. After building LiteAPI in mod_perl and running into issues with performance and those wacky implicit closures, I decided to try an event driven framework, Mojolicious.

The site is https://bitk.in, and it allows you to spend bitcoin & litecoin to get fair-valued gift cards at big online stores like Amazon, Walmart, etc.

It uses the API from http://liteapi.org for Litecoin and Blockchain's wallet API for Bitcoin.

Bitcoin/litecoin integration was trivial. Mojolicious allowed me to spend more time focusing on integration with vendors and payment systems, and just about no time worrying about HTTP.

To be honest, I spent more time thinking about the color than the back end.

[View/Post Comments] [Digg] [Del.icio.us] [Stumble]

UTF8 Encoding and Postgres Dump
Posted 5:52 AM

If you back up a database from one version of postgres using pg_dumpall, then restore to another version, it's very possible that the strictness or some other aspect of the UTF8 library has changed. The end result is that you will see an error "invalid character for encoding" while restoring. The utility "iconv" can help, and it was really hard to find the answer online, which is why I'm reposting for my own sanity:

gunzip -c dumpall.gz | iconv | sudo -u postgres psql > restore.errs 2>&1 

Miraculously, iconv worked without options, cleaning up version incompatibilities, while maintaining all the data... even the funny French city names. If you're changing encodings while also upgrading, that's fine too. iconv takes a "from" and "to" argument, so log in to the source db and destination db, and use:

SHOW client_encoding;

Then use the results as the "-f FROM" and "-t TO" arguments to iconv. No conversion is perfect, but iconv seems to do as good a job as possible.
[View/Post Comments] [Digg] [Del.icio.us] [Stumble]