idea: calculate the hash tree on the fly #1

If a hash is 20 bytes, and I have hashes for 1 million records computed already,
then I only need to hash through n=20mb; n*log(n) of data to compute a hash tree.

I can hash a 300mb file in 1.7 seconds, on my laptop, so there is a lot of breathing room here.

calculating hash sets on the fly would enable replicating subsets dynamically.

Example, I could A and B could replicate data from their mutual friends.
Every pair of nodes would have different overlapping interests,
but if these sets are smallish, then it's probably not a problem to filter out a portion of the total set.

wikipedia is 30 million articles, but to replicate it properly, you'd have to replicate edits.
1.896 billion as of today... http://toolserver.org/~emijrp/wikimediacounter/

Just storing the hashes for that much many edits would be 37.8 gb
hmm, you'd probably not put all of wikipedia on a single machine...

but, if you computed the hash tree on the fly, you could replicate arbitary subsets.