downloading mail batches (synchronizing large files efficiently)

[Kragen Javier Sitaker]

On Sat, 21 Oct 2006 13:22:43 +0200, Dave Long wrote:
> When does the head of a mailbox change?  Why not sequence your inbox
> as a TCP-style stream? ... send the length of your laptop's
> mailbox. ...

That's probably a better idea.  It makes me feel good to know that the
whole mailbox gets checked for differences from the server, but it's
probably not worth the extra minute.

> (how easy is it to generate HTTP Content-Encoded Range Requests from 
> python?)

You mean, byte-range requests that advertise they'll accept gzip
content-encoding?  Probably fairly easy, but so far I have been
avoiding running my own code on internet-cafe Microsoft Windows boxes,
and I'm kind of leaning in the direction of encrypting stuff
end-to-end --- from my laptop to my server --- rather than putting
more smarts on the cafe machine.

> > control over the data I'm rsyncing; with a naive implementation of the
> > above algorithm, they could send me a chunk of data with the same CRC
> > as some later email they expect I will get, in order to prevent me
> > from getting the later email.
> 
> This scheme has a similar problem, in that Mallet could send a bogus 
> update.

In the problem I describe above, Mallet only needs to be able to send
me email and guess what CRC polynomial and rsync block size I'll be
using.  To send a bogus update, Mallet has to be able to MITM my
connection with my mailserver --- while that's not all that difficult
at the moment, it's still more difficult than sending me email.

> A slight modification would be to request a sequence starting a few
> (from a suitable unbounded random distribution?) pages before the
> end of your current mailbox, and signalling an error if the overlaps
> ever failed to match.

It's probably better just to shell out to "openssl aes-256-cbc" and
"openssl rsautl".