We look at file storage in the cloud, its key benefits and how it can complement on-prem storage, in particular to burst to the cloud in times of increased application demand ...

Spidr - web spidering library that can spider a site, multiple domains, certain links or infinitely. cobweb - web crawler that can use resque to cluster crawls to quickly crawl extremely large sites.