Assuming that you are a part of large-scale international collaboration. In this collaboration, you are focusing on analyzing large amount of data. To read the data efficiently, the data should be stored as near as your computer. By the way, parties locate on different network and different country. As a result, the data must be mirrored from the origin server to each party dedicated server. There are so many options to satisfy this requirement. For example, rsync and mirror. Anyway, both tools do not scale well for big network. We are talking about content distribution problem that initially there are only one complete source. BitTorrent fits in this situation perfectly.

For more detail, we have a complete source located on an origin server. What we want is to distribute this file to other 10 servers as fast as possible. In other words, we have a seeder and 10 leechers. BitTorrent can do it very easy and fast. Anyway, there are several issues need to be solved.

• There must not be just only one file at a time. How to distribute multiple files efficiently? Fortunately, BitTorrent supports distributing multiple files by just one metadata.
• What if existing file changed? It is very easy to add new file. In case the existing files must be updated, BitTorrent may be confusing the current state of these files. Many BitTorrent client will store the latest state of each files in separate file for fast resume on next startup. To handle updating correctly, we must force BitTorrent client to recheck the file by removing state file.
• What if existing file removed? This is the real problem. BitTorrent is designed for downloading, not synchronize, so it doesn’t natively support removing existing files.

For the last problem, my idea is to distribute a kind of MANIFEST file to list current files on the origin server. We then just need to walk through the directory structure and remove unknown files according to the given list. This solution sounds good. I will manage my time to add this feature to BTQueue.

Technorati Tags: English, Product, BTQueue, Tips and Tricks, BitTorrent, Mirror, Rsync