Architecture A MouseFS instance is composed by a metadata server, a number of chunk servers, and a number of clients. The metadata server stores the full file system hieararchy, except instead of storing the file contents, it stores an array of hashes of the chunks of each file. A "chunk" is a file range that is fixed for a single file but may vary between files. Chunk servers hold an array of chunks that are identified by their hash. The metadata server keeps track of which chunks each chunk server is holding. Clients are users of the file system that can read and write metadata and files. They are assumed to behave correctly. Any read and write operation that doesn't involve file contents can be performed by clients by talking to the metadata server directly. Such operations include creating an empty file or a directory, deleting a file or directory, listing files. If a client wants to read a range of bytes from a file, it sends the metadata server the file name and range. The metadata server responds with the chunk size of that file, the list of hashes for the chunks involved in the read, and the IP addresses of the chunk servers that hold each chunk. The metadata server also adds the IP addresses of three chunk servers any new chunks should be written to. The client can then download the chunks from the chunk servers and reassemble the result. If a client wants to write at a range of bytes of a file, it starts by reading that range from the metadata server, getting the list of hashes it will modify, their locations, and locations for any new chunks. The client then modifies the chunk by sending to each chunk server the hash to modify and the patch (a range of bytes within a chunk plus the new data). The chunk server creates a new modified chunk and keeps the old version, then returns the new hash. If all modifications are successful, the client holds the set of old hashes and new hashes for that file range. It completes the write by telling the metadata server to swap the old hashes with the new ones. If the old hashes don't match, another write succeded in the mean time and touched that range, therefore the write fails. If the old hashes match, the write succeded. If the client fails to modify any chunks, it doesn't commit the write with the metadata server. Note that write failures may cause chunks to be orphaned on chunk servers. This is solved by a garbage collection algorithm implemented by the synchronization messages between metadata and chunk server. Note that clients may cache chunks and index them by their hash. When they read a file and receive its hashes, they may avoid reaching for the chunk servers if they already cached the chunks with those hashes. This allows reading files with only one round trip at no cost of correctness. If getting the up-to-date contents is not a concern, clients may also cache file metadata. Metadata and chunk server exchange: The metadata server is only aware of each chunk server as long as they have a TCP connection. When a chunk server first connects to the metadata server, it authenticates itself and sends its own IP addresses. If the server is authentic, the metadata server requests the full list of chunks the chunk server is holding. Upon receiving the state of chunk server, the metadata server adds all useful chunks to the "old_list" and all useless chunks to the "rem_list", then sends the rem_list to the chunk server which removes those chunks. When writes are committed to the metadata server involving new chunks to a chunk server, the metadata server adds those hashes to an "add_list" and any hashes that are not useful anymore to the rem_list. Periodically, the metadata server sends the add_list and rem_list to the chunk server. These list tell the chunk server the ideal state it should have from the point of view of the metadata server. Elements in the add_list should already be in the chunk servers, and elements from the rem_list are to be removed. A chunk server marks any chunk in the rem_list as to be removed and checks that hashes in the add list are present. If a chunk in the add list is marked as to be removed, it is unmarked. When a chunk is marked as to be removed for a certain amount of time, it is permanently deleted. When the synchronization is complete, the metadata server merges the add_list into the old_list and clears the rem_list. If chunks in the add_list are not present in the chunk server, it responds with an error message containing the list of missing chunks. The metadata server then responds with a list of chunk server addresses where the chunk server with the missing chunk can download it from. Each chunk server goes through its download list one at the time downloading the missing chunks. Note that if the chunk server finds that its holding some chunks that are not in the hash list of the metadata server, that does not mean they are orphaned. It's possible that some writes are being performed by clients that have uploaded chunks to that chunk server but didn't yet acknowledge it to the metadata server. If all goes well and the write succeded, the metadata server will add those hashes to the hash list. Chunk servers should only drop chunks if they are not referenced by the metadata server for a period of time (say, 30 minutes). Security All nodes of the system share a secret key and use it to authenticate each other and encrypt messages. This allows the server to accept new chunk servers and clients with no prior setup Reliability The metadata server is a single point of failure. To reduce the impact of crashes, the metadata server stores all write operations into a write-ahead log that is replayed any time the process goes online. Chunk Management: Chunks are added to the system when: 1. A chunk server connects 2. A write operation on metadata occurs (adding chunks) They are removed when: 1. A chunk server disconnects 2. A write operation on metadata occurs (overwriting old chunks) 3. A delete operation on metadata occurs 4. A chunk is corrupted or removed forcefully from the chunk server The system must make sure that chunks are not over-replicated or under-replicated. If they are over-replicated, some chunk servers need to forget some copies. If they are under-replicated, some chunk servers need to copy chunks from elsewhere. The metadata server holds an authoritative list of hashes that each chunk server needs to hold. These lists are enforced periodically and automatically during the state updates, therefore the metadata server only needs to reorganize the chunks in its in-memory lists. The changes will propagate over the chunk servers in time. The metadata server's in-memory representation is the ideal state the system is converging to, which means every chunk if always replicated correctly. Given this setup, a chunk can only be under-replicated when there are not enough chunk servers. Case 1: The first chunk server connects with no chunks CS connects MS accepts CS sends auth message MS validates auth MS requests list of chunks CS sends the chunk list MS notices it's empty Case 2: The first chunk server connects with some chunks CS connects MS accepts CS sends auth message MS valudates auth MS requests list of chunks CS sends the chunk list MS iterates over each chunk. For each chunk it determines whether it's useless, under-replicated, over-replicated, or properly replicated. MS adds all useless or over-replicated chunks to the rem_list of the newly connected chunk server. Note that it's only possible for a chunk to be over-replicated by 1, so removing it from the new chunk server will fix the issue. MS If a chunk is under-replicated, it means that there are not enough chunk servers to hold all the replicas, so there is nothing to be done. Case 3: Chunk server with some chunks disconnects TODO Event 3: A chunk server disconnects A number of chunks are lost, but there is a chance the server will come back in a short while (10s or so). => Wait for about 10s. If the chunk server doesn't come back, distribute its hashes to other chunk servers Event 4: A write operation on metadata occurs (overwriting old chunks) Assuming the write overwrites some chunks: - The chunks may still be referenced by some other file or section of the same file => Do nothing - The chunks are not used anymore => Find all chunk servers holding the chunk and mark them for removal Event 5: A delete operation on metadata occurs Do the same as event 4 Event 6: A chunk is corrupted or removed forcefully from the chunk server The chunk server adds the chunk name to a list of lost chunks and sends them to the metadata server at the next periodic update.