From 1d3ce1247e764b75c831028336278ba59b3d8e80 Mon Sep 17 00:00:00 2001 From: Francesco Cozzuto Date: Mon, 10 Nov 2025 13:36:28 +0100 Subject: [PATCH] Add DESIGN.txt --- DESIGN.txt | 126 +++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 126 insertions(+) create mode 100644 DESIGN.txt diff --git a/DESIGN.txt b/DESIGN.txt new file mode 100644 index 0000000..8dbfaa0 --- /dev/null +++ b/DESIGN.txt @@ -0,0 +1,126 @@ +Architecture + A TinyDFS instance is composed by a metadata server, a number + of chunk servers, and a number of clients. + + The metadata server stores the full file system hieararchy, + except instead of storing the file contents, it stores an + array of hashes of the chunks of each file. A "chunk" is a + file range that is fixed for a single file but may vary + between files. Chunk servers hold an array of chunks that + are identified by their hash. The metadata server keeps + track of which chunks each chunk server is holding. + + Clients are users of the file system that can read and + write metadata and files. They are assumed to behave + correctly. + + Any read and write operation that doesn't involve file + contents can be performed by clients by talking to the + metadata server directly. Such operations include creating + an empty file or a directory, deleting a file or directory, + listing files. + + If a client wants to read a range of bytes from a file, + it sends the metadata server the file name and range. + The metadata server responds with the chunk size of that + file, the list of hashes for the chunks involved in the + read, and the IP addresses of the chunk servers that hold + each chunk. The metadata server also adds the IP addresses + of three chunk servers any new chunks should be written + to. The client can then download the chunks from the chunk + servers and reassemble the result. + + If a client wants to write at a range of bytes of a file, + it starts by reading that range from the metadata server, + getting the list of hashes it will modify, their locations, + and locations for any new chunks. The client then modifies + the chunk by sending to each chunk server the hash to modify + and the patch (a range of bytes within a chunk plus the new + data). The chunk server creates a new modified chunk and + keeps the old version, then returns the new hash. If all + modifications are successful, the client holds the set of + old hashes and new hashes for that file range. It completes + the write by telling the metadata server to swap the old + hashes with the new ones. If the old hashes don't match, + another write succeded in the mean time and touched that + range, therefore the write fails. If the old hashes match, + the write succeded. If the client fails to modify any + chunks, it doesn't commit the write with the metadata server. + Note that write failures may cause chunks to be orphaned + on chunk servers. This is solved by a garbage collection + algorithm implemented by the synchronization messages + between metadata and chunk server. + + Note that clients may cache chunks and index them by their + hash. When they read a file and receive its hashes, they may + avoid reaching for the chunk servers if they already cached + the chunks with those hashes. This allows reading files with + only one round trip at no cost of correctness. If getting + the up-to-date contents is not a concern, clients may also + cache file metadata. + +Metadata and chunk server exchange: + + The metadata server is only aware of each chunk server + as long as they have a TCP connection. When a chunk server + first connects to the metadata server, it authenticates + itself and sends its own IP addresses. If the server is + authentic, the metadata server requests the full list + of chunks the chunk server is holding. Upon receiving the + state of chunk server, the metadata server adds all useful + chunks to the "old_list" and all useless chunks to the + "rem_list", then sends the rem_list to the chunk server + which removes those chunks. + + When writes are committed to the metadata server involving + new chunks to a chunk server, the metadata server adds those + hashes to an "add_list" and any hashes that are not useful + anymore to the rem_list. + + Periodically, the metadata server sends the add_list and + rem_list to the chunk server. These list tell the chunk + server the ideal state it should have from the point of + view of the metadata server. Elements in the add_list should + already be in the chunk servers, and elements from the + rem_list are to be removed. A chunk server marks any chunk + in the rem_list as to be removed and checks that hashes + in the add list are present. If a chunk in the add list + is marked as to be removed, it is unmarked. When a chunk + is marked as to be removed for a certain amount of time, + it is permanently deleted. When the synchronization is + complete, the metadata server merges the add_list into + the old_list and clears the rem_list. If chunks in the + add_list are not present in the chunk server, it responds + with an error message containing the list of missing chunks. + The metadata server then responds with a list of chunk + server addresses where the chunk server with the missing + chunk can download it from. Each chunk server goes + through its download list one at the time downloading + the missing chunks. + + Note that if the chunk server finds that its holding some + chunks that are not in the hash list of the metadata server, + that does not mean they are orphaned. It's possible that + some writes are being performed by clients that have uploaded + chunks to that chunk server but didn't yet acknowledge it + to the metadata server. If all goes well and the write + succeded, the metadata server will add those hashes to the + hash list. Chunk servers should only drop chunks if they + are not referenced by the metadata server for a period of + time (say, 30 minutes). + +Security + All nodes of the system share a secret key and use it to + authenticate each other and encrypt messages. This allows + the server to accept new chunk servers and clients with + no prior setup + +Reliability + The metadata server is a single point of failure. To reduce + the impact of crashes, the metadata server stores all write + operations into a write-ahead log that is replayed any time + the process goes online. + +TODO: When a write occurs, the written to chunks must be marked + as orphaned or "to-be-deleted" unless they are used by + someone else