cozis/ToastyFS
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Clean up chunk management section of DESIGN.txt

Browse Source
This commit is contained in:
cozis
2025-11-13 12:09:29 +01:00
parent 14cd2f1ac6
commit 4ec47e66b4
2 changed files with 52 additions and 52 deletions
Download Patch File Download Diff File Expand all files Collapse all files
DESIGN.txt
+46 -52
View File
@@ -137,62 +137,56 @@ Chunk Management:
servers need to forget some copies. If they are under-replicated, servers need to forget some copies. If they are under-replicated,
some chunk servers need to copy chunks from elsewhere. some chunk servers need to copy chunks from elsewhere.
The metadata server holds an authoritative list of hashes that each Metadata server variables for a chunk server:
chunk server needs to hold. These lists are enforced periodically and ms_old_list: List of chunks that are known to be held by CS
automatically during the state updates, therefore the metadata server ms_add_list: List of chunks that should be held by CS
only needs to reorganize the chunks in its in-memory lists. The changes ms_rem_list: List of chunks that may be held by CS but should removed from it
will propagate over the chunk servers in time. The metadata server's
in-memory representation is the ideal state the system is converging
to, which means every chunk if always replicated correctly. Given this
setup, a chunk can only be under-replicated when there are not enough
chunk servers.
Case 1: The first chunk server connects with no chunks Chunk server variables:
CS connects cs_add_list: List of chunks added since the last update
MS accepts cs_rem_list: List of chunks marked for removal after a timeout
CS sends auth message
MS validates auth
MS requests list of chunks
CS sends the chunk list
MS notices it's empty
Case 2: The first chunk server connects with some chunks Metadata change for write:
CS connects When clients commit a write by adding new hashes to the metadata,
MS accepts MS adds those hashes to the ms_add_list for the CS where the client
CS sends auth message uploaded the chunks. If a hash was overwritten and became useless,
MS valudates auth that hash is added to the ms_rem_list for all CS holding it.
MS requests list of chunks
CS sends the chunk list
MS iterates over each chunk. For each chunk it determines
whether it's useless, under-replicated, over-replicated,
or properly replicated.
MS adds all useless or over-replicated chunks to the rem_list
of the newly connected chunk server. Note that it's only
possible for a chunk to be over-replicated by 1, so removing
it from the new chunk server will fix the issue.
MS If a chunk is under-replicated, it means that there are not
enough chunk servers to hold all the replicas, so there is
nothing to be done.
Case 3: Chunk server with some chunks disconnects Metadata change for delete:
TODO All hashes that are no longer reachable by the file tree are added
to the ms_rem_list of their holders
Event 3: A chunk server disconnects Chunk upload:
A number of chunks are lost, but there is a chance the server will come back When a chunk is uploaded to a chunk server, its hash is added to
in a short while (10s or so). the cs_add_list.
=> Wait for about 10s. If the chunk server doesn't come back, distribute
its hashes to other chunk servers
Event 4: A write operation on metadata occurs (overwriting old chunks) Periodically on the chunk server:
Assuming the write overwrites some chunks: Elements in the cs_rem_list have a 30 minute timeout after which they are deleted
- The chunks may still be referenced by some other file or section of the same file permanently.
=> Do nothing
- The chunks are not used anymore
=> Find all chunk servers holding the chunk and mark them for removal
Event 5: A delete operation on metadata occurs Periodically:
Do the same as event 4 CS sends cs_add_list to MS
MS may add a subset of cs_add_list to ms_add_list based on the chunk replication and distribution policy
MS sends ms_add_list and ms_rem_list to CS
CS (1) Adds all elements from ms_rem_list to cs_rem_list
(2) Adds elements in cs_add_list that are not in ms_add_list to cs_rem_list
(3) Removes elements that are in ms_add_list and in cs_rem_list from cs_rem_list
(4) Elements in ms_add_list that are not held by the chunk server are added to
a temporary list tmp_list are sent to the metadata server
MS (1) Receives tmp_list and sends download locations to CS for those chunks
(2) Moves elements from ms_add_list that are not in tmp_list into ms_old_list
(3) Sets ms_add_list equal to tmp_list
Event 6: A chunk is corrupted or removed forcefully from the chunk server Chunk replication and distribution policy:
The chunk server adds the chunk name to a list of lost chunks and During an update, when CS reports a new chunk to MS, MS has to decide whether
sends them to the metadata server at the next periodic update. to allow the CS to keep it or not.
There are 4 cases:
- The chunk is useless (not referenced by any file)
- The chunk is under-replicated even counting the new copy
- The chunk is properly replicated with the new copy
- The chunk is over-replicated with the new copy
If the chunk is useless, it is added to the cs_rem_list. If the chunk is
over-replicated, it is added to the cs_rem_list of this chunk or any other
holder of that chunk. Otherwise, add the chunk to the cs_add_list.
TODO.txt
+6
View File
@@ -8,3 +8,9 @@
- find a way to remove over-replicated chunks and formalize the chunk management policy - find a way to remove over-replicated chunks and formalize the chunk management policy
- add logging of: - add logging of:
- when chunk servers connect and disconnect to the metadata server - when chunk servers connect and disconnect to the metadata server
- Should list scenarios that need testing, like those where chunks would be dropped
Roadmap:
[ ] Complete all endpoints
[ ] Add fault injections to the simulation
[ ] Implement WAL