Add notes on the chunk management strategy to DESIGN.txt

This commit is contained in:
2025-11-11 10:48:23 +01:00
parent 012dc6bdb3
commit 641c1999ad
+50 -3
View File
@@ -121,6 +121,53 @@ Reliability
operations into a write-ahead log that is replayed any time operations into a write-ahead log that is replayed any time
the process goes online. the process goes online.
TODO: When a write occurs, the written to chunks must be marked Chunk Management:
as orphaned or "to-be-deleted" unless they are used by Chunks are added to the system when:
someone else 1. A chunk server connects
2. A write operation on metadata occurs (adding chunks)
They are removed when:
1. A chunk server disconnects
2. A write operation on metadata occurs (overwriting old chunks)
3. A delete operation on metadata occurs
4. A chunk is corrupted or removed forcefully from the chunk server
The system must make sure that chunks are not over-replicated
or under-replicated. If they are over-replicated, some chunk
servers need to forget some copies. If they are under-replicated,
some chunk servers need to copy chunks from elsewhere.
Event 1: A chunk server connects
If the chunk server is holding some chunks, for each chunk
one of the following must be true:
- The chunk is not used by the file system
=> Mark the chunk for removal
- The chunk is used and, with this copy, perfectly replicated
=> Do nothing
- The chunk is used and under-replicated
=> ???
- The chunk is used and over-replicated
=> Mark the chunk for removal
Event 2: A write operation on metadata (adding chunks)
TODO
Event 3: A chunk server disconnects
A number of chunks are lost, but there is a chance the server will come back
in a short while (10s or so).
TODO
Event 4: A write operation on metadata occurs (overwriting old chunks)
Assuming the write overwrites some chunks:
- The chunks may still be referenced by some other file or section of the same file
=> Do nothing
- The chunks are not used anymore
=> Find all chunk servers holding the chunk and mark them for removal
Event 5: A delete operation on metadata occurs
Do the same as event 4
Event 6: A chunk is corrupted or removed forcefully from the chunk server
The chunk server adds the chunk name to a list of lost chunks and
sends them to the metadata server at the next periodic update.