193 lines
9.1 KiB
Plaintext
193 lines
9.1 KiB
Plaintext
Architecture
|
|
A MouseFS instance is composed by a metadata server, a number
|
|
of chunk servers, and a number of clients.
|
|
|
|
The metadata server stores the full file system hieararchy,
|
|
except instead of storing the file contents, it stores an
|
|
array of hashes of the chunks of each file. A "chunk" is a
|
|
file range that is fixed for a single file but may vary
|
|
between files. Chunk servers hold an array of chunks that
|
|
are identified by their hash. The metadata server keeps
|
|
track of which chunks each chunk server is holding.
|
|
|
|
Clients are users of the file system that can read and
|
|
write metadata and files. They are assumed to behave
|
|
correctly.
|
|
|
|
Any read and write operation that doesn't involve file
|
|
contents can be performed by clients by talking to the
|
|
metadata server directly. Such operations include creating
|
|
an empty file or a directory, deleting a file or directory,
|
|
listing files.
|
|
|
|
If a client wants to read a range of bytes from a file,
|
|
it sends the metadata server the file name and range.
|
|
The metadata server responds with the chunk size of that
|
|
file, the list of hashes for the chunks involved in the
|
|
read, and the IP addresses of the chunk servers that hold
|
|
each chunk. The metadata server also adds the IP addresses
|
|
of three chunk servers any new chunks should be written
|
|
to. The client can then download the chunks from the chunk
|
|
servers and reassemble the result.
|
|
|
|
If a client wants to write at a range of bytes of a file,
|
|
it starts by reading that range from the metadata server,
|
|
getting the list of hashes it will modify, their locations,
|
|
and locations for any new chunks. The client then modifies
|
|
the chunk by sending to each chunk server the hash to modify
|
|
and the patch (a range of bytes within a chunk plus the new
|
|
data). The chunk server creates a new modified chunk and
|
|
keeps the old version, then returns the new hash. If all
|
|
modifications are successful, the client holds the set of
|
|
old hashes and new hashes for that file range. It completes
|
|
the write by telling the metadata server to swap the old
|
|
hashes with the new ones. If the old hashes don't match,
|
|
another write succeded in the mean time and touched that
|
|
range, therefore the write fails. If the old hashes match,
|
|
the write succeded. If the client fails to modify any
|
|
chunks, it doesn't commit the write with the metadata server.
|
|
Note that write failures may cause chunks to be orphaned
|
|
on chunk servers. This is solved by a garbage collection
|
|
algorithm implemented by the synchronization messages
|
|
between metadata and chunk server.
|
|
|
|
Note that clients may cache chunks and index them by their
|
|
hash. When they read a file and receive its hashes, they may
|
|
avoid reaching for the chunk servers if they already cached
|
|
the chunks with those hashes. This allows reading files with
|
|
only one round trip at no cost of correctness. If getting
|
|
the up-to-date contents is not a concern, clients may also
|
|
cache file metadata.
|
|
|
|
Metadata and chunk server exchange:
|
|
|
|
The metadata server is only aware of each chunk server
|
|
as long as they have a TCP connection. When a chunk server
|
|
first connects to the metadata server, it authenticates
|
|
itself and sends its own IP addresses. If the server is
|
|
authentic, the metadata server requests the full list
|
|
of chunks the chunk server is holding. Upon receiving the
|
|
state of chunk server, the metadata server adds all useful
|
|
chunks to the "old_list" and all useless chunks to the
|
|
"rem_list", then sends the rem_list to the chunk server
|
|
which removes those chunks.
|
|
|
|
When writes are committed to the metadata server involving
|
|
new chunks to a chunk server, the metadata server adds those
|
|
hashes to an "add_list" and any hashes that are not useful
|
|
anymore to the rem_list.
|
|
|
|
Periodically, the metadata server sends the add_list and
|
|
rem_list to the chunk server. These list tell the chunk
|
|
server the ideal state it should have from the point of
|
|
view of the metadata server. Elements in the add_list should
|
|
already be in the chunk servers, and elements from the
|
|
rem_list are to be removed. A chunk server marks any chunk
|
|
in the rem_list as to be removed and checks that hashes
|
|
in the add list are present. If a chunk in the add list
|
|
is marked as to be removed, it is unmarked. When a chunk
|
|
is marked as to be removed for a certain amount of time,
|
|
it is permanently deleted. When the synchronization is
|
|
complete, the metadata server merges the add_list into
|
|
the old_list and clears the rem_list. If chunks in the
|
|
add_list are not present in the chunk server, it responds
|
|
with an error message containing the list of missing chunks.
|
|
The metadata server then responds with a list of chunk
|
|
server addresses where the chunk server with the missing
|
|
chunk can download it from. Each chunk server goes
|
|
through its download list one at the time downloading
|
|
the missing chunks.
|
|
|
|
Note that if the chunk server finds that its holding some
|
|
chunks that are not in the hash list of the metadata server,
|
|
that does not mean they are orphaned. It's possible that
|
|
some writes are being performed by clients that have uploaded
|
|
chunks to that chunk server but didn't yet acknowledge it
|
|
to the metadata server. If all goes well and the write
|
|
succeded, the metadata server will add those hashes to the
|
|
hash list. Chunk servers should only drop chunks if they
|
|
are not referenced by the metadata server for a period of
|
|
time (say, 30 minutes).
|
|
|
|
Security
|
|
All nodes of the system share a secret key and use it to
|
|
authenticate each other and encrypt messages. This allows
|
|
the server to accept new chunk servers and clients with
|
|
no prior setup
|
|
|
|
Reliability
|
|
The metadata server is a single point of failure. To reduce
|
|
the impact of crashes, the metadata server stores all write
|
|
operations into a write-ahead log that is replayed any time
|
|
the process goes online.
|
|
|
|
Chunk Management:
|
|
Chunks are added to the system when:
|
|
1. A chunk server connects
|
|
2. A write operation on metadata occurs (adding chunks)
|
|
|
|
They are removed when:
|
|
1. A chunk server disconnects
|
|
2. A write operation on metadata occurs (overwriting old chunks)
|
|
3. A delete operation on metadata occurs
|
|
4. A chunk is corrupted or removed forcefully from the chunk server
|
|
|
|
The system must make sure that chunks are not over-replicated
|
|
or under-replicated. If they are over-replicated, some chunk
|
|
servers need to forget some copies. If they are under-replicated,
|
|
some chunk servers need to copy chunks from elsewhere.
|
|
|
|
Metadata server variables for a chunk server:
|
|
ms_old_list: List of chunks that are known to be held by CS
|
|
ms_add_list: List of chunks that should be held by CS
|
|
ms_rem_list: List of chunks that may be held by CS but should removed from it
|
|
|
|
Chunk server variables:
|
|
cs_add_list: List of chunks added since the last update
|
|
cs_rem_list: List of chunks marked for removal after a timeout
|
|
|
|
Metadata change for write:
|
|
When clients commit a write by adding new hashes to the metadata,
|
|
MS adds those hashes to the ms_add_list for the CS where the client
|
|
uploaded the chunks. If a hash was overwritten and became useless,
|
|
that hash is added to the ms_rem_list for all CS holding it.
|
|
|
|
Metadata change for delete:
|
|
All hashes that are no longer reachable by the file tree are added
|
|
to the ms_rem_list of their holders
|
|
|
|
Chunk upload:
|
|
When a chunk is uploaded to a chunk server, its hash is added to
|
|
the cs_add_list.
|
|
|
|
Periodically on the chunk server:
|
|
Elements in the cs_rem_list have a 30 minute timeout after which they are deleted
|
|
permanently.
|
|
|
|
Periodically:
|
|
CS sends cs_add_list to MS
|
|
MS may add a subset of cs_add_list to ms_add_list based on the chunk replication and distribution policy
|
|
MS sends ms_add_list and ms_rem_list to CS
|
|
CS (1) Adds all elements from ms_rem_list to cs_rem_list
|
|
(2) Adds elements in cs_add_list that are not in ms_add_list to cs_rem_list
|
|
(3) Removes elements that are in ms_add_list and in cs_rem_list from cs_rem_list
|
|
(4) Elements in ms_add_list that are not held by the chunk server are added to
|
|
a temporary list tmp_list are sent to the metadata server
|
|
MS (1) Receives tmp_list and sends download locations to CS for those chunks
|
|
(2) Moves elements from ms_add_list that are not in tmp_list into ms_old_list
|
|
(3) Sets ms_add_list equal to tmp_list
|
|
|
|
Chunk replication and distribution policy:
|
|
During an update, when CS reports a new chunk to MS, MS has to decide whether
|
|
to allow the CS to keep it or not.
|
|
|
|
There are 4 cases:
|
|
- The chunk is useless (not referenced by any file)
|
|
- The chunk is under-replicated even counting the new copy
|
|
- The chunk is properly replicated with the new copy
|
|
- The chunk is over-replicated with the new copy
|
|
|
|
If the chunk is useless, it is added to the cs_rem_list. If the chunk is
|
|
over-replicated, it is added to the cs_rem_list of this chunk or any other
|
|
holder of that chunk. Otherwise, add the chunk to the cs_add_list.
|