ToastyFS/misc/DESIGN.txt

Architecture
  A ToastyFS instance is composed by a metadata server, a number
  of chunk servers, and a number of clients.

  The metadata server stores the full file system hieararchy,
  except instead of storing the file contents, it stores an
  array of hashes of the chunks of each file. A "chunk" is a
  file range that is fixed for a single file but may vary
  between files. Chunk servers hold an array of chunks that
  are identified by their hash. The metadata server keeps
  track of which chunks each chunk server is holding.

  Clients are users of the file system that can read and
  write metadata and files. They are assumed to behave
  correctly.

  Any read and write operation that doesn't involve file
  contents can be performed by clients by talking to the
  metadata server directly. Such operations include creating
  an empty file or a directory, deleting a file or directory,
  listing files.

  If a client wants to read a range of bytes from a file,
  it sends the metadata server the file name and range.
  The metadata server responds with the chunk size of that
  file, the list of hashes for the chunks involved in the
  read, and the IP addresses of the chunk servers that hold
  each chunk. The metadata server also adds the IP addresses
  of three chunk servers any new chunks should be written
  to. The client can then download the chunks from the chunk
  servers and reassemble the result.

  If a client wants to write at a range of bytes of a file,
  it starts by reading that range from the metadata server,
  getting the list of hashes it will modify, their locations,
  and locations for any new chunks. The client then modifies
  the chunk by sending to each chunk server the hash to modify
  and the patch (a range of bytes within a chunk plus the new
  data). The chunk server creates a new modified chunk and
  keeps the old version, then returns the new hash. If all
  modifications are successful, the client holds the set of
  old hashes and new hashes for that file range. It completes
  the write by telling the metadata server to swap the old
  hashes with the new ones. If the old hashes don't match,
  another write succeded in the mean time and touched that
  range, therefore the write fails. If the old hashes match,
  the write succeded. If the client fails to modify any
  chunks, it doesn't commit the write with the metadata server.
  Note that write failures may cause chunks to be orphaned
  on chunk servers. This is solved by a garbage collection
  algorithm implemented by the synchronization messages
  between metadata and chunk server.

  Note that clients may cache chunks and index them by their
  hash. When they read a file and receive its hashes, they may
  avoid reaching for the chunk servers if they already cached
  the chunks with those hashes. This allows reading files with
  only one round trip at no cost of correctness. If getting
  the up-to-date contents is not a concern, clients may also
  cache file metadata.

Metadata and chunk server exchange:

  The metadata server is only aware of each chunk server
  as long as they have a TCP connection. When a chunk server
  first connects to the metadata server, it authenticates
  itself and sends its own IP addresses. If the server is
  authentic, the metadata server requests the full list
  of chunks the chunk server is holding. Upon receiving the
  state of chunk server, the metadata server adds all useful
  chunks to the "old_list" and all useless chunks to the
  "rem_list", then sends the rem_list to the chunk server
  which removes those chunks.

  When writes are committed to the metadata server involving
  new chunks to a chunk server, the metadata server adds those
  hashes to an "add_list" and any hashes that are not useful
  anymore to the rem_list.

  Periodically, the metadata server sends the add_list and
  rem_list to the chunk server. These list tell the chunk
  server the ideal state it should have from the point of
  view of the metadata server. Elements in the add_list should
  already be in the chunk servers, and elements from the
  rem_list are to be removed. A chunk server marks any chunk
  in the rem_list as to be removed and checks that hashes
  in the add list are present. If a chunk in the add list
  is marked as to be removed, it is unmarked. When a chunk
  is marked as to be removed for a certain amount of time,
  it is permanently deleted. When the synchronization is
  complete, the metadata server merges the add_list into
  the old_list and clears the rem_list. If chunks in the
  add_list are not present in the chunk server, it responds
  with an error message containing the list of missing chunks.
  The metadata server then responds with a list of chunk
  server addresses where the chunk server with the missing
  chunk can download it from. Each chunk server goes
  through its download list one at the time downloading
  the missing chunks.

  Note that if the chunk server finds that its holding some
  chunks that are not in the hash list of the metadata server,
  that does not mean they are orphaned. It's possible that
  some writes are being performed by clients that have uploaded
  chunks to that chunk server but didn't yet acknowledge it
  to the metadata server. If all goes well and the write
  succeded, the metadata server will add those hashes to the
  hash list. Chunk servers should only drop chunks if they
  are not referenced by the metadata server for a period of
  time (say, 30 minutes).

Security
  All nodes of the system share a secret key and use it to
  authenticate each other and encrypt messages. This allows
  the server to accept new chunk servers and clients with
  no prior setup

Reliability
  The metadata server is a single point of failure. To reduce
  the impact of crashes, the metadata server stores all write
  operations into a write-ahead log that is replayed any time
  the process goes online.

Chunk Management:
    Chunks are added to the system when:
        1. A chunk server connects
        2. A write operation on metadata occurs (adding chunks)

    They are removed when:
        1. A chunk server disconnects
        2. A write operation on metadata occurs (overwriting old chunks)
        3. A delete operation on metadata occurs
        4. A chunk is corrupted or removed forcefully from the chunk server

    The system must make sure that chunks are not over-replicated
    or under-replicated. If they are over-replicated, some chunk
    servers need to forget some copies. If they are under-replicated,
    some chunk servers need to copy chunks from elsewhere.

    Metadata server variables for a chunk server:
        ms_old_list: List of chunks that are known to be held by CS
        ms_add_list: List of chunks that should be held by CS
        ms_rem_list: List of chunks that may be held by CS but should removed from it

    Chunk server variables:
        cs_add_list: List of chunks added since the last update
        cs_rem_list: List of chunks marked for removal after a timeout
        cs_lst_list: List of chunks that were lost due to errors or forceful removals of chunk files

    Metadata change for write:
        When clients commit a write by adding new hashes to the metadata,
        MS adds those hashes to the ms_add_list for the CS where the client
        uploaded the chunks. If a hash was overwritten and became useless,
        that hash is added to the ms_rem_list for all CS holding it.

    Metadata change for delete:
        All hashes that are no longer reachable by the file tree are added
        to the ms_rem_list of their holders

    Chunk upload:
        When a chunk is uploaded to a chunk server, its hash is added to
        the cs_add_list.

    Periodically on the chunk server:
        Elements in the cs_rem_list have a 30 minute timeout after which they are deleted
        permanently.

    Periodically:
        CS sends cs_add_list to MS
        MS may add a subset of cs_add_list to ms_add_list based on the chunk replication and distribution policy
        MS sends ms_add_list and ms_rem_list to CS
        CS (1) Adds all elements from ms_rem_list to cs_rem_list
           (2) Elements in ms_add_list that are not held by the chunk server are added to
               a temporary list tmp_list
           (3) Removes elements in ms_add_list from cs_add_list and cs_rem_list, then merges cs_add_list into cs_rem_list and clears cs_add_list.
           (4) Elements in cs_lst_list are added to tmp_list, then cs_lst_list is cleared.
           (6) tmp_list is sent to MS
           (7) cs_add_list is cleared
        MS (1) Receives tmp_list and sends download locations to CS for those chunks
           (2) Merges ms_add_list into ms_old_list, then removes all items in tmp_list from ms_old_list
           (3) Sets ms_add_list equal to tmp_list

    Chunk replication and distribution policy:
        During an update, when CS reports a new chunk to MS, MS has to decide whether
        to allow the CS to keep it or not.

        There are 4 cases:
            - The chunk is useless (not referenced by any file)
            - The chunk is under-replicated even counting the new copy
            - The chunk is properly replicated with the new copy
            - The chunk is over-replicated with the new copy

        If the chunk is not referenced by the file tree, do nothing.

        If the chunk is properly replicated or under-replicated, add it to the ms_add_list.

        If the chunk is over-replicated, either don't add it to the ms_add_list or add it to the ms_rem_list of some other holder.

    TODO: The way chunk servers tell about the chunks they are holding
          recently changed. Now instead of sending a full chunk list when
          connecting, they send batches of chunks to the metadata server
          during state updates. It also changed that now chunk servers
          initiate updates.

    When a chunk server connects (and authenticates)
    it sends alongside the auth message the hash of
    the hash list of all chunks. The metadata server
    then replies with a message saying whether it
    already cached that chunk list or not. If it didn't, the chunk server sends the entire list
    in chunks during state updates, with an increased
    update frequency.