diff --git a/spec/design.txt b/spec/design.txt new file mode 100644 index 0000000..7415df2 --- /dev/null +++ b/spec/design.txt @@ -0,0 +1,233 @@ +Architecture + A ToastyFS instance is composed by a metadata server, a number + of chunk servers, and a number of clients. + + The metadata server stores the full file system hieararchy, + except instead of storing the file contents, it stores an + array of hashes of the chunks of each file. A "chunk" is a + file range that is fixed for a single file but may vary + between files. Chunk servers hold an array of chunks that + are identified by their hash. The metadata server keeps + track of which chunks each chunk server is holding. + + Clients are users of the file system that can read and + write metadata and files. They are assumed to behave + correctly. + + Any read and write operation that doesn't involve file + contents can be performed by clients by talking to the + metadata server directly. Such operations include creating + an empty file or a directory, deleting a file or directory, + listing files. + + If a client wants to read a range of bytes from a file, + it sends the metadata server the file name and range. + The metadata server responds with the chunk size of that + file, the list of hashes for the chunks involved in the + read, and the IP addresses of the chunk servers that hold + each chunk. The metadata server also adds the IP addresses + of three chunk servers any new chunks should be written + to. The client can then download the chunks from the chunk + servers and reassemble the result. + + If a client wants to write at a range of bytes of a file, + it starts by reading that range from the metadata server, + getting the list of hashes it will modify, their locations, + and locations for any new chunks. The client then modifies + the chunk by sending to each chunk server the hash to modify + and the patch (a range of bytes within a chunk plus the new + data). The chunk server creates a new modified chunk and + keeps the old version, then returns the new hash. If all + modifications are successful, the client holds the set of + old hashes and new hashes for that file range. It completes + the write by telling the metadata server to swap the old + hashes with the new ones (optionally including write flags + such as TOASTY_WRITE_CREATE_IF_MISSING). If the old hashes + don't match, another write succeded in the mean time and + touched that range, therefore the write fails. If the old + hashes match, the write succeded. If the client fails to + modify any chunks, it doesn't commit the write with the + metadata server. + + If the file doesn't exist and the TOASTY_WRITE_CREATE_IF_MISSING + flag is set, the metadata server atomically creates the file + with a default chunk size (4096 bytes) and retries the write. + This operation is logged to the WAL to ensure crash consistency. + + If the TOASTY_WRITE_TRUNCATE_AFTER flag is set, the file is + truncated after the write, setting its size to exactly offset+length + and discarding any data beyond that point. This is useful for HTTP + PUT semantics where the entire file content should be replaced. + + Note that write failures may cause chunks to be orphaned + on chunk servers. This is solved by a garbage collection + algorithm implemented by the synchronization messages + between metadata and chunk server. + + Note that clients may cache chunks and index them by their + hash. When they read a file and receive its hashes, they may + avoid reaching for the chunk servers if they already cached + the chunks with those hashes. This allows reading files with + only one round trip at no cost of correctness. If getting + the up-to-date contents is not a concern, clients may also + cache file metadata. + +Metadata and chunk server exchange: + + The metadata server is only aware of each chunk server + as long as they have a TCP connection. When a chunk server + first connects to the metadata server, it authenticates + itself and sends its own IP addresses. If the server is + authentic, the metadata server requests the full list + of chunks the chunk server is holding. Upon receiving the + state of chunk server, the metadata server adds all useful + chunks to the "old_list" and all useless chunks to the + "rem_list", then sends the rem_list to the chunk server + which removes those chunks. + + When writes are committed to the metadata server involving + new chunks to a chunk server, the metadata server adds those + hashes to an "add_list" and any hashes that are not useful + anymore to the rem_list. + + Periodically, the metadata server sends the add_list and + rem_list to the chunk server. These list tell the chunk + server the ideal state it should have from the point of + view of the metadata server. Elements in the add_list should + already be in the chunk servers, and elements from the + rem_list are to be removed. A chunk server marks any chunk + in the rem_list as to be removed and checks that hashes + in the add list are present. If a chunk in the add list + is marked as to be removed, it is unmarked. When a chunk + is marked as to be removed for a certain amount of time, + it is permanently deleted. When the synchronization is + complete, the metadata server merges the add_list into + the old_list and clears the rem_list. If chunks in the + add_list are not present in the chunk server, it responds + with an error message containing the list of missing chunks. + The metadata server then responds with a list of chunk + server addresses where the chunk server with the missing + chunk can download it from. Each chunk server goes + through its download list one at the time downloading + the missing chunks. + + Note that if the chunk server finds that its holding some + chunks that are not in the hash list of the metadata server, + that does not mean they are orphaned. It's possible that + some writes are being performed by clients that have uploaded + chunks to that chunk server but didn't yet acknowledge it + to the metadata server. If all goes well and the write + succeded, the metadata server will add those hashes to the + hash list. Chunk servers should only drop chunks if they + are not referenced by the metadata server for a period of + time (say, 30 minutes). + +Security + All nodes of the system share a secret key and use it to + authenticate each other and encrypt messages. This allows + the server to accept new chunk servers and clients with + no prior setup + +Reliability + The metadata server is a single point of failure. To reduce + the impact of crashes, the metadata server stores all write + operations into a write-ahead log that is replayed any time + the process goes online. + +Chunk Management: + Chunks are added to the system when: + 1. A chunk server connects + 2. A write operation on metadata occurs (adding chunks) + + They are removed when: + 1. A chunk server disconnects + 2. A write operation on metadata occurs (overwriting old chunks) + 3. A delete operation on metadata occurs + 4. A chunk is corrupted or removed forcefully from the chunk server + + The system must make sure that chunks are not over-replicated + or under-replicated. If they are over-replicated, some chunk + servers need to forget some copies. If they are under-replicated, + some chunk servers need to copy chunks from elsewhere. + + Metadata server variables for a chunk server: + ms_old_list: List of chunks that are known to be held by CS + ms_add_list: List of chunks that should be held by CS + ms_rem_list: List of chunks that may be held by CS but should removed from it + + Chunk server variables: + cs_add_list: List of chunks added since the last update + cs_rem_list: List of chunks marked for removal after a timeout + cs_lst_list: List of chunks that were lost due to errors or forceful removals of chunk files + + Metadata change for write: + When clients commit a write by adding new hashes to the metadata, + MS adds those hashes to the ms_add_list for the CS where the client + uploaded the chunks. If a hash was overwritten and became useless, + that hash is added to the ms_rem_list for all CS holding it. + + Metadata change for delete: + All hashes that are no longer reachable by the file tree are added + to the ms_rem_list of their holders + + Chunk upload: + When a chunk is uploaded to a chunk server, its hash is added to + the cs_add_list. + + Periodically on the chunk server: + Elements in the cs_rem_list have a 30 minute timeout after which they are deleted + permanently. + + Periodically: + CS sends cs_add_list to MS + MS may add a subset of cs_add_list to ms_add_list based on the chunk replication and distribution policy + MS sends ms_add_list and ms_rem_list to CS + CS (1) Adds all elements from ms_rem_list to cs_rem_list + (2) Elements in ms_add_list that are not held by the chunk server are added to + a temporary list tmp_list + (3) Removes elements in ms_add_list from cs_add_list and cs_rem_list, then merges cs_add_list into cs_rem_list and clears cs_add_list. + (4) Elements in cs_lst_list are added to tmp_list, then cs_lst_list is cleared. + (6) tmp_list is sent to MS + (7) cs_add_list is cleared + MS (1) Receives tmp_list and sends download locations to CS for those chunks + (2) Merges ms_add_list into ms_old_list, then removes all items in tmp_list from ms_old_list + (3) Sets ms_add_list equal to tmp_list + + Chunk replication and distribution policy: + During an update, when CS reports a new chunk to MS, MS has to decide whether + to allow the CS to keep it or not. + + There are 4 cases: + - The chunk is useless (not referenced by any file) + - The chunk is under-replicated even counting the new copy + - The chunk is properly replicated with the new copy + - The chunk is over-replicated with the new copy + + If the chunk is not referenced by the file tree, do nothing. + + If the chunk is properly replicated or under-replicated, add it to the ms_add_list. + + If the chunk is over-replicated, either don't add it to the ms_add_list or add it to the ms_rem_list of some other holder. + + TODO: The way chunk servers tell about the chunks they are holding + recently changed. Now instead of sending a full chunk list when + connecting, they send batches of chunks to the metadata server + during state updates. It also changed that now chunk servers + initiate updates. + + When a chunk server connects (and authenticates) + it sends alongside the auth message the hash of + the hash list of all chunks. The metadata server + then replies with a message saying whether it + already cached that chunk list or not. If it didn't, the chunk server sends the entire list + in chunks during state updates, with an increased + update frequency. + +Metadata Persistence & Crash Recovery: + The metadata server uses a write-ahead log (WAL) file to store its state on disk. + + Log files start with a full snapshot of the metadata and continues with operation + log entries. + + When a file gets too big, the metadata server creates a new WAL file by writing + a snapshot to it and continuing logging there. \ No newline at end of file diff --git a/spec/protocol.txt b/spec/protocol.txt new file mode 100644 index 0000000..4b32e7a --- /dev/null +++ b/spec/protocol.txt @@ -0,0 +1,281 @@ +1. Introduction & Notation + +The DESIGN.txt file gives an overview of the system +and how nodes of a cluster interact with each other. +This file documents the specific binary format used +to exchange information between nodes. + +All messages start with a shared header, defined as: + + struct Header { + uint16_t version; + uint16_t type; + uint32_t length; + }; + +1.1 Generation Counter Special Values + +Generation counters (uint64_t) have two special values: + + NO_GENERATION (0): + When used in expect_gen, means "skip generation check entirely". + Files/directories are never assigned this generation value. + + MISSING_FILE_GENERATION (UINT64_MAX): + When used in expect_gen, means "expect file/directory to NOT exist". + - For DELETE: succeeds if file doesn't exist (no-op) + - For WRITE: fails with BADGEN if file exists, fails with NOENT if missing + - For existing operations: causes generation mismatch if file exists + Files/directories are never assigned this generation value. + +2. Client Messages + +2.1 Client to Metadata Server messages + +Let's start from the interactions between a client and +the metadata server: + +[ CREATE | C -> MS ] + Upon file creation, the client sends a "CREATE" message with the following layout: + + struct CreateMessage { + Header header; // type=CREATE + uint16_t path_len; + char path[path_len]; + uint8_t is_dir; + if (is_dir != 0) { + uint32_t chunk_size; + } + }; + + Note that in general only paths up to 65K bytes are + supported and that the layout of fields may depend + on the value of others. + + The server then responds with a CREATE_SUCCESS or CREATE_ERROR message. + +[ DELETE | C -> MS ] + When a client deletes a file, it sends a DELETE + message with the following layout: + + struct DeleteMessage { + Header header; // type=DELETE + uint64_t expect_gen; + uint16_t path_len; + char path[path_len]; + }; + + The file/directory at the given path is only deleted if its generation counter matches expect_gen. If expect_gen is 0, the file/directory is deleted regardless. + + The server then responds with a DELETE_SUCCESS or DELETE_ERROR message. + +[ LIST | C -> MS ] + When a client requests a directory listing, it sends a LIST message whith the following format: + + struct ListMessage { + Header header; // type=LIST + uint64_t expect_gen; + uint16_t path_len; + char path[path_len]; + }; + + If the expect_gen field doesn't match the generation counter of the directory, the operation fails. If expect_gen is 0, the check is skipped. + + The server then responds with a LIST_SUCCESS or LIST_ERROR message. + +[ READ | C -> MS ] + When a client requests to read a file, it sends a READ message with the following format: + + struct ReadMessage { + Header header; // type=READ + uint64_t expect_gen; + uint16_t path_len; + char path[path_len]; + uint32_t offset; + uint32_t length; + }; + + The expect_gen field is the expected generation counter for the target resource. If it doesn't match, the operation fails. If expect_gen is 0, the check is skipped. + + The offset and length fields determine the region to be read from the file. + +[ WRITE | C -> MS ] + When a client wants to write to a file, it sends a WRITE message with the following layout: + + struct Location { + uint8_t is_ipv4; + if (is_ipv4) { + uint32_t ipv4; + } else { + uint128_t ipv6; + } + uint16_t port; + }; + + struct WriteChunk { + SHA256 hash; + uint32_t num_locations; + Location locations[num_locations]; + }; + + struct WriteMessage { + Header header; // type=WRITE + uint64_t expect_gen; + uint32_t flags; + uint16_t path_len; + char path[path_len]; + uint32_t offset; + uint32_t length; + uint32_t num_chunks; + WriteChunk chunks[num_chunks]; + }; + + If the expect_gen field doesn't match the generation of the target file, the operation fails. Note that unlike other operations, the expect_gen CAN'T be 0. This is due to the assumption that the chunk size hasn't change for that file since the writer originally retrieved the file's metadata. + + The flags field contains write operation flags. Currently defined flags: + - TOASTY_WRITE_CREATE_IF_MISSING (0x01): If the file doesn't exist, + the metadata server will atomically create it with a default chunk + size of 4096 bytes and retry the write operation. The creation is + logged to the WAL for crash consistency. + - TOASTY_WRITE_TRUNCATE_AFTER (0x02): Truncate the file after the write + operation, setting the file size to exactly offset+length. Any data + beyond this point is discarded. Useful for HTTP PUT semantics where + the entire file content should be replaced. + + The offset and length mark the region that is being written to. + + Then comes an array of num_chunks sections each specifying where a given chunk was written to. Note that the number of chunks is equal to + + num_chunks == length / chunk_size + + Where chunk_size is the one for the target file at the specified generation. + + Each WriteChunk lists the new hashes for the file in the write range and for each one it lists all the chunk servers that are now holding a copy of it. + +2.2 Client to Chunk Server messages + + TODO + +3. Metadata Server messages + +The following is the list of messages the Metadata Server may send to a Client: + +[ CREATE_ERROR | MS -> C ] + When a client sends a CREATE request to the metadata server and the operation fails, the metadata server responds with a CREATE_ERROR message with the following layout: + + struct CreateErrorMessage { + Header header; // type=CREATE_ERROR + uint16_t message_len; + char message[message_len]; + }; + +[ CREATE_SUCCESS | MS -> C ] + When a client sends a CREATE requests to the metadata server which succedes, the metadata server replies with a CREATE_SUCCESS message with the following layout: + + struct CreateSuccessMessage { + Header header; // type=CREATE_SUCCESS + uint64_t gen; + }; + + The gen field is the generation counter given to the file. + +[ DELETE_ERROR | MS -> C ] + See CREATE_ERROR + +[ DELETE_SUCCESS | MS -> C ] + When a client sends a DELETE operation to the metadata server which succedes, a DELETE_SUCCESS message is sent back with the following layout: + + struct DeleteSuccessMessage { + Header header; // type=DELETE_SUCCESS + }; + + It does not store any fields other than the header. + +[ LIST_ERROR | MS -> C ] + When a client sends a LIST request to the metadata server and it fails, the server replies with a LIST_ERROR message with the following layout: + + struct ListErrorMessage { + Header header; // type=LIST_ERROR + uint16_t message_len; + char message[message_len]; + }; + +[ LIST_SUCCESS | MS -> C ] + When a client sends a LIST request to the metadata server and it succedes, the server replies with a LIST_SUCCESSS message with the following layout: + + struct ListItem { + uint64_t gen; + uint8_t is_dir; + uint16_t name_len; + char name[name_len]; + }; + + struct ListSuccessMessage { + Header header; // type=LIST_SUCCESS + uint64_t gen; + uint8_t truncated; + uint32_t item_count; + ListItem items[item_count]; + }; + + The ListSuccessMessage gen field contains the generation counter for the directory, while the gen field in ListItem is the counter for that specific child. + + If the truncated field is non-zero, the actual item count for this directory is greater than the one sent in the message. + +[ READ_ERROR | MS -> C ] + TODO + +[ READ_SUCCESS | MS -> C ] + + struct IPv4AndPort { + uint32_t ipv4; + uint16_t port; + }; + + struct IPv6AndPort { + uint128_t ipv6; + uint16_t port; + }; + + struct ChunkServerAddrList { + uint32_t num_ipv4; + IPv4AndPort ipv4s[num_ipv4]; + uint32_t num_ipv6; + IPv6AndPort ipv6s[num_ipv6]; + }; + + struct ReadChunk { + SHA256 hash; + uint32_t num_holders; + ChunkServerAddrList holders[num_holders]; + }; + + struct ReadSuccessMessage { + Header header; // type=READ_SUCCESS + uint64_t gen; + uint32_t chunk_size; + uint32_t file_length; + uint32_t num_hashes; + ReadChunk chunks[num_hashes]; + uint32_t num_write_locations; + ChunkServerAddrList write_locations[num_write_locations]; + }; + + The message returns general information about the file such as its generation counter, length in bytes, and chunk size. + + The chunks list contains the hashes of the chunks touched by the read and the list of chunk servers that are holding them. + + The write locations are a list of chunk servers that clients may write new chunks to. + +[ WRITE_ERROR | MS -> C ] + See CREATE_ERROR + +[ WRITE_SUCCESS | MS -> C ] + When a client sends a WRITE message which succedes, the metadata server responds with a WRITE_SUCCESS message with the following layout: + + struct WriteSuccessMessage { + Header header; // type=WRITE_SUCCESS + uint64_t gen; + }; + + The gen field is the new generation counter for that file. \ No newline at end of file diff --git a/spec/spec.txt b/spec/spec.txt index 59604cb..21059b2 100644 --- a/spec/spec.txt +++ b/spec/spec.txt @@ -35,15 +35,94 @@ metadata_server_tick: if the peer is a client: if the message has type CREATE: - TODO + Read path_len, path, and is_dir from the message + If is_dir is false: + Read chunk_size from message + Else: + Set chunk_size to 0 + + Append the create operation to the WAL + Attempt to create entity in the file_tree + + If file_tree creation fails: + Send CREATE_ERROR with the error description + Else: + Send CREATE_SUCCESS containing the new generation number + if the message has type DELETE: - TODO + Read expect_gen, path_len, and path from message + Append delete operation to the WAL + Attempt to delete entity in the file_tree + + If file_tree deletion fails: + Send DELETE_ERROR with the error description + Else: + Send DELETE_SUCCESS (no payload) + if the message has type LIST: - TODO + Read expect_gen, path_len, and path from message + Attempt to list directory contents from file_tree + + If listing fails: + Send LIST_ERROR with error description + Else: + Send LIST_SUCCESS containing: + - Directory generation + - Truncated flag (if results exceed MAX_LIST_SIZE) + - Item count + - List of items (generation, is_dir, name_len, name) + if the message has type READ: - TODO + Read expect_gen, path_len, path, offset, and length + Attempt to read file info from file_tree + + If read fails: + Send READ_ERROR containing: + - Error description + - A list of suggested chunk server addresses for writing (load balanced) + NOTE: Providing write locations on read error allows clients to handle + "missing file" scenarios by immediately writing if desired. + Else: + Send READ_SUCCESS containing: + - File generation + - Chunk size + - Actual bytes available (may be less than requested length) + - Number of chunks + - List of chunks, where each contains: + - SHA256 Hash + - List of chunk servers holding this chunk (holders) + - A list of suggested chunk server addresses for writing new chunks + (load balanced via choose_servers_for_write) + if the message has type WRITE: - TODO + Read expect_gen, flags, path_len, path + Read offset, length, num_chunks + + For each chunk in num_chunks: + Read the hash and the list of Chunk Server addresses (locations) that hold it + (This implies the client has already uploaded the data to these servers) + + Append write operation to WAL + + Attempt to apply write to file_tree (returns new_gen and a list of removed_hashes) + + If result is NOENT (File Not Found) AND flags has TOASTY_WRITE_CREATE_IF_MISSING: + - Append create operation (default chunk_size 4096) to WAL + - Create entity in file_tree + - Retry the write operation to file_tree + + If write fails: + Send WRITE_ERROR with error description + Else: + For each new chunk hash provided in the message: + Find the chunk servers associated with that hash in the message + Add the hash to those servers' ms_add_list + + For each removed_hash returned by the file_tree: + Find all chunk servers currently holding this hash + Add the hash to their ms_rem_list (marking it for deletion) + + Send WRITE_SUCCESS containing the new generation number if the peer is a CS: if the message has type AUTH: @@ -175,3 +254,229 @@ chunk_server_tick: === CLIENT LIBRARY ====================================== ========================================================= +# ------------------------------------------------------------------------------ +# 1. DATA STRUCTURES +# ------------------------------------------------------------------------------ + +class ToastyClient: + metadata_server_conn: Connection + chunk_server_pool: Map + pending_operations: Map + +class Operation: + id: Integer + type: Enum(READ, WRITE, CREATE, DELETE, LIST) + status: Enum(PENDING, COMPLETED, FAILED) + + # Context + path: String + user_buffer: Byte[] + offset: Integer + length: Integer + flags: Bitmask + expected_generation: Integer + + # Internal State + chunks_to_process: List + bytes_transferred: Integer + result_data: Any + +class ChunkTask: + # Used for tracking individual chunk uploads/downloads + chunk_index: Integer + hash: String + target_servers: List
+ status: Enum(WAITING, IN_PROGRESS, DONE) + final_hash: String # Returned by CS after upload/patch + +# ------------------------------------------------------------------------------ +# 2. PUBLIC BLOCKING API (User Facing) +# ------------------------------------------------------------------------------ + +function toasty_connect(address, port): + client = new ToastyClient() + client.metadata_server_conn = tcp_connect(address, port) + return client + +function toasty_read(client, path, offset, length): + op_handle = begin_read_op(client, path, offset, length) + return wait_for_result(client, op_handle) + +function toasty_write(client, path, data, offset, flags): + op_handle = begin_write_op(client, path, data, offset, flags) + return wait_for_result(client, op_handle) + +# (Create, Delete, and List follow the same pattern: begin_op -> wait_for_result) + +# ------------------------------------------------------------------------------ +# 3. INTERNAL ASYNCHRONOUS LOGIC +# ------------------------------------------------------------------------------ + +# --- READ LOGIC --- + +function begin_read_op(client, path, offset, length): + op = new Operation(type=READ, path, offset, length) + + # Step 1: Ask Metadata Server (MS) for the file layout + send_message(client.metadata_server_conn, + type=READ_REQUEST, + path=path, range=(offset, length)) + + return register_operation(client, op) + +function handle_read_metadata_response(op, msg): + if msg.type == ERROR: + op.status = FAILED + return + + # Step 2: Calculate chunk plan + chunk_size = msg.chunk_size + op.chunks_to_process = calculate_required_chunks(op.offset, op.length, chunk_size) + + # Step 3: Queue downloads from Chunk Servers (CS) + for chunk in op.chunks_to_process: + # MS returns a list of holders for each chunk hash + best_cs = load_balance_select(chunk.holders) + conn = get_cs_connection(best_cs) + + send_message(conn, + type=DOWNLOAD_CHUNK, + hash=chunk.hash) + +function handle_chunk_download_success(op, msg): + # Step 4: Assemble data + copy_bytes(src=msg.data, dest=op.user_buffer, at_offset=msg.chunk_index) + + mark_task_complete(op, msg.chunk_index) + + if all_tasks_complete(op): + op.status = COMPLETED + +# --- WRITE LOGIC (The Complex Part) --- + +function begin_write_op(client, path, data, offset, flags): + op = new Operation(type=WRITE, path, data, offset, flags) + + # Step 1: Send READ to MS first. + # We need the current generation number and existing chunk hashes to perform + # patches or ensure consistency, even if we are overwriting. + send_message(client.metadata_server_conn, + type=READ_REQUEST, + path=path, range=(offset, len(data))) + + return register_operation(client, op) + +function handle_write_metadata_ready(op, msg): + # Step 2: Prepare Upload Tasks + # If file missing & CREATE_IF_MISSING flag is set, we use defaults (size 0, gen 0). + current_generation = msg.generation + chunk_size = msg.chunk_size + write_targets = msg.write_locations # Recommended CS IPs from MS + + # Slice user data into chunks + sliced_chunks = split_into_chunks(op.user_buffer, chunk_size) + + for slice in sliced_chunks: + task = new ChunkTask() + task.target_servers = write_targets # We must replicate to 3 servers + + # Determine if this is a Create (New) or Update (Patch) + if slice.index >= len(msg.existing_hashes): + task.type = NEW_CHUNK + else: + task.type = PATCH_CHUNK + task.base_hash = msg.existing_hashes[slice.index] + + op.chunks_to_process.add(task) + + # Step 3: Start Uploads (Parallel up to limit) + pump_upload_queue(op) + +function pump_upload_queue(op): + while active_uploads(op) < MAX_PARALLEL_UPLOADS: + task = get_next_waiting_task(op) + if not task: break + + # We must upload to N replicas (usually 3) + for replica_addr in task.target_servers: + conn = get_cs_connection(replica_addr) + + if task.type == NEW_CHUNK: + # Full upload + send_message(conn, type=CREATE_CHUNK, data=task.data) + else: + # Differential patch (Rsync-style or offset-based) + send_message(conn, type=UPLOAD_CHUNK, + base_hash=task.base_hash, + patch_data=task.data) + +function handle_upload_ack(op, msg): + # Step 4: Collect new hashes + task = get_task(op, msg.task_id) + + # The CS computes the new SHA256 after applying the patch/data + task.final_hash = msg.new_hash + task.replicas_confirmed++ + + if task.replicas_confirmed >= REQUIRED_REPLICATION: + task.status = DONE + + if all_tasks_complete(op): + commit_write(op) + else: + pump_upload_queue(op) # Start next batch + +function commit_write(op): + # Step 5: Atomic Commit at Metadata Server + commit_msg = new Message(type=WRITE_REQUEST) + commit_msg.path = op.path + commit_msg.expected_generation = op.expected_generation # Optimistic Locking + commit_msg.flags = op.flags + + # Map the linear chunk index to the new Hash + Location list + commit_msg.chunks = map_chunks_to_hashes_and_locations(op.chunks_to_process) + + send_message(client.metadata_server_conn, commit_msg) + +function handle_commit_response(op, msg): + if msg.type == WRITE_SUCCESS: + op.status = COMPLETED + op.result_data = msg.new_generation + elif msg.type == ERROR and msg.code == GENERATION_MISMATCH: + # Optimistic concurrency failed (someone else wrote). + # Retry the whole operation from Step 1 (READ). + reset_and_retry(op) + else: + op.status = FAILED + +# ------------------------------------------------------------------------------ +# 4. EVENT LOOP (Driver) +# ------------------------------------------------------------------------------ + +function wait_for_result(client, op_handle): + while get_op_status(op_handle) == PENDING: + # Wait for network activity (Poll/Select) + events = wait_for_network_events(timeout=DEFAULT_TIMEOUT) + + for event in events: + if event.type == DISCONNECT: + handle_disconnect(client, event.conn) + continue + + # Route message to the specific Operation ID embedded in the packet tag + op = client.pending_operations[event.tag.op_id] + msg = read_message(event.conn) + + # Dispatcher + switch (op.type, msg.type): + case (READ, READ_SUCCESS): handle_read_metadata_response(op, msg) + case (READ, CHUNK_DATA): handle_chunk_download_success(op, msg) + + case (WRITE, READ_SUCCESS): handle_write_metadata_ready(op, msg) + case (WRITE, UPLOAD_ACK): handle_upload_ack(op, msg) + case (WRITE, WRITE_SUCCESS): handle_commit_response(op, msg) + + case (_, ERROR): handle_error(op, msg) + + return get_op_result(op_handle) +} \ No newline at end of file