cozis/ToastyFS
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Add library pseudocode

Browse Source
This commit is contained in:
cozis
2026-01-29 00:07:29 +01:00
parent fa5cdca459
commit 6e3a5c2fff
3 changed files with 824 additions and 5 deletions
Download Patch File Download Diff File Expand all files Collapse all files
spec/design.txt
+233
View File
@@ -0,0 +1,233 @@
Architecture
A ToastyFS instance is composed by a metadata server, a number
of chunk servers, and a number of clients.
The metadata server stores the full file system hieararchy,
except instead of storing the file contents, it stores an
array of hashes of the chunks of each file. A "chunk" is a
file range that is fixed for a single file but may vary
between files. Chunk servers hold an array of chunks that
are identified by their hash. The metadata server keeps
track of which chunks each chunk server is holding.
Clients are users of the file system that can read and
write metadata and files. They are assumed to behave
correctly.
Any read and write operation that doesn't involve file
contents can be performed by clients by talking to the
metadata server directly. Such operations include creating
an empty file or a directory, deleting a file or directory,
listing files.
If a client wants to read a range of bytes from a file,
it sends the metadata server the file name and range.
The metadata server responds with the chunk size of that
file, the list of hashes for the chunks involved in the
read, and the IP addresses of the chunk servers that hold
each chunk. The metadata server also adds the IP addresses
of three chunk servers any new chunks should be written
to. The client can then download the chunks from the chunk
servers and reassemble the result.
If a client wants to write at a range of bytes of a file,
it starts by reading that range from the metadata server,
getting the list of hashes it will modify, their locations,
and locations for any new chunks. The client then modifies
the chunk by sending to each chunk server the hash to modify
and the patch (a range of bytes within a chunk plus the new
data). The chunk server creates a new modified chunk and
keeps the old version, then returns the new hash. If all
modifications are successful, the client holds the set of
old hashes and new hashes for that file range. It completes
the write by telling the metadata server to swap the old
hashes with the new ones (optionally including write flags
such as TOASTY_WRITE_CREATE_IF_MISSING). If the old hashes
don't match, another write succeded in the mean time and
touched that range, therefore the write fails. If the old
hashes match, the write succeded. If the client fails to
modify any chunks, it doesn't commit the write with the
metadata server.
If the file doesn't exist and the TOASTY_WRITE_CREATE_IF_MISSING
flag is set, the metadata server atomically creates the file
with a default chunk size (4096 bytes) and retries the write.
This operation is logged to the WAL to ensure crash consistency.
If the TOASTY_WRITE_TRUNCATE_AFTER flag is set, the file is
truncated after the write, setting its size to exactly offset+length
and discarding any data beyond that point. This is useful for HTTP
PUT semantics where the entire file content should be replaced.
Note that write failures may cause chunks to be orphaned
on chunk servers. This is solved by a garbage collection
algorithm implemented by the synchronization messages
between metadata and chunk server.
Note that clients may cache chunks and index them by their
hash. When they read a file and receive its hashes, they may
avoid reaching for the chunk servers if they already cached
the chunks with those hashes. This allows reading files with
only one round trip at no cost of correctness. If getting
the up-to-date contents is not a concern, clients may also
cache file metadata.
Metadata and chunk server exchange:
The metadata server is only aware of each chunk server
as long as they have a TCP connection. When a chunk server
first connects to the metadata server, it authenticates
itself and sends its own IP addresses. If the server is
authentic, the metadata server requests the full list
of chunks the chunk server is holding. Upon receiving the
state of chunk server, the metadata server adds all useful
chunks to the "old_list" and all useless chunks to the
"rem_list", then sends the rem_list to the chunk server
which removes those chunks.
When writes are committed to the metadata server involving
new chunks to a chunk server, the metadata server adds those
hashes to an "add_list" and any hashes that are not useful
anymore to the rem_list.
Periodically, the metadata server sends the add_list and
rem_list to the chunk server. These list tell the chunk
server the ideal state it should have from the point of
view of the metadata server. Elements in the add_list should
already be in the chunk servers, and elements from the
rem_list are to be removed. A chunk server marks any chunk
in the rem_list as to be removed and checks that hashes
in the add list are present. If a chunk in the add list
is marked as to be removed, it is unmarked. When a chunk
is marked as to be removed for a certain amount of time,
it is permanently deleted. When the synchronization is
complete, the metadata server merges the add_list into
the old_list and clears the rem_list. If chunks in the
add_list are not present in the chunk server, it responds
with an error message containing the list of missing chunks.
The metadata server then responds with a list of chunk
server addresses where the chunk server with the missing
chunk can download it from. Each chunk server goes
through its download list one at the time downloading
the missing chunks.
Note that if the chunk server finds that its holding some
chunks that are not in the hash list of the metadata server,
that does not mean they are orphaned. It's possible that
some writes are being performed by clients that have uploaded
chunks to that chunk server but didn't yet acknowledge it
to the metadata server. If all goes well and the write
succeded, the metadata server will add those hashes to the
hash list. Chunk servers should only drop chunks if they
are not referenced by the metadata server for a period of
time (say, 30 minutes).
Security
All nodes of the system share a secret key and use it to
authenticate each other and encrypt messages. This allows
the server to accept new chunk servers and clients with
no prior setup
Reliability
The metadata server is a single point of failure. To reduce
the impact of crashes, the metadata server stores all write
operations into a write-ahead log that is replayed any time
the process goes online.
Chunk Management:
Chunks are added to the system when:
1. A chunk server connects
2. A write operation on metadata occurs (adding chunks)
They are removed when:
1. A chunk server disconnects
2. A write operation on metadata occurs (overwriting old chunks)
3. A delete operation on metadata occurs
4. A chunk is corrupted or removed forcefully from the chunk server
The system must make sure that chunks are not over-replicated
or under-replicated. If they are over-replicated, some chunk
servers need to forget some copies. If they are under-replicated,
some chunk servers need to copy chunks from elsewhere.
Metadata server variables for a chunk server:
ms_old_list: List of chunks that are known to be held by CS
ms_add_list: List of chunks that should be held by CS
ms_rem_list: List of chunks that may be held by CS but should removed from it
Chunk server variables:
cs_add_list: List of chunks added since the last update
cs_rem_list: List of chunks marked for removal after a timeout
cs_lst_list: List of chunks that were lost due to errors or forceful removals of chunk files
Metadata change for write:
When clients commit a write by adding new hashes to the metadata,
MS adds those hashes to the ms_add_list for the CS where the client
uploaded the chunks. If a hash was overwritten and became useless,
that hash is added to the ms_rem_list for all CS holding it.
Metadata change for delete:
All hashes that are no longer reachable by the file tree are added
to the ms_rem_list of their holders
Chunk upload:
When a chunk is uploaded to a chunk server, its hash is added to
the cs_add_list.
Periodically on the chunk server:
Elements in the cs_rem_list have a 30 minute timeout after which they are deleted
permanently.
Periodically:
CS sends cs_add_list to MS
MS may add a subset of cs_add_list to ms_add_list based on the chunk replication and distribution policy
MS sends ms_add_list and ms_rem_list to CS
CS (1) Adds all elements from ms_rem_list to cs_rem_list
(2) Elements in ms_add_list that are not held by the chunk server are added to
a temporary list tmp_list
(3) Removes elements in ms_add_list from cs_add_list and cs_rem_list, then merges cs_add_list into cs_rem_list and clears cs_add_list.
(4) Elements in cs_lst_list are added to tmp_list, then cs_lst_list is cleared.
(6) tmp_list is sent to MS
(7) cs_add_list is cleared
MS (1) Receives tmp_list and sends download locations to CS for those chunks
(2) Merges ms_add_list into ms_old_list, then removes all items in tmp_list from ms_old_list
(3) Sets ms_add_list equal to tmp_list
Chunk replication and distribution policy:
During an update, when CS reports a new chunk to MS, MS has to decide whether
to allow the CS to keep it or not.
There are 4 cases:
- The chunk is useless (not referenced by any file)
- The chunk is under-replicated even counting the new copy
- The chunk is properly replicated with the new copy
- The chunk is over-replicated with the new copy
If the chunk is not referenced by the file tree, do nothing.
If the chunk is properly replicated or under-replicated, add it to the ms_add_list.
If the chunk is over-replicated, either don't add it to the ms_add_list or add it to the ms_rem_list of some other holder.
TODO: The way chunk servers tell about the chunks they are holding
recently changed. Now instead of sending a full chunk list when
connecting, they send batches of chunks to the metadata server
during state updates. It also changed that now chunk servers
initiate updates.
When a chunk server connects (and authenticates)
it sends alongside the auth message the hash of
the hash list of all chunks. The metadata server
then replies with a message saying whether it
already cached that chunk list or not. If it didn't, the chunk server sends the entire list
in chunks during state updates, with an increased
update frequency.
Metadata Persistence & Crash Recovery:
The metadata server uses a write-ahead log (WAL) file to store its state on disk.
Log files start with a full snapshot of the metadata and continues with operation
log entries.
When a file gets too big, the metadata server creates a new WAL file by writing
a snapshot to it and continuing logging there.
spec/protocol.txt
+281
View File
@@ -0,0 +1,281 @@
1. Introduction & Notation
The DESIGN.txt file gives an overview of the system
and how nodes of a cluster interact with each other.
This file documents the specific binary format used
to exchange information between nodes.
All messages start with a shared header, defined as:
struct Header {
uint16_t version;
uint16_t type;
uint32_t length;
};
1.1 Generation Counter Special Values
Generation counters (uint64_t) have two special values:
NO_GENERATION (0):
When used in expect_gen, means "skip generation check entirely".
Files/directories are never assigned this generation value.
MISSING_FILE_GENERATION (UINT64_MAX):
When used in expect_gen, means "expect file/directory to NOT exist".
- For DELETE: succeeds if file doesn't exist (no-op)
- For WRITE: fails with BADGEN if file exists, fails with NOENT if missing
- For existing operations: causes generation mismatch if file exists
Files/directories are never assigned this generation value.
2. Client Messages
2.1 Client to Metadata Server messages
Let's start from the interactions between a client and
the metadata server:
[ CREATE | C -> MS ]
Upon file creation, the client sends a "CREATE" message with the following layout:
struct CreateMessage {
Header header; // type=CREATE
uint16_t path_len;
char path[path_len];
uint8_t is_dir;
if (is_dir != 0) {
uint32_t chunk_size;
}
};
Note that in general only paths up to 65K bytes are
supported and that the layout of fields may depend
on the value of others.
The server then responds with a CREATE_SUCCESS or CREATE_ERROR message.
[ DELETE | C -> MS ]
When a client deletes a file, it sends a DELETE
message with the following layout:
struct DeleteMessage {
Header header; // type=DELETE
uint64_t expect_gen;
uint16_t path_len;
char path[path_len];
};
The file/directory at the given path is only deleted if its generation counter matches expect_gen. If expect_gen is 0, the file/directory is deleted regardless.
The server then responds with a DELETE_SUCCESS or DELETE_ERROR message.
[ LIST | C -> MS ]
When a client requests a directory listing, it sends a LIST message whith the following format:
struct ListMessage {
Header header; // type=LIST
uint64_t expect_gen;
uint16_t path_len;
char path[path_len];
};
If the expect_gen field doesn't match the generation counter of the directory, the operation fails. If expect_gen is 0, the check is skipped.
The server then responds with a LIST_SUCCESS or LIST_ERROR message.
[ READ | C -> MS ]
When a client requests to read a file, it sends a READ message with the following format:
struct ReadMessage {
Header header; // type=READ
uint64_t expect_gen;
uint16_t path_len;
char path[path_len];
uint32_t offset;
uint32_t length;
};
The expect_gen field is the expected generation counter for the target resource. If it doesn't match, the operation fails. If expect_gen is 0, the check is skipped.
The offset and length fields determine the region to be read from the file.
[ WRITE | C -> MS ]
When a client wants to write to a file, it sends a WRITE message with the following layout:
struct Location {
uint8_t is_ipv4;
if (is_ipv4) {
uint32_t ipv4;
} else {
uint128_t ipv6;
}
uint16_t port;
};
struct WriteChunk {
SHA256 hash;
uint32_t num_locations;
Location locations[num_locations];
};
struct WriteMessage {
Header header; // type=WRITE
uint64_t expect_gen;
uint32_t flags;
uint16_t path_len;
char path[path_len];
uint32_t offset;
uint32_t length;
uint32_t num_chunks;
WriteChunk chunks[num_chunks];
};
If the expect_gen field doesn't match the generation of the target file, the operation fails. Note that unlike other operations, the expect_gen CAN'T be 0. This is due to the assumption that the chunk size hasn't change for that file since the writer originally retrieved the file's metadata.
The flags field contains write operation flags. Currently defined flags:
- TOASTY_WRITE_CREATE_IF_MISSING (0x01): If the file doesn't exist,
the metadata server will atomically create it with a default chunk
size of 4096 bytes and retry the write operation. The creation is
logged to the WAL for crash consistency.
- TOASTY_WRITE_TRUNCATE_AFTER (0x02): Truncate the file after the write
operation, setting the file size to exactly offset+length. Any data
beyond this point is discarded. Useful for HTTP PUT semantics where
the entire file content should be replaced.
The offset and length mark the region that is being written to.
Then comes an array of num_chunks sections each specifying where a given chunk was written to. Note that the number of chunks is equal to
num_chunks == length / chunk_size
Where chunk_size is the one for the target file at the specified generation.
Each WriteChunk lists the new hashes for the file in the write range and for each one it lists all the chunk servers that are now holding a copy of it.
2.2 Client to Chunk Server messages
TODO
3. Metadata Server messages
The following is the list of messages the Metadata Server may send to a Client:
[ CREATE_ERROR | MS -> C ]
When a client sends a CREATE request to the metadata server and the operation fails, the metadata server responds with a CREATE_ERROR message with the following layout:
struct CreateErrorMessage {
Header header; // type=CREATE_ERROR
uint16_t message_len;
char message[message_len];
};
[ CREATE_SUCCESS | MS -> C ]
When a client sends a CREATE requests to the metadata server which succedes, the metadata server replies with a CREATE_SUCCESS message with the following layout:
struct CreateSuccessMessage {
Header header; // type=CREATE_SUCCESS
uint64_t gen;
};
The gen field is the generation counter given to the file.
[ DELETE_ERROR | MS -> C ]
See CREATE_ERROR
[ DELETE_SUCCESS | MS -> C ]
When a client sends a DELETE operation to the metadata server which succedes, a DELETE_SUCCESS message is sent back with the following layout:
struct DeleteSuccessMessage {
Header header; // type=DELETE_SUCCESS
};
It does not store any fields other than the header.
[ LIST_ERROR | MS -> C ]
When a client sends a LIST request to the metadata server and it fails, the server replies with a LIST_ERROR message with the following layout:
struct ListErrorMessage {
Header header; // type=LIST_ERROR
uint16_t message_len;
char message[message_len];
};
[ LIST_SUCCESS | MS -> C ]
When a client sends a LIST request to the metadata server and it succedes, the server replies with a LIST_SUCCESSS message with the following layout:
struct ListItem {
uint64_t gen;
uint8_t is_dir;
uint16_t name_len;
char name[name_len];
};
struct ListSuccessMessage {
Header header; // type=LIST_SUCCESS
uint64_t gen;
uint8_t truncated;
uint32_t item_count;
ListItem items[item_count];
};
The ListSuccessMessage gen field contains the generation counter for the directory, while the gen field in ListItem is the counter for that specific child.
If the truncated field is non-zero, the actual item count for this directory is greater than the one sent in the message.
[ READ_ERROR | MS -> C ]
TODO
[ READ_SUCCESS | MS -> C ]
struct IPv4AndPort {
uint32_t ipv4;
uint16_t port;
};
struct IPv6AndPort {
uint128_t ipv6;
uint16_t port;
};
struct ChunkServerAddrList {
uint32_t num_ipv4;
IPv4AndPort ipv4s[num_ipv4];
uint32_t num_ipv6;
IPv6AndPort ipv6s[num_ipv6];
};
struct ReadChunk {
SHA256 hash;
uint32_t num_holders;
ChunkServerAddrList holders[num_holders];
};
struct ReadSuccessMessage {
Header header; // type=READ_SUCCESS
uint64_t gen;
uint32_t chunk_size;
uint32_t file_length;
uint32_t num_hashes;
ReadChunk chunks[num_hashes];
uint32_t num_write_locations;
ChunkServerAddrList write_locations[num_write_locations];
};
The message returns general information about the file such as its generation counter, length in bytes, and chunk size.
The chunks list contains the hashes of the chunks touched by the read and the list of chunk servers that are holding them.
The write locations are a list of chunk servers that clients may write new chunks to.
[ WRITE_ERROR | MS -> C ]
See CREATE_ERROR
[ WRITE_SUCCESS | MS -> C ]
When a client sends a WRITE message which succedes, the metadata server responds with a WRITE_SUCCESS message with the following layout:
struct WriteSuccessMessage {
Header header; // type=WRITE_SUCCESS
uint64_t gen;
};
The gen field is the new generation counter for that file.
spec/spec.txt
+310 -5
View File
@@ -35,15 +35,94 @@ metadata_server_tick:
if the peer is a client:
if the message has type CREATE:
TODO
Read path_len, path, and is_dir from the message
If is_dir is false:
Read chunk_size from message
Else:
Set chunk_size to 0
Append the create operation to the WAL
Attempt to create entity in the file_tree
If file_tree creation fails:
Send CREATE_ERROR with the error description
Else:
Send CREATE_SUCCESS containing the new generation number
if the message has type DELETE:
TODO
Read expect_gen, path_len, and path from message
Append delete operation to the WAL
Attempt to delete entity in the file_tree
If file_tree deletion fails:
Send DELETE_ERROR with the error description
Else:
Send DELETE_SUCCESS (no payload)
if the message has type LIST:
TODO
Read expect_gen, path_len, and path from message
Attempt to list directory contents from file_tree
If listing fails:
Send LIST_ERROR with error description
Else:
Send LIST_SUCCESS containing:
- Directory generation
- Truncated flag (if results exceed MAX_LIST_SIZE)
- Item count
- List of items (generation, is_dir, name_len, name)
if the message has type READ:
TODO
Read expect_gen, path_len, path, offset, and length
Attempt to read file info from file_tree
If read fails:
Send READ_ERROR containing:
- Error description
- A list of suggested chunk server addresses for writing (load balanced)
NOTE: Providing write locations on read error allows clients to handle
"missing file" scenarios by immediately writing if desired.
Else:
Send READ_SUCCESS containing:
- File generation
- Chunk size
- Actual bytes available (may be less than requested length)
- Number of chunks
- List of chunks, where each contains:
- SHA256 Hash
- List of chunk servers holding this chunk (holders)
- A list of suggested chunk server addresses for writing new chunks
(load balanced via choose_servers_for_write)
if the message has type WRITE:
TODO
Read expect_gen, flags, path_len, path
Read offset, length, num_chunks
For each chunk in num_chunks:
Read the hash and the list of Chunk Server addresses (locations) that hold it
(This implies the client has already uploaded the data to these servers)
Append write operation to WAL
Attempt to apply write to file_tree (returns new_gen and a list of removed_hashes)
If result is NOENT (File Not Found) AND flags has TOASTY_WRITE_CREATE_IF_MISSING:
- Append create operation (default chunk_size 4096) to WAL
- Create entity in file_tree
- Retry the write operation to file_tree
If write fails:
Send WRITE_ERROR with error description
Else:
For each new chunk hash provided in the message:
Find the chunk servers associated with that hash in the message
Add the hash to those servers' ms_add_list
For each removed_hash returned by the file_tree:
Find all chunk servers currently holding this hash
Add the hash to their ms_rem_list (marking it for deletion)
Send WRITE_SUCCESS containing the new generation number
if the peer is a CS:
if the message has type AUTH:
@@ -175,3 +254,229 @@ chunk_server_tick:
=== CLIENT LIBRARY ======================================
=========================================================
# ------------------------------------------------------------------------------
# 1. DATA STRUCTURES
# ------------------------------------------------------------------------------
class ToastyClient:
metadata_server_conn: Connection
chunk_server_pool: Map<Address, Connection>
pending_operations: Map<OpID, Operation>
class Operation:
id: Integer
type: Enum(READ, WRITE, CREATE, DELETE, LIST)
status: Enum(PENDING, COMPLETED, FAILED)
# Context
path: String
user_buffer: Byte[]
offset: Integer
length: Integer
flags: Bitmask
expected_generation: Integer
# Internal State
chunks_to_process: List<ChunkTask>
bytes_transferred: Integer
result_data: Any
class ChunkTask:
# Used for tracking individual chunk uploads/downloads
chunk_index: Integer
hash: String
target_servers: List<Address>
status: Enum(WAITING, IN_PROGRESS, DONE)
final_hash: String # Returned by CS after upload/patch
# ------------------------------------------------------------------------------
# 2. PUBLIC BLOCKING API (User Facing)
# ------------------------------------------------------------------------------
function toasty_connect(address, port):
client = new ToastyClient()
client.metadata_server_conn = tcp_connect(address, port)
return client
function toasty_read(client, path, offset, length):
op_handle = begin_read_op(client, path, offset, length)
return wait_for_result(client, op_handle)
function toasty_write(client, path, data, offset, flags):
op_handle = begin_write_op(client, path, data, offset, flags)
return wait_for_result(client, op_handle)
# (Create, Delete, and List follow the same pattern: begin_op -> wait_for_result)
# ------------------------------------------------------------------------------
# 3. INTERNAL ASYNCHRONOUS LOGIC
# ------------------------------------------------------------------------------
# --- READ LOGIC ---
function begin_read_op(client, path, offset, length):
op = new Operation(type=READ, path, offset, length)
# Step 1: Ask Metadata Server (MS) for the file layout
send_message(client.metadata_server_conn,
type=READ_REQUEST,
path=path, range=(offset, length))
return register_operation(client, op)
function handle_read_metadata_response(op, msg):
if msg.type == ERROR:
op.status = FAILED
return
# Step 2: Calculate chunk plan
chunk_size = msg.chunk_size
op.chunks_to_process = calculate_required_chunks(op.offset, op.length, chunk_size)
# Step 3: Queue downloads from Chunk Servers (CS)
for chunk in op.chunks_to_process:
# MS returns a list of holders for each chunk hash
best_cs = load_balance_select(chunk.holders)
conn = get_cs_connection(best_cs)
send_message(conn,
type=DOWNLOAD_CHUNK,
hash=chunk.hash)
function handle_chunk_download_success(op, msg):
# Step 4: Assemble data
copy_bytes(src=msg.data, dest=op.user_buffer, at_offset=msg.chunk_index)
mark_task_complete(op, msg.chunk_index)
if all_tasks_complete(op):
op.status = COMPLETED
# --- WRITE LOGIC (The Complex Part) ---
function begin_write_op(client, path, data, offset, flags):
op = new Operation(type=WRITE, path, data, offset, flags)
# Step 1: Send READ to MS first.
# We need the current generation number and existing chunk hashes to perform
# patches or ensure consistency, even if we are overwriting.
send_message(client.metadata_server_conn,
type=READ_REQUEST,
path=path, range=(offset, len(data)))
return register_operation(client, op)
function handle_write_metadata_ready(op, msg):
# Step 2: Prepare Upload Tasks
# If file missing & CREATE_IF_MISSING flag is set, we use defaults (size 0, gen 0).
current_generation = msg.generation
chunk_size = msg.chunk_size
write_targets = msg.write_locations # Recommended CS IPs from MS
# Slice user data into chunks
sliced_chunks = split_into_chunks(op.user_buffer, chunk_size)
for slice in sliced_chunks:
task = new ChunkTask()
task.target_servers = write_targets # We must replicate to 3 servers
# Determine if this is a Create (New) or Update (Patch)
if slice.index >= len(msg.existing_hashes):
task.type = NEW_CHUNK
else:
task.type = PATCH_CHUNK
task.base_hash = msg.existing_hashes[slice.index]
op.chunks_to_process.add(task)
# Step 3: Start Uploads (Parallel up to limit)
pump_upload_queue(op)
function pump_upload_queue(op):
while active_uploads(op) < MAX_PARALLEL_UPLOADS:
task = get_next_waiting_task(op)
if not task: break
# We must upload to N replicas (usually 3)
for replica_addr in task.target_servers:
conn = get_cs_connection(replica_addr)
if task.type == NEW_CHUNK:
# Full upload
send_message(conn, type=CREATE_CHUNK, data=task.data)
else:
# Differential patch (Rsync-style or offset-based)
send_message(conn, type=UPLOAD_CHUNK,
base_hash=task.base_hash,
patch_data=task.data)
function handle_upload_ack(op, msg):
# Step 4: Collect new hashes
task = get_task(op, msg.task_id)
# The CS computes the new SHA256 after applying the patch/data
task.final_hash = msg.new_hash
task.replicas_confirmed++
if task.replicas_confirmed >= REQUIRED_REPLICATION:
task.status = DONE
if all_tasks_complete(op):
commit_write(op)
else:
pump_upload_queue(op) # Start next batch
function commit_write(op):
# Step 5: Atomic Commit at Metadata Server
commit_msg = new Message(type=WRITE_REQUEST)
commit_msg.path = op.path
commit_msg.expected_generation = op.expected_generation # Optimistic Locking
commit_msg.flags = op.flags
# Map the linear chunk index to the new Hash + Location list
commit_msg.chunks = map_chunks_to_hashes_and_locations(op.chunks_to_process)
send_message(client.metadata_server_conn, commit_msg)
function handle_commit_response(op, msg):
if msg.type == WRITE_SUCCESS:
op.status = COMPLETED
op.result_data = msg.new_generation
elif msg.type == ERROR and msg.code == GENERATION_MISMATCH:
# Optimistic concurrency failed (someone else wrote).
# Retry the whole operation from Step 1 (READ).
reset_and_retry(op)
else:
op.status = FAILED
# ------------------------------------------------------------------------------
# 4. EVENT LOOP (Driver)
# ------------------------------------------------------------------------------
function wait_for_result(client, op_handle):
while get_op_status(op_handle) == PENDING:
# Wait for network activity (Poll/Select)
events = wait_for_network_events(timeout=DEFAULT_TIMEOUT)
for event in events:
if event.type == DISCONNECT:
handle_disconnect(client, event.conn)
continue
# Route message to the specific Operation ID embedded in the packet tag
op = client.pending_operations[event.tag.op_id]
msg = read_message(event.conn)
# Dispatcher
switch (op.type, msg.type):
case (READ, READ_SUCCESS): handle_read_metadata_response(op, msg)
case (READ, CHUNK_DATA): handle_chunk_download_success(op, msg)
case (WRITE, READ_SUCCESS): handle_write_metadata_ready(op, msg)
case (WRITE, UPLOAD_ACK): handle_upload_ack(op, msg)
case (WRITE, WRITE_SUCCESS): handle_commit_response(op, msg)
case (_, ERROR): handle_error(op, msg)
return get_op_result(op_handle)
}