Initialize *gen to NO_GENERATION at the start of file_tree_read so that
error paths (FILETREE_BADPATH, FILETREE_NOENT, FILETREE_ISDIR) have a
well-defined value. Previously, when file_tree_read failed, gen was
uninitialized, causing the assertion in process_client_read to fire.
The assertion checking gen != NO_GENERATION before the error check was
removed since:
1. On error paths, gen is not used
2. On success paths, the assertion at line 482 already validates gen
Implements a new special generation counter value (UINT64_MAX) that
allows clients to assert that a file should NOT exist when performing
operations.
Primary use case: Atomic "create-if-not-exists" operations
When combined with TOASTY_WRITE_CREATE_IF_MISSING flag:
- File doesn't exist → server creates file and writes (SUCCESS)
- File exists → gen_match fails → BADGEN error (prevents overwrite)
This enables race-condition-free file creation where you want to
create a NEW file or fail if it already exists.
Key changes:
- Added MISSING_FILE_GENERATION constant to file_tree.h
- Updated gen_match() to return false when checking existing entities
against MISSING_FILE_GENERATION
- Modified file_tree_delete_entity() to succeed (no-op) when file is
missing and MISSING_FILE_GENERATION is specified (idempotent delete)
- Modified file_tree_write() to validate against the new value
- Added documentation in metadata_server.c explaining how
MISSING_FILE_GENERATION works WITH CREATE_IF_MISSING
- Updated protocol documentation in PROTOCOL.txt
- Added comprehensive feature documentation in MISSING_FILE_GENERATION.md
Additional use cases:
- Idempotent delete operations (delete only if not already gone)
- Detecting unexpected file creation in validation/testing scenarios
The implementation is backward compatible and type-safe. Files are
never assigned MISSING_FILE_GENERATION as their actual generation value.
Adds support for truncating files after write operations, enabling proper
HTTP PUT semantics where the entire file content should be replaced.
Implementation:
- Added TOASTY_WRITE_TRUNCATE_AFTER flag (0x02) to ToastyFS API
- Updated file_tree_write() to accept truncate_after parameter:
* When true, sets file size to exactly offset+length
* Removes chunks beyond the new file size
* Marks removed chunks for garbage collection
- Metadata server extracts and passes truncate flag to file_tree_write()
- WAL replay passes false for truncate_after (already handled in original write)
- Web proxy PUT now uses both CREATE_IF_MISSING and TRUNCATE_AFTER flags
- Updated documentation in PROTOCOL.txt, DESIGN.txt, and web/DESIGN.txt
Benefits:
- Proper HTTP PUT semantics (replace entire file content)
- Efficient: single write operation truncates and updates file
- Works with both sync and async APIs
- Garbage collection automatically removes orphaned chunks
Adds support for automatically creating files when write operations target
non-existent files. This is essential for REST PUT operations which should
be able to create new resources.
Implementation:
- Added TOASTY_WRITE_CREATE_IF_MISSING flag to ToastyFS API header
- Updated toasty_write() and toasty_begin_write() to accept flags parameter
- Client sends write flags in MESSAGE_TYPE_WRITE message to metadata server
- Metadata server parses flags and handles file auto-creation atomically:
* When write fails with FILETREE_NOENT and flag is set
* Logs creation to WAL for crash consistency
* Creates file with default 4096-byte chunk size
* Retries write with newly created file's generation tag
- Updated web proxy PUT handler to use TOASTY_WRITE_CREATE_IF_MISSING
- Updated examples and simulation client for new API signature
Benefits:
- Works for both sync and async APIs
- Single round trip (atomic server-side operation)
- Prevents race conditions
- Proper WAL logging ensures crash consistency
Updated both client.c and metadata_server.c to match the protocol
specification in misc/PROTOCOL.txt:
Client-to-Server Messages:
- Added expect_gen field to DELETE, LIST, READ, and WRITE messages
- Removed old_hash and chunk_size from WRITE message chunks
Server-to-Client Responses:
- Added generation counter to CREATE_SUCCESS response
- Added generation counter to LIST_SUCCESS response
- Fixed LIST_SUCCESS field order (gen, truncated, item_count)
- Added generation counter to READ_SUCCESS response
- Added generation counter to WRITE_SUCCESS response
All changes compile successfully and pass valgrind memory checks.
Add functionality to determine the actual number of bytes read during
read operations, making it possible to detect when a read was truncated
because it went past the end of the file.
Changes:
- Modified file_tree_read() to calculate and return actual_bytes via
new output parameter
- Updated metadata server to send actual_bytes in READ_SUCCESS messages
- Added bytes_read field to ToastyResult structure
- Modified client to parse, store, and report actual bytes read
- Updated toasty_read() to return the actual number of bytes read
instead of always returning 0
- Fixed web server to use bytes_read field instead of non-existent
count field
This allows clients to distinguish between:
- Reading zeros because the file is sparse (has holes)
- Reading past the end of the file (truncated read)
Fixed critical bugs in the DOWNLOAD_LOCATIONS message exchange between
metadata server and chunk server:
Metadata server changes:
- Fixed bug using wrong index (j instead of holders[j]) when writing
chunk server addresses
- Added missing hash field to the message (was only read, never written)
- Added trace output for STATE_UPDATE_ERROR messages
- Pre-read all missing chunks before building response
Chunk server changes:
- Updated message parser to match metadata server's message format
- Changed from group-based format to per-chunk format
- Updated type sizes (uint32_t for counts instead of uint8_t/uint16_t)
- Correctly accumulates addresses from all holders before adding to
pending download list
The message now follows this structure:
- num_missing (uint32_t)
- For each missing chunk:
- num_holders (uint32_t)
- For each holder: server addresses (num_ipv4, num_ipv6, addresses)
- hash (SHA256)
Tested successfully with mousefs_random_test under valgrind with no
memory errors detected.
This implements the periodic STATE_UPDATE mechanism described in DESIGN.txt
lines 80-99, where the metadata server periodically sends synchronization
messages to chunk servers.
Changes:
1. Added timing fields to ChunkServerPeer:
- last_sync_time: tracks when STATE_UPDATE was last sent
- last_response_time: tracks when chunk server last responded
2. Added health check configuration constants (config.h):
- HEALTH_CHECK_INTERVAL: 30 seconds between STATE_UPDATE messages
- HEALTH_CHECK_TIMEOUT: 90 seconds before marking server as unhealthy
3. Implemented send_state_update() function:
- Serializes and sends add_list and rem_list to chunk servers
- Updates last_sync_time when messages are sent
4. Added message handlers for chunk server responses:
- process_chunk_server_state_update_success(): merges add_list into old_list,
clears pending lists, updates last_response_time
- process_chunk_server_state_update_error(): handles missing chunks,
updates last_response_time, logs errors
5. Implemented periodic timer logic in metadata_server_step():
- Calculates when next STATE_UPDATE should be sent to each chunk server
- Sends updates when HEALTH_CHECK_INTERVAL has elapsed
- Logs warnings when chunk servers haven't responded within HEALTH_CHECK_TIMEOUT
- Sets appropriate poll() timeout to wake up for next health check
The implementation ensures that:
- Chunk servers receive regular synchronization messages with add_list/rem_list
- The metadata server tracks chunk server health based on response times
- STATE_UPDATE messages are sent periodically (every 30 seconds by default)
- Failed/missing chunks are detected and logged for recovery
This completes the health check mechanism described in the architecture design,
enabling the metadata server to maintain accurate state with chunk servers and
detect unhealthy servers.
Fixed multiple uninitialized memory issues detected by valgrind:
1. SHA256 struct size mismatch: Changed from 64 bytes to 32 bytes to
match the actual output of sha256() function (which produces 256
bits = 32 bytes).
2. File tree chunk allocation: Fixed loop that wasn't initializing
newly allocated chunks because num_chunks was updated before the
initialization loop. Now saves old_num_chunks first.
3. Metadata server process_client_write: Initialized results[i].num_addrs
to 0 before use to prevent reading uninitialized value.
4. Address struct initialization: Zero-initialize Address structs before
setting fields to ensure the entire union is initialized (union
contains either IPv4 or IPv6, leaving unused bytes uninitialized).
5. Fixed bug in IPv6 address creation: Changed incorrect is_ipv4=true
to is_ipv4=false for IPv6 addresses.
6. Fixed all_chunk_servers_holding_chunk: Function was incrementing
counter for all chunk servers instead of only those containing the
hash, causing access to uninitialized array elements.
All valgrind errors resolved: ERROR SUMMARY: 0 errors from 0 contexts
Unified the function signatures across all process types (metadata server,
chunk server, and simulation client) to accept an int *timeout parameter.
This provides consistency in the simulation framework and allows all
process types to control their wakeup times.
**Changes:**
**Headers (metadata_server.h, chunk_server.h):**
- Updated metadata_server_init() to accept int *timeout parameter
- Updated metadata_server_step() to accept int *timeout parameter
- Updated chunk_server_init() to accept int *timeout parameter
- Updated chunk_server_step() to accept int *timeout parameter
**Implementations (metadata_server.c, chunk_server.c):**
- Modified init functions to set *timeout = -1 (no timeout needed)
- Modified step functions to set *timeout = -1 (no timeout needed)
- Both server types currently don't require periodic wakeups, but the
parameter allows for future timeout-based behavior
**System Integration (system.c):**
- Updated spawn_simulated_process() to declare timeout variable once
and pass it to all process init functions uniformly
- Updated update_simulation() to declare timeout variable once and
pass it to all process step functions uniformly
- Simplified control flow by removing special case for client timeout
handling - now all process types use the same timeout logic
- Consolidated timeout-to-wakeup_time conversion after switch statements
**Benefits:**
- Consistent API across all process types
- Cleaner code with reduced duplication
- Future-proof for server timeout requirements
- All processes now managed uniformly by simulation framework
The simulation continues to run correctly with all process types
handling timeout parameter appropriately.
cozis
Claude