Add mock implementation for MoveFileExW Windows API function following
the existing pattern in the codebase. The mock forwards calls to the
real Windows API, allowing for future interception in the simulation
framework if needed.
CRITICAL BUG FIX: The previous Windows implementation had a fatal flaw
where it deleted the old WAL file before renaming, creating a window
where all data could be lost if the rename failed.
Previous (BROKEN) Windows code:
remove_file_or_dir(wal->file_path); // Delete old file
rename(...); // <-- If this fails, we've lost all data!
New implementation uses MoveFileExW with MOVEFILE_REPLACE_EXISTING:
- Windows: MoveFileExW(..., MOVEFILE_REPLACE_EXISTING) atomically
replaces the destination file, matching Unix semantics
- Unix/Linux: rename() continues to atomically replace as before
- Both platforms now have atomic file replacement with no data loss window
This ensures durability on both platforms - if the operation fails at
any point, we still have either the old file or the new file, never
losing all data.
This commit addresses several important robustness issues in the WAL
implementation:
1. Fixed file_path lifetime issue:
- The file_path argument to wal_open() may not have the same lifetime
as the WAL structure
- Now allocate and copy the path string to ensure WAL owns its data
- Properly free the allocated path in wal_close()
- Added error cleanup path in wal_open() to prevent memory leaks
2. Replaced static arrays with dynamic allocation in next_entry():
- Static arrays were not thread-safe and had limited lifetimes
- Now dynamically allocate path buffer and hash buffers using sys_malloc
- Caller must free allocated fields after using the entry
- Added proper error cleanup to free allocations on failure
- Updated wal_open() to free entry fields after processing each entry
3. Improved swap_file() for cross-platform compatibility:
- On Unix/Linux: rename() atomically replaces the destination file
- On Windows: rename() doesn't overwrite, so delete old file first
- Added platform-specific handling with #ifdef _WIN32
- Ensures WAL rotation works correctly on both platforms
4. Added system.h include for sys_malloc/sys_free definitions
These changes ensure proper memory management, prevent leaks, and make
the WAL implementation more robust across different platforms.
This commit completes the Write-Ahead Log (WAL) implementation for the
metadata server, including all missing functionality and error fixes:
File System Changes (src/file_system.c):
- Implemented file_set_offset() using lseek (Linux) and SetFilePointer (Windows)
- Implemented file_get_offset() to get current file position
WAL Header Changes (src/wal.h):
- Added file_tree pointer to WAL structure for snapshot serialization
- Added file_path to WAL structure for rotation operations
WAL Implementation (src/wal.c):
- Fixed handle assignment bug in wal_open (was using wal->handle before assignment)
- Fixed missing return statement in append_begin()
- Implemented serialize_callback for writing file tree snapshots to WAL
- Implemented deserialize_callback for reading file tree snapshots from WAL
- Implemented next_entry() to read WAL entries from file during recovery
- Implemented wal_append_write() to write file modifications to WAL
- Implemented swap_file() for complete WAL file rotation:
* Creates temporary file with new snapshot
* Writes WAL header and serialized file tree
* Atomically replaces old WAL file
* Resets entry counter for new rotation cycle
- Added WAL file initialization for newly created files
- Added helper functions: read_exact, read_u8, read_u16, read_u32, read_u64, write_u32
- Fixed typo in comment (Not -> Now)
The WAL now supports:
- Full crash recovery by replaying logged operations
- Automatic file rotation when entry limit is reached
- Atomic file replacement to ensure durability
- Proper file locking throughout rotation process
This commit adds tooling to measure how many branches the random simulation
reaches during execution, which helps understand code coverage and identify
untested code paths.
Changes:
- Add coverage build target to Makefile with --coverage flags
- Create measure_coverage.sh script to build, run, and report branch coverage
- Add signal handlers (SIGINT/SIGTERM) to main_test.c for clean shutdown
- Update clean target to remove coverage files (*.gcda, *.gcno)
The script runs the simulation for a specified duration (default 5 seconds),
then uses gcov to analyze which branches were executed. In a 3-second run,
the simulation reaches approximately 32% of all branches (781/2431).
Usage:
./measure_coverage.sh [duration_in_seconds]
Example output shows per-file and total branch coverage statistics.
Fixed compilation errors:
- Added <dirent.h> include to file_system.h for DIR type on Linux
- Replaced xxx placeholder in chunk_server.c with actual chunks_dir path
using state->store.path
Fixed several bugs in the DirectoryScanner implementation:
Windows fixes:
- Fixed snprintf call missing buffer size parameter
- Fixed FindFirstFileA to use pattern (with wildcard) instead of path
- Changed sys_GetLastError() to GetLastError()
- Added scanner->first = false after using first result
- Removed TODO comment from cFileName usage (it's the correct field)
Linux fixes:
- Fixed typo: sys_reddir → sys_readdir
- Added missing 'done' field to DirectoryScanner struct
- Initialize scanner->done = false on successful opendir
- Fixed return value when scanner->done (should return 1, not -1)
Added mock implementations for file/directory search operations to
support both Windows and Linux in the simulation environment.
Windows mocks:
- FindFirstFileA: Wraps real search handle in descriptor
- FindNextFileA: Forwards to real API
- FindClose: Properly cleans up search handles
Linux mocks:
- opendir: Wraps real DIR* handle in descriptor
- readdir: Forwards to real API
- closedir: Properly cleans up directory handles
Common changes:
- Added DESC_DIRECTORY descriptor type to track directory handles
- Added real_d field to Descriptor for directory handles (HANDLE on Windows, DIR* on Linux)
- Updated close_desc to handle directory cleanup on both platforms
- Added sys_* macros for both BUILD_TEST and non-BUILD_TEST modes
- Added <dirent.h> include for Linux directory operations
Fixed critical bugs in the DOWNLOAD_LOCATIONS message exchange between
metadata server and chunk server:
Metadata server changes:
- Fixed bug using wrong index (j instead of holders[j]) when writing
chunk server addresses
- Added missing hash field to the message (was only read, never written)
- Added trace output for STATE_UPDATE_ERROR messages
- Pre-read all missing chunks before building response
Chunk server changes:
- Updated message parser to match metadata server's message format
- Changed from group-based format to per-chunk format
- Updated type sizes (uint32_t for counts instead of uint8_t/uint16_t)
- Correctly accumulates addresses from all holders before adding to
pending download list
The message now follows this structure:
- num_missing (uint32_t)
- For each missing chunk:
- num_holders (uint32_t)
- For each holder: server addresses (num_ipv4, num_ipv6, addresses)
- hash (SHA256)
Tested successfully with mousefs_random_test under valgrind with no
memory errors detected.
- Fix inverted binary_read error checks in process_metadata_server_download_locations
- Fix array indexing bugs (use correct loop variables j and k instead of i)
- Fix IPv4 port array type from uint8_t to uint16_t
- Fix reader cursor advancement in process_client_upload_chunk
- Enable periodic download retry mechanism by calling start_download_if_necessary
Fix how chunk servers handle state updates from the metadata server
based on the design in DESIGN.txt. The implementation now:
1. Checks if chunks in the add_list exist in the chunk store and
collects any missing chunks
2. Unmarks chunks in the add_list that were previously marked for removal
3. Marks chunks in the rem_list for removal with timestamps
4. Responds with SUCCESS if all chunks are present, or ERROR with the
list of missing chunks
Added:
- RemovalList data structure to track chunks marked for deletion with
timestamps
- chunk_store_exists() to check if a chunk file exists
- Helper functions for removal list management (init, free, add, remove, find)
This implements the garbage collection mechanism described in DESIGN.txt
where chunks are marked for removal and can be unmarked if they appear
in the add_list again.
This implements the periodic STATE_UPDATE mechanism described in DESIGN.txt
lines 80-99, where the metadata server periodically sends synchronization
messages to chunk servers.
Changes:
1. Added timing fields to ChunkServerPeer:
- last_sync_time: tracks when STATE_UPDATE was last sent
- last_response_time: tracks when chunk server last responded
2. Added health check configuration constants (config.h):
- HEALTH_CHECK_INTERVAL: 30 seconds between STATE_UPDATE messages
- HEALTH_CHECK_TIMEOUT: 90 seconds before marking server as unhealthy
3. Implemented send_state_update() function:
- Serializes and sends add_list and rem_list to chunk servers
- Updates last_sync_time when messages are sent
4. Added message handlers for chunk server responses:
- process_chunk_server_state_update_success(): merges add_list into old_list,
clears pending lists, updates last_response_time
- process_chunk_server_state_update_error(): handles missing chunks,
updates last_response_time, logs errors
5. Implemented periodic timer logic in metadata_server_step():
- Calculates when next STATE_UPDATE should be sent to each chunk server
- Sends updates when HEALTH_CHECK_INTERVAL has elapsed
- Logs warnings when chunk servers haven't responded within HEALTH_CHECK_TIMEOUT
- Sets appropriate poll() timeout to wake up for next health check
The implementation ensures that:
- Chunk servers receive regular synchronization messages with add_list/rem_list
- The metadata server tracks chunk server health based on response times
- STATE_UPDATE messages are sent periodically (every 30 seconds by default)
- Failed/missing chunks are detected and logged for recovery
This completes the health check mechanism described in the architecture design,
enabling the metadata server to maintain accurate state with chunk servers and
detect unhealthy servers.
cozis