Adds support for automatically creating files when write operations target
non-existent files. This is essential for REST PUT operations which should
be able to create new resources.
Implementation:
- Added TOASTY_WRITE_CREATE_IF_MISSING flag to ToastyFS API header
- Updated toasty_write() and toasty_begin_write() to accept flags parameter
- Client sends write flags in MESSAGE_TYPE_WRITE message to metadata server
- Metadata server parses flags and handles file auto-creation atomically:
* When write fails with FILETREE_NOENT and flag is set
* Logs creation to WAL for crash consistency
* Creates file with default 4096-byte chunk size
* Retries write with newly created file's generation tag
- Updated web proxy PUT handler to use TOASTY_WRITE_CREATE_IF_MISSING
- Updated examples and simulation client for new API signature
Benefits:
- Works for both sync and async APIs
- Single round trip (atomic server-side operation)
- Prevents race conditions
- Proper WAL logging ensures crash consistency
Per PROTOCOL.txt specification, WRITE operations must provide a valid
generation counter and cannot use expect_gen=0 (unlike other operations).
This ensures the client has retrieved the file's metadata and knows the
correct chunk size before writing.
Changes:
- Added FILETREE_BADGEN error code (-8)
- Modified file_tree_write() to reject expect_gen=0
- Added error message for FILETREE_BADGEN
Updated both client.c and metadata_server.c to match the protocol
specification in misc/PROTOCOL.txt:
Client-to-Server Messages:
- Added expect_gen field to DELETE, LIST, READ, and WRITE messages
- Removed old_hash and chunk_size from WRITE message chunks
Server-to-Client Responses:
- Added generation counter to CREATE_SUCCESS response
- Added generation counter to LIST_SUCCESS response
- Fixed LIST_SUCCESS field order (gen, truncated, item_count)
- Added generation counter to READ_SUCCESS response
- Added generation counter to WRITE_SUCCESS response
All changes compile successfully and pass valgrind memory checks.
Fixed two bugs in the HTTP server that could cause hangs:
1. web/chttp.c:4060 - Fixed incorrect error buffer check
- Was checking output buffer for errors after a read operation
- Should check input buffer instead
- This could cause the server to miss read errors and not close
bad connections properly
2. web/main.c:334 - Added loop to process multiple queued requests
- Previously only processed one request per event loop iteration
- Now processes all queued requests before returning to poll()
- This prevents requests from being stuck in the queue when the
socket is in ESTABLISHED_READY state
These fixes improve the server's ability to handle multiple requests
on Keep-Alive connections.
- Add web server integration to cluster_demo.sh script
- Configure web server port (8090)
- Update build_if_needed() to check for toastyfs_web binary
- Add get_web_binary() function
- Start web server after chunk servers
- Update status display to show web server
- Update help text to show HTTP interface
- Fix bug in web/main.c where --local-port incorrectly set
upstream_port instead of local_port
The cluster demo now starts a full cluster with:
- Metadata server on port 8080 (ToastyFS protocol)
- N chunk servers starting from port 8081
- Web server on port 8090 (HTTP interface)
This script provides a convenient way to spawn and manage a ToastyFS cluster
for demo and testing purposes. Features include:
- Start a cluster with configurable number of chunk servers
- Automatic building if binary is not present
- Process management with PID tracking
- Status checking for all cluster nodes
- Easy cleanup of all cluster processes
- Separate log files for each server component
- Colorized output for better readability
Usage:
./scripts/cluster_demo.sh start [num_servers] - Start cluster
./scripts/cluster_demo.sh stop - Stop cluster
./scripts/cluster_demo.sh status - Show status
./scripts/cluster_demo.sh clean - Clean data/logs
Previously, file_size was calculated as num_chunks * chunk_size, which
incorrectly treated the file size as the full capacity of all allocated
chunks. The actual file size should be the offset of the last byte
written plus 1.
Changes:
- Added file_size field to File structure to track actual file extent
- Initialize file_size to 0 when creating new files
- Update file_size in file_tree_write() based on write offset + length
- Modified file_tree_read() to use file_size instead of num_chunks * chunk_size
- Updated serialization/deserialization to handle file_size field
This ensures that actual_bytes calculations correctly reflect the true
file size, not just the allocated chunk capacity.
Add functionality to determine the actual number of bytes read during
read operations, making it possible to detect when a read was truncated
because it went past the end of the file.
Changes:
- Modified file_tree_read() to calculate and return actual_bytes via
new output parameter
- Updated metadata server to send actual_bytes in READ_SUCCESS messages
- Added bytes_read field to ToastyResult structure
- Modified client to parse, store, and report actual bytes read
- Updated toasty_read() to return the actual number of bytes read
instead of always returning 0
- Fixed web server to use bytes_read field instead of non-existent
count field
This allows clients to distinguish between:
- Reading zeros because the file is sparse (has holes)
- Reading past the end of the file (truncated read)
- Added mock_lseek declaration and implementation for Linux
- Added mock_SetFilePointer declaration and implementation for Windows
- Added sys_lseek and sys_SetFilePointer macros for both BUILD_TEST and production modes
- Mocks follow existing pattern: validate descriptors, check type, forward to real functions
- Both mocks properly handle error cases and set errno/SetLastError
Add mock implementation for MoveFileExW Windows API function following
the existing pattern in the codebase. The mock forwards calls to the
real Windows API, allowing for future interception in the simulation
framework if needed.
CRITICAL BUG FIX: The previous Windows implementation had a fatal flaw
where it deleted the old WAL file before renaming, creating a window
where all data could be lost if the rename failed.
Previous (BROKEN) Windows code:
remove_file_or_dir(wal->file_path); // Delete old file
rename(...); // <-- If this fails, we've lost all data!
New implementation uses MoveFileExW with MOVEFILE_REPLACE_EXISTING:
- Windows: MoveFileExW(..., MOVEFILE_REPLACE_EXISTING) atomically
replaces the destination file, matching Unix semantics
- Unix/Linux: rename() continues to atomically replace as before
- Both platforms now have atomic file replacement with no data loss window
This ensures durability on both platforms - if the operation fails at
any point, we still have either the old file or the new file, never
losing all data.
This commit addresses several important robustness issues in the WAL
implementation:
1. Fixed file_path lifetime issue:
- The file_path argument to wal_open() may not have the same lifetime
as the WAL structure
- Now allocate and copy the path string to ensure WAL owns its data
- Properly free the allocated path in wal_close()
- Added error cleanup path in wal_open() to prevent memory leaks
2. Replaced static arrays with dynamic allocation in next_entry():
- Static arrays were not thread-safe and had limited lifetimes
- Now dynamically allocate path buffer and hash buffers using sys_malloc
- Caller must free allocated fields after using the entry
- Added proper error cleanup to free allocations on failure
- Updated wal_open() to free entry fields after processing each entry
3. Improved swap_file() for cross-platform compatibility:
- On Unix/Linux: rename() atomically replaces the destination file
- On Windows: rename() doesn't overwrite, so delete old file first
- Added platform-specific handling with #ifdef _WIN32
- Ensures WAL rotation works correctly on both platforms
4. Added system.h include for sys_malloc/sys_free definitions
These changes ensure proper memory management, prevent leaks, and make
the WAL implementation more robust across different platforms.
This commit completes the Write-Ahead Log (WAL) implementation for the
metadata server, including all missing functionality and error fixes:
File System Changes (src/file_system.c):
- Implemented file_set_offset() using lseek (Linux) and SetFilePointer (Windows)
- Implemented file_get_offset() to get current file position
WAL Header Changes (src/wal.h):
- Added file_tree pointer to WAL structure for snapshot serialization
- Added file_path to WAL structure for rotation operations
WAL Implementation (src/wal.c):
- Fixed handle assignment bug in wal_open (was using wal->handle before assignment)
- Fixed missing return statement in append_begin()
- Implemented serialize_callback for writing file tree snapshots to WAL
- Implemented deserialize_callback for reading file tree snapshots from WAL
- Implemented next_entry() to read WAL entries from file during recovery
- Implemented wal_append_write() to write file modifications to WAL
- Implemented swap_file() for complete WAL file rotation:
* Creates temporary file with new snapshot
* Writes WAL header and serialized file tree
* Atomically replaces old WAL file
* Resets entry counter for new rotation cycle
- Added WAL file initialization for newly created files
- Added helper functions: read_exact, read_u8, read_u16, read_u32, read_u64, write_u32
- Fixed typo in comment (Not -> Now)
The WAL now supports:
- Full crash recovery by replaying logged operations
- Automatic file rotation when entry limit is reached
- Atomic file replacement to ensure durability
- Proper file locking throughout rotation process
Claude
cozis