cozis/ToastyFS
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Add TOASTY_WRITE_CREATE_IF_MISSING flag for server-side file auto-creation

Browse Source 
Adds support for automatically creating files when write operations target
non-existent files. This is essential for REST PUT operations which should
be able to create new resources.

Implementation:
- Added TOASTY_WRITE_CREATE_IF_MISSING flag to ToastyFS API header
- Updated toasty_write() and toasty_begin_write() to accept flags parameter
- Client sends write flags in MESSAGE_TYPE_WRITE message to metadata server
- Metadata server parses flags and handles file auto-creation atomically:
  * When write fails with FILETREE_NOENT and flag is set
  * Logs creation to WAL for crash consistency
  * Creates file with default 4096-byte chunk size
  * Retries write with newly created file's generation tag
- Updated web proxy PUT handler to use TOASTY_WRITE_CREATE_IF_MISSING
- Updated examples and simulation client for new API signature

Benefits:
- Works for both sync and async APIs
- Single round trip (atomic server-side operation)
- Prevents race conditions
- Proper WAL logging ensures crash consistency
This commit is contained in:
Claude
2025-12-03 18:48:42 +00:00
parent a082f46b78
commit 8160c5ac51
9 changed files with 80 additions and 16 deletions
Download Patch File Download Diff File Expand all files Collapse all files
examples/blocking_api.c
+1 -1
View File
@@ -22,7 +22,7 @@ int main(void)
}
char data[] = "Hello, world!";
ret = toasty_write(toasty, path, 0, data, sizeof(data)-1, 0);
ret = toasty_write(toasty, path, 0, data, sizeof(data)-1, NULL, 0);
if (ret < 0) {
printf("Couldn't write to file\n");
toasty_disconnect(toasty);
inc/ToastyFS.h
+13 -2
View File
@@ -56,6 +56,9 @@ typedef uint64_t ToastyVersionTag;
// TODO: comment
#define TOASTY_VERSION_TAG_EMPTY ((ToastyVersionTag) 0)
// Write operation flags
#define TOASTY_WRITE_CREATE_IF_MISSING (1 << 0) // Create file if it doesn't exist
// Creates a directory at the specified path.
// Returns 0 on success, -1 on error.
//
@@ -125,8 +128,12 @@ int toasty_read(ToastyFS *toasty, ToastyString path, int off,
// on success, or -1 on error.
//
// For how vtag works, see toasty_read.
//
// The flags parameter can be set to TOASTY_WRITE_CREATE_IF_MISSING
// to automatically create the file if it doesn't exist. A default
// chunk size of 4096 bytes will be used for the created file.
int toasty_write(ToastyFS *toasty, ToastyString path, int off,
void *src, int len, ToastyVersionTag *vtag);
void *src, int len, ToastyVersionTag *vtag, uint32_t flags);
//////////////////////////////////////////////////////////////////////////////////
// ASYNCHRONOUS API
@@ -177,8 +184,12 @@ ToastyHandle toasty_begin_read(ToastyFS *toasty, ToastyString path,
//
// If vtag is not 0, the operation only succedes if the
// tag matches the remote entity's.
//
// The flags parameter can be set to TOASTY_WRITE_CREATE_IF_MISSING
// to automatically create the file if it doesn't exist. A default
// chunk size of 4096 bytes will be used for the created file.
ToastyHandle toasty_begin_write(ToastyFS *toasty, ToastyString path,
int off, void *src, int len, ToastyVersionTag vtag);
int off, void *src, int len, ToastyVersionTag vtag, uint32_t flags);
// Associate the pointer "user" to the handle. The user
// pointer will be returned in the ToastyResult when the
misc/DESIGN.txt
+13 -5
View File
@@ -41,11 +41,19 @@ Architecture
modifications are successful, the client holds the set of
old hashes and new hashes for that file range. It completes
the write by telling the metadata server to swap the old
hashes with the new ones. If the old hashes don't match,
another write succeded in the mean time and touched that
range, therefore the write fails. If the old hashes match,
the write succeded. If the client fails to modify any
chunks, it doesn't commit the write with the metadata server.
hashes with the new ones (optionally including write flags
such as TOASTY_WRITE_CREATE_IF_MISSING). If the old hashes
don't match, another write succeded in the mean time and
touched that range, therefore the write fails. If the old
hashes match, the write succeded. If the client fails to
modify any chunks, it doesn't commit the write with the
metadata server.
If the file doesn't exist and the TOASTY_WRITE_CREATE_IF_MISSING
flag is set, the metadata server atomically creates the file
with a default chunk size (4096 bytes) and retries the write.
This operation is logged to the WAL to ensure crash consistency.
Note that write failures may cause chunks to be orphaned
on chunk servers. This is solved by a garbage collection
algorithm implemented by the synchronization messages
misc/PROTOCOL.txt
+7
View File
@@ -106,6 +106,7 @@ the metadata server:
struct WriteMessage {
Header header; // type=WRITE
uint64_t expect_gen;
uint32_t flags;
uint16_t path_len;
char path[path_len];
uint32_t offset;
@@ -116,6 +117,12 @@ the metadata server:
If the expect_gen field doesn't match the generation of the target file, the operation fails. Note that unlike other operations, the expect_gen CAN'T be 0. This is due to the assumption that the chunk size hasn't change for that file since the writer originally retrieved the file's metadata.
The flags field contains write operation flags. Currently defined flags:
- TOASTY_WRITE_CREATE_IF_MISSING (0x01): If the file doesn't exist,
the metadata server will atomically create it with a default chunk
size of 4096 bytes and retry the write operation. The creation is
logged to the WAL for crash consistency.
The offset and length mark the region that is being written to.
Then comes an array of num_chunks sections each specifying where a given chunk was written to. Note that the number of chunks is equal to
src/client.c
+7 -3
View File
@@ -124,6 +124,7 @@ typedef struct {
uint32_t num_chunks;
uint32_t chunk_size;
uint64_t expect_gen;
uint32_t flags; // Write operation flags (e.g., TOASTY_WRITE_CREATE_IF_MISSING)
UploadSchedule *uploads;
int num_uploads;
int cap_uploads;
@@ -744,7 +745,7 @@ unlock_and_exit:
ToastyHandle toasty_begin_write(ToastyFS *toasty,
ToastyString path, int off, void *src, int len,
ToastyVersionTag vtag)
ToastyVersionTag vtag, uint32_t flags)
{
int opidx = -1;
ToastyHandle handle = TOASTY_INVALID;
@@ -759,6 +760,7 @@ ToastyHandle toasty_begin_write(ToastyFS *toasty,
goto unlock_and_exit;
toasty->operations[opidx].path = path; // TODO: must be a copy
toasty->operations[opidx].flags = flags;
if (send_read_message(toasty, opidx, TAG_RETRIEVE_METADATA_FOR_WRITE, path, off, len, vtag) < 0) {
free_operation(toasty, opidx);
@@ -1974,6 +1976,7 @@ static void process_event_for_write(ToastyFS *toasty,
ToastyString path = toasty->operations[opidx].path;
uint32_t offset = toasty->operations[opidx].off;
uint32_t length = toasty->operations[opidx].len;
uint32_t flags = toasty->operations[opidx].flags;
if (path.len > UINT16_MAX) {
assert(0); // TODO
@@ -1984,6 +1987,7 @@ static void process_event_for_write(ToastyFS *toasty,
uint32_t num_chunks = num_upload_results;
message_write(&writer, &expect_gen, sizeof(expect_gen));
message_write(&writer, &flags, sizeof(flags));
message_write(&writer, &path_len, sizeof(path_len));
message_write(&writer, path.ptr, path.len);
message_write(&writer, &offset, sizeof(offset));
@@ -2449,9 +2453,9 @@ int toasty_read(ToastyFS *toasty, ToastyString path,
}
int toasty_write(ToastyFS *toasty, ToastyString path,
int off, void *src, int len, ToastyVersionTag *vtag)
int off, void *src, int len, ToastyVersionTag *vtag, uint32_t flags)
{
ToastyHandle handle = toasty_begin_write(toasty, path, off, src, len, vtag ? *vtag : 0);
ToastyHandle handle = toasty_begin_write(toasty, path, off, src, len, vtag ? *vtag : 0, flags);
if (handle == TOASTY_INVALID)
return -1;
src/metadata_server.c
+26
View File
@@ -537,6 +537,10 @@ process_client_write(MetadataServer *state, int conn_idx, ByteView msg)
if (!binary_read(&reader, &expect_gen, sizeof(expect_gen)))
return -1;
uint32_t flags;
if (!binary_read(&reader, &flags, sizeof(flags)))
return -1;
char path_mem[1<<10];
uint16_t path_len;
@@ -634,6 +638,28 @@ process_client_write(MetadataServer *state, int conn_idx, ByteView msg)
int ret = file_tree_write(&state->file_tree, path, offset, length,
num_chunks, expect_gen, &new_gen, new_hashes, removed_hashes, &num_removed);
// If write failed because file doesn't exist and CREATE_IF_MISSING flag is set,
// create the file and retry the write
#define TOASTY_WRITE_CREATE_IF_MISSING (1 << 0)
if (ret == FILETREE_NOENT && (flags & TOASTY_WRITE_CREATE_IF_MISSING)) {
// Create the file with default chunk size of 4096 bytes
uint64_t chunk_size = 4096;
// Log the creation in the WAL
if (wal_append_create(&state->wal, path, false, chunk_size) < 0) {
assert(0); // TODO
}
uint64_t create_gen;
int create_ret = file_tree_create_entity(&state->file_tree, path, false, chunk_size, &create_gen);
if (create_ret == 0) {
// File created successfully, retry the write with the new generation
ret = file_tree_write(&state->file_tree, path, offset, length,
num_chunks, create_gen, &new_gen, new_hashes, removed_hashes, &num_removed);
}
}
if (ret < 0) {
string desc = file_tree_strerror(ret);
src/simulation_client.c
+1 -1
View File
@@ -209,7 +209,7 @@ int simulation_client_step(SimulationClient *client, void **contexts,
ptr = malloc(len);
if (ptr == NULL) assert(0);
memset(ptr, 'a', len);
handle = toasty_begin_write(client->toasty, entry.path, off, ptr, len, TOASTY_VERSION_TAG_EMPTY);
handle = toasty_begin_write(client->toasty, entry.path, off, ptr, len, TOASTY_VERSION_TAG_EMPTY, 0);
//printf("[Client] submit write (path=%s, off=%d, len=%d)\n", entry.path, off, len);
break;
}
web/DESIGN.txt
+8 -3
View File
@@ -34,9 +34,14 @@ PUT
??? (TODO) is returned. If the payload the request is not empty, the
operation fails with code 400.
If the path refers to a file, it creates that file with the specified
contents. If a file or directory already existed at that path, it is
overwritten unless the header "X-ToastyFS-Overwrite: no" is sent.
If the path refers to a file, the file is automatically created if it
doesn't exist, then the specified contents are written to it. This is
implemented using the TOASTY_WRITE_CREATE_IF_MISSING flag which performs
file creation and write atomically on the metadata server. Files are
created with a default chunk size of 4096 bytes.
If a file or directory already existed at that path, it is overwritten
unless the header "X-ToastyFS-Overwrite: no" is sent.
(TODO error codes)
DELETE
web/src/proxy_put.c
+4 -1
View File
@@ -7,8 +7,11 @@ bool process_request_put(ProxyState *state, ProxyOperation *operation,
request->url.path.ptr,
request->url.path.len,
};
// Use TOASTY_WRITE_CREATE_IF_MISSING to automatically create files
// when they don't exist, enabling PUT operations to create new files
ToastyHandle handle = toasty_begin_write(state->backend, path, 0,
request->body.ptr, request->body.len, TOASTY_VERSION_TAG_EMPTY);
request->body.ptr, request->body.len, TOASTY_VERSION_TAG_EMPTY,
TOASTY_WRITE_CREATE_IF_MISSING);
if (handle == TOASTY_INVALID) {
http_response_builder_status(builder, 500); // Internal Server Error
http_response_builder_send(builder);