Add proxy options --addr, --max-ops, --client-id

README.md

@@ -9,6 +9,10 @@ ToastyFS is a simple, fault-tolerant, highly available object storage featuring:
 
 I initially started this project to learn about distributed systems. I asked myself what it would take to build my own Dropbox. A rabbit-hole later and here is my distributed storage system!
 
+Here is a cool pic of the hardware I'm building to run ToastyFS :D
+
+![photo of an 8 Raspberry PI cluster mouted in a small rack connected via a switch](pic.jpg)
+
 ## Table of Contents
 
 * [ToastyFS](#toastyfs)
@@ -34,15 +38,13 @@ I initially started this project to learn about distributed systems. I asked mys
 This project should be considered a robust proof-of-concept at this time. Features that would be required for long-running instances are missing, such as:
 
 * Log compaction
-* Log persistence on disk
+* Ability to shut down all nodes without resetting the system
-
-If a majority of nodes is turned off, the system's state will be lost. This is in accordance with disk-free version of the Viewstamped Replication protocol.
 
 ## Getting Started
 
 ### Building
 
-ToastyFS supports Windows and Linux. It can be compiled by calling the `build.bat` script on Windows and `build.sh` on Linux
+ToastyFS supports Windows and Linux. Each platform has its own build script:
 
 ```
 # Windows
@@ -54,12 +56,14 @@ ToastyFS supports Windows and Linux. It can be compiled by calling the `build.ba
 
 The build script will produce the following executables:
 
-toasty_simulation: Runs a ToastyFS cluster in-memory, with sped-up time and serving a number of random operations. See [Testing](#testing) section for details.
+| Name | Description |
-toasty: The actual ToastyFS program. This is what you need to run to use ToastyFS.
+| :--- | :---------- |
-toasty_random_client: An utility client which spams random requests towards a ToastyFS cluster. Useful for testing.
+| toasty_simulation | Runs a ToastyFS cluster in-memory, with sped-up time and serving a number of random operations. See [Testing](#testing) section for details. |
-toasty_proxy: An HTTP proxy that translates HTTP request to the ToastyFS-specific request protocol.
+| toasty | The actual ToastyFS program. This is what you need to run to use ToastyFS. |
+| toasty_random_client | An utility client which spams random requests towards a ToastyFS cluster. Useful for testing. |
+| toasty_proxy | An HTTP proxy that translates HTTP request to the ToastyFS-specific request protocol. |
 
-If you are on Windows, the executable names will have the .exe extension (so you will get `toasty_simulation.exe` instead of `toasty_simulation` for instance).
+On Windows the executables will have the `.exe` extension.
 
 ### Running a Cluster
 
@@ -92,15 +96,13 @@ I'm the first object
 
 ### The Need for Replication
 
-In order to build fault tolerant systems, it is necessary to build them as distributed systems: multiple nodes that coordinate to offer a single service. This allows for the system to stay alive in case some nodes fail. The system must be able to detect when some nodes become unavailable and choose other nodes to take their places to ensure the service is not interrupted. In practice, this is achieved via replication algorithms, such as Raft and Viewstamped Replication.
+To build a fault-tolerant service, it's necessary for it to run on multiple machines in such a way that if a machine dies, the others can continue its work. This is referred to as replication. Designing and implementing a replication system that properly manages all edge cases is **incredibly hard**. Such an algorithm needs to account for any number of node crashes at any phase of the protocol, it needs to account for arbitrary network partitions (network failures that cause the system to be split in multiple groups of nodes) making it impossible for the state of each group to diverge. This is such a hard problem that not only few general algorithms have been designed, but also the number of their implementations is vanishingly low. The common replication algorithms are Raft and Paxos (*). This project uses a less known algorithm called Viewstamped Replication (VSR).
 
-Replication algorithms, which are related but not exactly the same as consensus algorithms, allow the creation of a group of nodes (called replicas) that are synchronized maintaining the same state. The replication algorithm ensures that if a node of the group dies, it's impossible for a node that takes its place to have a stale state, causing an inconsistency in the overall system from the perspective of the user.
+(*) Strictly speaking, Paxos is a consensus algorithm. Replication can be built on top of it.
-
-This is generally considered as a very hard problem, since the value of such algorithms is in the mathematical certainty that once a request is accepted by the system, that information will never be lost. The algorithm needs both be designed and implemented correctly. ToastyFS solves this problem using the Viewstamped Replication algorithm.
 
 ### Raft VS Viewstamped Replication
 
-For historical reasons, the established algorithm to achieve replication is Paxos (strictly speaking, it's a consensus algorithm on top of which replication is built). Paxos (1989) is known for being hard to understand, and therefore to implement, so in 2014 a new simpler algorithm called Raft was introduced. Since then it has been the go-to algorithm for new open source projects.
+For historical reasons, Paxos (1989) is the established algorithm to achieve replication. Paxos is notorious for being hard to understand and implement, so in 2014 a more understandable algorithm called Raft was introduced. Since then, Raft has been the go-to algorithm from new open source projects.
 
 There is also a lesser known replication algorithm called Viewstamped Replication (VSR), which never got much attention from the industry even though it was invented around the same time as Paxos (and arguably, before it). In 2012 the "Viewstamped Replication Revisited" paper was published which offered a modernized design of the protocol. The later paper is the reference for VSR used in this project.
 

build.zig

@@ -0,0 +1,85 @@
+const std = @import("std");
+
+const include_paths: []const []const u8 = &.{
+    ".",
+    "src",
+    "include",
+    "quakey/include",
+};
+
+const files: []const []const u8 = &.{
+
+    "src/chunk_store.c",
+    "src/client.c",
+    "src/client_table.c",
+    "src/http_proxy.c",
+    "src/invariant_checker.c",
+    "src/log.c",
+    "src/main.c",
+    "src/metadata.c",
+    "src/random_client.c",
+    "src/server.c",
+
+    "lib/basic.c",
+    "lib/byte_queue.c",
+    "lib/file_system.c",
+    "lib/http_parse.c",
+    "lib/http_server.c",
+    "lib/message.c",
+    "lib/tcp.c",
+    "lib/tls_openssl.c",
+    "lib/tls_schannel.c",
+
+    "quakey/src/mockfs.c",
+    "quakey/src/quakey.c",
+};
+
+fn buildTarget(b: *std.Build, name: []const u8, macros: []const []const u8, flags: []const []const u8) void {
+    const exe = b.addExecutable(.{
+        .name = name,
+        .root_module = b.createModule(.{
+            .target = b.graph.host,
+        }),
+    });
+    for (macros) |m| {
+        exe.root_module.addCMacro(m, "");
+    }
+    for (include_paths) |p| {
+        exe.root_module.addIncludePath(b.path(p));
+    }
+    exe.root_module.addCSourceFiles(.{ .files = files, .flags = flags });
+    exe.root_module.link_libc = true;
+
+    b.installArtifact(exe);
+}
+
+pub fn build(b: *std.Build) !void {
+
+    // Server
+    buildTarget(b,
+        "toasty",
+        &.{ "MAIN_SERVER" },
+        &.{ "-Wall", "-Wextra", "-ggdb", "-O0" },
+    );
+
+    // Client
+    buildTarget(b,
+        "toasty_random_client",
+        &.{ "MAIN_CLIENT" },
+        &.{ "-Wall", "-Wextra", "-ggdb", "-O0" },
+    );
+
+    // Proxy
+    buildTarget(b,
+        "toasty_proxy",
+        &.{ "MAIN_HTTP_PROXY" },
+        &.{ "-Wall", "-Wextra", "-ggdb", "-O0" },
+    );
+
+    // Simulation
+    buildTarget(b,
+        "toasty_simulation",
+        &.{ "MAIN_SIMULATION", "FAULT_INJECTION" },
+        &.{ "-Wall", "-Wextra", "-ggdb", "-O0" },
+    );
+}

src/http_proxy.c

@@ -19,6 +19,10 @@ int http_proxy_init(void *state, int argc, char **argv,
     char *addrs[NODE_LIMIT];
     int num_addrs = 0;
 
+    int      max_opers = 128;
+    string   http_addr = S("127.0.0.1:3000");
+    uint64_t client_id = 999;
+
     for (int i = 1; i < argc; i++) {
         if (!strcmp(argv[i], "--server")) {
             i++;
@@ -32,16 +36,40 @@ int http_proxy_init(void *state, int argc, char **argv,
             }
             addrs[num_addrs] = argv[i];
             num_addrs++;
+        } else if (!strcmp(argv[i], "--addr")) {
+            i++;
+            if (i == argc) {
+                fprintf(stderr, "Option --addr missing value\n");
+                return -1;
+            }
+            http_addr = (string) { argv[i], strlen(argv[i]) };
+        } else if (!strcmp(argv[i], "--max-ops")) {
+            i++;
+            if (i == argc) {
+                fprintf(stderr, "Option --max-ops missing value\n");
+                return -1;
+            }
+            max_opers = atoi(argv[i]); // TODO: don't use atoi
+            if (max_opers == 0) {
+                fprintf(stderr, "Invalid value for option --max-ops\n");
+                return -1;
+            }
+        } else if (!strcmp(argv[i], "--client-id")) {
+            i++;
+            if (i == argc) {
+                fprintf(stderr, "Invalid value for option --client-id\n");
+                return -1;
+            }
+            client_id = atoi(argv[i]);
+            if (client_id == 0) {
+                fprintf(stderr, "Invalid value for option --client-id\n");
+                return -1;
+            }
         } else {
             // Ignore unknown options
         }
     }
 
-    // TODO: Make these configurable
-    int      max_opers = 128;
-    string   http_addr = S("127.0.0.1:3000");
-    uint64_t client_id = 999;
-
     proxy->max_opers = max_opers;
     proxy->opers = malloc(max_opers * sizeof(ProxyOper));
     if (proxy->opers == NULL)

src/server.c

@@ -75,7 +75,7 @@ static void node_log_impl(Server *state, const char *event, const char *detail)
 static int count_set(uint32_t word)
 {
     int n = 0;
-    for (int i = 0; i < (int) sizeof(word) * 8; i++)
+    for (int i = 0; i < (int) sizeof(word) * 8 - 1; i++)
         if (word & (1 << i))
             n++;
     return n;