readme: update README.md and add DST.md

DST.md

@@ -0,0 +1,48 @@
+
+# Deterministic Simulation Testing
+
+Deterministic Simulation Testing consists of running a program in an environment that is completely controlled to exercise all its behaviors that would otherwise hard to reach. This allows running programs in scenarios that are much more hostile than production. This can be done deterministically or not. What makes determinism so great, is that any time a bug presents itself, it's possible to replay the scenarios that caused it. This is great because it's not always trivial to understand the causes of a bug from its symptoms.
+
+By running a program inside a "simulation", we mean that any time the program interacts with the outside world (the disk, network, reads the time) we should be able to arbitrarily decide what is returned to the program. This allows us to test all the complex corser-cases that would otherwise hardly happen while testing. For instance we could decide to flip a bit in a file to see whether checksums are implemented correctly. This can be called Simulation Testing, which may be deterministic but usually isn't.
+
+For instance say you want to test a program that uses the file system. You may test it by creating a wrapper over the file system calls to return random errors 10% of the time and call the real system the remaining 90%. This would be a non-deterministic approach since the kernel is not deterministic. If instead we created a mock of the file system that runs in memory and doesn't go through the kernel, that would make this approach deterministic, although it would be so only regarding file operations and not other things such network and time.
+
+To achieve true determinism it's necessary to control all interactions with kernel and hardware. This includes intercepting any system calls or interactions with the hardware that would bypass the kernel (such as reading the CPU's timestamp counter). The other cause of non-determinism is scheduling. If our program uses threads or multiple processes that interact with each other, multiple runs of the program will produce different interleavings of those flows of execution, leading to differences in the output.
+
+Overall, the main challenges of DST are
+1. Mocking the world accurately
+2. Controlling the scheduling
+
+Unfortunately there is no silver bullet here. There are a number of solutions that make more or less sense given the specific situation.
+
+Two notable examples of DST are TigerBeetle and Antithesis.
+
+## DST in TigerBeetle
+
+TigerBeetle is an open-source database for online transaction processing designed to be incredibly scalable and robust. The way it emplyes DST is by wrapping any interaction with the system with functions that may be switched with a mocked version.
+
+The "mock the world" problem is solved by (1) only mocking the specific I/O operations needed by the system and (2) by not mocking system calls but their own I/O library with a simplified contract that is easier to reason about since less general.
+
+TiberBeetle is replicated, which means a single instance of the database has multiple nodes, while each node is single-threaded. The interleaving of nodes is controlled by running all nodes in a single process and scheduling them in userspace.
+
+The downside of this approach is that it's tailored to the application. The testing framework used by TigerBeetle can only be used for TigerBeetle or very similar applications, which means that with this approach the tested application is better be worth the work to mock the world.
+
+## DST in Antithesis
+
+Another interesting application of DST is that offered by Antithesis.
+
+They decided to fix the mocking and scheduling problems in one swoop by making existing kernels determministic. Once you have a deterministic kernel, you can run and application on it unmodified, and the output will be completely predictable. Once you have a deterministic system, you can intercept system calls at runtime and inject arbitrary fauls.
+
+This system is proprietary, so my understanding of it is based on what they made public. But my understanding is that they implemented a deterministic machine (as an hypervisor) that is deterministic, to then run the kernel on top of it. Since the underlying machine is deterministic, the kernel will be too.
+
+The advantage of this approach is huge. You can perform DST on any program unmodified. The downside is creating a deterministic hypervisor, which requires a lot of work.
+
+# How to Find Bugs
+
+Okay so let's say you managed to set up a DST framework with fault injection. Now how do you find all the bugs?
+
+Some bugs will merely present themselves by causing the program to crash. Some bugs will cause an unrecoverable state like requesting the value from an unmapped page or dividing by zero. This may be cause by the bug directly, or downstream of its effects: the bug will cause the program to run in an incoherent state until a hard stop condition is created. The way I visualize this is by throwing a fish net over a river. To catch a fish, you don't have to throw the net on the fish, you can instead throw the net side to side over the river and wait for the fish to reach the net.
+
+A second class of bugs don't cause the program to crash but cause it to behave incorrectly and output an incorrect value. If the program is behaving incorrectly, it must mean that one or more invariants of its state have been violated. We can cause artificial crashes when such events occur via assertions. This effectively turns such bugs in the first kind of bugs.
+
+The problem then becomes which invariants to check.

README.md

@@ -1,18 +1,12 @@
 # Zigmulator
 
-Zigmulator is a Deterministic Simulation Testing framework for Zig.
+Zigmulator is an experimental Deterministic Simulation Testing framework for Zig 0.16
 
-It allows you to run one or more zig programs in a controlled environment with the ability to inject arbitrary I/O faults. The zigmulation is fully deterministic, which allows you to replay any scenario. It's ideal for testing distributed systems where bugs lay in the interleavings of different nodes.
+It allows you to run one or more zig programs in a controlled environment with the ability to inject arbitrary I/O faults. The zigmulation is fully deterministic, which allows you to replay any scenario. It's ideal for testing distributed systems where bugs lay in the interleavings of different nodes. You can learn more about DST [here](DST.md).
 
-It's still under construction, but the basic idea is that you write your programs in the form:
+# Getting Started
 
-```zig
+To run your program in Zigmulator, you need to create a driver program that imports your main() functions and plugs them in a simulator.
-pub fn main(init: std.process.Init) anyerror!void {
-    // ...
-}
-```
-
-Then import them in the simulation program:
 
 ```zig
 const std = @import("std");
@@ -20,6 +14,11 @@ const Io = std.Io;
 
 const Simulator = @import("zigmulator").Simulator;
 
+// Import entry points of the programs to simulate.
+// 
+// The expected interface is:
+//     pub fn main(init: std.process.Init) anyerror!void
+// 
 const programA = @import("projectA/main.zig").main;
 const programB = @import("projectB/main.zig").main;
 const programC = @import("projectC/main.zig").main;
@@ -51,15 +50,7 @@ pub fn main(init: std.process.Init) !void {
 }
 ```
 
-To see this example in action:
+## How it Works
-
-```sh
-zig build run
-```
-
-which will build examples/example.zig
-
-## How it works
 
 Zigmulator implements an in-memory kernel, disk, network and plugs it into your application via the new IO interface (since Zig 0.16). Multiple programs can run in the same simulation, in which case network traffic is routed between their mocks. This is a bruteforce approach but the only one that allows a complete control of the environment the applications are running in.
 
@@ -69,12 +60,14 @@ To avoid non-determinism of kernel-level scheduling, all simulated programs run
 
 Since Zigmulator mocks the entire world, the main limitation is in how much of it is mocked and how it differs from the real implementation. This project takes a best-effort approach: the model of the external world is simplified, and applications tested with Zigmulator are expected to only rely on functionality that is mocked. For instance, in the Zigmulator model, each process owns its own machine. If you spawn multiple nodes, each will have its own NIC and disk.
 
+Adapting mature projects to DST frameworks is a generally hard thing. For new project it's a good idea to develop inside Zigmulator to only rely on available features and then test the project on real hardware as a sanity check.
+
 ## Current Status
 
-Currently Zigmulator is a proof of concept. A basic version of network, disk, scheduling have been implemented and only the write operations have been mocked.
+Currently Zigmulator implements mocks for the most common functions of `std.Io` like file system operations, network operations, asynchronous tasks. Concurrent tasks and process management are considered out of scope.
 
 ## Visualization
 
-You can also use the simulation trace to create visualizations of what your program is doing
+You can also use the simulation trace to create visualizations of what your program is doing. This is still work-in-progress, but here's a timeline diagram I created the other day:
 
 ![timeline image](timeline.png)