MystenLabs · mlegner · Jun 12, 2024 · Jun 11, 2024 · Jun 12, 2024 · Jun 12, 2024
diff --git a/.gitignore b/.gitignore
@@ -24,5 +24,8 @@ config.yml
 working_dir
 *.log
 
-# Walrus binary
+# Walrus binary and configuration
+walrus
 examples/CONFIG/bin/walrus
+client_config.yaml
+examples/CONFIG/config_dir/client_config.yaml
diff --git a/docs/SUMMARY.md b/docs/SUMMARY.md
@@ -7,26 +7,26 @@
 
 # Overview
 
-- [Objectives and use-cases](./objectives_use_cases.md)
-- [Overview](./overview.md)
-  - [Properties](./properties.md)
-  - [Architecture](./architecture.md)
-  - [Encoding](./encoding.md)
-- [Operations](./operations.md)
-  - [Sui operations](./operations-sui.md)
-  - [Off-chain operations](./operations-off-chain.md)
-- [Future discussion](./future.md)
+- [Objectives and use-cases](./overview/objectives_use_cases.md)
+- [Overview](./overview/overview.md)
+  - [Properties](./overview/properties.md)
+  - [Architecture](./overview/architecture.md)
+  - [Encoding](./overview/encoding.md)
+- [Operations](./overview/operations.md)
+  - [Sui operations](./overview/operations-sui.md)
+  - [Off-chain operations](./overview/operations-off-chain.md)
+- [Future discussion](./overview/future.md)
 
 # Usage
 
-- [Setup]()
-  - [Prerequisites]()
-  - [Installation]()
-  - [Configuration]()
-- [Interacting with Walrus](interacting.md)
-  - [Using the client CLI](client-cli.md)
-  - [Using the client JSON API](json-api.md)
-  - [Using the client HTTP API](web-api.md)
+- [Setup](./usage/setup.md)
+  - [Prerequisites](./usage/prerequisites.md)
+  - [Installation](./usage/installation.md)
+  - [Configuration](./usage/configuration.md)
+- [Interacting with Walrus](./usage/interacting.md)
+  - [Using the client CLI](./usage/client-cli.md)
+  - [Using the client JSON API](./usage/json-api.md)
+  - [Using the client HTTP API](./usage/web-api.md)
 - [Examples]()
 
 # Walrus sites

diff --git a/docs/architecture.md → docs/overview/architecture.md b/docs/architecture.md → docs/overview/architecture.md
diff --git a/docs/encoding.md → docs/overview/encoding.md b/docs/encoding.md → docs/overview/encoding.md
diff --git a/docs/future.md → docs/overview/future.md b/docs/future.md → docs/overview/future.md
diff --git a/docs/objectives_use_cases.md → docs/overview/objectives_use_cases.md b/docs/objectives_use_cases.md → docs/overview/objectives_use_cases.md
diff --git a/docs/operations-off-chain.md → docs/overview/operations-off-chain.md b/docs/operations-off-chain.md → docs/overview/operations-off-chain.md
@@ -5,7 +5,7 @@ cycle.
 
 ## Write paths
 
-![Write paths of Walrus](assets/WriteFlow.png)
+![Write paths of Walrus](../assets/WriteFlow.png)
 
 Systems overview of writes, illustrated above:
 

diff --git a/docs/operations-sui.md → docs/overview/operations-sui.md b/docs/operations-sui.md → docs/overview/operations-sui.md
diff --git a/docs/operations.md → docs/overview/operations.md b/docs/operations.md → docs/overview/operations.md
diff --git a/docs/overview.md → docs/overview/overview.md b/docs/overview.md → docs/overview/overview.md
diff --git a/docs/properties.md → docs/overview/properties.md b/docs/properties.md → docs/overview/properties.md
diff --git a/docs/client-cli.md → docs/usage/client-cli.md b/docs/client-cli.md → docs/usage/client-cli.md
diff --git a/docs/usage/configuration.md b/docs/usage/configuration.md
@@ -0,0 +1,67 @@
+# Configuration
+
+## Configuration file
+
+You can configure the Walrus client through a `client_config.yaml` file. By default, the CLI will
+look for it in the current directory or in `~/.walrus/`, but you can specify a custom path through
+the `--client` option.
+
+The configuration file currently supports the following parameters:
+
+```yaml
+# This is the only mandatory field. The system object is specific for a particular Walrus
+# deployment.
+#
+# NOTE: THE VALUE INCLUDED HERE IS AN EXAMPLE VALUE.
+# You can get the object ID for the current devnet deployment as described below.
+system_object: 0x3243f6a8885a308d313198a2e03707344a4093822299f31d0082efa98ec4e6c8
+
+# You can define a custom path to your Sui wallet configuration here. If this is unset or `null`,
+# the wallet is configured from `./sui_config.yaml` (relative to your current working directory), or
+# the system-wide wallet at `~/.sui/sui_config/client.yaml` in this order.
+wallet_config: null
+
+# The following parameters can be used to tune the networking behavior of the client. There is no
+# risk in playing around with these values. In the worst case, you may not be able to store/read
+# blob due to timeouts or other networking errors.
+communication_config:
+  max_concurrent_writes: null
+  max_concurrent_sliver_reads: null
+  max_concurrent_metadata_reads: 3
+  max_concurrent_status_reads: null
+  reqwest_config:
+    total_timeout:
+      secs: 180
+      nanos: 0
+    pool_idle_timeout: null
+    http2_keep_alive_timeout:
+      secs: 5
+      nanos: 0
+    http2_keep_alive_interval:
+      secs: 30
+      nanos: 0
+    http2_keep_alive_while_idle: true
+  request_rate_config:
+    max_node_connections: 10
+    max_retries: 5
+    min_backoff:
+      secs: 2
+      nanos: 0
+    max_backoff:
+      secs: 60
+      nanos: 0
+```
+
+## System object ID
+
+You can get the system object ID of the current devnet deployment as follows:
+
+```sh
+curl https://storage.googleapis.com/mysten-walrus-binaries/walrus-configs/client_config.yaml
+```
+
+If you want, you can directly store this as a configuration file:
+
+```sh
+curl https://storage.googleapis.com/mysten-walrus-binaries/walrus-configs/client_config.yaml -o client_config.yaml
+```
diff --git a/docs/usage/installation.md b/docs/usage/installation.md
@@ -0,0 +1,19 @@
+# Installation
+
+We currently provide the `walrus` client binary for macOS (Intel and Apple CPUs) and Ubuntu. You can
+download the latest build from our Google Cloud Storage (GCS) bucket:
+
+```sh
+SYSTEM=macos-arm64 # or macos-x86_64 or ubuntu-x86_64
+curl https://storage.googleapis.com/mysten-walrus-binaries/latest/walrus-latest-$SYSTEM -o walrus
+chmod +x walrus
+```
+
+You can then run the CLI simply as `./walrus`, or, if it is in a different location, as
+`path/to/walrus`.  Alternatively, you can also place it into any directory that is in your `$PATH`
+and run it as `walrus`. See [the next chapter](./interacting.md) for further details on how to use
+it.
+
+In addition to the latest version of the `walrus` binary, the GCS bucket also contains previous
+versions. An overview in XML format is available at
+<https://storage.googleapis.com/mysten-walrus-binaries/>.
diff --git a/docs/interacting.md → docs/usage/interacting.md b/docs/interacting.md → docs/usage/interacting.md
diff --git a/docs/json-api.md → docs/usage/json-api.md b/docs/json-api.md → docs/usage/json-api.md
diff --git a/docs/usage/prerequisites.md b/docs/usage/prerequisites.md
@@ -0,0 +1,21 @@
+# Prerequisites
+
+Interacting with Walrus requires a valid Sui testnet wallet with some amount of SUI tokens. The
+easiest way to set this up is via the Sui CLI; see the [installation
+instructions](https://docs.sui.io/guides/developer/getting-started/sui-install) in the Sui
+documentation.
+
+After installing the Sui CLI, you need to set up a testnet wallet by running `sui client`, which
+prompts you to set up a new configuration. You can use the full node at
+`https://fullnode.testnet.sui.io:443`. See
+[here](https://docs.sui.io/guides/developer/getting-started/connect) for further details.
+
+Finally, you need to get at least two SUI testnet coins from the faucet:
+
+```sh
+sui client faucet && sui client faucet
+```
+
+The system-wide wallet will be used by Walrus if no other path is specified. If you want to use a
+different Sui wallet, you can specify this in the [Walrus configuration file](./configuration.md) or
+when [running the CLI](./interacting.md).
diff --git a/docs/usage/setup.md b/docs/usage/setup.md
@@ -0,0 +1,7 @@
+# Setup
+
+At this stage of the project, our Walrus code is not yet public. Instead, we provide a pre-compiled
+`walrus` client binary for macOS (Intel and Apple CPUs) and Ubuntu, which supports different usage
+patterns (see [the next chapter](./interacting.md)). This chapter describes the
+[prerequisites](./prerequisites.md), [installation](./installation.md), and
+[configuration](./configuration.md) of the Walrus client.
diff --git a/docs/web-api.md → docs/usage/web-api.md b/docs/web-api.md → docs/usage/web-api.md
diff --git a/examples/CONFIG/bin/README.md b/examples/CONFIG/bin/README.md
@@ -1,3 +1,7 @@
-# Configuration
+# Walrus binary
 
-Place the 'walrus' client binary for your system in this directory.
+Place the `walrus` client binary for your system in this directory.
+
+See [the installation
+documentation](https://mystenlabs.github.io/walrus-docs/usage/installation.html) for how to obtain
+the binary.
diff --git a/examples/CONFIG/config_dir/README.md b/examples/CONFIG/config_dir/README.md
@@ -0,0 +1,7 @@
+# Configuration
+
+Place the `client_config.yaml` configuration file in this directory.
+
+See [the configuration
+documentation](https://mystenlabs.github.io/walrus-docs/usage/configuration.html) for details of the
+available parameters and how to obtain the mandatory parameters.
diff --git a/examples/CONFIG/config_dir/client_config.yaml b/examples/CONFIG/config_dir/client_config.yaml
diff --git a/examples/python/README.md b/examples/python/README.md
@@ -2,12 +2,29 @@
 
 ## Prerequisites
 
-- [Configure Sui Client](https://docs.sui.io/guides/developer/getting-started/connect) to connect
-  to testnet, and some testnet Sui tokens.
-- Configure Walrus TODO(#12).
-- Update the paths PATH_TO_WALRUS and PATH_TO_WALRUS_CONFIG and other
-  constant in `utils.py`.
+- Set up Sui and Walrus as described
+  [here](https://mystenlabs.github.io/walrus-docs/usage/setup.html).
+- Optional: Set up a Python virtual environment:
+
+  ```sh
+  python -m venv .venv
+  source .venv/bin/activate
+  ```
+
+- Install the dependencies:
+
+  ```sh
+  pip install -r requirements.txt
+  ```
+
+- Update the paths `PATH_TO_WALRUS` and `PATH_TO_WALRUS_CONFIG` and other constant in `utils.py`.
 
 ## Index of examples
 
-- ...
+- `hello_walrus_jsonapi.py` shows how to store and read blobs using the JSON API of the Walrus
+  client.
+- `hello_walrus_sui_system.py` prints information on the Walrus deployment based on the system
+  object.
+- `hello_walrus_webapi.py` shows how to store and read blobs using the HTTP API of the Walrus
+  client.
+- `track_walrus_events.py` is a simple script to track all Walrus-related events on Sui.
diff --git a/examples/python/hello_walrus_jsonapi.py b/examples/python/hello_walrus_jsonapi.py
@@ -13,7 +13,7 @@
 from utils import num_to_blob_id, PATH_TO_WALRUS, PATH_TO_WALRUS_CONFIG, FULL_NODE_URL
 
 try:
-    # Create a 1MB file of random data
+    # Create a 1MiB file of random data
     random_data = os.urandom(1024 * 1024)
     tmp = tempfile.NamedTemporaryFile(delete=False)
     tmp.write(random_data)

diff --git a/examples/python/track_walrus_events.py b/examples/python/track_walrus_events.py
@@ -6,9 +6,7 @@
 import requests
 import re
 
-from utils import num_to_blob_id
-
-PATH_TO_WALRUS_CONFIG = "../CONFIG/config_dir/client_config.yaml"
+from utils import num_to_blob_id, PATH_TO_WALRUS_CONFIG
 
 system_object_id = re.findall(
     r"system_object:[ ]*(.*)", open(PATH_TO_WALRUS_CONFIG).read()