Improve Self-Hosting Guide

docs/python/tutorials/workflow-tutorial.md

@@ -253,6 +253,23 @@ async def example_workflow(friend: str):
     return result
 ```
 
+## Workflow Versioning and Recovery
+
+Because DBOS recovers workflows by re-executing them using information saved in the database, a workflow cannot safely be recovered if its code has changed since the workflow was started.
+To guard against this, DBOS _versions_ applications and their workflows.
+When DBOS is launched, it computes an application version from a hash of the source code of its workflows (this can be overridden by setting the `DBOS__APPVERSION` environment variable).
+All workflows are tagged with the application version on which they started.
+When DBOS tries to recover workflows, it only recovers workflows whose version matches the current application version.
+This prevents unsafe recovery of workflows that depend on different code.
+
+On DBOS Cloud, when an application is redeployed, executors running old versions are retained until they have completed all workflows that started on those versions.
+When self-hosting, to safely recover workflows started on an older version of your code, you should start a process running that code version.
+You can also manually recover a workflow on your current version with:
+
+```shell
+dbos workflow resume <workflow-id>
+```
+
 ## Workflow Management
 
 Because DBOS stores the execution state of workflows in Postgres, you can view and manage your workflows from the command line.

docs/self-hosting/self-hosting.md

@@ -0,0 +1,70 @@
+---
+sidebar_position: 1
+title: Self-Hosting DBOS Apps
+---
+
+Because DBOS is implemented in lightweight TypeScript and Python libraries, you can run your DBOS application anywhere as long as it has a Postgres server to connect to.
+This guide provides information on operating a self-hosted DBOS application.
+
+## Self-Hosting On A Single Server
+
+Self-hosting a DBOS application on a single server is simple: each time you restart your application's process, it recovers all workflows that were executing before the restart (all `PENDING` workflows).
+
+However, it is important to be careful when upgrading your application's code.
+When DBOS is launched, it computes an "application version" from a checksum of the code in your application's workflows (you can override this version through the `DBOS__APPVERSION` environment variable).
+Each workflow is tagged with the version of the application that started it.
+When a DBOS application starts, it does not recover workflows tagged with a different application version.
+Thus, to safely recover workflows started on an older version of your code, you should start a process running that code version.
+
+## Self-Hosting on Multiple Servers
+
+When self-hosting in a distributed setting, it is important to manage workflow recovery so that when an executor crashes, restarts, or is shut down, its workflows are recovered.
+You should assign each executor running a DBOS application an executor ID by setting the `DBOS__VMID` environment variable.
+Each workflow is tagged with the ID of the executor that started it.
+When an application with an executor ID restarts, it only recovers pending workflows assigned to that executor ID.
+You can also instruct your executor to recover workflows assigned to other executor IDs through the [workflow recovery endpoint of the admin API](#workflow-recovery).
+
+It is also important to be careful when upgrading your application's code.
+When DBOS is launched, it computes an "application version" from a checksum of the code in your application's workflows (you can override this version through the `DBOS__APPVERSION` environment variable).
+Each workflow is tagged with the version of the application that started it.
+When a DBOS application starts, it does not recover workflows tagged with a different application version.
+To safely recover workflows started on an older version of your code, you should start a process running that code version and use the [workflow recovery endpoint of the admin API](#workflow-recovery) to instruct it to recover workflows belonging to executors that ran old versions of DBOS.
+
+## Admin API Reference
+
+DBOS applications expose an admin API on port 3001.
+It provides the following endpoints:
+
+### Health Check
+
+- **Endpoint**: `/dbos-healthz`
+- **HTTP Method**: GET
+- **Description**: Performs a health check on the application.
+- **Response**:
+  - **Status Code**: 200 OK if the system is healthy; otherwise, appropriate error codes.
+
+### Workflow Recovery
+
+- **Endpoint**: `/dbos-workflow-recovery`
+- **Method**: POST
+- **Description**: Recover all pending workflows associated with input [executor IDs](#managing-workflow-recovery). Returns the IDs of all workflows queued for recovery.
+- **Request Body Format**: JSON list of executors whose pending workflows to recover.
+  - **Example**:
+    ```json
+    ["executor-id-1", "executor-id-2", "..."]
+    ```
+- **Response**:
+  - **Status Code**: 200 OK on successful recovery initiation; otherwise, appropriate error codes.
+  - **Body Format**: JSON list of the IDs of workflows queued for recovery.
+  - **Example**:
+    ```json
+    ["workflow-uuid-1", "workflow-uuid-2", "..."]
+    ```
+
+### Deactivate
+
+- **Endpoint**: `/deactivate`
+- **Method**: GET
+- **Description**: Deactivate an executor. A deactivated executor may complete active workflows and recover `PENDING` workflows, but may not start new workflows or dequeue workflows.
+- **Response**:
+  - **Status Code**: 200 OK if the request succeeeded; otherwise, appropriate error codes.

docs/typescript/tutorials/workflow-tutorial.md

@@ -224,6 +224,23 @@ All messages are persisted to the database, so if `send` completes successfully,
 If you're sending a message from a workflow, DBOS guarantees exactly-once delivery because [workflows are reliable](#reliability-guarantees).
 If you're sending a message from normal TypeScript code, you can specify an idempotency key for `send` or use [`DBOS.withNextWorkflowID`](../reference/transactapi/dbos-class.md#assigning-workflow-ids) to guarantee exactly-once delivery.
 
+## Workflow Versioning and Recovery
+
+Because DBOS recovers workflows by re-executing them using information saved in the database, a workflow cannot safely be recovered if its code has changed since the workflow was started.
+To guard against this, DBOS _versions_ applications and their workflows.
+When DBOS is launched, it computes an application version from a hash of the source code of its workflows (this can be overridden by setting the `DBOS__APPVERSION` environment variable).
+All workflows are tagged with the application version on which they started.
+When DBOS tries to recover workflows, it only recovers workflows whose version matches the current application version.
+This prevents unsafe recovery of workflows that depend on different code.
+
+On DBOS Cloud, when an application is redeployed, executors running old versions are retained until they have completed all workflows that started on those versions.
+When self-hosting, to safely recover workflows started on an older version of your code, you should start a process running that code version.
+You can also manually recover a workflow on your current version with:
+
+```shell
+npx dbos workflow resume <workflow-id>
+```
+
 ## Workflow Management
 
 Because DBOS stores the execution state of workflows in Postgres, you can view and manage your workflows from the command line.

docusaurus.config.js

@@ -128,7 +128,11 @@ const config = {
           },
           {
             from: '/typescript/tutorials/self-hosting',
-            to: '/typescript/tutorials/development/self-hosting',
+            to: '/self-hosting',
+          },
+          {
+            from: '/typescript/tutorials/development/self-hosting',
+            to: '/self-hosting',
           },
           {
             from: '/typescript/tutorials/using-libraries',

sidebars.js

@@ -59,6 +59,16 @@ const sidebars = {
         }
       ],
     },
+    {
+      type: 'category',
+      label: 'Self-Hosting',
+      items: [
+        {
+          type: 'autogenerated',
+          dirName: 'self-hosting',
+        }
+      ],
+    },
     {
       type: 'category',
       label: 'Integrations',