[WIP] Structurize, update and extend the documentation

docs/pyproject.toml

@@ -10,13 +10,15 @@ package-mode = false
 python = "^3.10"
 pygments = "^2.18.0"
 sphinx-scylladb-theme = "^1.8.1"
+#sphinx-substitution-extensions = "=2024.10.17"
 sphinx-sitemap = "^2.6.0"
+beartype = ">0.0.0"
 sphinx-autobuild = "^2024.4.19"
-Sphinx = "^7.3.7"
+Sphinx = "^8.1.3"
 sphinx-multiversion-scylla = "^0.3.1"
 redirects_cli ="^0.1.3"
-myst-parser = "^3.0.1"
-sphinx-design = "^0.5.0"
+myst-parser = "^4.0.0"
+sphinx-design = "^0.6.1"
 
 [build-system]
 requires = ["poetry>=1.8.0"]

docs/source/.internal/helm-crd-warning.md

@@ -0,0 +1,5 @@
+:::{warning}
+Helm doesn't support managing CustomResourceDefinition resources ([#5871](https://github.com/helm/helm/issues/5871), [#7735](https://github.com/helm/helm/issues/7735)).
+Helm only creates CRDs on the first install and never updates them, while keeping the CRDs up to date (with any update) is absolutely essential.
+In order to update them, users have to do it manually every time.
+:::

docs/source/.internal/manager-license-note.md

@@ -0,0 +1,5 @@
+:::{note}
+ScyllaDB Manager is available for ScyllaDB Enterprise customers and ScyllaDB Open Source users.
+With ScyllaDB Open Source, ScyllaDB Manager is limited to 5 nodes.
+See the ScyllaDB Manager [Proprietary Software License Agreement](https://www.scylladb.com/scylla-manager-software-license-agreement/) for details.
+:::

docs/source/.internal/tuning-warning.md

@@ -0,0 +1,4 @@
+:::{warning}
+We recommend that you first try out the performance tuning on a pre-production instance.
+Given the nature of the underlying tuning script, undoing the changes requires rebooting the Kubernetes node(s).
+:::

docs/source/architecture/index.md

@@ -0,0 +1,10 @@
+# Architecture
+
+:::{toctree}
+:maxdepth: 1
+
+overview
+storage/index
+tuning
+manager
+:::

docs/source/architecture/manager.md

@@ -0,0 +1,25 @@
+# ScyllaDB Manager
+
+{{productName}} has a basic integration with ScyllaDB Manager. At this point there is one global ScyllaDBManager instance that manages all [ScyllaClusters](../resources/scyllaclusters/overview.md) and a corresponding controller that automatically configures the ScyllaDB Manager to monitor the ScyllaDB instances and sync repair and backup tasks based on ScyllaCLuster definition. Unfortunately, the rest of the functionality is not yet implemented in ScyllaCluster APIs and e.g. a restore of a cluster from a backup needs to be performed by executing into the shared ScyllaDB manager deployment and using `sctool` directly by an administrator.
+
+:::{caution}
+Because ScyllaDB manager instance is shared by all users and their ScyllaClusters, only administrators should have privileges to access the `scylla-manager` namespace.
+:::
+
+
+:::{include} ../.internal/manager-license-note.md
+:::
+
+## Accessing ScyllaDB Manager
+
+For the operations that are not yet supported on ScyllaClusters, you can access the ScyllaDB manager manually.
+
+First, you need to find the ScyllaDB Maanager ID for your cluster:
+
+:::{code-block} bash
+kubectl -n=test get scyllacluster/scylla --template='{{ .status.managerId }}'
+:::
+
+:::{note}
+Note that some of the operations use *ScyllaDB Manager Agent* that runs within the ScyllaCluster that has to have access e.g. to buckets being used.
+:::

docs/source/architecture/overview.md

@@ -0,0 +1,31 @@
+# Overview
+
+## Foreword
+
+{{productName}} is a set of controllers and API extensions that need to be installed in your cluster.
+The Kubernetes API is extended using [CustomResourceDefinitions (CRDs)](https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/) and [dynamic admission webhooks](https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/) to provide new resources ([API reference](/api-reference/index)).
+These resources are reconciled by controllers embedded within {{productName}} deployment.
+
+ScyllaDB is a stateful application and {{productName}} requires you to have a storage provisioner installed in your cluster.
+To achieve the best performance, we recommend using a storage provisioner based on local NVMEs.
+You can learn more about different setups in [a dedicated storage section](./storage/overview.md).
+
+## Components
+
+{{productName}} deployment consists of several components that needs to be installed / present in you Kubernetes cluster.
+By design, some of the components need elevated permissions, but they are only accessible to the administrators.
+
+
+### Cluster scoped
+```{image} ./components-cluster_scoped.svg
+:name: components-cluster-scoped
+:align: center
+:scale: 75%
+```
+
+### Namespaced
+```{image} ./components-namespaced.svg
+:name: components-namespaced
+:align: center
+:scale: 75%
+```

docs/source/architecture/storage/index.md

@@ -0,0 +1,8 @@
+# Storage
+
+:::{toctree}
+:maxdepth: 1
+
+overview
+local-csi-driver
+:::

docs/source/architecture/storage/local-csi-driver.md

@@ -0,0 +1,14 @@
+# Local CSI Driver
+
+## About
+
+The Local Volume Provisioner implements the [Container Storage Interface (CSI)](https://kubernetes.io/blog/2019/01/15/container-storage-interface-ga/), a specification for container orchestrators to manage the lifecycle of volumes.
+
+It supports dynamic provisioning on local disks, so storage volumes can be created on-demand through managed directories on the local disk.
+
+## CSI Features
+
+Local CSI Driver implements the following CSI features:
+- Controller Service
+- Node Service
+- Identity Service

docs/source/architecture/storage/overview.md

@@ -0,0 +1,23 @@
+# Overview
+
+ScyllaDB works with both local and network attached storage provisioners, but our primary focus is around the local storage that can provide the best performance. We support using 2 local provisioners: [ScyllaDB Local CSI Driver](https://github.com/scylladb/local-csi-driver) and [Kubernetes SIG Storage Local Persistence Volume Static Provisioner](https://github.com/kubernetes-sigs/sig-storage-local-static-provisioner).
+
+## Setting up local disks
+
+When a Kubernetes node having local disk(s) is created, the storage is usually uninitialized. This heavily depends on you platform and its options, but even when provisioned with mounted disks, they usually don't set up *RAID*, nor have the option to choose a file system type. (ScyllaDB needs the storage to be formatted with `xfs`.)
+
+Setting up the RAID arrays, formatting the file system or mounting it in a declarative manner is challenging and that's one of the reasons we have created the [NodeConfig](../../resources/nodeconfigs.md) custom resource.
+
+## Supported local provisioners
+
+### ScyllaDB Local CSI driver
+
+ScyllaDB Local CSI Driver supports dynamic provisioning on local disks and sharing the storage capacity.
+It is based on dynamic directories and `xfs` `prjquota`.
+It allows [PersistentVolumes](https://kubernetes.io/docs/concepts/storage/persistent-volumes/) to be created dynamically for a corresponding [PersistentVolumeClaim](https://kubernetes.io/docs/concepts/storage/volumes/#persistentvolumeclaim) by automatically provisioning directories created on disks attached to instances. On supported filesystems, directories have quota limitations to ensure volume size limits.
+
+At this point, the Local CSI Driver doesn't know how to provision block devices which would give you a more dedicated and isolated dynamic storage.
+
+### Kubernetes SIG Storage Static Local Volume Provisioner
+
+The local volume static provisioner is a Kubernetes SIG Storage project that can turn your disks into dedicated and isolated Persistent volumes but all of them have to be created manually.

docs/source/architecture/tuning.md

@@ -0,0 +1,36 @@
+# Tuning
+
+ScyllaDB works best when it's pinned to the CPUs and not interrupted.
+To get the best performance and latency {{productName}} 
+
+One of the most common causes of context-switching are network interrupts.
+Packets coming to a Kubernetes node need to be processed which requires using CPU shares.  
+
+On K8s we always have at least a couple of processes running on the node: kubelet, kubernetes provider applications, daemons etc. 
+These processes require CPU shares, so we cannot dedicate entire node processing power to Scylla, we need to leave space for others.  
+We take advantage of it, and we pin IRQs to CPUs not used by any Scylla Pods exclusively.
+
+Performance tuning is enabled by default **when you create a corresponding [NodeConfig](../resources/nodeconfigs.md) for your nodes**.
+
+Because some of the operations it needs to perform are not multitenant or require elevated privileges, the tuning scripts are run in a dedicated system namespace called `scylla-operator-node-tuning`.
+This namespace is created and entirely managed by {{productName}} and only administrators can access it.
+
+The tuning is based around `perftune` script that comes from [scyllaDBUtilsImage](../api-reference/groups/scylla.scylladb.com/scyllaoperatorconfigs.rst#api-scylla-scylladb-com-scyllaoperatorconfigs-v1alpha1-status). `perftune` executes the performance optmizations like tuning the kernel, network, disk devices, spreading IRQs across CPUs and more. Conceptually this is run in 2 parts: tuning the [Kubernetes nodes](#kubernetes-nodes) and tuning for [ScyllaDB Pods](#scylladb-pods).
+
+:::{include} ../.internal/tuning-warning.md
+:::
+
+## Kubernetes nodes
+
+`perftune` script is executed on the target nodes and tunes kernel, network, disk devices and more.
+This is executed right after the tuning is enabled using a [NodeConfig](../resources/nodeconfigs.md)
+
+## ScyllaDB Pods
+
+When a [ScyllaCluster](../resources/scyllaclusters/overview.md) Pod is created (and performance tuning is enabled), the Pod initializes but waits until {{productName}} runs an on-demand Job that will configure the host and the ScyllaDB process accordingly (e.g. spreading IRQs across other CPUs).
+Only after that it will actually start running ScyllaDB.
+
+:::{caution}
+Only Pods with [`Guaranteed` QoS class](https://kubernetes.io/docs/concepts/workloads/pods/pod-qos/#guaranteed) are eligible to be tuned, otherwise they would not have pinned CPUs.
+
+Always verify that your [ScyllaCluster](../resources/scyllaclusters/overview.md) resource specifications meat [all the criteria](https://kubernetes.io/docs/concepts/workloads/pods/pod-qos/#criteria).

docs/source/conf.py

@@ -50,8 +50,18 @@
 
 # -- Options for myst parser
 
-myst_enable_extensions = ["colon_fence"]
+myst_enable_extensions = ["colon_fence", "attrs_inline", "substitution"]
 myst_heading_anchors = 6
+myst_substitutions = {
+  "productName": "Scylla Operator",
+  "repository": "scylladb/scylla-operator",
+  "revision": "master",
+  "imageRepository": "docker.io/scylladb/scylla",
+  "imageTag": "6.2.0",
+  "enterpriseImageRepository": "docker.io/scylladb/scylla-enterprise",
+  "enterpriseImageTag": "2024.1.12",
+  "agentVersion": "3.4.0",
+}
 
 # -- Options for not found extension
 
@@ -78,7 +88,6 @@
 # Defines which version is considered to be the latest stable version.
 # Must be listed in smv_tag_whitelist or smv_branch_whitelist.
 smv_latest_version = 'v1.14'
-smv_rename_latest_version = 'stable'
 # Whitelist pattern for remotes (set to None to use local branches only)
 smv_remote_whitelist = r"^origin$"
 # Pattern for released versions