From 4805bf5bdfadde666778f21b0d4e0fe4cecb7aa5 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Rapha=C3=ABl=20Vinot?= Date: Thu, 8 Aug 2024 12:13:48 +0200 Subject: [PATCH] chg: Use valkey by default --- modules/ROOT/pages/dev-docs.adoc | 10 +++--- .../ROOT/pages/implementation-details.adoc | 4 +-- modules/ROOT/pages/install-lookyloo.adoc | 35 +++++++++++-------- modules/ROOT/pages/lookyloo-auth.adoc | 2 +- modules/ROOT/pages/update-lookyloo.adoc | 4 +-- 5 files changed, 31 insertions(+), 24 deletions(-) diff --git a/modules/ROOT/pages/dev-docs.adoc b/modules/ROOT/pages/dev-docs.adoc index eb20a81..541f21b 100644 --- a/modules/ROOT/pages/dev-docs.adoc +++ b/modules/ROOT/pages/dev-docs.adoc @@ -11,7 +11,7 @@ and gave it a try. == General guidelines * After editing a file, you can run the `stop` and `start` scripts to restart the platform and see the changes. - It will restart everything, including the redis databases, and can take a bot of time. See below if you're + It will restart everything, including the valkey databases, and can take a bot of time. See below if you're working on a specific part of the tool and want to stop/start a specific module. * You can disable individual services by commenting them out in `bin/start.py` * All the services can be started/killed manually (`CTRL+c` to kill) @@ -19,11 +19,11 @@ and gave it a try. == Scripts * `start`: Starts all the services -* `stop`: Triggers `shutdown`, stop redis databases. -* `shutdown`: Requests the services to stop by putting a `shutdown` key in redis, wait for all of them to stop. -* `run_backend`: Starts/stops redis databases +* `stop`: Triggers `shutdown`, stop valkey databases. +* `shutdown`: Requests the services to stop by putting a `shutdown` key in valkey, wait for all of them to stop. +* `run_backend`: Starts/stops valkey databases * `async_capture`: Triggers a capture from the queue. Multiple instances of this process can be run in parallel -* `background_indexer`: Caches the captures in redis, creates the tree pickles, adds the urls, cookies, +* `background_indexer`: Caches the captures in valkey, creates the tree pickles, adds the urls, cookies, and body hashes in indexes for fast lookup * `archiver`: Archives the old captures (default: 180+ days), maintain the indexes files in the capture directories * `processing`: Triggers actions happening once a day. Currently: creates the user-agent file of the users of the platform (if enabled) diff --git a/modules/ROOT/pages/implementation-details.adoc b/modules/ROOT/pages/implementation-details.adoc index 508362e..f835123 100644 --- a/modules/ROOT/pages/implementation-details.adoc +++ b/modules/ROOT/pages/implementation-details.adoc @@ -6,9 +6,9 @@ developed by others. == Backend -=== Redis +=== Valkey -link:https://redis.io/[Redis] is the database used to store the cached information visualized on the web interface. +link:https://valkey.io/[Valkey] is the database used to store the cached information visualized on the web interface. === Playwright diff --git a/modules/ROOT/pages/install-lookyloo.adoc b/modules/ROOT/pages/install-lookyloo.adoc index 72dbf58..c567857 100644 --- a/modules/ROOT/pages/install-lookyloo.adoc +++ b/modules/ROOT/pages/install-lookyloo.adoc @@ -3,33 +3,31 @@ [IMPORTANT] ==== -Requires Python 3.8 or higher (3.10 prefered), expects Ubuntu 22.04 or more recent. +Requires Python 3.8 or higher (3.12 prefered), expects Ubuntu 22.04 or more recent (24.04 prefered). ==== == Install Lookyloo Dependencies === Install Poetry -link:https://python-poetry.org/[Poetry] is tool to handle dependency installation +link:https://python-poetry.org/[Poetry] handles dependency installation as well as building and packaging of Python packages. -```bash -curl -sSL https://install.python-poetry.org | python3 - -``` +See the link:https://python-poetry.org/docs/#installation[installation guide] for more details. -More details in the the link:https://github.com/python-poetry/poetry/#installation[installation guide]. +=== Install Valkey -=== Install Redis +NOTE: Lacus supports valkey or redis, but valkey is prefered now due to the change of license. -link:https://redis.io/[Redis]: An open source (BSD licensed), in-memory data structure +link:https://valkey.io/[Valkey]: An open source (BSD licensed), in-memory data structure store, used as a database, cache and message broker. -NOTE: Redis should be installed from the source, and the repository must be in +NOTE: Valkey should be installed from the source, and the repository must be in the same directory as the one you will be cloning Lookyloo in to. -NOTE: Since Lookyloo 1.15, redis 7.0 is required. +NOTE: Since Lookyloo 1.25, valkey 7.2 is required. -In order to compile and test redis, you will need a few packages: +In order to compile and test valkey, you will need a few packages: ```bash sudo apt-get update @@ -37,8 +35,8 @@ sudo apt install build-essential tcl ``` ```bash -git clone https://github.com/redis/redis.git -cd redis +git clone https://github.com/valkey-io/valkey +cd valkey git checkout 7.2 make # Optionally, you can run the tests: @@ -51,9 +49,10 @@ cd .. === Prerequisites * Poetry is installed. -* Redis is installed from the source. The repository must be in the same directory +* Valkey is installed from the source. The repository must be in the same directory as the one you will be cloning Lookyloo in to. + * Python3 development package ```bash @@ -68,6 +67,14 @@ sudo apt install python3-dev git clone https://github.com/Lookyloo/lookyloo.git ``` +The directory tree must look like that: + +``` +. +├── valkey => cloned valkey +└── lookyloo => cloned lookyloo +``` + . Change directory + ``` diff --git a/modules/ROOT/pages/lookyloo-auth.adoc b/modules/ROOT/pages/lookyloo-auth.adoc index 16a66c0..af18b25 100644 --- a/modules/ROOT/pages/lookyloo-auth.adoc +++ b/modules/ROOT/pages/lookyloo-auth.adoc @@ -30,7 +30,7 @@ To authenticate your session, open `http://your.lookyloo.instance/login`. When logged in, you have access to the following URLs: * `/hidden`: show the tree hidden on the index -* `/rebuild_cache`: rebuild only the redis cache. Can take a while, but not as much as `/rebuild_all` +* `/rebuild_cache`: rebuild only the valkey cache. Can take a while, but not as much as `/rebuild_all` * `/rebuild_all`: rebuild all the tree (drop all the pickles of all the trees). **warning**: it can take a very, very long time. * `/logout`: drop the session * `/tree//rebuild`: rebuild the tree diff --git a/modules/ROOT/pages/update-lookyloo.adoc b/modules/ROOT/pages/update-lookyloo.adoc index 65fbbaa..89dc4ba 100644 --- a/modules/ROOT/pages/update-lookyloo.adoc +++ b/modules/ROOT/pages/update-lookyloo.adoc @@ -40,7 +40,7 @@ The sections below are only required if the tools have been updated, or if upgra poetry self update ``` -=== Update Redis +=== Update Valkey It requires to shutdown lookyloo completely. @@ -50,7 +50,7 @@ From the lookyloo repository: poetry run stop ``` -From the directory you cloned redis into: +From the directory you cloned valkey into: ```bash # Optional: git checkout