feature: implement PostgreSQL for storage of per-server settings #19

RalphORama · 2023-12-28T08:10:02Z

Summary

As LinkFix gets added to more servers, several people have reached out requesting features that would require per-server configuration. We need to store those configurations.

Methodology

Firstly, I am going to try to avoid using ORMs as much as possible. the pg package offers the ability to execute raw queries which should be good enough for our requirements. If, in the future, the database gets more complicated, I wouldn't have an issue moving to something like Sequelize.

Secondly, I want LinkFix to store as little data as possible. I think we shouldn't create records for guilds until someone in the guild actually sets configuration options. This keeps us in the clear for crap like GDPR, plus I personally believe in data privacy myself and don't want to become a vector of attack for other people.

Database Structire

https://dbdiagram.io/d/LinkFix-658b15e289dea6279992fb74

To-Do

After we move into production:

Benchmark/optimize queries
Check for pool leaks

- added postgres service to docker-compose - started using Docker secrets to manage database password, bot token - add barebones "configure" command to experiment with commands - installed and configured pgtyped

realized I accidentally deleted .eslintrc with the previous commit so I took some time to re-configure it to properly work with all the files in our repo - update .eslintrc.js - add prettier config and ignore files - update pre-commit configuration to parity with eslintrc and prettierrc

- move prettier and eslint to separate jobs within the same workflow - use actions/setup-node's built in caching instead of a manual step

@ts-expect-error

using .mts was a folley... instead, it appears we need to prepend import statements in *.queries.ts files with `// @ts-expect-error moduleResolution`

we were using this functionality for getting the Discord token, we will also use this for getting the database password.

- make `id` columns properly autoincrement - flesh out table relations - add "pg" and "@types/pg" dependencies

I *think* this will allow for easy access to the database with the IP `postgres.docker`

- dropped PgTyped and related files - set up pg for raw Postgres queries - catch Ctrl+C SIGTERM so we can cleanly shut down database connections - add pgPool member to Discord Client instance - update linter configs to reflect dropping PgTyped

- docker-compose: Use a shared array of environment variables in docker compose - discord-bot: Run a test command on startup to check our database

we're going to store all the database logic in src/bot/database. also flesh out the CustomPool object to add event handlers.

- delete seed/index.ts (don't need to initialize db in software) - rename seed/ to initdb/ - change initdb path in docker-compose - use smaller postgres:16-alpine image in docker-compose

man oh man it's ugly as sin but I got some database crap working. next step is to abstract db logic out into its own files to simplify application command code.

makes merging in from main easier

problems to solve: - abstract database logic out to src/bot/database - build paramaterized query strings with DEFAULT (cannot pass DEFAULT in a parameter... just pass same value we got back from the db?) - finally get around to parsing all those boolean options from the command

wholly crap lois whole lotta bug testing I gotta do on this. also introduced some issues with AppCommandsCLI but I should really break that out into its own module anyway

shoegaze · 2024-05-19T10:41:59Z

Add permission checks - admins only?

It's possible to set permissions for a command using SlashCommandBuilder#setDefaultMemberPermissions(): https://discordjs.guide/slash-commands/permissions.html#member-permissions

e.g.

new SlashCommandBuilder()
  .setName(...)
  // ...
  .setDefaultMemberPermissions(PermissionFlagsBits.Administrator)

should make the command admin-only.

shoegaze · 2024-05-19T12:29:17Z

Should settings.guild be an FK (with REFERENCES guilds(id))?
Additionally, should it have ON DELETE CASCADE so that the settings rows are deleted when /forget is run? (Or should we keep the rows so that we can restore settings on /configure ?)

Also FYI: guilds.id is a bigint while settings.guild is an integer

RalphORama and others added 14 commits December 26, 2023 12:45

initial commit to pgtyped branch

da78648

initial database development

abbcb66

- added postgres service to docker-compose - started using Docker secrets to manage database password, bot token - add barebones "configure" command to experiment with commands - installed and configured pgtyped

style: formatted with prettier

d75492a

pgtyped: renamed config and formatted with prettier

06c4e6f

ci (linting): split prettier and eslint jobs

4d4b41b

- move prettier and eslint to separate jobs within the same workflow - use actions/setup-node's built in caching instead of a manual step

style: add comments to .env.example

ca06e39

Merge branch 'main' into feature-pgtyped

4848926

use regular .ts extensions for queries

7d7a2cb

using .mts was a folley... instead, it appears we need to prepend import statements in *.queries.ts files with `// @ts-expect-error moduleResolution`

refactor: move getFromEnvOrFile to its own file

1196aee

we were using this functionality for getting the Discord token, we will also use this for getting the database password.

db: update sql schema

938b57c

- make `id` columns properly autoincrement - flesh out table relations - add "pg" and "@types/pg" dependencies

feat(docker-compose): add links to bot

38cc337

I *think* this will allow for easy access to the database with the IP `postgres.docker`

refactor: docker secrets suck! + pgPool test

2ff7410

- docker-compose: Use a shared array of environment variables in docker compose - discord-bot: Run a test command on startup to check our database

RalphORama added enhancement New feature or request in progress Work in progress labels Dec 28, 2023

RalphORama added this to the v2.0.0 milestone Dec 28, 2023

RalphORama self-assigned this Dec 28, 2023

RalphORama marked this pull request as draft December 28, 2023 08:10

RalphORama added 9 commits December 28, 2023 15:16

refactor: move CustomPool to submodule

1c6fa9b

we're going to store all the database logic in src/bot/database. also flesh out the CustomPool object to add event handlers.

style (eslint): enable eslint/sort-imports rule

4406bd9

refactor: oops I forgot we're using initdb

e11a627

- delete seed/index.ts (don't need to initialize db in software) - rename seed/ to initdb/ - change initdb path in docker-compose - use smaller postgres:16-alpine image in docker-compose

npm run format: use cache for eslint and prettier

58396d5

feat: initial work on /config command

92617e2

man oh man it's ugly as sin but I got some database crap working. next step is to abstract db logic out into its own files to simplify application command code.

Merge branch 'main' into feature-postgresql

63607e5

Merge branch 'main' into feature-postgresql

637cf89

style: update prettierrc on our side

20cee19

makes merging in from main easier

Merge branch 'main' into feature-postgresql

b262c22

RalphORama mentioned this pull request Dec 29, 2023

📌 database implementation #11

Open

RalphORama added the database PostgreSQL and all its glory label Dec 29, 2023

RalphORama added 7 commits December 29, 2023 18:09

fix: run prettier first in npm format script

3b93258

ci: fix call to prettier linting task

d6e4b4f

refactor: move @types up one directory

78877f3

feature: add missing pg types

f23c388

fix: refactor tsconfig.json after moving @types

20d8fde

feat: /configure reads from/writes to database

e82c9ad

wholly crap lois whole lotta bug testing I gotta do on this. also introduced some issues with AppCommandsCLI but I should really break that out into its own module anyway

RalphORama mentioned this pull request Dec 31, 2023

pixiv.net -> phixiv.net replacement no longer working #23

Closed

feat: add created and updated fields to initdb

f7aba51

This was linked to issues Dec 31, 2023

📌 database implementation #11

Open

Add commands for configuring the bot per-server #4

Open

Feature Request - Delete Original Message #7

Open

RalphORama added 6 commits December 31, 2023 03:40

Merge branch 'main' into feature-postgresql

7551929

Merge branch 'main' into feature-postgresql

f78d4ed

Merge branch 'main' into feature-postgresql

f765432

Merge branch 'main' into feature-postgresql

3f66a15

fix: schema updates

887d07f

docker: only expose postgres to localhost

309dc49

RalphORama mentioned this pull request Jan 5, 2024

Tiktok embed not working #37

Closed

Deyzeria mentioned this pull request May 7, 2024

Request for per-server option for specific users to opt-out #86

Open

RalphORama closed this Oct 30, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feature: implement PostgreSQL for storage of per-server settings #19

feature: implement PostgreSQL for storage of per-server settings #19

RalphORama commented Dec 28, 2023 •

edited

Loading

shoegaze commented May 19, 2024 •

edited

Loading

shoegaze commented May 19, 2024 •

edited

Loading

feature: implement PostgreSQL for storage of per-server settings #19

feature: implement PostgreSQL for storage of per-server settings #19

Conversation

RalphORama commented Dec 28, 2023 • edited Loading

Summary

Methodology

Database Structire

To-Do

shoegaze commented May 19, 2024 • edited Loading

shoegaze commented May 19, 2024 • edited Loading

RalphORama commented Dec 28, 2023 •

edited

Loading

shoegaze commented May 19, 2024 •

edited

Loading

shoegaze commented May 19, 2024 •

edited

Loading