feat(server): faster folder responses

[mertalev]

Description

• Adds separate asset_folders table (many assets to one folder) containing unique parent folders. This ends up being much faster than indexing the main assets table for our access patterns when the library is large.
  • Changes asset creation to first insert into this table
  • Nightly job to remove folders that are no longer used by any assets
• Sort responses by asset_folder's path and assets's originalFileName, treating numbers as numbers (1, 2, 10 instead of 1, 10, 2)
• Distinguish between absolute and relative paths (/upload and upload are not the same)
  • Added a few helper functions to handle / and make path manipulation more ergonomic
• Only query for assets for folders that have assets, using a Symbol in the tree denoting a leaf folder
  • Symbols are unique, don't conflict with string properties and don't appear when listing object keys or entries, making them ideal here

originalPath should ideally be removed to have a single source of truth for the file path and make rows narrower in the assets table. However, in the interest of making downgrading possible and giving us more options to fix any possible bugs, I think it's better to keep it in the short-term.

How Has This Been Tested?

Tested with a large instance containing an external library, storage template assets and a few corrupt images in the upload folder. They all display in the folder sidebar, and navigating each is instant (numbers are response times from the server):

• 7000-7300ms -> 75ms when getting 35k unique paths for 2m assets in the initial page load
• 400-600ms -> 8-10ms when going to a folder that contains assets
• 180-200ms -> 0ms when going to a folder that doesn't contain any assets

Folders and files are in natural sorted order as expected.

The tag page is unchanged.

In draft, pending handling updates to originalPath and adding E2E tests.

[alextran1502]

If you want to update the folders instantly, do we have a job to do that?

[mertalev]

It doesn't handle updates that change originalPath right now (so mainly storage template migration). That will just be modifying the update query and won't be tied to a particular job.

[jrasm91]

This seems to add a lot of edge case handling in the frontend. Can we simplify it by handling more of them server-side?

[mertalev]

We can remove normalizeTreePath since the server response will never have trailing slashes. They won't happen in code either as long as joinPaths is used instead of basic string concatenation.

[jrasm91]

Then why do we have this logic here?

[jrasm91]

Do you mind moving the vacuum logic into a separate pull request?

[jrasm91]

Lots of conflicts, so closing for now. If you get around to working on this in the future we can always revisit it.