Persistency layer

[kettanaito]

I suggest we add a client-side synchronization layer to keep multiple clients that use the same DB in sync.

Implementation

Each factory call attaches a sync middleware. That middleware establishes a BroadcastChannel with a reproducible ID that allows other DB to communicate via the same channel.

Whenever there's a change to a DB (create/update/delete), that change is signaled to the channel. Other DB subscribe to channel events and apply the occurred change to their own instances.

Motivation

Multiple tabs of the same app should treat DB as a source of truth, meaning operations performed in one tab should be reflected when communicating with the DB in another tab. Since each JavaScript runtime will instantiate its own DB instance, the DB updates must be synchronized between clients.

Does this work in Node.js?

No. The synchronization layer is designed only for client-side usage. There is nothing to sync between the client and Node. That connection should be covered in the persistance layer.

Roadmap

• Synchornize database operations between active tabs.
• Persist and hydrate database state in sessionStorage.

[kettanaito]

I have an intention to replace the storage library with the synchronization layer of the data library. They are going to work very similarly under the hood, the exception being is that the data library encapsulates the synchronization, making it an implicit internal mechanics.

[kettanaito]

Actually, I've come to the conclusion that using the @mswjs/storage library would be beneficial.

[kettanaito]

This logic can be applied internally in two separate middleware:

• Synchronization middleware. Utilizes BroadcastChannel to sync database operations between currently open tabs.
• Persistency middleware. Flushes the latest database state into sessionStorage and hydrates from it upon page load.

Those two are independent functionalities that together contribute to a good client-side experience.

[easybird]

Hi,

Thank you for this library. It's great.
But this is the killer feature I need it to be usable..we want to use it during frontend development and some sort of quick demo mode.

@kettanaito , why did you stop working on #87 ? Any plans on finishing it?
Or anything I can do to help finishing it?

[kettanaito]

Hey, @easybird. Thank you for your kind words.

I think I've stopped working on the feature simply due to the lack of time. Technically, the last task I recall tackling was the proper serialization of the entire database instance to the session storage. Now, with the migration to Symbols for internal properties, it'd have to have an additional step to serialize those as well.

I wouldn't advise following that particular branch, the library has changed quite significantly since then, including some internal refactoring that may pose more challenges than it's worth rebase upon.

If this feature is crucial for you, consider supporting me so I could dedicate proper time to work on it and have it released sooner. You can do so via GitHub Sponsors or Open Collective. Sponsorships are entirely voluntary, I just explain that there isn't time for everything and features do get overlooked (not forgotten, though!). Thanks for understanding.

That being said, I'm planning some time off until the rest of the year, and I won't be working on any features in my open-source projects. I do think the persistency layer is a core feature for the Data library, and I'm planning on working on it somewhen next year.

[arekucr]

Hi any new on the persistency layer?

[kettanaito]

@are, hi. Nope, you can see any updates in the PR #87. For now, there's no plans to continue with this.

[noveogroup-amorgunov]

Hello, @kettanaito thank you for great ecosystem around msw. What news about #87 ? Do you have some plans about this one?

[Kamahl19]

@kettanaito I have been using the above PR #277 for a month now, works great!

[Kamahl19]

According to this #285 no new features will be merged. I have modified @noveogroup-amorgunov 's code to be used outside of mswjs-data internal code.

You can see it being used in this project https://github.com/Kamahl19/react-starter/blob/main/src/mocks/persist.ts . I will keep the most up-to-date version there.

Usage:

import { factory, primaryKey } from '@mswjs/data';
const db = factory({ ... });
persist(db);

Create persist.ts with this code

import debounce from 'lodash/debounce';
import {
  DATABASE_INSTANCE,
  ENTITY_TYPE,
  PRIMARY_KEY,
  type FactoryAPI,
  type Entity,
  type ModelDictionary,
  type PrimaryKeyType,
} from '@mswjs/data/lib/glossary';
import {
  type SerializedEntity,
  SERIALIZED_INTERNAL_PROPERTIES_KEY,
} from '@mswjs/data/lib/db/Database';
import { inheritInternalProperties } from '@mswjs/data/lib/utils/inheritInternalProperties';

const STORAGE_KEY_PREFIX = 'mswjs-data';

// Timout to persist state with some delay
const DEBOUNCE_PERSIST_TIME_MS = 10;

type Models<Dictionary extends ModelDictionary> = Record<
  keyof Dictionary,
  Map<PrimaryKeyType, Entity<Dictionary, any>> // eslint-disable-line @typescript-eslint/no-explicit-any
>;

type SerializedModels<Dictionary extends ModelDictionary> = Record<
  keyof Dictionary,
  Map<PrimaryKeyType, SerializedEntity>
>;

// eslint-disable-next-line @typescript-eslint/no-explicit-any
export default function persist<Dictionary extends ModelDictionary>(
  factory: FactoryAPI<Dictionary>,
) {
  if (typeof window === 'undefined' || typeof sessionStorage === 'undefined') {
    return;
  }

  const db = factory[DATABASE_INSTANCE];

  const key = `${STORAGE_KEY_PREFIX}/${db.id}`;

  const persistState = debounce(function persistState() {
    // eslint-disable-next-line @typescript-eslint/dot-notation, @typescript-eslint/consistent-type-assertions
    const models = db['models'] as Models<Dictionary>;
    // eslint-disable-next-line @typescript-eslint/dot-notation, @typescript-eslint/consistent-type-assertions
    const serializeEntity = db['serializeEntity'] as (
      entity: Entity<Dictionary, any>, // eslint-disable-line @typescript-eslint/no-explicit-any
    ) => SerializedEntity;

    const json = Object.fromEntries(
      Object.entries(models).map(([modelName, entities]) => [
        modelName,
        Array.from(entities, ([, entity]) => serializeEntity(entity)),
      ]),
    );

    sessionStorage.setItem(key, JSON.stringify(json));
  }, DEBOUNCE_PERSIST_TIME_MS);

  function hydrateState() {
    const initialState = sessionStorage.getItem(key);

    if (initialState) {
      // eslint-disable-next-line @typescript-eslint/consistent-type-assertions
      const data = JSON.parse(initialState) as SerializedModels<Dictionary>;

      for (const [modelName, entities] of Object.entries(data)) {
        for (const entity of entities.values()) {
          db.create(modelName, deserializeEntity(entity));
        }
      }
    }

    // Add event listeners only after hydration
    db.events.on('create', persistState);
    db.events.on('update', persistState);
    db.events.on('delete', persistState);
  }

  if (document.readyState === 'loading') {
    document.addEventListener('DOMContentLoaded', hydrateState);
  } else {
    hydrateState();
  }
}

function deserializeEntity(entity: SerializedEntity) {
  const { [SERIALIZED_INTERNAL_PROPERTIES_KEY]: internalProperties, ...publicProperties } = entity;

  inheritInternalProperties(publicProperties, {
    [ENTITY_TYPE]: internalProperties.entityType,
    [PRIMARY_KEY]: internalProperties.primaryKey,
  });

  // eslint-disable-next-line @typescript-eslint/consistent-type-assertions, @typescript-eslint/no-explicit-any
  return publicProperties as Entity<any, any>;
}