Skip to content

Commit

Permalink
update arch doc
Browse files Browse the repository at this point in the history
  • Loading branch information
danReynolds committed Sep 25, 2024
1 parent 1730615 commit 3d105c6
Showing 1 changed file with 34 additions and 40 deletions.
74 changes: 34 additions & 40 deletions architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ final messageDoc = Loon.collection('users').doc('1').subcollection('messages').d
messageDoc.create('Hello');
```

In this example, a user message is stored under its associated user in the document tree. The data is modeled as a tree data structure that groups values under a given path, enabling quick access to all values of a particular collection. Given the example above, the [ValueStore] representation is shown below:
In this example, a user message is stored under its associated user in the document tree. The data is modeled as a tree data structure that groups values under a given path, enabling quick access to all values of a particular collection. Given the example above, the `ValueStore` representation is shown below:

```dart
{
Expand Down Expand Up @@ -44,19 +44,19 @@ Loon.collection('users').where((snap) => snap.data.name == 'Luke Skywalker').str

### Broadcasts

Each type of observable ([ObservableDocument], [ObservableQuery]) implements the [BroadcastObserver] interface, and are stored in a set of broadcast observers
on the [BroadcastManager].
Each observable (`ObservableDocument`, `ObservableQuery`) implements the `BroadcastObserver` interface, and are stored in a set of broadcast observers
on the `BroadcastManager`.

When changes occur to documents in the store, the change event, such as an [EventTypes.added], [EventTypes.modified] or [EventTypes.removed]
event, is recorded in a broadcast tree, which is another instance of a [ValueStore].
When changes occur to documents in the store, the change event such as a `BroadcastEvents.added`, `BroadcastEvents.modified` or `BroadcastEvents.removed`
event, is recorded in the broadcast tree, which is another instance of a `ValueStore`.

All changes that occur in the same task of the event loop are batched into the current broadcast tree and are scheduled to be processed asynchronously by broadcast observers. The handing off of the batched change events to each broadcast observer for processing is called the *broadcast*.
All changes that occur in the same task of the event loop are batched into the current broadcast tree and are scheduled to be processed asynchronously by the broadcast observers. The handing off of the batched change events to each broadcast observer for processing is called the *broadcast*.

On broadcast, each broadcast observer checks if it is affected by any of the changes in the current broadcast.

* For an [ObservableDocument], this involves checking if the broadcast tree contains an event for itself.
* For an [ObservableQuery], this involves iterating through the changes at its collection path in the broadcast tree
and determining if any of those changes affect its result set.
* For an `ObservableDocument`, this involves checking if the broadcast tree contains an event for itself.
* For an `ObservableQuery`, this involves iterating through the changes at its collection path in the broadcast tree
and determining if any of those changes affect its result set. The complexity of this check scales with the size of the *changes* to its collection, rather than the size of the query's existing result set.

If the broadcast observer has changes, then it emits its updated data to its listeners.

Expand All @@ -65,7 +65,6 @@ If the broadcast observer has changes, then it emits its updated data to its lis
Documents can specify that they depend on other documents and that they should react to changes to those documents.

```dart
Loon.collection(
'posts',
dependenciesBuilder: (postSnap) {
Expand All @@ -77,38 +76,36 @@ Loon.collection(
```

In this example, each post specifies that it has a dependency on its associated user. This means that when a post's user changes,
any observers of the post, meaning an [ObservableDocument] for that specific post or an [ObservableQuery] that currently includes
any observers of the post, such as an `ObservableDocument` for that specific post or an `ObservableQuery` that currently includes
that post in its list of documents, should also notify its listeners.

In this example, if a post was to be displayed along with its user's profile picture, then without dependencies it would need to observe
both the post and user document for changes. With dependencies, it only needs to observe the post and the post will react to any changes to the user automatically.
If a post is displayed alongside its user's profile picture, then without dependencies the code for the post would need to observe
both the post and user document for changes separately. With dependencies, it only needs to observe the post and the post will react to any changes to the user automatically.

Document dependencies are modeled using a [ValueStore] for indexing a document's dependencies by path, and a flat map of documents to their set of dependents.
Document dependencies are modeled using a `ValueStore` for indexing a document's dependencies by path, and a flat map of documents to their set of dependents.

```dart
final dependenciesStore = ValueStore<Set<Document>>();
final Map<Document, Set<Document>> dependentsStore = {};
```

When a document is updated, it iterates through its set of dependents and marks each of them for broadcast with a [BroadcastEvent.touched] event.
When a document is updated, it iterates through its set of dependents and marks each of them for broadcast with a `BroadcastEvent.touched` event.

When a document or collection is deleted, each broadcast observer with dependencies checks to see if the deleted path is present in its dependency tree and if it is, then the broadcast observer notifies its listeners.

Each broadcast observer has its own cached dependency tree which maintains a ref count of the number of times a path exists in the tree. This tree is implemented using the [PathRefStore]. When a path's ref count goes to 0, it is removed from the tree. This ref store is used to determine if a deleted path exists in an observer's dependencies.
Each broadcast observer has its own cached dependency tree which maintains a ref count of the number of times a path exists in the tree using the `PathRefStore`. When a path's ref count goes to 0, it is removed from the tree. This ref store is used to determine if a deleted path exists in an observer's dependencies.

## Persistence Layer

Loon comes with a default [FilePersistor] implementation. To enable persistence, clients can make a call to `configure()` with the persistor as shown below:
Loon comes with a default `FilePersistor` implementation. To enable persistence, clients call to `configure()` with the persistor as shown below:

```dart
Loon.configure(
persistor: FilePersistor(),
);
Loon.configure(persistor: FilePersistor());
```

By default, the [FilePersistor] serializes and persists all documents in a single `__store__.json` file.
By default, the `FilePersistor` serializes and persists all documents in a single `__store__.json` file.

In order to support persistence, collections must specify a fromJson/toJson serialization implementation:
Non-primitive collections (not a string, bool, number) must specify a fromJson/toJson serialization implementation:

```dart
class UserModel {
Expand Down Expand Up @@ -138,7 +135,7 @@ class UserModel {
}
```

The [FilePersistor] extends a base [Persistor] class that automatically handles throttling and synchronization of the four types of persistence operations:
The `FilePersistor` extends a base `Persistor` class that automatically handles throttling and synchronization of the four types of persistence operations:

```dart
/// Persist function called with the bath of documents that have changed (including been deleted) within the last throttle window
Expand All @@ -157,13 +154,10 @@ Future<void> clear(Collection collection);
Future<void> clearAll();
```

Any custom persistor can be implemented using just these four operations. In the case of the [FilePersistor], it packages each of these operations
into messages that it sends to the [FilePersistorWorker], which runs on a separate isolate.

The worker receives persistence operation messages from the main isolate and passes them off to the [FileDataStoreManager] for processing. The [FileDataStoreManager]
maintains a set of hydrated [FileDataStore] objects, which map 1:1 with a persistence file and are responsible for hydrating and persisting documents to and from their associated files.
Any custom persistor can be implemented using just these four operations. In the case of the `FilePersistor`, it packages each of these operations
into messages that it sends to the `FilePersistorWorker` running on a background isolate.

Each [FileDataStore] can contain any arbitrary subset of document data from across various collections. By default, all document data is stored in a single [FileDataStore] that writes documents to a `__store__.json` file.
The worker receives these persistence operation messages from the main isolate and passes them off to the [FileDataStoreManager] for processing. The `FileDataStoreManager` maintains a set of hydrated `FileDataStore` objects, which map 1:1 between persistence keys and file stores and are responsible for hydrating and persisting documents to and from their associated files.

Individual collections can customize their persistence file by specifying a persistor key:

Expand Down Expand Up @@ -198,7 +192,7 @@ class UserModel {
}
```

In the above example, the users collection has specified that all user documents should be stored in a separate [FileDataStore] and persisted
In the above example, the users collection has specified that all user documents should be stored in a separate `FileDataStore` and persisted
to a file named after its key called `users.json`.

File persistence can be specified even more granularly at the individual document-level using a `FilePersistor.keyBuilder`:
Expand Down Expand Up @@ -236,16 +230,16 @@ class UserModel {
}
```

Whenever a document is updated, it will evaluate the `keyBuilder` function with its latest data in order to determine the file in which it should be stored. If the resolved persistence key of an existing document changes, then the document and its subcollections will be moved from the previous file to the one specified by its updated key.
Documents with a `keyBuilder` recalculate their resolved persistence key whenever they are updated. If the resolved persistence key of an existing document changes, then the document and its subcollections (provided they don't specify their own persistence key) will be moved from the previous file store to the one specified by its updated key.

This level of persistence granularity is done to support features like sharding of large collections across multiple files (like breaking one big messages collection into `messages_shard_1`, `messages_shard_2`, etc) and grouping of small subcollections into the same file (like grouping subcollections of posts with document paths `posts__1__reactions__2`, `posts__2__reactions__1`, etc into one large reactions file).

Since any given collection can be spread across multiple [FileDataStore] objects, a [FileDataStoreResolver] is used to lookup the set of files that contain data for a given store path. The [FileDataStoreResolver] uses a variant of the [ValueStore] implementation called a [ValueRefStore] which extends the [ValueStore] with ref counting
Since any given collection can be spread across multiple `FileDataStore` objects, the `FileDataStoreResolver` is used to lookup the set of files that contain data for a given store path. The `FileDataStoreResolver` uses a variant of the `ValueStore` implementation called a `ValueRefStore` which extends `ValueStore` with ref counting
of the values that exist under a given path.

To illustrate how this works, consider the case of hydrating a users collection that is spread across multiple files in the format `users_1`, `users_2`, etc. using a `FilePersistor.keyBuilder`.
To illustrate how this works, consider the case of hydrating a users collection that is spread across multiple files in the format `users_1`, `users_2`, ... using a `FilePersistor.keyBuilder`.

The resolver mapping for this collection could look like the following:
The file data store resolver would look like the following:

```dart
{
Expand All @@ -268,9 +262,9 @@ The resolver mapping for this collection could look like the following:
}
```

Each node of the resolver contains a ref count of the values that exist under its path in the tree. Because the number of files should be small (assumption that storing data across 100s is files is an edge case not desirable for users), the references at each node will also be small and the performance benefit of being able to quickly resolve the set of stores that exist under a given path is worth the memory trade-off.
Each node of the resolver contains a ref count of the values that exist under its path in the tree. Because the number of files should be small (assumption that storing data across 100s is files is an edge case not desirable for users), the references at each node will also be small and the benefit of being able to quickly resolve the set of stores that exist under a given path is currently thought to be worth the memory trade-off.

The resolver persists this mapping to a `__resolver__.json` file that is hydrated automatically when the [FilePersistor] is initialized.
The resolver persists this mapping to a `__resolver__.json` file that is hydrated automatically when the `FilePersistor` is initialized.

Here's a full example of how it all comes together. In this scenario, the client has specified to hydrate **just** the `users` collection by calling `hydrate()` with a collection path.

Expand All @@ -282,12 +276,12 @@ Future<void> main() async {
}
```

When the hydration operation for the users collection is executed, the [FilePersistor] sends a [hydrate] message for the given collection to the [FilePersistorWorker] on the background isolate.
When the hydration operation for the users collection is executed, the `FilePersistor` sends a `hydrate` message for the given collection to the `FilePersistorWorker` on the background isolate.

The worker parses the message and tells the [FileDataStoreManager] to hydrate the appropriate [FileDataStore] objects containing data for the given collection. The manager uses the [FileDataStoreResolver] to look up all of the file key refs for the `users` collection and immediately finds two values: `users_1` and `users_2`.
The worker parses the message and tells the `FileDataStoreManager` to hydrate all `FileDataStore` objects containing data for the given collection. The manager uses the `FileDataStoreResolver` to look up all of the file key refs for the `users` collection and immediately finds two values: `users_1` and `users_2`.

The [FileDataStoreManager] then iterates over each resolved [FileDataStore] and hydrates its file from disk. The manager returns only the requested collection data hydrated from the file to the [FileDataStoreWorker], which packages it into a message and sends its response back to the [FilePersistor] on the main isolate.
The `FileDataStoreManager` then iterates over each resolved `FileDataStore` and hydrates its file from disk. The manager returns only the subset of paths requested by the hydration call to the `FileDataStoreWorker`, which packages it into a message and sends its response back to the `FilePersistor` on the main isolate.

The documents are then written into the Loon document store on the main isolate and are available to the client.
The documents are then written into the Loon document store on the main isolate and broadcast to observers.


0 comments on commit 3d105c6

Please sign in to comment.