Spec Core Microservices

Welcome to Spec's core microservices :)
https://spec.dev

Requirements

• node >= 16
• npm >= 8
• postgres >= 14
• redis >= 6
• docker
• Rust
• Deno >= 1.3 (+recommend the Deno/Denoland VSCode extension)

Helpful Links

Installing Node.js with nvm on Mac:
https://collabnix.com/how-to-install-and-configure-nvm-on-mac-os/

Installing Postgres with brew:
https://gist.github.com/ibraheem4/ce5ccd3e4d7a65589ce84f2a3b7c23a3

Installing Redis:
https://redis.io/docs/getting-started/installation/install-redis-on-mac-os/

Installing Docker:
https://docs.docker.com/desktop/install/mac-install/

Installing Rust with rustup:

$ curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh

Installing Deno:
https://deno.com/[email protected]/getting_started/installation)

Data Pipeline

The backbone of Spec is its indexing pipeline, which ingests data from a variety of blockchains, uses that data to curate data higher-level data models, and then publishes updates downstream to customers' databases.

Components

• Head Reporter
• Indexer
• Event Sorter
• Event Generator
• Event Relay
• Tables API
• Gap Detector
• Live Object Deno Entrypoint
• Shared Tables
• Indexer DB
• Indexer Redis
• Spec Client

Full Stack App

The infrastructure that end-users interact with follows more of the classic Client -> Server -> Database pattern and resembles the following:

Components

• CLI
• Desktop App
• Core API
• Delayed Jobs
• Log Relay
• Indexer Redis
• ABI Redis
• Core Redis
• Core DB
• Shared Tables
• Spec Client

Storage Infrastructure

Spec has 3 postgres databases and 3 redis instances that all work together to power its indexing pipeline and infrastructure for data delivery. All Postgres databases are hosted as RDS instances on AWS, and all Redis instances are hosted as Elasticache instances on AWS.

Indexer DB

The Indexer database keeps track of index-block jobs as well as any chain reorgs that occur.
[RDS]

• indexed_blocks - A block indexed by Spec.
• reorgs - A chain reorg.

Local Setup

Create spec user if you haven't already:

$ createuser spec

Create the Indexer database:

$ createdb indexer -O spec

Run migrations:

$ cd shared
$ npm install
$ bin/migrate indexer

Shared Tables DB

The Shared Tables database stores all blockchain data. This includes all chain-specific primitives (blocks, transactions, logs, etc.), all cross-chain token data (tokens, balances, etc.), and all Live Object tables. Every Live Object on Spec exists under a specific namespace, and each namespace has its own corresponding schema within the Shared Tables DB. For example, a Live Object for the Uniswap protocol would exist under the uniswap namespace and would have its own corresponding table in the uniswap schema.
[RDS]

Local Setup

Create spec user if you haven't already:

$ createuser spec

Create the Shared Tables database:

$ createdb shared-tables -O spec

Run migrations:

$ cd shared
$ chmod u+x bin/*
$ bin/init-shared-tables

Core DB

The Core database stores all users, namespaces, projects, Live Objects, events, contracts, and other ecosystem data.
[RDS]

• users - A user on Spec.
• sessions - An authed user session.
• namespaces - A globally unique namespace serving as an umbrella to other resources on Spec.
• namespace_users - A user that belongs to a particular namespace, with associated permissions.
• projects - A customer project on Spec.
• project_roles - A way of specifying owners, admins, and members of a project.
• contracts - A group of smart contracts that all share the same ABI.
• contract_instances - A smart contract deployed to a specific chain/address.
• events - An event on Spec that represents something that happened on-chain.
• event_versions - Version control for events.
• live_objects - A data model representing some live data on-chain.
• live_object_versions - Version control for Live Objects.
• live_event_versions - A join-table specifying which event versions are associated with which live object versions, either as inputs or outputs.
• live_call_handlers - A smart contract function whose handler is used as an input to a live object version.

Local Setup

Create spec user if you haven't already:

$ createuser spec

Create the Core database:

$ createdb core -O spec

Run migrations:

$ cd shared
$ npm install
$ npm run build
$ bin/migrate core

Indexer Redis

The Indexer Redis instance is primarily used for communicating between microservices in the data pipeline, leveraging redis queues, streams, and hashes, and more.
[Elasticache]

Core Redis

The Core Redis instance is primarily used for storing logs sent to it from the various Spec clients (customers running Spec). These logs are stored in Redis streams, which can then be easily pulled down and tailed from the CLI when requested.
[Elasticache]

ABI Redis

The ABI Redis instance is in charge of mapping smart contracts to their associated ABIs.
[Elasticache]

In practice, this looks something like:

{
    "eth-contracts": {
        "0x123.": "<abi>",
        "0x456.": "<abi>"
    },
    "polygon-contracts": {
        "0x789": "<abi>",
        ...
    }
}

Local Development (CLI -> Core API)

Follow the steps below to get up and running with a local version of the CLI, Core API, Delayed Jobs worker, and the databases they interact with.

CLI

Prerequisites:

• Install Deno
• Local Postgres installation

1) Clone the CLI

$ git clone https://github.com/spec-dev/cli && cd cli

2) Install dependencies

$ npm install
$ npm install -g @spec.dev/spec

3) Create a local installation script

$ mkdir bin
$ touch bin/install
$ chmod u+x bin/*

4) Add the following contents to bin/install

#!/bin/bash

npm run build
mkdir ./dist/files
cp -r ./src/files ./dist
npm install -g . --force
chmod u+x /usr/local/bin/spec

5) Run your installation script

$ bin/install

You'll want to re-run this anytime you make changes and want to test them out.

6) Set environment variables to point CLI -> local Core API

export SPEC_API_ORIGIN=http://localhost:7777

Core API

Follow setup instructions here

Delayed Jobs

Follow setup instructions here

License

Copyright (c) 2023 Spec Development Inc, All rights reserved.