Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Rollup candlesticks via @CandlestickColumn #20

Merged
merged 40 commits into from
Feb 12, 2025
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
40 commits
Select commit Hold shift + click to select a range
52cd626
feat: init rollup
danstarns Feb 6, 2025
a114b21
*
danstarns Feb 6, 2025
0da682c
feat: more rollup stuff
danstarns Feb 6, 2025
59fa994
test: fix *
danstarns Feb 7, 2025
26243fe
*
danstarns Feb 7, 2025
4e7655f
fix: *
danstarns Feb 10, 2025
0f1f8a4
change rollup column name
danstarns Feb 10, 2025
e570a7b
docs: *
danstarns Feb 10, 2025
b062dc1
test: add timeout
danstarns Feb 10, 2025
4a022fa
feat: add coverage for bucket column enforcing
danstarns Feb 10, 2025
cd42f00
docs: *
danstarns Feb 10, 2025
def0000
feat: add rollup example to sequelize
danstarns Feb 10, 2025
aa7449b
feat: add checks for nested rollups
danstarns Feb 10, 2025
a83b482
docs: *
danstarns Feb 10, 2025
ec5fa97
feat: more rollup stuff
danstarns Feb 6, 2025
0df1532
feat: init candlestick rollup and candlestick columns
danstarns Feb 10, 2025
4d1d0c5
Merge branch 'feat/rollups' into candlestick-rollup
danstarns Feb 10, 2025
94cf153
dev: gitignore
danstarns Feb 10, 2025
b833052
refactor: *
danstarns Feb 10, 2025
897654d
refactor: use same aggregate type schema
danstarns Feb 10, 2025
6bb2e53
refactor: lowercase stick in type name
danstarns Feb 10, 2025
7c7d9ca
feat: add TimeColumn and timestampz
danstarns Feb 11, 2025
747c286
docs: *
danstarns Feb 11, 2025
610298a
docs: *
danstarns Feb 11, 2025
445afb6
docs: *
danstarns Feb 11, 2025
a2cc9ab
further candlestick
danstarns Feb 11, 2025
229f680
feat: further candlestick support on views
danstarns Feb 11, 2025
755cda9
*
danstarns Feb 11, 2025
ffb8310
feat: add candelstick query for rollups
danstarns Feb 11, 2025
6b780ac
test: throw
danstarns Feb 11, 2025
f76065e
more updates
danstarns Feb 11, 2025
d5ac560
up
danstarns Feb 11, 2025
92184c4
up timer in tests
danstarns Feb 11, 2025
3dfce53
docs: *
danstarns Feb 11, 2025
4e99e1b
ci: run typeorm first
danstarns Feb 12, 2025
36f9175
test: fix migrations always run
danstarns Feb 12, 2025
a18e6da
test: *
danstarns Feb 12, 2025
948ff3e
test: make less flaky
danstarns Feb 12, 2025
28b0520
test: fix more flaky
danstarns Feb 12, 2025
9566f89
refactor: remove comments
danstarns Feb 12, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 10 additions & 10 deletions .github/workflows/test.yml
Original file line number Diff line number Diff line change
Expand Up @@ -60,22 +60,22 @@ jobs:
- name: Run TypeORM Lib Tests
run: pnpm run --filter @timescaledb/typeorm test

- name: Run Sequelize Migration
env:
DATABASE_URL: postgres://postgres:password@localhost:5432/sequelize
run: pnpm run --filter @timescaledb/example-node-sequelize migrate

- name: Run TypeORM Migration
env:
DATABASE_URL: postgres://postgres:password@localhost:5432/typeorm
run: pnpm run --filter @timescaledb/example-node-typeorm migrate

- name: Run Sequelize Example Tests
env:
DATABASE_URL: postgres://postgres:password@localhost:5432/sequelize
run: pnpm run --filter @timescaledb/example-node-sequelize test

- name: Run TypeORM Example Tests
env:
DATABASE_URL: postgres://postgres:password@localhost:5432/typeorm
run: pnpm run --filter @timescaledb/example-node-typeorm test

- name: Run Sequelize Migration
env:
DATABASE_URL: postgres://postgres:password@localhost:5432/sequelize
run: pnpm run --filter @timescaledb/example-node-sequelize migrate

- name: Run Sequelize Example Tests
env:
DATABASE_URL: postgres://postgres:password@localhost:5432/sequelize
run: pnpm run --filter @timescaledb/example-node-sequelize test
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -5,3 +5,4 @@ package-lock.json
node_modules/
dist/
.env
repomix-output.txt
5 changes: 3 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@ If you are looking to setup this project locally, you can follow the instruction

- [Getting Started](./docs/guides/getting-started.md) - A guide to getting started with TimescaleDB and this library.
- [Working with Energy Data](./docs/guides/energy-data.md) - A guide to working with energy data in TimescaleDB.
- [Working with Candlesticks](./docs/guides/candlesticks.md) - A guide to working with candlestick data in TimescaleDB.

## Feature Compatibility

Expand Down Expand Up @@ -66,15 +67,15 @@ Then you can use the `@Hypertable` decorator to define your hypertables:

```diff
import { Entity, PrimaryColumn } from 'typeorm';
+ import { Hypertable } from '@timescaledb/typeorm';
+ import { Hypertable, TimeColumn } from '@timescaledb/typeorm';

+ @Hypertable({ ... })
@Entity('page_loads')
export class PageLoad {
@PrimaryColumn({ type: 'varchar' })
user_agent!: string;

@PrimaryColumn({ type: 'timestamp' })
+ @TimeColumn()
time!: Date;
}
```
Expand Down
246 changes: 246 additions & 0 deletions docs/guides/candlesticks.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,246 @@
# Candlesticks and Rollups with TimescaleDB and TypeORM

## Introduction

Candlesticks are a powerful way to analyze time-series data, particularly in financial applications. TimescaleDB provides advanced functionality for generating candlestick data, and with TypeORM, we can easily create and query these aggregations.

## Prerequisites

- Node.js >= 22.13.0
- TypeORM
- @timescaledb/typeorm package
- PostgreSQL with TimescaleDB extension

## Setting Up a Stock Price Entity

Let's create a stock price entity that will serve as our base for candlestick and rollup operations:

```typescript
import { Entity, Column } from 'typeorm';
import { Hypertable, TimeColumn } from '@timescaledb/typeorm';

@Entity('stock_prices')
@Hypertable({
compression: {
compress: true,
compress_orderby: 'timestamp',
compress_segmentby: 'symbol',
policy: {
schedule_interval: '7 days',
},
},
})
export class StockPrice {
@PrimaryColumn({ type: 'varchar' })
symbol!: string;

@TimeColumn()
timestamp!: Date;

@Column({ type: 'decimal', precision: 10, scale: 2 })
price!: number;

@Column({ type: 'decimal', precision: 10, scale: 2 })
volume!: number;
}
```

## Generating 1-Minute Candlesticks

First, let's create a continuous aggregate for 1-minute candlesticks:

```typescript
import { ContinuousAggregate, BucketColumn, CandlestickColumn } from '@timescaledb/typeorm';
import { StockPrice } from './StockPrice';
import { Candlestick } from '@timescaledb/schemas';

@ContinuousAggregate(StockPrice, {
name: 'stock_candlesticks_1m',
bucket_interval: '1 minute',
refresh_policy: {
start_offset: '1 day',
end_offset: '1 minute',
schedule_interval: '1 minute',
},
})
export class StockPrice1M {
@BucketColumn({
source_column: 'timestamp',
})
bucket!: Date;

@PrimaryColumn()
symbol!: string;

@CandlestickColumn({
time_column: 'timestamp',
price_column: 'price',
volume_column: 'volume',
})
candlestick!: Candlestick;
}
```

## Creating 1-Hour Rollups

Now, let's create a rollup that aggregates the 1-minute candlesticks into 1-hour candlesticks:

```typescript
import { Rollup, BucketColumn, CandlestickColumn } from '@timescaledb/typeorm';
import { StockPrice1M } from './StockPrice1M';
import { Candlestick } from '@timescaledb/schemas';

@Rollup(StockPrice1M, {
name: 'stock_candlesticks_1h',
bucket_interval: '1 hour',
refresh_policy: {
start_offset: '7 days',
end_offset: '1 hour',
schedule_interval: '1 hour',
},
})
export class StockPrice1H {
@BucketColumn({
source_column: 'bucket',
})
bucket!: Date;

@PrimaryColumn()
symbol!: string;

@CandlestickColumn({
source_column: 'candlestick',
})
candlestick!: Candlestick;
}
```

## Querying Candlesticks

### 1-Minute Candlesticks

```typescript
import { AppDataSource } from './data-source';
import { StockPrice1M } from './models/StockPrice1M';

async function get1MinuteCandlesticks() {
const repository = AppDataSource.getRepository(StockPrice1M);

const candlesticks = await repository
.createQueryBuilder()
.where('bucket >= :start', { start: new Date('2025-01-01') })
.andWhere('bucket < :end', { end: new Date('2025-01-02') })
.andWhere('symbol = :symbol', { symbol: 'AAPL' })
.orderBy('bucket', 'ASC')
.getMany();

console.log(JSON.stringify(candlesticks, null, 2));
// Example output:
// [
// {
// "bucket": "2025-01-01T00:00:00.000Z",
// "symbol": "AAPL",
// "candlestick": {
// "open": 150.25,
// "high": 152.30,
// "low": 149.80,
// "close": 151.45,
// "volume": 1250000,
// "open_time": "2025-01-01T00:00:15.000Z",
// "close_time": "2025-01-01T00:59:45.000Z"
// }
// },
// ...
// ]
}
```

### 1-Hour Rollup Candlesticks

```typescript
import { AppDataSource } from './data-source';
import { StockPrice1H } from './models/StockPrice1H';

async function get1HourCandlesticks() {
const repository = AppDataSource.getRepository(StockPrice1H);

const candlesticks = await repository
.createQueryBuilder()
.where('bucket >= :start', { start: new Date('2025-01-01') })
.andWhere('bucket < :end', { end: new Date('2025-02-01') })
.andWhere('symbol = :symbol', { symbol: 'AAPL' })
.orderBy('bucket', 'ASC')
.getMany();

console.log(JSON.stringify(candlesticks, null, 2));
// Example output:
// [
// {
// "bucket": "2025-01-01T00:00:00.000Z",
// "symbol": "AAPL",
// "candlestick": {
// "open": 150.25,
// "high": 155.60,
// "low": 149.50,
// "close": 153.20,
// "volume": 8750000,
// "open_time": "2025-01-01T00:00:15.000Z",
// "close_time": "2025-01-01T00:59:45.000Z"
// }
// },
// ...
// ]
}
```

### Using Repository Method for Candlesticks

You can also use the repository's `getCandlesticks` method directly on the base `StockPrice` entity:

```typescript
import { AppDataSource } from './data-source';
import { StockPrice } from './models/StockPrice';

async function getCandlesticksDirectly() {
const repository = AppDataSource.getRepository(StockPrice);

const candlesticks = await repository.getCandlesticks({
timeRange: {
start: new Date('2025-01-01'),
end: new Date('2025-01-02'),
},
config: {
price_column: 'price',
volume_column: 'volume',
bucket_interval: '1 hour',
},
where: {
symbol: 'AAPL',
},
});

console.log(JSON.stringify(candlesticks, null, 2));
// Example output:
// [
// {
// "bucket_time": "2025-01-01T00:00:00.000Z",
// "open": 150.25,
// "high": 155.60,
// "low": 149.50,
// "close": 153.20,
// "volume": 8750000,
// "vwap": 152.75,
// "open_time": "2025-01-01T00:00:15.000Z",
// "close_time": "2025-01-01T00:59:45.000Z"
// },
// ...
// ]
}
```

## Example Use Cases

- Stock price analysis
- Cryptocurrency trading
- IoT sensor data aggregation
- Performance metrics tracking
7 changes: 2 additions & 5 deletions docs/guides/energy-data.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,13 +8,10 @@ First, let's set up our energy metrics model:

```typescript
import { Entity, PrimaryColumn, Column } from 'typeorm';
import { Hypertable } from '@timescaledb/typeorm';
import { Hypertable, TimeColumn } from '@timescaledb/typeorm';

@Entity('energy_metrics')
@Hypertable({
by_range: {
column_name: 'timestamp',
},
compression: {
compress: true,
compress_orderby: 'timestamp',
Expand All @@ -28,7 +25,7 @@ export class EnergyMetric {
@PrimaryColumn({ type: 'varchar' })
meter_id!: string;

@PrimaryColumn({ type: 'timestamp' })
@TimeColumn()
timestamp!: Date;

@Column({ type: 'float' })
Expand Down
8 changes: 2 additions & 6 deletions docs/guides/getting-started.md
Original file line number Diff line number Diff line change
Expand Up @@ -96,13 +96,10 @@ Create `src/models/CryptoPrice.ts`:

```typescript
import { Entity, PrimaryColumn, Column } from 'typeorm';
import { Hypertable } from '@timescaledb/typeorm';
import { Hypertable, TimeColumn } from '@timescaledb/typeorm';

@Entity('crypto_prices')
@Hypertable({
by_range: {
column_name: 'timestamp',
},
compression: {
compress: true,
compress_orderby: 'timestamp',
Expand All @@ -116,7 +113,7 @@ export class CryptoPrice {
@PrimaryColumn({ type: 'varchar' })
symbol!: string;

@PrimaryColumn({ type: 'timestamp' })
@TimeColumn()
timestamp!: Date;

@Column({ type: 'decimal', precision: 18, scale: 8 })
Expand Down Expand Up @@ -190,7 +187,6 @@ async function analyzeBTC() {
end: new Date('2025-01-02T00:00:00Z'),
},
config: {
time_column: 'timestamp',
price_column: 'price',
volume_column: 'volume',
bucket_interval: '1 hour',
Expand Down
2 changes: 1 addition & 1 deletion examples/node-sequelize/tests/daily.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -56,7 +56,7 @@ describe('GET /api/daily', () => {
});

expect(response.status).toBe(200);
expect(response.body).toHaveLength(3);
expect(response.body.length).toBeCloseTo(3);

const firstDay = response.body[0];
expect(firstDay).toHaveProperty('bucket');
Expand Down
Loading