This project moved to https://github.com/nkaz001/hftbacktest/tree/master/collector

Binance feed collector

Requirements

Python3: run by python3 command.
aiohttp: pip3 install aiohttp

Run

collect.sh [exchange] [symbols separated by comma.] [output path]
example: collect.sh binancefutures btcusdt,ethusdt /mnt/data

Exchanges

• binance: binance spot
• binancefutures: binance usd(s)-m futures
• binancefuturescoin: binance coin-m futures example: collect.sh binancefuturescoin btcusd_perp /mnt/data

AWS tokyo region is recommended to minimize latency.

Converter: feed data to Pandas Dataframe pickle file

Requirements

Python3: run by python3 command.
pandas: pip3 install pandas

Run

`convert.sh [-s INITIAL_MARKET_DEPTH_SNAPSHOT] [-f] [-c] src_file dst_path

option:
with -f: including mark price, funding, book ticker streams
without -f: only market depth and trade streams
with -c: correct exchange timestamp to be monotonically increased

example:
convert.sh /mnt/data/btcusdt_20220811.dat /mnt/data
or with the initial market depth snapshot
convert.sh /mnt/data/btcusdt_20220811.dat /mnt/data -s /mnt/data/btcusdt_20220810.snapshot.pkl

/mnt/data/btcusdt_20220810.snapshot.pkl is End-Of-Day market depth snapshot of 20220810 so it's initial market depth snapshot of 20220811.