Data IO¶
Read and save candles, ticks and order books in NumPy .npz (the base binary
format), CSV, Parquet and HDF5. Parquet needs tradetropy[parquet] and HDF5 needs
tradetropy[hdf5]; .npz and CSV need no extra.
io ¶
read_ticks ¶
read_ticks(path: str | Path, symbol: 'str | None' = None, format: 'Literal["csv", "parquet", "hdf5", "npz"] | None' = None, source: str = 'auto', hdf5_key: str = 'data', col_datetime: str = 'datetime', tick_size: float = 0.01, tick_value: float = 0.01, contract_size: float = 1.0, digits: int = 2, avg_spread: float = 0.0, volume_min: float = 0.01, volume_max: float = 100.0, volume_step: float = 0.01) -> 'TickData'
Read ticks from file and return TickData.
Accepts external files (broker CSV) and files generated by save_ticks().
Required columns: datetime, bid, ask Optional columns: volume, flags, volume_real, price (filled by normalize_ticks() if missing)
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
path
|
str | Path
|
File path to read from. |
required |
symbol
|
'str | None'
|
Trading symbol. If None, read from HDF5 attrs (saved by TickData.save()). Required for CSV/Parquet or non-tradetropy files. |
None
|
format
|
'Literal["csv", "parquet", "hdf5", "npz"] | None'
|
File format ('csv', 'parquet', 'hdf5'). None -> auto-detect. |
None
|
source
|
str
|
Tick file schema. 'auto' (default) reads the generic
datetime/bid/ask/volume/flags/price schema; on failure it sniffs
for a known broker-specific schema and raises an actionable error
suggesting the right |
'auto'
|
hdf5_key
|
str
|
Key inside HDF5 file. |
'data'
|
col_datetime
|
str
|
Datetime column name. |
'datetime'
|
tick_size
|
float
|
Minimum price move. |
0.01
|
tick_value
|
float
|
Value of one tick. |
0.01
|
contract_size
|
float
|
Size of contract. |
1.0
|
digits
|
int
|
Number of decimal places. |
2
|
avg_spread
|
float
|
Average bid-ask spread. |
0.0
|
volume_min
|
float
|
Minimum order volume. |
0.01
|
volume_max
|
float
|
Maximum order volume. |
100.0
|
volume_step
|
float
|
Minimum volume increment (lot step). |
0.01
|
Returns:
| Name | Type | Description |
|---|---|---|
TickData |
'TickData'
|
Normalized tick data object. |
Raises:
| Type | Description |
|---|---|
DataError
|
If source='mt5' is used on a file that is not an MT5 tick export, or if the generic schema is missing required columns and no known broker-specific schema is detected. |
Example
ticks = read_ticks('ticks.parquet', 'BTCUSDT') ticks = read_ticks('session.h5') # symbol read from attrs ticks = read_ticks('broker_export.csv', 'BTCUSDT') ticks = read_ticks('MESU26_ticks.csv', 'MESU26', source='mt5')
Source code in src/tradetropy/io/_ticks.py
213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 | |
read_klines ¶
read_klines(path: str | Path, symbol: 'str | None' = None, timeframe: 'str | int | None' = None, format: 'Literal["csv", "parquet", "hdf5", "npz"] | None' = None, hdf5_key: str = 'data', col_datetime: str = 'datetime', tick_size: float = 0.01, tick_value: float = 0.01, contract_size: float = 1.0, digits: int = 2, avg_spread: float = 0.0, volume_min: float = 0.01, volume_max: float = 100.0, volume_step: float = 0.01) -> 'KlineData'
Read klines from file and return KlineData.
Accepts external files (broker CSV) and files generated by save_klines().
Required columns: datetime, open, high, low, close, volume Optional columns: turnover (filled with NaN if missing)
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
path
|
str | Path
|
File path to read from. |
required |
symbol
|
'str | None'
|
Trading symbol. If None, read from HDF5 attrs (saved by KlineData.save()). Required for CSV/Parquet or non-tradetropy files. |
None
|
timeframe
|
'str | int | None'
|
Candle interval. Accepts a timeframe string ('1m', '5m', '1h', '1d', etc.) or an integer number of milliseconds, parsed via parse_timeframe(). If None, read from HDF5 attrs. |
None
|
format
|
'Literal["csv", "parquet", "hdf5", "npz"] | None'
|
File format ('csv', 'parquet', 'hdf5'). None -> auto-detect. |
None
|
hdf5_key
|
str
|
Key inside HDF5 file. |
'data'
|
col_datetime
|
str
|
Datetime column name. |
'datetime'
|
tick_size
|
float
|
Minimum price move. |
0.01
|
tick_value
|
float
|
Value of one tick. |
0.01
|
contract_size
|
float
|
Size of contract. |
1.0
|
digits
|
int
|
Number of decimal places. |
2
|
volume_min
|
float
|
Minimum order volume. |
0.01
|
volume_max
|
float
|
Maximum order volume. |
100.0
|
volume_step
|
float
|
Minimum volume increment (lot step). |
0.01
|
Returns:
| Name | Type | Description |
|---|---|---|
KlineData |
'KlineData'
|
Kline data object. |
Example
klines = read_klines('ohlc.parquet', 'BTCUSDT', timeframe='1m') klines = read_klines('session.h5') # all read from attrs klines = read_klines('bybit_export.csv', 'BTCUSDT', timeframe=60_000)
Source code in src/tradetropy/io/_klines.py
69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 | |
read_book ¶
read_book(path: 'str | Path', symbol: 'str | None' = None, levels: 'int | None' = None, format: 'Literal["csv", "parquet", "hdf5", "npz"] | None' = None, layout: Literal['auto', 'wide', 'long'] = 'auto', hdf5_key: str = 'book', col_datetime: str = 'datetime', tick_size: float = 0.01) -> BookData
Read L2 order-book rows and return a BookData (multi-format, multi-layout).
Round-trips files written by save_book() / BookData.save() in any format (csv, parquet, hdf5) and the append-only HDF5 written by the live record= path (_append_book_hdf5). Two on-disk layouts are supported and normalized to the same wide BookData in memory:
- 'wide' (tradetropy native): grouped bid_px_/bid_sz_/ask_px_/ask_sz_ columns, one book event per row.
- 'long' (Binance bookDepth-style): 'timestamp'/'percentage'/'depth'/ 'notional' columns, one row per price level per side, with cumulative depth/notional. De-cumulated back to per-level price/size on read (exact for tradetropy-written files, approximate per-band VWAP for a real Binance bookDepth file, which carries no explicit per-level price).
The 'ts' column is accepted directly, or rebuilt from a 'datetime' / 'timestamp' column. For the wide layout the level count K is taken from the argument, from HDF5 metadata, or inferred from the bid_px_* columns; for the long layout it is inferred from the largest |percentage| present.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
path
|
'str | Path'
|
File path (as written by save_book / BookData.save / record=). |
required |
symbol
|
'str | None'
|
Trading symbol. If None, read from HDF5 attrs when available. |
None
|
levels
|
'int | None'
|
Number of book levels K (wide only). If None, from metadata or inferred. |
None
|
format
|
'Literal["csv", "parquet", "hdf5", "npz"] | None'
|
File format ('csv', 'parquet', 'hdf5'). None -> auto-detect. |
None
|
layout
|
Literal['auto', 'wide', 'long']
|
On-disk layout ('auto', 'wide', 'long'). 'auto' detects from the columns (long if percentage/depth/notional present, else wide). |
'auto'
|
hdf5_key
|
str
|
Key inside the HDF5 file (default 'book'). |
'book'
|
col_datetime
|
str
|
Datetime column name (when no 'ts' column is present). |
'datetime'
|
tick_size
|
float
|
Minimum price step propagated to BookData. |
0.01
|
Returns:
| Name | Type | Description |
|---|---|---|
BookData |
BookData
|
Reconstructed order-book data (always the wide in-memory form). |
Raises:
| Type | Description |
|---|---|
DataError
|
If the level count cannot be determined. |
ValueError
|
If the symbol cannot be resolved. |
Source code in src/tradetropy/io/_book.py
226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 | |
read_mbo ¶
read_mbo(path: 'str | Path', symbol: 'str | None' = None, hdf5_key: str = 'mbo', tick_size: float = 0.01)
Read recorded L3 / MBO event rows and return an MboData.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
path
|
'str | Path'
|
HDF5 file path (as written by _append_mbo_hdf5). |
required |
symbol
|
'str | None'
|
Trading symbol. If None, read from HDF5 attrs (persisted by the live record= path). Required when the file carries no attrs. |
None
|
hdf5_key
|
str
|
Key inside the HDF5 file (default 'mbo'). |
'mbo'
|
tick_size
|
float
|
Minimum price step propagated to MboData. |
0.01
|
Returns:
| Name | Type | Description |
|---|---|---|
MboData |
Reconstructed market-by-order data. |
Raises:
| Type | Description |
|---|---|
ValueError
|
If the symbol cannot be resolved from arg or attrs. |
Source code in src/tradetropy/io/_mbo.py
read_trades ¶
read_trades(path: str | Path, symbol: str, source: str = 'binance', format: 'Literal["csv", "parquet", "hdf5", "npz"] | None' = None, hdf5_key: str = 'data', tick_size: float = 0.01, tick_value: float = 0.01, contract_size: float = 1.0, digits: int = 2, avg_spread: float = 0.0, volume_min: float = 0.01, volume_max: float = 100.0, volume_step: float = 0.01) -> 'TickData'
Read raw exchange trades (times and sales) and return TickData.
Normalizes a venue-specific trades export into the tradetropy tick array, filling bid/ask from the trade price (a trades file carries no quotes) and mapping the aggressor side onto the 'flags' column (+1 buy / -1 sell). The supported schemas live in the _TRADE_SCHEMAS registry, keyed by 'source'; 'binance' handles the columns id, price, qty, quote_qty, time, is_buyer_maker (the id and quote_qty columns are ignored).
The returned TickData is a standard object, so it is saved to any format with its usual .save() (parquet / hdf5 / csv) and re-read with read_ticks().
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
path
|
str | Path
|
File path to read from. |
required |
symbol
|
str
|
Trading symbol (required; a trades file has no tradetropy attrs). |
required |
source
|
str
|
Trades schema key. One of the keys in _TRADE_SCHEMAS. |
'binance'
|
format
|
'Literal["csv", "parquet", "hdf5", "npz"] | None'
|
File format ('csv', 'parquet', 'hdf5'). None -> auto-detect. |
None
|
hdf5_key
|
str
|
Key inside the HDF5 file. |
'data'
|
tick_size
|
float
|
Minimum price move. |
0.01
|
tick_value
|
float
|
Value of one tick. |
0.01
|
contract_size
|
float
|
Size of contract. |
1.0
|
digits
|
int
|
Number of decimal places. |
2
|
avg_spread
|
float
|
Average bid-ask spread. |
0.0
|
volume_min
|
float
|
Minimum order volume. |
0.01
|
volume_max
|
float
|
Maximum order volume. |
100.0
|
volume_step
|
float
|
Minimum volume increment (lot step). |
0.01
|
Returns:
| Name | Type | Description |
|---|---|---|
TickData |
'TickData'
|
Normalized tick data object. |
Raises:
| Type | Description |
|---|---|
DataError
|
If the source is unknown or a required column is missing. |
Example
ticks = read_trades('ADAUSDT-trades.csv', 'ADAUSD', source='binance') ticks.save('ada_ticks.parquet')
Source code in src/tradetropy/io/_ticks.py
409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 | |
read_klines_csv ¶
Compatibility alias for read_klines(format='csv'). Use read_klines() for new code.
read_ticks_csv ¶
Compatibility alias for read_ticks(format='csv'). Use read_ticks() for new code.
save_ticks ¶
save_ticks(data: 'TickProxy | np.ndarray | pd.DataFrame', path: str | Path, format: 'Literal["csv", "parquet", "hdf5", "npz"] | None' = None, ts_format: Literal['iso', 'ms', 's'] = 'iso', hdf5_key: str = 'data', compression: str = 'snappy', metadata: 'dict | None' = None) -> pd.DataFrame
Save ticks to file.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
data
|
'TickProxy | np.ndarray | pd.DataFrame'
|
TickProxy, ndarray [N×7] or DataFrame with standard tick columns. |
required |
path
|
str | Path
|
Destination file path. |
required |
format
|
'Literal["csv", "parquet", "hdf5", "npz"] | None'
|
Output format ('csv', 'parquet', 'hdf5', 'npz'). None (default) infers from the extension, falling back to 'npz'. |
None
|
ts_format
|
Literal['iso', 'ms', 's']
|
Timestamp format for CSV ('iso', 'ms', 's'). |
'iso'
|
hdf5_key
|
str
|
Key inside the HDF5 file. |
'data'
|
compression
|
str
|
Parquet compression method ('snappy', 'gzip', 'zstd', None). |
'snappy'
|
metadata
|
'dict | None'
|
Optional dict of tradetropy metadata to store as HDF5 attrs. |
None
|
Returns:
| Type | Description |
|---|---|
DataFrame
|
pd.DataFrame: The saved tick data as DataFrame. |
Example
save_ticks(self.btc_tick, 'ticks.npz') save_ticks(self.btc_tick, 'ticks.parquet', format='parquet') save_ticks(self.btc_tick, 'session.h5', format='hdf5', hdf5_key='btc/ticks') save_ticks(my_array, 'ticks.csv', format='csv')
Source code in src/tradetropy/io/_ticks.py
save_klines ¶
save_klines(data: 'OhlcProxy | np.ndarray | pd.DataFrame', path: str | Path, format: 'Literal["csv", "parquet", "hdf5", "npz"] | None' = None, include_partial: bool = True, ts_format: Literal['iso', 'ms', 's'] = 'iso', hdf5_key: str = 'data', compression: str = 'snappy', metadata: 'dict | None' = None) -> pd.DataFrame
Save klines to file.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
data
|
'OhlcProxy | np.ndarray | pd.DataFrame'
|
OhlcProxy, ndarray [N×6] or DataFrame with standard OHLC columns. |
required |
path
|
str | Path
|
Destination file path. |
required |
format
|
'Literal["csv", "parquet", "hdf5", "npz"] | None'
|
Output format ('csv', 'parquet', 'hdf5', 'npz'). None (default) infers from the extension, falling back to 'npz'. |
None
|
include_partial
|
bool
|
Include partial candle (partial=1) in OhlcProxy. |
True
|
ts_format
|
Literal['iso', 'ms', 's']
|
Timestamp format for CSV ('iso', 'ms', 's'). |
'iso'
|
hdf5_key
|
str
|
Key inside the HDF5 file. |
'data'
|
compression
|
str
|
Parquet compression method ('snappy', 'gzip', 'zstd', None). |
'snappy'
|
metadata
|
'dict | None'
|
Optional dict of tradetropy metadata to store as HDF5 attrs. |
None
|
Returns:
| Type | Description |
|---|---|
DataFrame
|
pd.DataFrame: The saved kline data as DataFrame. |
Example
save_klines(self.btc_1m, 'ohlc.npz') save_klines(self.btc_1m, 'ohlc.parquet', format='parquet') save_klines(self.btc_1m, 'session.h5', format='hdf5', hdf5_key='btc/1m')
Source code in src/tradetropy/io/_klines.py
save_book ¶
save_book(data, path: str | Path, format: 'Literal["csv", "parquet", "hdf5", "npz"] | None' = None, *, levels: 'int | None' = None, layout: Literal['auto', 'wide', 'long'] = 'wide', ts_format: Literal['iso', 'ms', 's'] = 'iso', hdf5_key: str = 'book', compression: str = 'snappy', metadata: 'dict | None' = None) -> pd.DataFrame
Save L2 order-book rows to file (symmetric to save_ticks / save_klines).
Two on-disk layouts are supported (both round-trip through read_book):
- layout='wide' (default, tradetropy native): grouped bid_px_/bid_sz_/ ask_px_/ask_sz_ columns, one book event per row.
- layout='long' (Binance bookDepth-style): 'ts'/'percentage'/'depth'/ 'notional' columns, one row per price level per side with CUMULATIVE depth/notional (percentage = signed 1-based level index; negative = bid, positive = ask). This is a lossless re-encoding of the wide data that de-cumulates back exactly on read.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
data
|
BookData, ndarray [N x (2+4*levels)] or DataFrame with book cols. |
required | |
path
|
str | Path
|
Destination file path. |
required |
format
|
'Literal["csv", "parquet", "hdf5", "npz"] | None'
|
Output format ('csv', 'parquet', 'hdf5'). |
None
|
levels
|
'int | None'
|
Book levels K (required for a raw ndarray; taken from BookData). |
None
|
layout
|
Literal['auto', 'wide', 'long']
|
On-disk layout ('wide' or 'long'). |
'wide'
|
ts_format
|
Literal['iso', 'ms', 's']
|
Timestamp format for CSV ('iso', 'ms', 's'). |
'iso'
|
hdf5_key
|
str
|
Key inside the HDF5 file (default 'book'). |
'book'
|
compression
|
str
|
Parquet compression method ('snappy', 'gzip', 'zstd', None). |
'snappy'
|
metadata
|
'dict | None'
|
Optional dict of tradetropy metadata to store as HDF5 attrs (merged over the defaults derived from a BookData). |
None
|
Returns:
| Type | Description |
|---|---|
DataFrame
|
pd.DataFrame: The saved book data as DataFrame (in the chosen layout). |
Example
book.save('book.h5') # via BookData.save() save_book(book, 'book.parquet', format='parquet') save_book(arr, 'book.csv', format='csv', levels=20) save_book(book, 'bookdepth.csv', layout='long')
Source code in src/tradetropy/io/_book.py
save_proxy ¶
Save TickProxy or OhlcProxy to file.
Automatically dispatches to save_ticks() or save_klines() based on proxy type.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
proxy
|
TickProxy or OhlcProxy instance. |
required | |
path
|
Destination file path. |
required | |
format
|
Output format ('csv', 'parquet', 'hdf5'). |
'parquet'
|
|
**kwargs
|
Additional arguments passed to save_ticks() or save_klines(). |
{}
|
Raises:
| Type | Description |
|---|---|
TradingError
|
If proxy type is not TickProxy or OhlcProxy. |
Source code in src/tradetropy/io/_compat.py
convert_book ¶
convert_book(src: 'str | Path', dst: 'str | Path', *, symbol: 'str | None' = None, to_layout: Literal['wide', 'long'] = 'wide', src_layout: Literal['auto', 'wide', 'long'] = 'auto', src_format: 'Literal["csv", "parquet", "hdf5", "npz"] | None' = None, dst_format: 'Literal["csv", "parquet", "hdf5", "npz"] | None' = None, levels: 'int | None' = None, tick_size: float = 0.01, **save_kwargs)
Convert an order-book file between layouts and/or container formats.
Reads src (any supported layout/format) into a BookData - which is always
the wide in-memory form - and writes it to dst in to_layout. Layout
and container format are auto-detected from the file contents/extension
unless overridden. This is thin sugar over read_book() + save_book().
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
src
|
'str | Path'
|
Source order-book file path. |
required |
dst
|
'str | Path'
|
Destination file path. |
required |
symbol
|
'str | None'
|
Trading symbol (required if src carries no HDF5 metadata). |
None
|
to_layout
|
Literal['wide', 'long']
|
Output layout ('wide' or 'long'). |
'wide'
|
src_layout
|
Literal['auto', 'wide', 'long']
|
Input layout ('auto', 'wide', 'long'). |
'auto'
|
src_format
|
'Literal["csv", "parquet", "hdf5", "npz"] | None'
|
Source format ('csv'/'parquet'/'hdf5'); None -> auto-detect. |
None
|
dst_format
|
'Literal["csv", "parquet", "hdf5", "npz"] | None'
|
Destination format; None -> auto-detect from dst extension. |
None
|
levels
|
'int | None'
|
Book levels K for the source (wide only; inferred otherwise). |
None
|
tick_size
|
float
|
Minimum price step propagated to the intermediate BookData. |
0.01
|
**save_kwargs
|
Extra arguments forwarded to save_book (ts_format, hdf5_key, compression, metadata). |
{}
|
Returns:
| Name | Type | Description |
|---|---|---|
BookData |
The intermediate (wide) BookData, for convenience. |
Example
convert_book('btc_bookdepth.csv', 'btc_book.h5', symbol='BTCUSDT') convert_book('btc_book.h5', 'btc_bookdepth.csv', to_layout='long')
Source code in src/tradetropy/io/_book.py
ticks_to_file ¶
klines_to_file ¶
book_to_file ¶
klines_from_file ¶
Compatibility alias for read_klines(). Use read_klines() for new code.