11. Timeserie tools

The toolbox

tse_timeserie.h exports the engine's own CSV reader and writer as first-class handles, plus the timeserie utilities the engine uses internally — head/tail subranges, duration-based splitting, and the scalar transforms diff, lag and log. Everything operates over flat, caller-owned arrays: the same TseTickOHLCV, TseTickBidAsk and TseTickTrade arrays you feed into the market adapter, the TseRetained arrays you extract from results, the TseBookMessage arrays of a book serie, and parallel timestamp/value arrays for scalar series.

The toolbox lives next to the engine rather than in it for one reason: preparing data before a run and slicing data after it are the two chores that surround every backtest, and doing them with the engine's own parser and its own monetary arithmetic guarantees that what you inspect is exactly what the engine saw. The header is distinct from the specialized tick loaders of tse_dataplane.h (tse_load_ohlcv_csv, tse_load_ohlcv_csv_ex, tse_load_bidask_csv, tse_load_trade_csv): those parse straight into tick structs, while the reader here is a general-purpose handle for arbitrary columnar files. Both share the TseCsvOptions struct, which is declared in tse_dataplane.h.

The reader and the writer each carry a per-handle error text (tse_csv_reader_error, tse_csv_writer_error). The utility functions carry no handle at all and fail with a bare tse_error status, mirroring the silent refusals of the engine's own utilities. Any borrowed char const* returned by a handle stays valid until the next read on the same handle or its destruction.

The two-phase protocol

Every extraction call in this header follows the count-then-buffer protocol shared with the rest of the library. The call is made twice. On the first call the output buffer is NULL and the capacity is zero: the call touches nothing, and writes into the final out-parameter the number of elements the extraction would produce. The caller then obtains storage for that many elements, by whatever means its language provides, and calls a second time with the buffer and its capacity; this time the call fills the buffer and writes the number of elements actually written. Nothing is allocated by the library and nothing has to be released to it: every buffer the engine writes into is the caller's own, for its whole lifetime. The two wrappers perform both calls internally and hand back a container, so the protocol is visible only at the C surface — the CSV reader section below shows it end to end on a real column.

Two refinements apply throughout:

• On a short buffer a truncated count is written. Matrix extraction truncates whole rows only, so the written count stays a multiple of the requested column count.
• Timestamps at the boundary are always int64_t nanoseconds since the epoch, in every direction: the reader's index columns come out as nanoseconds, the writer accepts nanoseconds, and the splitting functions take their fromNs and stepNs arguments in nanoseconds.

The cell accessors are the one variation: cells are stored unterminated inside the reader, so the text is copied out. A NULL output stores the cell length (without terminator) into length; otherwise up to capacity - 1 bytes plus a NUL terminator are written and length receives the copied byte count.

CSV reader

A reader is created from TseCsvOptions and destroyed explicitly. Passing NULL options falls back to the engine reader's own defaults: dateFormat "%m/%d/%Y %H:%M", separator ',', no header row, parse errors not ignored, engine-chosen buffer size. Of the options struct, only dateFormat, separator, hasHeader, ignoreParseErrors and bufferSize (zero means the engine default) matter here — the role-column fields exist for the tick loaders of tse_dataplane.h and play no part in the general reader.

tse_csv_reader_read_file parses a file from disk; tse_csv_reader_read_buffer parses an in-memory buffer through the same machinery. After a successful read the shape is available through tse_csv_reader_column_count and tse_csv_reader_row_count, and — when the file has a header row — tse_csv_reader_header_count and tse_csv_reader_header_name enumerate the header names.

Column addressing comes in two families: by header name, which requires a header row, and by zero-based column index, which works either way. Each extraction exists in both forms:

• tse_csv_reader_read_index_by_name / tse_csv_reader_read_index_by_index — parse a column as timestamps using the configured dateFormat and return int64_t nanoseconds.
• tse_csv_reader_read_doubles_by_name / tse_csv_reader_read_doubles_by_index — parse a column as doubles.
• tse_csv_reader_read_matrix_by_name / tse_csv_reader_read_matrix_by_index — extract several columns at once as a row-major flat double matrix; capacity and the written count are in double elements, and truncation drops whole rows only.
• tse_csv_reader_cell_by_name / tse_csv_reader_cell_by_index — copy out the raw text of a single cell, with the copy semantics described above.

The reader itself extracts by column; slicing an extracted serie by time window is the job of the splitting utilities below, which return index ranges over the arrays you have already pulled out.

TseCsvReaderHandle tse_csv_reader_create(TseCsvOptions const* options);
void tse_csv_reader_destroy(TseCsvReaderHandle reader);
char const* tse_csv_reader_error(TseCsvReaderHandle reader);
TseStatus tse_csv_reader_read_file(TseCsvReaderHandle reader, char const* filePath);
TseStatus tse_csv_reader_read_buffer(TseCsvReaderHandle reader, char const* data, size_t length);

TseStatus tse_csv_reader_read_index_by_name(TseCsvReaderHandle reader, char const* column, int64_t* out, size_t capacity, size_t* written);
TseStatus tse_csv_reader_read_matrix_by_index(TseCsvReaderHandle reader, size_t const* columns, size_t columnCount, double* out, size_t capacity, size_t* written);
TseStatus tse_csv_reader_cell_by_name(TseCsvReaderHandle reader, char const* column, size_t row, char* out, size_t capacity, size_t* length);

Put together, one column comes out of a file like this — the first call sizes the array, the second fills it, and both the reader and the storage are released by the caller that made them.

TseCsvOptions options = {0};
options.structSize = sizeof(TseCsvOptions);
options.separator = ',';
options.hasHeader = 1;

TseCsvReaderHandle reader = tse_csv_reader_create(&options);
tse_csv_reader_read_file(reader, "prices.csv");

size_t needed = 0;
tse_csv_reader_read_doubles_by_name(reader, "close", NULL, 0, &needed);

double* closes = calloc(needed, sizeof(double));
size_t written = 0;
tse_csv_reader_read_doubles_by_name(reader, "close", closes, needed, &written);

free(closes);
tse_csv_reader_destroy(reader);

tse::CsvReader reader {tse::CsvOptions {}};
reader.readFile("prices.csv");
std::vector<double> const closes {reader.readDoubles("close")};

reader = tse.CsvReader(has_header=True)
reader.read_file("prices.csv")
closes = reader.read_doubles_by_name("close")

CSV writer

A writer is created from TseCsvWriterOptions and writes one file per call. NULL options mean separator ',' and raw nanosecond timestamps.

tse_csv_writer_write_bidask and tse_csv_writer_write_trade complete the family with the same shape as the OHLCV form. A NULL headers (or zero headerCount) writes no header row; otherwise headerCount must equal the file's column count or the call refuses with the reason in the error text. Column order equals the tick layout, and the trade side is written as its textual name, the exact mirror of tse_side_to_string.

Doubles are written in the shortest representation that round-trips back to the same value, so a write–read cycle is lossless.

TseCsvWriterHandle tse_csv_writer_create(TseCsvWriterOptions const* options);
void tse_csv_writer_destroy(TseCsvWriterHandle writer);
char const* tse_csv_writer_error(TseCsvWriterHandle writer);
TseStatus tse_csv_writer_write_ohlcv(TseCsvWriterHandle writer, char const* filePath, char const* const* headers, size_t headerCount, TseTickOHLCV const* ticks, size_t count);
TseStatus tse_csv_writer_write_scalar(TseCsvWriterHandle writer, char const* filePath, char const* const* headers, size_t headerCount, int64_t const* ts, double const* values, size_t count);
TseStatus tse_csv_writer_write_matrix(TseCsvWriterHandle writer, char const* filePath, char const* const* headers, size_t headerCount, int64_t const* ts, double const* values, size_t rows, size_t columns);

Slicing and transforms

The utilities come one family per element kind: _ohlcv, _bidask, _trade, _executed (over TseRetained) and _book (over TseBookMessage) run on the respective tick arrays, and _scalar runs on parallel timestamp/value arrays. All slicing results are expressed through TseSlice:

The head and tail functions yield the first or last n elements of the caller's array as an offset plus a length and copy nothing — the result is a view into memory you already own.

tse_timeserie_split_by_duration_* mirrors the engine's own splitting: the timestamps must be sorted, the sign of stepNs sets the direction, buckets align to the |stepNs| grid, fromNs clamps the start, and a zero step yields zero segments. Segments arrive as TseSlice pairs under the count-then-buffer protocol.

The scalar transforms run on the engine's monetary arithmetic of the given currency, so their rounding matches the engine's bookkeeping exactly. out receives exactly count values whose first offset (or lag) positions are zero — the engine's own convention. An offset or lag below one, or a serie not longer than it, is refused.

diff produces the arithmetic change over offset positions, log the logarithmic return over offset positions, and lag shifts the serie forward by lag positions.

TseStatus tse_timeserie_head_ohlcv(TseTickOHLCV const* ticks, size_t count, size_t n, size_t* outOffset, size_t* outLength);
TseStatus tse_timeserie_tail_scalar(int64_t const* ts, double const* values, size_t count, size_t n, size_t* outOffset, size_t* outLength);

TseStatus tse_timeserie_split_by_duration_scalar(int64_t const* ts, double const* values, size_t count, int64_t fromNs, int64_t stepNs, TseSlice* out, size_t capacity, size_t* written);

TseStatus tse_timeserie_diff_scalar(int64_t const* ts, double const* values, size_t count, TseCurrency currency, int offset, double* out);
TseStatus tse_timeserie_lag_scalar(int64_t const* ts, double const* values, size_t count, TseCurrency currency, int lag, double* out);
TseStatus tse_timeserie_log_scalar(int64_t const* ts, double const* values, size_t count, TseCurrency currency, int offset, double* out);

C++ and Python

C++

The C++ wrapper exposes the reader and writer as the RAII classes tse::CsvReader and tse::CsvWriter. Both hide the count-then-buffer protocol behind vector returns, and failures throw tse::Error carrying the handle's error text. CsvReader is constructed from tse::CsvOptions — note that the C++ struct defaults hasHeader to true, unlike the C NULL-options default of no header row — and offers name/index overloads in place of the _by_name/_by_index pairs.

CsvReader additionally offers readBuffer, columnCount, rowCount and headers; CsvWriter offers writeOhlcv, writeBidAsk, writeTrade, writeScalar and writeMatrix, each taking an optional headers vector that defaults to empty (no header row). The free functions mirror the utilities for every element kind — tse::head and tse::tail are overloaded for OhlcvTick, BidAskTick, TradeTick, Retained, BookMessage and scalar vectors, defaulting n to 5; tse::splitByDuration carries the same overload set and returns std::vector<TseSlice>; tse::diff, tse::lag and tse::log default their offset or lag to 1 and the currency to Currency::usd.

tse::CsvReader reader {options};
reader.readFile("quotes.csv");
std::vector<std::int64_t> ts {reader.readIndex("Date")};
std::vector<double> closes {reader.readDoubles("close")};
std::vector<double> matrix {reader.readMatrix(std::vector<std::string> {"open", "close"})};
std::string cellText {reader.cell("close", 0)};

tse::CsvWriter writer {tse::CsvWriterOptions {}};
writer.writeScalar("out.csv", ts, closes);

std::pair<std::size_t, std::size_t> firstFive {tse::head(ticks)};
std::vector<TseSlice> days {tse::splitByDuration(ts, values, fromNs, stepNs)};
std::vector<double> returns {tse::log(ts, values)};

Python

The Python module offers the same surface as the classes tse.CsvReader and tse.CsvWriter plus the module-level functions head, tail, split_by_duration, diff, lag and log. Failures raise tse.TseError with the handle's error text. The Python reader constructor exposes exactly the options that matter to the general reader — date_format, separator (default ","), has_header (default False, matching the C default), ignore_parse_errors (default False) and buffer_size (default 0, the engine default). The extraction methods keep the explicit _by_name/_by_index naming of the C API and return lists.

The Python timeserie functions cover the scalar family: they take parallel timestamps and values sequences, head and tail default n=5 and return an (offset, length) tuple, split_by_duration returns a list of (begin, length) tuples, and diff, lag and log default their offset or lag to 1 and the currency to tse.Currency.Usd, returning a list of exactly len(values) floats. Both CsvReader and CsvWriter release their handle in close(), which is also invoked on garbage collection.

reader = tse.CsvReader(date_format="%Y-%m-%d %H:%M:%S", has_header=True)
reader.read_file("quotes.csv")
ts = reader.read_index_by_name("Date")
closes = reader.read_doubles_by_name("close")
matrix = reader.read_matrix_by_index([1, 4])
text = reader.cell_by_name("close", 0)

writer = tse.CsvWriter(separator=",", format_timestamps=False)
writer.write_scalar("out.csv", ts, closes)

offset, length = tse.tail(ts, closes, n=10)
segments = tse.split_by_duration(ts, closes, from_ns, step_ns)
returns = tse.log(ts, closes)

Version 5.0.0.0