WFDB-x

A fast, lightweight implementation of the WFDB library and command-line tools, with identical behavior across seven programming languages

View the Project on GitHub alphanumericslab/wfdb-x

WFDB-x for Kotlin: implementation notes

These notes describe how the Kotlin Multiplatform implementation is organized, and the decisions that later work must follow. They complement docs/porting-guide.md (the C semantics every implementation must reproduce). The Rust implementation (rust/) is the model: most Kotlin files are close ports of the Rust module of the same name; cpp/ settles doubts, and python/src/wfdbx/ is the reference for the high-level API.

Requirements and build

cd kotlin
./gradlew jvmTest                       # unit tests on the JVM
./gradlew jvmTest -Pwfdbx.linux=true linuxX64Test   # also on Linux/Native, where the toolchain downloads
./gradlew compileCommonMainKotlinMetadata           # checks that common code uses only common APIs

The code must build without warnings.

Layout

Path Contents
src/commonMain/kotlin/wfdbx/Wfdb.kt the Wfdb class (all library state), the default instance (wfdbDefault(), withDefault)
.../wfdbx/Types.kt constants and structures of wfdb.h (Siginfo, Calinfo, Anninfo, Annotation, Seginfo, WFDB_*, DEFWFDB, …)
.../wfdbx/WfdbFile.kt WfdbFile, C’s FILE * / WFDB_FILE (buffered, C stdio semantics)
.../wfdbx/Wfdbio.kt port of lib/wfdbio.c: input/output directories, network access, legacy WFDB path, wfdb_open, wfdb_fopen, errors, freopen_stdin, run_sortann, and the legacy switch (wfdbx_set_legacy, wfdbx_get_legacy, wfdbx_legacy_args)
.../wfdbx/Signal.kt port of lib/signal.c (through rust/src/signal.rs): headers (including EDF), opening and closing signals, getvec/getframe/getvecs (bulk path), putvec, seeks, multi-segment and variable-layout records, skew, setheader/newheader/setmsheader, info strings, sample
.../wfdbx/SignalTime.kt the time, date and amplitude conversions of signal.c (strtim, timstr, mstimstr, datstr, strdat, setbasetime, counter frequency, adumuv, …), and the local time (TZif files, as in Rust)
.../wfdbx/Flac.kt FLAC decoder and encoder (port of rust/src/flac.rs), MD5
.../wfdbx/Annot.kt port of lib/annot.c, the in-process sort (AnnList, used by the sortann tool too)
.../wfdbx/Calib.kt port of lib/calib.c
.../wfdbx/Wfdbinit.kt port of lib/wfdbinit.c
.../wfdbx/Ecgcodes.kt ecgcodes.h codes and the ecgmap.h functions
.../wfdbx/cstd/ C standard library emulation (port of rust/src/cstd*): Printf.kt + Float.kt (glibc-exact printf), Parse.kt (strtol, strtod, sscanf, strtok, fgets lines), Rand.kt (glibc rand), Stdio.kt (buffered stdout, stderr, exit, environment), ByteVec.kt (byte buffer and byte-string helpers), Platform.kt (the platform layer: expect declarations)
src/jvmMain/ JVM platform layer (cstd/PlatformJvm.kt) and the thread-local default instance
src/nativeMain/, src/linuxMain/, src/iosMain/ native platform layer (POSIX; Linux: curl and posix_spawnp; iOS: NSURLSession)
src/commonTest/ unit tests (run on every target), with helpers in TestSupport.kt (DataDir copies tests/data to a scratch directory)
src/jvmTest/, src/nativeTest/ platform parts of the test helpers; JVM-only tests (network, against a local HTTP server)
src/commonMain/kotlin/wfdbx/apps/ the 33 tools, one file per tool (Rdsamp.kt, …), and Common.kt (see “Tools”)
src/jvmMain/kotlin/wfdbx/cli/Main.kt wfdbx.cli.MainKt: the command-line driver (TOOL ARGS..., --list, --version)
bin/wfdbx the launcher (POSIX shell): java -cp build/libs/wfdbx-all.jar wfdbx.cli.MainKt TOOL ARGS...
src/commonMain/kotlin/wfdbx/Record.kt the high-level API (port of python/src/wfdbx/highlevel.py, through rust/src/record.rs): readRecord, readHeader, readAnnotations, writeRecord, writeAnnotations, Record, Header, Annotations, TimeSpec, WfdbException, wfdbxSetInputDir, …
src/commonMain/kotlin/wfdbx/Check.kt checkRecord (port of python/src/wfdbx/check.py, through rust/src/check.rs), CheckReport, Issue, Severity
../benchmarks/kotlin/Bench.kt the benchmark driver, compiled by the bench compilation into build/libs/wfdbx-bench.jar (./gradlew benchJar; see Performance)

The library object and the C-style API

All the state of the C library is held by one class, Wfdb. Its fields are the states of the C source files, each a class defined in the file that ports it (as in Rust):

field class defined in
io IoState Wfdbio.kt
sig SigState Signal.kt
ann AnnState Annot.kt
cal CalState Calib.kt
maps EcgMaps Ecgcodes.kt
legacy Boolean (Wfdb.kt; the legacy flag is used by every file, so it is not in SigState as in Rust)

The state classes and their fields are internal; the files use each other’s state directly where the C code shares globals, and otherwise call the library functions.

Each C library function is an extension function of Wfdb with the C name and arguments, defined in the file that ports its C source (fun Wfdb.annopen(record: ByteArray?, aiarray: Array<Anninfo>, nann: Int): Int), and also a top-level function with the same name and arguments that runs it on the default instance (fun annopen(...) = wfdbDefault().annopen(...)), for line-by-line ports of C programs. The default instance is one per thread (ThreadLocal on the JVM, @ThreadLocal on Native). A Wfdb is not thread-safe: use each instance from one thread at a time; separate instances share nothing but the process-wide structures guarded by withGlobalLock (open output streams, fetched URLs).

Rules for library code (they matter, because a mistake compiles and calls the wrong instance):

Data representation

C standard library emulation (wfdbx.cstd)

Files: WfdbFile

Port of Rust’s WfdbFile on top of the platform’s unbuffered RawFile (file descriptors on Native, FileChannels on the JVM): fopen(name, mode), stdin(), stdout(), stderr(), fromBytes(...); getc, ungetc, read, fread, getline(ByteVec), fgets(ByteVec, size), readToEnd, g16, g32, putc, write, printf, p16, p32, flush, tell, seek, eof, error, clearerr, close. End of file is sticky until clearerr or a seek. Reads go through a buffer of the file’s block size (4096 on the JVM, st_blksize on Native), as glibc reads, which matters in legacy mode when a file is read while it is rewritten. There are no destructors: a WfdbFile must be closed (its buffered output is otherwise written only by exit). A file on the standard input never closes it and gives back unconsumed buffered bytes to the next reader; freopen_stdin(name) is C’s freopen(name, "rb", stdin).

Inside the library, open files with wfdb_open/wfdb_fopen (the settings apply), localFopen for a local file that must not depend on the settings, and wfdbx_open_in(headerDir, type, name, record) for files named in a header (containment check). pathId(name) and the internal WfdbFile.fileId() give file identities (device and inode) for the “output file is an input file” check.

Platform layer

cstd/Platform.kt declares everything common code needs from the system: RawFile (read, write, seek, position, size, blksize, fileId, close) and Platform (open, stdin, freopenStdin, writeStdout, writeStderr, stdoutIsTty, stdoutTell, getenv, setenv, isDirectory, mkdir, pathId, readFile, currentUid, timeNow, fetchUrl, runSortann, exit, currentDir, nameBytes, nameString), plus withGlobalLock and the per-thread default instances. Keep platform code minimal; add a declaration there rather than platform code elsewhere.

Native code

nativeMain, linuxMain, iosMain and nativeTest have not been compiled. Where the Kotlin/Native toolchain is available, build with -Pwfdbx.linux=true and fix what the compiler reports (likely: cinterop type conversions of struct stat fields and of posix_spawn arguments, and Platform.timeNow(), platform.posix.time(null)).

Signals (Signal.kt, SignalTime.kt, Flac.kt)

Close ports of rust/src/signal.rs and rust/src/flac.rs, with every WFDB-x divergence and addition (no assumed frequency, gain, resolution or units; files named in a header in its directory; segments of a multi-segment record next to its header; the output-is-input guard of osigfopen/osigopen; hea_name: names written in a header relative to its directory; wfdb_checkname on the last component; checksum messages recorded without printing during a high-level read; legacy mode).

The C-style signatures, where C’s types do not map directly:

C Kotlin
isigopen(rec, siarray, nsig) (also osigopen, wfdbinit) isigopen(rec: ByteArray?/String, siarray: Array<Siginfo>?, nsig: Int); null is C’s NULL; the elements are filled with Siginfo.set (allocate them: Array(n) { Siginfo() })
osigfopen(siarray, nsig) osigfopen(Array<Siginfo>?, Int)
getvec(v), getframe(v), putvec(v) IntArray; a shorter array receives what fits (as in Rust)
WFDB_Signal, WFDB_Group (unsigned) Int; negative values are compared as C’s unsigned values (invalid signal numbers)
getseginfo(&sarray) getseginfo(sarray: MutableList<Seginfo>?): the list is cleared and receives copies of the segments
setmsheader(rec, names, n) setmsheader(ByteArray, Array<ByteArray>, Int) or (String, Array<String>, Int)
timstr, mstimstr, datstr, getinfo return a new ByteArray (a copy of C’s static buffer); getinfo returns null at the end
sample(s, t), tnextvec(s, t), isgsettime(g, t) Int signal or group, Long time

WFDB-x additions, not in C: getvecs(out, n): Long and getvecs_status(out, n): GetvecsResult(count, status) (n vectors at once, as n getvec calls; the bulk path decodes the complete frames already in the input buffers in one pass, with identical results, which BulkTest verifies against the per-frame path); wfdbx_bulk_read(on) (turns the bulk path off, for tests); wfdbx_time_needs_freq. The bulk path (bulkFrames) handles formats 8, 16, 24, 32, 61, 80, 160, 212, 310 and 311 (one format per signal group), and null signals (format 0) of known length, without deskewing or resampling: records of several sampling frequencies, in low-resolution mode (averaging the oversampled signals with meansamp) and in high-resolution mode (from the start of a frame), and the segments of multi-segment records, of fixed or variable layout (mapped with sigmap), null segments included. Each group is decoded in tiles of 1024 frames (into bulkRaw, which stays in the cache), then turned into samples signal by signal (invalid samples, format 8 differences, the last sample and the checksum of the signal), as C++’s bulk_frames does; in low-resolution mode the vectors are then made signal by signal too. A frame that crosses the end of an input buffer, the frame at which checksums are verified or a null signal ends, the change from one segment to the next and everything else go through getvec, so the files are read in the same blocks, and the statuses, messages and state (samples, checksums, counts, gvc, tvector, ovec) are those of frame-by-frame reading (wfdbxBulkVectors and wfdbxInputState are for the tests). Internal accessors for the high-level API and check_record: wfdbxHeaderNsamples, wfdbxRecordNsamples, wfdbxIsEdf, wfdbxCksumResiduals, wfdbxGroupFileSizes, wfdbxBaseTimeDate, wfdbxSetCksumQuiet (getvec’s checksum message recorded, not printed).

Rules followed by the port (keep them when changing it):

Tools

Each tool is a function fun Wfdb.<tool>Main(argv: Array<ByteArray>): Int in wfdbx/apps/<Tool>.kt, a close port of rust/src/bin/<tool>.rs. argv is C’s argv (argv[0] is the tool name, as for a C tool found in the PATH). The function starts with val argv = toolArgv(argv0) (Common.kt: a mutable copy after wfdbx_legacy_args, which handles a first argument --legacy), and ends with exit(status) (wfdbx.cstd.exit, C’s exit, which may also be called anywhere) or returns the status. The driver (wfdbx.cli.MainKt) runs it on the default instance and exits with the status; an uncaught exception is reported as wfdbx: error: ... (status 1). Helpers that need the library state are Wfdb extensions, as in the library. Files the tool opens are closed explicitly (Rust closes them when they are dropped).

cd kotlin && ./gradlew fatJar        # build/libs/wfdbx-all.jar
export WFDBX_INPUT_DIR=../tests/data # the sample record 100s
bin/wfdbx rdsamp -r 100s -t 5        # bin/wfdbx --list, bin/wfdbx --version

(100s, in tests/data of the repository, is the first minute of record 100 of the MIT-BIH Arrhythmia Database.)

bin/wfdbx runs the jar named by WFDBX_KOTLIN_JAR, or build/libs/wfdbx-all.jar; it moves JAVA_TOOL_OPTIONS to the command line (the JVM would otherwise report it on the standard error) and starts the JVM with options for a short start-up (-XX:TieredStopAtLevel=1 -XX:+UseSerialGC, and the class data sharing archive wfdbx-all.jsa next to the jar if there is one: ./gradlew fatJar makes it, by running rdsamp once with -XX:ArchiveClassesAtExit, task cdsArchive; a JVM that cannot use it ignores it silently, -Xlog:cds=off). A link to bin/wfdbx named after a tool runs that tool. rdsamp formats its numbers without printf or divisions (RdOut: digits by multiplication, sample numbers by incrementing the digits of the last one), with output identical to printf’s (FastPathTest). In legacy mode, the library runs sortann as java -cp <class path> wfdbx.cli.MainKt sortann --legacy ....

Conformance (from the repository root; the harness runs kotlin/bin/wfdbx TOOL ..., and passes WFDBX_KOTLIN_JAR through):

python3 tests/conformance/harness.py check --impl kotlin -j8 [TOOL|CASE ...]
python3 tests/conformance/harness.py check --impl kotlin -j8 --legacy [TOOL|CASE ...]

The high-level API (Record.kt, Check.kt)

Testing

Unit tests are in commonTest and run on every target; expected values come from the C library (through the Rust tests and tests/reference/api_expect.c). The tests never change the current directory (the JVM cannot): they copy tests/data to a scratch directory (DataDir) and use the input and output directories. Environment changes go through setenv/unsetenv and are restored. The path of tests/data is passed by Gradle (wfdbx.testdata system property, WFDBX_TESTDATA for native tests).

Bytes that the C programs produce (sorted annotation files) were checked against rust/target/release/sortann, which matches the C program.

Test file Contents
cstd/PrintfTest.kt, cstd/ParseTest.kt C formatting and parsing
WfdbioTest.kt, FilesTest.kt files, directories, WFDB path, files named in headers, multi-segment records
AnnotTest.kt annotations, code tables, calibration, sorting, wfdbinit/wfdbquit
SignalTest.kt the signal functions (port of the C-style parts of rust/tests/signals.rs: values of api_expect.c, no-defaults and legacy behavior, EDF, FLAC seeks, round trips in every format, output-is-input guard, checksum errors, record names with directories)
BulkTest.kt getvecs and readRecord: bulk path against the per-frame path, for single- and multi-frequency, multi-segment and all tests/data records (port of rust/tests/bulk.rs)
FlacTest.kt FLAC decoder and encoder (port of the unit tests of rust/src/flac.rs)
jvmTest/NetworkTest.kt URLs and remote records, against a local HTTP server
jvmTest/SignalJvmTest.kt an output file that is a hard link to an input file
RecordTest.kt, AnnotationsTest.kt the high-level API (ports of the high-level parts of python/tests/test_signals.py, test_annotations.py and rust/tests/signals.rs, annotations.rs): round trips in every format, bulk reading and writing against the per-frame paths, the bulk annotation encoder against putann, checksums, truncated signal and annotation files, the sort option, record names, legacy mode
jvmTest/CheckCorpusTest.kt checkRecord on the corpus of python/tests/check_corpus.py (an empty diff against its expected.txt; skipped with a message without python3)
jvmTest/HighLevelJvmTest.kt the top-level functions, the input options and network access (local HTTP server), checkRecord without the corpus
jvmTest/ReadmeExamplesTest.kt the examples of README.md

The parts of the Rust tests that use the high-level API (read_record, write_record, read_header, …) are written with the C-style functions in SignalTest.kt where they test the library; the high-level API itself is tested in RecordTest.kt and the files after it. Beyond the unit tests, every record of tests/data (.hea and EDF) was read with getvec in low- and high-resolution mode and compared, line by line, with rust/target/release/rdsamp and rdsamp -H: identical.

Performance

The benchmarks of WFDB-x are in benchmarks/ at the root of the repository: benchmarks/run.py runs the same operations in the seven languages (and wfdb-python and the original C tools) on the same records, and docs/BENCHMARKS.md reports the results. The Kotlin driver is benchmarks/kotlin/Bench.kt (package wfdbx.bench); the bench compilation of build.gradle.kts compiles it from ../benchmarks/kotlin, and ./gradlew benchJar builds build/libs/wfdbx-bench.jar, which run.py runs as java -jar kotlin/build/libs/wfdbx-bench.jar OP DATADIR REPS (see benchmarks/README.md).

readRecord and writeRecord decode and encode the samples in bulk (writeRecord encodes the frames of the common formats in one loop, byte for byte as putvec; writeAnnotations encodes the annotations into one buffer, byte for byte as putann); both are tested against the per-frame paths. The first call of a function in a JVM process is slower than the next ones, which run code compiled by the JIT compiler; the benchmarks report both. A tool run (bin/wfdbx rdsamp ...) includes the start-up of the JVM; the launcher limits the JVM to its first compiler, for a fast start of the short runs that are typical of the tools.