Repository navigation
chore: release v1.1.0 - #4
Merged
Merged
Conversation
calculate_precision_recall_f1 returned NaN for F1 whenever TP was 0, so a recording with real bouts and no match (nothing predicted, or only false bouts) left the mean over recordings. F1 is now 0 when TP = 0 and FP + FN > 0, the value of 2 TP / (2 TP + FP + FN). It stays NaN for TP = FP = FN = 0 and every other value is unchanged. The MATLAB function BANOScalculatePrecisionRecallF1 gets the same rule. The goldens move to golden_python_v110.json and golden_matlab_v110.json (the v0.2.0 files are kept). Two values change: Recording_8 mount F1 goes from null to 0 and Recording_8 overall F1 from 0.6062 to 0.4546.
count_switches_within_bout read pred_sequence[i] on a pandas Series, which is a lookup by index label, while bouts are frame positions. Data indexed other than 0..n-1 (frames numbered from 1, reversed or float labels, strings on pandas 3) got a wrong IC or a KeyError. count_switches_within_bout and calculate_ic now read a Series by position, and compute_behavior_metrics passes the column values. No golden value changes; the MATLAB code has no index labels.
calculate_ic averaged 1 - switches / (length - 1) over every real bout. A missed real bout has no predicted frame, hence no switch, and scored 1.0: missing a bout raised IC, and a model that predicted nothing got IC 1.0. IC now averages over the real bouts that contain at least one predicted frame (the overlap rule of SO and TP) and is NaN when no real bout is found. One-frame real bouts stay out of IC. The MATLAB function BANOScalculateIC gets the same rule. 39 golden IC values change, in every recording. Recording_8 mount IC goes from 1.0 to null: its two real bouts are both missed.
SO and TP average over the overlapping (pred, GT) bout pairs and IC
over the found real bouts, so a real bout with no predicted frame inside
it leaves all three averages. The new missed_bouts option keeps this as
the default ("exclude") or adds one term equal to 0 per missed real bout
to SO, TP and IC ("zero"); one-frame real bouts stay out of IC.
Precision, recall, F1 and the absent-behavior rule do not depend on it.
calculate_so_terms, calculate_tp_terms and calculate_ic_terms return
the terms behind each average; calculate_so, calculate_tp_metric and
calculate_ic return their mean. They take the option as a keyword, as
do compute_behavior_metrics, calculate_banos_for_each_file and score(),
which check it on entry: an unknown value raises even when no column
reaches the metrics.
MATLAB takes it as the 'missed_bouts' name-value pair, read and
validated by the new BANOSparseOptions: an unknown option name or value
now raises an error. BANOScalculateSO, BANOScalculateTP and
BANOScalculateIC return their terms as a second output. Default results
are unchanged; six MATLAB golden IC values move by one or two units in
the last place, IC being now the mean of its terms instead of a nanmean.
score() averages per-file metrics (aggregation="per_file", default): each recording weighs the same whatever its number of bouts, and a recording without the behavior scores 1.0 or 0.0 on all six metrics. The new aggregation="pooled" sums, per behavior, the TP, FP and FN counts of all recordings for precision, recall and F1, and averages all their SO, TP and IC terms (under the chosen missed_bouts). A recording without the behavior adds its false bouts to FP and no term; a behavior with no real and no predicted bout anywhere is NaN. overall_metrics is the mean over the behaviors. The result has the same form in both modes. compute_behavior_components and calculate_components_for_each_file return the per-file counts and terms, and pool_components combines them. compute_behavior_metrics now derives its metrics from the components, with unchanged results. score() checks matching, missed_bouts and aggregation on entry, and the pipeline functions check matching and missed_bouts on entry, so an unknown value raises even when no column reaches the bout matching. In MATLAB, BANOS_score takes the 'aggregation' option, BANOScomputeBehaviorMetrics and BANOScalculateBANOSForEachFile return the components as a second output, and BANOSpoolComponents pools them.
preprocess_data, and so score(), now raises ValueError when the prediction and the ground truth of a file differ in number of frames (a shorter prediction used to raise KeyError, a longer one was scored) or when a file has no frame. It emits a UserWarning naming the file and the column for each column dropped (values other than 0 and 1, such as probabilities or inf, or a column missing or invalid on the other side) and for each column whose NaN values are replaced by 0; score() used to discard the drop report without a word. The warnings point at the line that called preprocess_data() or score(). Valid input gives the same results and no warning. BANOSpreprocessData raises and warns the same way, with the identifiers BANOS:lengthMismatch, BANOS:noFrame, BANOS:columnDropped and BANOS:nanReplaced.
The per-recording goldens score each recording alone, so the means across recordings and the pooled metrics were never pinned. The generators now also score the 10 bundled recordings together with score() and BANOS_score, once per combination of missed_bouts and aggregation, under "all_recordings" in golden_python_v110.json and golden_matlab_v110.json. test_regression checks these values at 1e-10 and test_parity checks Python against MATLAB at 1e-6.
test_preprocess_data_does_not_mutate_caller_dict now compares the caller's dict and DataFrames before and after preprocess_data; it compared the dict's id with itself and could not fail. It runs twice, once with the prediction and once with the ground truth as the side that needs every cleaning step, so that a step applied in place to either side fails it. test_score_version compares banos.__version__ with the version in pyproject.toml. Remove tests/test_comparison_report.py. It looked for golden_matlab_current.json, which does not exist, so it was always skipped. Its check, the per-behavior metrics of each recording against Matlab at 1e-6, is made by test_parity_group_metrics, which also requires NaN on both sides where it accepted a Python NaN against any Matlab number. test_parity_group_metrics now also fails when Python and Matlab report different behaviors, as the removed test did; it skipped a behavior missing on the Python side and ignored an extra one. The removed test was the only user of pytest-html, which leaves the dev dependencies; the lock drops pytest-html and pytest-metadata. Correct test docstrings that said NaN comes from behaviors with no GT bout (such a recording scores 1.0 or 0.0), that optimal matching maximizes total overlap (it maximizes the number of matched pairs), or that pointed to the removed test. generate_archive_fixtures.py now states that the v0.1.5 fixture is frozen, how it was produced, and that the BANOS 0.1.5 package from PyPI writes the same file; like the other generator, it ends the file with a newline.
compare_fixtures.py tagged a value that becomes NaN as NULL->VALUE and the reverse as VALUE->NULL. The v0.1.5 / v0.2.0 legend then explained the four Recording_8 mount cells that go from 0 to NaN (precision, F1, SO, TP) as correct absence now scoring 1, and every other difference as the IC fix. Swap the tags and give each comparison a meaning for each tag. From v0.1.5 to v0.2.0: a metric with a zero denominator is NaN instead of 0; correct absence scores 1.0 instead of 0 on all six metrics; IC counts the last frame pair of a bout and leaves one-frame bouts out; the overall means move with these. Add mode 3, Python v0.2.0 against v1.1.0: one F1 from NaN to 0, one IC from 1.0 to NaN, 39 other values changed (IC and overall means), 259 equal. Only the Recording_N entries are compared; the all_recordings entry of the v1.1.0 file has no counterpart in older files. Legends use ASCII arrows, which a Windows console can print. Regenerate reports/fixture_comparison.html with the corrected script. The report now ends with a newline, as the committed file did before; the script never wrote one. tests/test_compare_fixtures.py runs the script: mode 3 must tag exactly the 41 values changed by v1.1.0, and the committed report must equal what the script writes (skipped where reports/ is absent, as in the sdist). The output depends only on the fixture files, so a fixture edited without regenerating the report, or a report edited by hand, fails the test.
The test matrix no longer ignores tests/test_parity.py. These tests compare the Python results with the committed Matlab golden values, need no Matlab, and are the only check that both implementations agree. actions/checkout no longer leaves the job token in the git config (persist-credentials: false); no step pushes or fetches. The workflow can now be called by another workflow (workflow_call), so that the publish workflow can require the whole matrix. Job names are unchanged.
publish.yml now has three jobs. check-version reads the version in pyproject.toml and fails unless the tag is v<version>. ci calls ci.yml, so lint and the test matrix on Python 3.11 to 3.14, parity tests included, must pass. build-and-publish runs only after both: it builds with uv build and publishes with trusted publishing, as before. The workflow token is read-only by default; only the publishing job gets id-token: write, in the pypi environment. Checkouts do not persist credentials. The matrix replaces the single test run on Python 3.11 that preceded the build.
Pre-commit runs ruff v0.15.7, the version in uv.lock and CI; v0.4.4 would rewrite imports and formatting that CI considers clean. Its lint hook id becomes ruff-check, and ruff-format leaves the notebooks alone. CI lints and format-checks demo/quickstart.py too; its import order is fixed. In the two notebooks, the ruff findings are fixed in the code cells (import order, a variable named l, a trailing space); outputs, cell ids and metadata are unchanged. mypy, pandas-stubs and scipy-stubs join the dev group; the lock only gains packages, no existing version moves. The stubs are bounded to the pandas 3.0 and scipy 1.17 of the lock. A new CI job, Type check (mypy), runs mypy on src/. Its two errors are fixed without changing behavior: is_logical_column returns a bool rather than a numpy bool, and drop_non_logical_columns is annotated with list[Hashable], the column labels it returns. src/banos/py.typed tells the type checkers of users that the package is typed.
matplotlib and seaborn move from the runtime dependencies to the notebook extra: the package never imports them, the Python tutorial does. pip install banos now pulls numpy, pandas and scipy only. The dev extra is removed; the dev dependency group, which uv sync installs by default, is kept. SETUP.md runs uv sync instead of uv sync --extra dev. Metadata: authors and maintainers, and the license as an SPDX expression with its file (license = "MIT", license-files). hatchling 1.27 is the first version that writes them as License-Expression and License-File, so the build backend is bounded to >=1.27,<2. The MIT license classifier is removed: the expression replaces it (PEP 639). The sdist lists what it includes: src, tests with their fixtures, the bundled recordings the tests read, demo/quickstart.py, README, LICENSE and CHANGELOG. The Keras model, notebooks, Matlab code, HTML report, images and repository files are left out (about 62 kB instead of 1.2 MB). The wheel keeps the same files; only its metadata changes. The lock changes only in the banos entry; no pinned version moves.
The README now defines bouts, overlap, greedy matching and each metric as the code computes them: IC is 1 - switches / (bout length - 1) over the found real bouts, SO and TP average over every overlapping pair, and F1 is 0 when no bout is matched. It says when each value is NaN and that NaN values are left out of the means, and that optimal matching gives the same counts as greedy on bouts read from 0/1 columns. New sections cover the aggregation and missed_bouts options with an example each, the input format and return shapes, the errors and warnings, the pooled chain of the lower-level API and the MATLAB options. The per-file mean is shown to mix detection quality with the share of recordings without the behavior. Installation lists numpy, pandas and scipy and the notebook and calms21 extras. The citation section gives the published article first and the arXiv preprint second. Links point to absolute GitHub URLs, so that they work on the PyPI page.
CITATION.cff (format 1.2.0) describes the software, BANOS 1.1.0 under the MIT license, and gives the article in Neuroscience & Biobehavioral Reviews as preferred citation, with the arXiv preprint as a reference. It has no release date yet. The README points to it.
The 1.1.0 entry, not yet dated, lists the numerical changes (F1 0 instead of NaN when no real bout is matched, IC over the found real bouts, IC read by position, the 41 golden values that move and the figures of the bundled data), the added options and functions, the changed errors, warnings, packaging and CI, and the fixes. The 0.2.0 entry gains the two changes it did not list: for a recording with real bouts and no match, F1, SO and TP, and precision when nothing was predicted, went from 0 to NaN; and a one-frame real bout, which scored 0 in IC, is left out of IC. Earlier dates and versions are unchanged.
RELEASE.md: the version is bumped on a branch and merged by pull request, then the merged commit is tagged; the publish workflow checks the tag against pyproject.toml and runs the whole CI, parity tests included, before publishing. The fixture table names the v1.1.0 files as current and the v0.2.0 and v0.1.5 files as archives, and the regenerate commands add the comparison report and the v0.1.5 command, which reproduces that file identically with Python 3.12 or later. SECURITY.md: BANOS reads no file, and uv.lock pins the development and CI environment, not the versions users install. The runtime closure is numpy, pandas and scipy. SETUP.md: the repository allows selected actions pinned by SHA, not all actions; the required check names, the protected tags and the pypi environment are listed, with the CI, notebook and build commands. The CalMS21 section matches the notebook: it downloads the zip, and loads the model from demo/models. CONTRIBUTING.md: there is no issue template; the CI checks to run before a pull request are listed.
score() no longer calls optimal matching "scientifically optimal": it gives the largest number of matched pairs, which greedy matching already reaches on bouts read from 0/1 columns. The docstring of test_matching.py says when the two can differ: only on bouts given directly, when those of one side overlap one another or are not in time order. score() and BANOS_score list F1 among the values that can be NaN when pooled, and BANOS_score names the missing-value error. compute_behavior_metrics and BANOScomputeBehaviorMetrics state the absent-behavior rule. score() and preprocess_data say that a warning is emitted per column, not per NaN value. BANOScalculateIC says that IC is averaged over the found bouts and that one-frame bouts are left out, and BANOSidentifyBouts describes its [start, end] vectors. generate_archive_fixtures.py says that it writes the v0.1.5 file identically with Python 3.12 or later only. No code changes.
The tutorials said that the prediction shifted by 15 frames still detects both real bouts (DA 1.0) and that a small shift gives a TP near 1.0. That prediction shares no frame with the real bouts, so DA is 0 and SO and TP are undefined; the 2-frame shift gets TP 0.2 and SO 0.667. The interpretation now says so, and the comparison with CalMS21 mentions the pooled aggregation. The MATLAB tutorial computes the frame F1 of this example as the Python one does (0, not NaN, when undefined), names the behaviors it uses, and states the frame F1 it prints (0.815). create_tutorial_notebook.py writes the imports in the order ruff expects and the wording of the committed notebook, which is generated again from it and executed: the printed metrics are those of this code (overall F1 0.7486, IC 0.9921; shifted prediction DA 0.000). The CalMS21 notebook is not run here, as it downloads its data: a note after its title says that its stored outputs come from BANOS 0.2.0 and that F1 and IC differ with 1.1.0. Its comparison table no longer calls SO a measure of matched bouts.
pyproject.toml, banos.__version__, the banos entry of uv.lock and the version line of the MATLAB Contents.m. Its date, required by the format of that line, is the date of this change; it is set again, with the changelog entry, on release day. The tutorial notebook prints the new version.
BANOS_score raised three errors without an identifier: optimal matching requested, a single prediction without its ground truth, and a first argument that is neither a cell array nor a struct. They are BANOS:optimalNotImplemented, BANOS:missingGroundTruth and BANOS:invalidData, in the scheme of the other errors and warnings; its help lists them. BANOSaggregateMetrics and BANOSpoolComponents averaged with nanmean, a function of the Statistics and Machine Learning Toolbox, so scoring several recordings needed that toolbox. They use mean(..., 'omitnan'), which is base MATLAB and gives the same values: the MATLAB golden fixture regenerates byte for byte. The BANOS toolbox now needs base MATLAB only; the README and the changelog say so, and the help texts no longer name nanmean.
The CalMS21 tutorial is executed again with this code, TensorFlow 2.21.0 and the other versions of uv.lock (Python 3.12), so its stored outputs are those of 1.1.0: detection F1 of mount 0.7006 -> 0.6638 and overall 0.6398 -> 0.6286; IC attack 0.7287 -> 0.7228, investigation 0.9734 -> 0.9652, mount 0.9374 -> 0.9308, overall 0.8798 -> 0.8719. The predictions are those of the earlier run: the 1.0.0 code gives the stored values on them. Precision, recall, SO and TP are unchanged. The note that the outputs came from 0.2.0 is removed, and the comparison table prints the corrected SO wording. The setup cell printed the absolute paths of the model and of the data cache on the machine that ran it; it prints them relative to the repository. data.caltech.edu answers 403 to the default user agent of urllib, so the automatic download failed for every user; the requests now send a User-Agent that names the tutorial and the repository. The stored output is that of a first run, which downloads the data, with its stream outputs merged as Jupyter displays them (one final line for the progress bar).
Eight keywords follow the existing six: computational ethology, animal behavior, behavioral neuroscience, action segmentation, bout detection, evaluation metrics, inter-annotator agreement and rodent. They are written in the Keywords field of the package metadata, and CITATION.cff lists the same fourteen.
The changelog entry and CITATION.cff (date-released) are dated 2026-10-30, the date already in the version line of Contents.m. RELEASE.md adds two steps: the release pull request is merged as a single squashed commit, subject "chore: release vX.Y.Z (#N)" and an empty body, and once the package is on PyPI the GitHub release of the tag is created with the changelog entry as notes.
Upgrade pillow, virtualenv, filelock, tornado, nbconvert, certifi and setuptools in uv.lock to releases without the published advisories; the new virtualenv also moves distlib and packaging. None of them is a runtime dependency of banos: they come from the notebook and calms21 extras and from the development tools. numpy, pandas and scipy are unchanged.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Release 1.1.0. Details in
CHANGELOG.md.Numerical changes
that recording stays in the mean (2 of the 300 per-recording golden values change).
averages over the real bouts that are found (39 golden values change).
Added
missed_bouts("exclude","zero") andaggregation("per_file","pooled")options, in Python and MATLAB, with the same defaults as before.
of the two options.
CITATION.cffandpy.typed.Changed
no frame, a warning for every dropped column and every column whose NaN are replaced.
notebookextra.to match the package version and the whole CI to pass.
Documentation