Skip to content

fix: correct pvalue logic and add discretization step#54

Open
drbh wants to merge 1 commit into
ArcInstitute:mainfrom
drbh:revise-experimental-impl
Open

fix: correct pvalue logic and add discretization step#54
drbh wants to merge 1 commit into
ArcInstitute:mainfrom
drbh:revise-experimental-impl

Conversation

@drbh
Copy link
Copy Markdown
Contributor

@drbh drbh commented Sep 23, 2025

This PR supersedes #53 and ensures that the experimental output is identical to the current output with a relative accuracy of <1e-6 and absolute accuracy of <1e-12.

Additionally this PR renames and improves the tests/bench_expr.py file. The new tests/test_experimental_correctness.py file will run with the other test (prefixed with test_) and compares the output between the reference and experimental outputs to ensure they are within a extremely small tolerance

note these changes slow down the overall computation by a small amount (mainly when the discretization step is needed) which can be avoid but providing a the raw integer data

uv run tests/test_experimental_correctness.py
======================================================== test session starts ========================================================
platform darwin -- Python 3.10.15, pytest-8.4.1, pluggy-1.6.0
benchmark: 5.1.0 (defaults: timer=time.perf_counter disable_gc=False min_rounds=5 min_time=0.000005 max_time=1.0 calibration_precision=10 warmup=False warmup_iterations=100000)
rootdir: /Users/drbh/Projects/pdex
configfile: pyproject.toml
plugins: benchmark-5.1.0
collected 9 items

tests/test_experimental_correctness.py .........                                                                              [100%]

========================================================= warnings summary ==========================================================
tests/test_experimental_correctness.py::test_correctness_comparison
tests/test_experimental_correctness.py::test_detailed_correctness_metrics
tests/test_experimental_correctness.py::test_edge_cases_numerical_stability
tests/test_experimental_correctness.py::test_edge_cases_numerical_stability
  /Users/drbh/Projects/pdex/.venv/lib/python3.10/site-packages/anndata/_core/aligned_df.py:68: ImplicitModificationWarning: Transforming to str index.
    warnings.warn("Transforming to str index.", ImplicitModificationWarning)

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html

------------------------------------------------------------------------------------------------------------ benchmark: 6 tests ------------------------------------------------------------------------------------------------------------
Name (time in ms)                                                    Min                   Max                  Mean             StdDev                Median                IQR            Outliers       OPS            Rounds  Iterations
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_benchmark_parameterized_datasets[True-500-100-10]            7.3009 (1.0)          8.3150 (1.0)          7.9079 (1.0)       0.4193 (1.0)          7.9405 (1.0)       0.6697 (1.0)           1;0  126.4562 (1.0)           5           1
test_benchmark_parameterized_datasets[True-1000-300-100]         81.2446 (11.13)       83.1028 (9.99)        82.1008 (10.38)     0.7036 (1.68)        82.1472 (10.35)     0.9652 (1.44)          2;0   12.1802 (0.10)          5           1
test_benchmark_parameterized_datasets[True-2000-500-50]         113.6737 (15.57)      124.0537 (14.92)      118.8101 (15.02)     4.5535 (10.86)      119.7459 (15.08)     8.1712 (12.20)         2;0    8.4168 (0.07)          5           1
test_benchmark_parameterized_datasets[False-500-100-10]       1,121.4055 (153.60)   1,163.2690 (139.90)   1,146.4066 (144.97)   15.3490 (36.61)    1,150.1841 (144.85)   13.6819 (20.43)         2;0    0.8723 (0.01)          5           1
test_benchmark_parameterized_datasets[False-2000-500-50]      3,598.1983 (492.84)   3,636.3652 (437.33)   3,620.5203 (457.84)   14.3509 (34.23)    3,620.4431 (455.95)   17.2623 (25.78)         2;0    0.2762 (0.00)          5           1
test_benchmark_parameterized_datasets[False-1000-300-100]     3,952.2674 (541.34)   4,037.7101 (485.59)   3,976.9548 (502.91)   34.6742 (82.70)    3,966.9077 (499.58)   30.1742 (45.06)         1;1    0.2514 (0.00)          5           1
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Legend:
  Outliers: 1 Standard Deviation from Mean; 1.5 IQR (InterQuartile Range) from 1st Quartile and 3rd Quartile.
  OPS: Operations Per Second, computed as 1 / Mean
============================================= 9 passed, 4 warnings in 71.21s (0:01:11) ==============================================

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant