-
-
Notifications
You must be signed in to change notification settings - Fork 8.9k
Quantile test failures. #12139
Copy link
Copy link
Open
Description
Tested on b2f15e6
Python:
tests/python-gpu/test_gpu_data_iterator.py:31:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _tests/python-gpu/test_gpu_data_iterator.py:48: in test_gpu_data_iterator
run_data_iterator(
tests/python/test_data_iterator.py:203: in run_data_iterator
_assert_cut_rank_error_within_tolerance(indptr_it, cuts_it, x_np, w_np)
> np.testing.assert_array_less(
np.array([abs(expected_rank - sum_weight)]),
np.array([acceptable_error + 1e-12]),
)
E AssertionError:
E Arrays are not strictly ordered `x < y`
E
E Mismatched elements: 1 / 1 (100%)
E Max absolute difference among violations: 0.14805494
E Max relative difference among violations: 0.12515479
E x: array([1.33103])
E y: array([1.182975])
E Falsifying example: test_gpu_data_iterator(
E n_samples_per_batch=3,
E n_features=3,
E n_batches=4,
E subsample=False,
E use_cupy=False,
E on_host=False,
E )
E
E You can reproduce this example by temporarily adding @reproduce_failure('6.151.10', b'AEEDQQNBBAAAAA==') as a decorator on your test case
CPP with 2 GPUs:
[ RUN ] MGPUQuantileTest.AllReduceBasic
[17:31:30] Insecure federated server listening on 0.0.0.0:0, world size 2
/home/jiamingy/ws/xgboost_dev/xgboost/tests/cpp/common/test_hist_util.h:219: Failure
Expected: (stats.max_normalized_error) <= (max_normalized_rank_error), actual: 22.349999999999941 vs 20
feature=6, cut=902, normalized_error=22.349999999999941, absolute_error=39.210526315789366, target_rank=1584.2105263157894, rank_lo=1544, rank_hi=1545, total_weight=2000, num_cuts=1140
/home/jiamingy/ws/xgboost_dev/xgboost/tests/cpp/common/test_hist_util.h:219: Failure
Expected: (stats.max_normalized_error) <= (max_normalized_rank_error), actual: 22.349999999999941 vs 20
feature=6, cut=902, normalized_error=22.349999999999941, absolute_error=39.210526315789366, target_rank=1584.2105263157894, rank_lo=1544, rank_hi=1545, total_weight=2000, num_cuts=1140
/home/jiamingy/ws/xgboost_dev/xgboost/tests/cpp/common/test_hist_util.h:219: Failure
Expected: (stats.max_normalized_error) <= (max_normalized_rank_error), actual: 20.498 vs 20
feature=61, cut=230, normalized_error=20.498000000000001, absolute_error=36.024604569420035, target_rank=405.97539543057997, rank_lo=442, rank_hi=443, total_weight=2000, num_cuts=1138
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
No labels