Fix test_shutdown_worker leaving stale callback that breaks later tests

Fix test_shutdown_worker leaving stale callback that breaks later tests #1664