Skip to content

Prevent trackio errors from crashing the user's training loop#496

Merged
abidlabs merged 10 commits into
mainfrom
safe-log-no-crash
Apr 15, 2026
Merged

Prevent trackio errors from crashing the user's training loop#496
abidlabs merged 10 commits into
mainfrom
safe-log-no-crash

Conversation

@abidlabs
Copy link
Copy Markdown
Member

@abidlabs abidlabs commented Apr 11, 2026 •
edited
Loading

  • hardens trackio.init() against corrupted/malformed sqlite metadata reads and Space/bootstrap metadata failures by falling back to safe defaults instead of raising into the caller
  • hardens the local and remote sender threads so sqlite/media/upload failures are converted into nonfatal warnings instead of killing the thread or escaping into user code
  • hardens local fallback buffering, artifact queueing, and finish() so shutdown/pending-buffer checks also stay nonfatal
  • makes degraded-mode warnings nonfatal even when the caller promotes warnings to exceptions
  • adds regression tests for corrupt-db init, local flush failure, finish() pending-data inspection failure, and warnings.simplefilter("error")

@abidlabs @claude
Prevent trackio errors from crashing the user's training loop
74464e5 
Wrap log(), log_system(), and alert() in try/except so that metric
processing failures emit a warning instead of propagating into the
caller's thread.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@gradio-pr-bot
Copy link
Copy Markdown
Contributor

gradio-pr-bot commented Apr 11, 2026 •
edited
Loading

🪼 branch checks and previews

• Name Status URL
🦄 Changes detected! Details

@gradio-pr-bot
Copy link
Copy Markdown
Contributor

gradio-pr-bot commented Apr 11, 2026

🦄 change detected

This Pull Request includes changes to the following packages.

Package Version
trackio minor
  • Prevent trackio errors from crashing the user's training loop

‼️ Changeset not approved. Ensure the version bump is appropriate for all packages before approving.

  • Maintainers can approve the changeset by checking this checkbox.

Something isn't right?
  • Maintainers can change the version label to modify the version bump.
  • If the bot has failed to detect any changes, or if this pull request needs to update multiple packages to different versions or requires a more comprehensive changelog entry, maintainers can update the changelog file directly.

@HuggingFaceDocBuilderDev
Copy link
Copy Markdown

HuggingFaceDocBuilderDev commented Apr 11, 2026 •
edited
Loading

🪼 branch checks and previews

• Name Status URL
Spaces ready! Spaces preview

Install Trackio from this PR (includes built frontend)

pip install "https://huggingface.co/buckets/trackio/trackio-wheels/resolve/4376bb5792ef7cc7e634f34d836e950156e02453/trackio-0.22.0-py3-none-any.whl"

@HuggingFaceDocBuilderDev
Copy link
Copy Markdown

HuggingFaceDocBuilderDev commented Apr 11, 2026

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@abidlabs abidlabs marked this pull request as ready for review April 15, 2026 00:14
znation
znation approved these changes Apr 15, 2026
View reviewed changes
Comment thread trackio/__init__.py Outdated
@abidlabs
Copy link
Copy Markdown
Member Author

abidlabs commented Apr 15, 2026

Thanks @znation for reviewing!

@abidlabs abidlabs merged commit af23d74 into main Apr 15, 2026
8 of 9 checks passed
@gradio-pr-bot gradio-pr-bot mentioned this pull request Apr 14, 2026
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@znation znation znation approved these changes

@Saba9 Saba9 Awaiting requested review from Saba9

@qgallouedec qgallouedec Awaiting requested review from qgallouedec

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@abidlabs @gradio-pr-bot @HuggingFaceDocBuilderDev @znation
⚡