Skip to content

Refactor conda build#3000

Open
markovskipetar wants to merge 35 commits intomasterfrom
refactor_conda_build
Open

Refactor conda build#3000
markovskipetar wants to merge 35 commits intomasterfrom
refactor_conda_build

Conversation

@markovskipetar
Copy link
Copy Markdown
Collaborator

This PR introduces GitHub Actions workflows for building and testing ArcticDB using conda environments across multiple platforms:

build_with_conda.yml — Parent workflow triggered on push to master, PRs, and manual dispatch. Orchestrates builds for Linux x64, Linux ARM64, macOS ARM64, and Windows x64.

compile_and_test_with_conda_unix.yml — Reusable workflow for Unix platforms (Linux/macOS). Handles compilation, C++ tests, and Python tests (unit, integration, hypothesis, stress, compat, enduser).

compile_and_test_with_conda_win.yml — Reusable workflow for Windows. Same structure as Unix but with Windows-specific shell handling and build configuration.

@github-actions
Copy link
Copy Markdown

Label error. Requires exactly 1 of: patch, minor, major. Found:

@claude
Copy link
Copy Markdown
Contributor

claude bot commented Mar 31, 2026

PLACEHOLDER_BODY

@claude
Copy link
Copy Markdown
Contributor

claude bot commented Apr 1, 2026

ArcticDB Code Review Summary

PR #3000 — Refactor conda build (CI/workflow-only change)
Last updated: 2026-04-03 · Covers full diff

This PR replaces a monolithic 1200+ line build_with_conda.yml with a hub-and-spoke design: a lightweight orchestrator that calls two reusable workflows (compile_and_test_with_conda_unix.yml and compile_and_test_with_conda_win.yml). No production code is changed.


API & Compatibility

  • ✅ No breaking changes to public Python API
  • ✅ On-disk format unchanged
  • ✅ Protobuf schema unchanged
  • ✅ Key types and ValueType enum unchanged
  • N/A Deprecation protocol (no Python API changes)

Memory & Safety

  • N/A (CI/workflow only)

Correctness

  • ✅ MongoDB hostname fixed: mongodb (Docker service) → localhost (locally installed)
  • STORAGE_TYPE env var now present in Windows python_tests step (was missing in prior push; fixed in latest commit — compile_and_test_with_conda_win.yml:217)
  • python/test.py scratch file removed (was accidentally committed in prior push; removed in latest commit)
  • run_on_arm_mac dispatch input intentionally removed (developer confirmed ARM Mac always runs now)
  • ⚠️ python_tests condition inconsistency between Unix and Windows workflows. The Windows python_tests job has !failure() in its if guard (skips if compile fails), but the Unix python_tests job only has always() && !cancelled() (always runs even if compile fails, wasting runner time on a doomed job). Noted by reviewer IvoDD — should be made consistent.

Code Quality

  • ✅ Consolidated duplicate compile/test logic into reusable workflows
  • ✅ Inline YAML input definitions are compact and consistent
  • ✅ New install_mongodb composite action correctly extracts MongoDB install logic for reuse

Testing

  • N/A (CI workflow refactor — the workflows themselves are the test infrastructure)

Build & Dependencies

  • ✅ No CMakeLists.txt changes
  • ✅ No dependency version changes
  • N/A vcpkg / conda (workflow only)

Security

  • ✅ No hardcoded credentials — secrets passed via ${{ secrets.* }}
  • ✅ No new credential exposure in workflow env blocks

PR Title & Description

  • ✅ Title is clear and uses imperative verb
  • ✅ Description explains the new three-workflow structure
  • ⚠️ No issue reference linked (minor)
  • ⚠️ PR has no labels — consider adding ci or enhancement

Documentation

  • N/A (no public API changes)

Summary: Both previously-flagged blocking issues are resolved in the latest push: STORAGE_TYPE is now correctly set in the Windows test step, and the python/test.py scratch file has been removed. One remaining nit: the Unix python_tests if guard should add !failure() to match Windows behaviour and avoid running against a failed compile.

name: Compile (${{inputs.platform}})
if: |
always() &&
!cancelled()
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why are there different files for _unix and _windows?

My main problem with this is the big amount of duplication between the two, so in the future when we change something we might forget to do it for the other workflow as well.

E.g. see this if: always() && !cancelled. It is missing on the windows build. Is there a reason for this?

I appreciate for some things we reqire completely different step (e.g. for Install Azurite). We can have Install Azurite (Unix) step and Install Azurite (Windows) step and skip them depending on the OS. E.g. see how we do this in build_steps.yml for reference.

However looking at the diff between the two the differences are not big:
E.g. windows doesn't build C++ tests. We can just pass run_cpp_tests=false or use if: matrix.os != "windows"
There are many others but we can go together through the diff and decide what can be easily unified.

if: |
always() &&
!cancelled() &&
!failure()
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is another place where we differ between unix and windows? Any particular reason?

Also I think always() && !cancelled() && !failures() is equivalent to the default success()? Might be better to leave it to the default in this case.

npm install -g azurite
npm_config_prefix=$(npm config get prefix)
npm_bin_dir="$npm_config_prefix"
if [[ "$RUNNER_OS" == "Windows" ]]; then
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need this if? This code should only be executed on windows.

post-cleanup: 'none'

- name: Install ArcticDB from artifacts
shell: cmd /C call {0}
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Out of curiosity why have we decided to use a non bash shell? I think this makes integrating the windonws and unix variants harder.

else
cd python
python -m pytest --timeout=3600 -v --tb=line -n logical --dist worksteal \
-m "not lmdb" \
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do we skip lmdb tests on windows?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants