Since 1.28.0, we have been finding subtle yet impossible to debug issues in our CI, where we get nonsensical results like this when trying to call Cargo:
Traceback (most recent call last):
File "<string>", line 24, in <module>
File "C:\Users\Administrator\runner\builds\amyspark\cerbero\cerbero\main.py", line 224, in main
Main(sys.argv[1:])
File "C:\Users\Administrator\runner\builds\amyspark\cerbero\cerbero\main.py", line 52, in __init__
self.run_command()
File "C:\Users\Administrator\runner\builds\amyspark\cerbero\cerbero\main.py", line 192, in run_command
res = commands.run(command, self.config, self.args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\Administrator\runner\builds\amyspark\cerbero\cerbero\commands\__init__.py", line 79, in run
return _commands[command].run(config, args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\Administrator\runner\builds\amyspark\cerbero\cerbero\commands\bootstrap.py", line 179, in run
run_until_complete(build_tools_task)
File "C:\Users\Administrator\runner\builds\amyspark\cerbero\cerbero\utils\__init__.py", line 763, in run_until_complete
result = loop.run_until_complete(tasks)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Python311\Lib\asyncio\base_events.py", line 653, in run_until_complete
return future.result()
^^^^^^^^^^^^^^^
File "C:\Users\Administrator\runner\builds\amyspark\cerbero\cerbero\bootstrap\build_tools.py", line 145, in fetch_recipes
await Fetch.fetch(self.cookbook, self.recipes, False, False, False, False, jobs)
File "C:\Users\Administrator\runner\builds\amyspark\cerbero\cerbero\commands\fetch.py", line 112, in fetch
await fetch_print_wrapper(recipe.name, stepfunc)
File "C:\Users\Administrator\runner\builds\amyspark\cerbero\cerbero\commands\fetch.py", line 100, in fetch_print_wrapper
await stepfunc()
File "C:\Users\Administrator\runner\builds\amyspark\cerbero\cerbero\build\recipe.py", line 102, in async_wrapped
ret = await stepfunc(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\Administrator\runner\builds\amyspark\cerbero\cerbero\build\source.py", line 456, in fetch
await self.extract_impl(fetching=True)
File "C:\Users\Administrator\runner\builds\amyspark\cerbero\cerbero\build\source.py", line 487, in extract_impl
await self.cargo_vendor(not fetching or self.offline)
File "C:\Users\Administrator\runner\builds\amyspark\cerbero\cerbero\build\source.py", line 137, in cargo_vendor
ct = await shell.async_call_output(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\Administrator\runner\builds\amyspark\cerbero\cerbero\utils\shell.py", line 345, in async_call_output
raise e
File "C:\Users\Administrator\runner\builds\amyspark\cerbero\cerbero\utils\shell.py", line 328, in async_call_output
proc = await asyncio.create_subprocess_exec(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Python311\Lib\asyncio\subprocess.py", line 218, in create_subprocess_exec
transport, protocol = await loop.subprocess_exec(
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Python311\Lib\asyncio\base_events.py", line 1694, in subprocess_exec
transport = await self._make_subprocess_transport(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Python311\Lib\asyncio\windows_events.py", line 399, in _make_subprocess_transport
transp = _WindowsSubprocessTransport(self, protocol, args, shell,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Python311\Lib\asyncio\base_subprocess.py", line 36, in __init__
self._start(args=args, shell=shell, stdin=stdin, stdout=stdout,
File "C:\Python311\Lib\asyncio\windows_events.py", line 921, in _start
self._proc = windows_utils.Popen(
^^^^^^^^^^^^^^^^^^^^
File "C:\Python311\Lib\asyncio\windows_utils.py", line 153, in __init__
super().__init__(args, stdin=stdin_rfd, stdout=stdout_wfd,
File "C:\Python311\Lib\subprocess.py", line 1024, in __init__
self._execute_child(args, executable, preexec_fn, close_fds,
File "C:\Python311\Lib\subprocess.py", line 1493, in _execute_child
hp, ht, pid, tid = _winapi.CreateProcess(executable, args,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [WinError 3] The system cannot find the path specified
C:/Users/Administrator/runner/builds/amyspark/cerbero/cb/rust/cargo/bin/cargo.exe vendor C:/Users/Administrator/runner/builds/amyspark/cerbero/cerbero-sources/bindgen-cli-0.71.1/cargo-vendor C:/Users/Administrator/runner/builds/amyspark/cerbero/cb/sources/build-tools/bindgen-cli-0.71.1
C:/Users/Administrator/runner/builds/amyspark/cerbero/cb/rust/cargo/bin/cargo.exe True
['cargo-clippy.exe', 'cargo-fmt.exe', 'cargo-miri.exe', 'cargo.exe', 'clippy-driver.exe', 'rls.exe', 'rust-analyzer.exe', 'rust-gdb.exe', 'rust-gdbgui.exe', 'rust-lldb.exe', 'rustc.exe', 'rustdoc.exe', 'rustfmt.exe', 'rustup.exe']
Where Cargo clearly exists, yet is impossible to be executed. It's simple to notice why, once the files are captured:
(We don't log `rustup --version` but I can verify it's 1.28.1 for x86_64-pc-windows-msvc)
(Same as above: stable-x86_64-pc-windows-msvc)
Verification
Problem
Since 1.28.0, we have been finding subtle yet impossible to debug issues in our CI, where we get nonsensical results like this when trying to call Cargo:
Where Cargo clearly exists, yet is impossible to be executed. It's simple to notice why, once the files are captured:
I tracked down the regression to this: rustup uses symlinks on Windows: #3677
Steps
Possible Solution(s)
Revert the commit 7e644c6.
Notes
Build log here: https://gitlab.freedesktop.org/amyspark/cerbero/-/jobs/74755751
Rustup version
(We don't log `rustup --version` but I can verify it's 1.28.1 for x86_64-pc-windows-msvc)Installed toolchains
(Same as above: stable-x86_64-pc-windows-msvc)OS version
Windows Server LTSC 2022