Skip to content

Fix: Remove glob character escaping for SCP RCP protocol#7335

Merged
khsrali merged 3 commits into
aiidateam:mainfrom
stillfast:fix/scp-glob-escaping
May 13, 2026
Merged

Fix: Remove glob character escaping for SCP RCP protocol#7335
khsrali merged 3 commits into
aiidateam:mainfrom
stillfast:fix/scp-glob-escaping

Conversation

@stillfast
Copy link
Copy Markdown
Contributor

Problem:

  • When using OpenSSH < 9.0, scp uses the RCP protocol
  • RCP protocol passes paths to remote shell for glob expansion
  • Escaping glob characters (*, ?, []) breaks file pattern matching
  • This causes SCP to fail with returncode=1 when copying files with glob paths

Solution:

  • Remove ALL glob characters from the special character escape set in _escape_for_rcp()
  • Glob characters should be passed to remote shell for proper expansion
  • All three glob metacharacters are preserved:
    • (asterisk) - matches any sequence of characters ? (question mark) - matches any single character [] (square brackets) - matches character classes/ranges

Changes:

  1. Modified _escape_for_rcp() to not escape glob characters (*, ?, [])
  2. Updated docstring in _escape_for_rcp() to explain why glob chars are preserved
  3. Enhanced docstring in _escape_for_scp() to clearly document version-specific behavior:
    • OpenSSH 9.0+ uses SFTP mode (no escaping needed)
    • OpenSSH < 9.0 uses RCP mode (preserve glob chars for shell expansion)

Testing:

  • Verified on OpenSSH 8.9 (Ubuntu 22.04)
  • Successfully copies remote files with glob patterns (*.txt, etc.)
  • OpenSSH 9.0+ unaffected (uses SFTP protocol)

Compatibility:

  • Fully compatible with both OpenSSH < 9.0 and >= 9.0
  • Version detection is based on is_openssh_9_or_higher flag

@codecov
Copy link
Copy Markdown

codecov Bot commented Apr 30, 2026

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 80.28%. Comparing base (2068a7d) to head (615e2c6).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #7335      +/-   ##
==========================================
- Coverage   80.28%   80.28%   -0.00%     
==========================================
  Files         577      577              
  Lines       45545    45545              
==========================================
- Hits        36562    36560       -2     
- Misses       8983     8985       +2     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@khsrali khsrali linked an issue Apr 30, 2026 that may be closed by this pull request
梦有三千州 and others added 2 commits April 30, 2026 11:56
Problem:
- When using OpenSSH < 9.0, scp uses the RCP protocol
- RCP protocol passes paths to remote shell for glob expansion
- Escaping glob characters (*, ?, []) breaks file pattern matching
- This causes SCP to fail with returncode=1 when copying files with glob paths

Solution:
- Remove ALL glob characters from the special character escape set in _escape_for_rcp()
- Glob characters should be passed to remote shell for proper expansion
- All three glob metacharacters are preserved:
  * (asterisk) - matches any sequence of characters
  ? (question mark) - matches any single character
  [] (square brackets) - matches character classes/ranges

Changes:
1. Modified _escape_for_rcp() to not escape glob characters (*, ?, [])
2. Updated docstring in _escape_for_rcp() to explain why glob chars are preserved
3. Enhanced docstring in _escape_for_scp() to clearly document version-specific behavior:
   - OpenSSH 9.0+ uses SFTP mode (no escaping needed)
   - OpenSSH < 9.0 uses RCP mode (preserve glob chars for shell expansion)

Testing:
- Verified on OpenSSH 8.9 (Ubuntu 22.04)
- Successfully copies remote files with glob patterns (*.txt, etc.)
- OpenSSH 9.0+ unaffected (uses SFTP protocol)

Compatibility:
- Fully compatible with both OpenSSH < 9.0 and >= 9.0
- Version detection is based on is_openssh_9_or_higher flag
@khsrali khsrali force-pushed the fix/scp-glob-escaping branch from 8a77ba6 to 4d74294 Compare April 30, 2026 09:56
Copy link
Copy Markdown
Collaborator

@khsrali khsrali left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks a lot @stillfast

@khsrali
Copy link
Copy Markdown
Collaborator

khsrali commented May 13, 2026

I thought we merged this, still here 😬

@khsrali khsrali disabled auto-merge May 13, 2026 07:42
@stillfast
Copy link
Copy Markdown
Contributor Author

Sorry, I will fix it as soon as possible. Let me try again @khsrali

@khsrali
Copy link
Copy Markdown
Collaborator

khsrali commented May 13, 2026

@stillfast
what are you trying to fix?

I think it's all good, the failing test might have been flaky. let's give it another 20 min, once passed I'll merge it

@stillfast
Copy link
Copy Markdown
Contributor Author

OK, I used to think we should change the code according to the failing test. It is fine if it pass.

@khsrali khsrali merged commit e3aae79 into aiidateam:main May 13, 2026
25 of 26 checks passed
@khsrali
Copy link
Copy Markdown
Collaborator

khsrali commented May 13, 2026

Thanks a lot @stillfast for finding the bug and for your contribution!

@stillfast stillfast deleted the fix/scp-glob-escaping branch May 13, 2026 08:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

SCP fails with glob patterns on OpenSSH < 9.0

2 participants