Merge pull request #20924 from davelopez/explore_url_fetch_with_headers

Add Support for HTTP Headers in URL Fetch Requests with Secure Storage for Landing Requests

Author: mvdbeek

client/src/api/schema/schema.ts

@@ -23533,6 +23533,13 @@ export interface components {
             extra_files?: components["schemas"]["ExtraFiles"] | null;
             /** Hashes */
             hashes?: components["schemas"]["FetchDatasetHash"][] | null;
+            /**
+             * Headers
+             * @description Optional headers to include in the URL fetch request
+             */
+            headers?: {
+                [key: string]: string;
+            } | null;
             /**
              * Info
              * @description Free text field that can be used to store arbitrary information about the dataset. This used to be prominently

config/url_headers_conf.yml.sample

@@ -0,0 +1 @@
+../lib/galaxy/config/sample/url_headers_conf.yml.sample

doc/source/admin/enable_headers_in_fetch_requests.md

@@ -0,0 +1,152 @@
+# Enabling HTTP Headers in Fetch Requests
+
+Galaxy allows users to **fetch remote data by URL** (for example via _Upload → Paste/Fetch data_ or via APIs that retrieve external resources).
+By default, Galaxy **does not forward any custom HTTP headers** when fetching URLs. This restriction is intentional and is part of Galaxy’s security model.
+
+Starting with recent Galaxy releases, administrators can **explicitly allow a controlled set of HTTP headers** to be sent with fetch requests, based on the target URL. This enables integrations with authenticated services (e.g. APIs requiring `Authorization` headers) while maintaining strict security boundaries.
+
+This document explains **how to safely enable HTTP headers for fetch requests**, how the allow‑list mechanism works, and how to configure it.
+
+## Why Header Allow‑Listing Is Required
+
+Allowing arbitrary headers in server‑side HTTP requests is dangerous. Without restrictions, users could:
+
+- Access internal services (SSRF attacks)
+- Exfiltrate credentials via forwarded headers
+- Abuse Galaxy as a proxy to privileged networks
+
+To prevent this, Galaxy implements **explicit header allow‑listing with URL pattern matching**:
+
+- **No headers are allowed by default**
+- Each allowed header must be explicitly configured
+- Headers are only sent to URLs that match defined patterns
+- Sensitive headers can be stored securely using Galaxy’s Vault
+
+## Configuration Overview
+
+Header forwarding for fetch requests is controlled via a dedicated configuration file:
+
+```yaml
+galaxy:
+  url_headers_config_file: url_headers_conf.yml
+```
+
+This file defines:
+
+- Which **HTTP headers** are allowed
+- For which **URL patterns** they may be sent
+- Whether headers are **sensitive** (stored encrypted in the Vault)
+
+If this configuration file is **not set or empty**, **no headers will ever be forwarded**.
+
+## url_headers_conf.yml Format
+
+The configuration file is a YAML list of rules. Each rule applies to one or more URL patterns.
+
+### Basic Structure
+
+```yaml
+- url_pattern: "https://api.example.org/.*"
+  headers:
+    - name: Authorization
+      sensitive: true
+    - name: X-API-Key
+      sensitive: true
+```
+
+### Fields
+
+| Field                 | Description                                              |
+| --------------------- | -------------------------------------------------------- |
+| `url_pattern`         | A regular expression matched against the full URL        |
+| `headers`             | List of allowed HTTP headers for matching URLs           |
+| `headers[].name`      | Exact HTTP header name (case‑insensitive)                |
+| `headers[].sensitive` | Whether the header value is stored securely in the Vault |
+
+## Sensitive vs Non‑Sensitive Headers
+
+### Sensitive Headers
+
+Sensitive headers (for example `Authorization`, `X-API-Key`, `Cookie`) are:
+
+- **Encrypted and stored in the Galaxy Vault**
+- Never logged or exposed in plaintext
+- Managed through Galaxy’s secure secrets infrastructure
+
+Example:
+
+```yaml
+- url_pattern: "https://protected.example.com/.*"
+  headers:
+    - name: Authorization
+      sensitive: true
+```
+
+### Non‑Sensitive Headers
+
+Non‑sensitive headers may be stored in plain configuration and are typically used for:
+
+- Feature flags
+- API versioning
+- Public metadata headers
+
+Example:
+
+```yaml
+- url_pattern: "https://public.example.com/.*"
+  headers:
+    - name: X-Client-Version
+      sensitive: false
+```
+
+## Multiple Rules and URL Matching
+
+Multiple rules may be defined. The first rule whose `url_pattern` matches the request URL is applied.
+
+```yaml
+- url_pattern: "https://api.github.com/.*"
+  headers:
+    - name: Authorization
+      sensitive: true
+
+- url_pattern: "https://raw.githubusercontent.com/.*"
+  headers:
+    - name: X-Client-Version
+      sensitive: false
+```
+
+```{note}
+Rules are evaluated in order. Be careful with overly broad patterns such as `.*`.
+```
+
+## Using Headers in Practice
+
+Once configured, users (or tools) may provide header values when performing fetch operations. Galaxy will:
+
+1. Validate the target URL against the allow‑list
+2. Filter headers to the allowed set
+3. Securely inject sensitive headers at request time
+
+Headers not explicitly allowed **will be silently dropped**.
+
+## Security Best Practices
+
+```{warning}
+Only allow headers and URL patterns that are strictly necessary.
+```
+
+Recommended practices:
+
+- Prefer **narrow URL patterns** over wildcards
+- Mark authentication headers as `sensitive: true`
+- Avoid allowing `Cookie` headers unless absolutely required
+- Never allow headers for internal or private network ranges
+
+## Troubleshooting
+
+If headers are not being forwarded as expected:
+
+1. Verify `url_headers_config_file` is configured in `galaxy.yml`
+2. Confirm the URL matches the configured `url_pattern`
+3. Check that the header name matches exactly
+4. Ensure Galaxy has access to the configured Vault

doc/source/admin/galaxy_options.rst

@@ -5638,6 +5638,26 @@
 :Type: str
 
 
+~~~~~~~~~~~~~~~~~~~~~~~~~~~
+``url_headers_config_file``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+:Description:
+    Configuration file for URL request headers allow-list with URL
+    pattern matching. This file defines which HTTP headers are allowed
+    in URL fetch requests based on URL patterns, and whether they
+    should be treated as sensitive (encrypted in the vault) or not. If
+    no allow-list is specified, no headers will be allowed in URL
+    requests. This provides fine-grained security control over what
+    headers can be sent when Galaxy fetches external URLs on behalf of
+    users, allowing different headers for different target domains or
+    services.
+    The value of this option will be resolved with respect to
+    <config_dir>.
+:Default: ``url_headers_conf.yml``
+:Type: str
+
+
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 ``display_builtin_converters``
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

doc/source/admin/index.rst

@@ -18,6 +18,7 @@ Galaxy Deployment & Administration
    jobs
    job_metrics
    authentication
+   enable_headers_in_fetch_requests
    tool_panel
    data_tables
    mq

lib/galaxy/app_unittest_utils/galaxy_mock.py

@@ -291,6 +291,7 @@ def __init__(self, **kwargs):
         self.monitor_thread_join_timeout = 1
         self.integrated_tool_panel_config = None
         self.vault_config_file = kwargs.get("vault_config_file")
+        self.url_headers_config_file = None
         self.max_discovered_files = 10000
         self.display_builtin_converters = True
         self.enable_notification_system = True

lib/galaxy/config/sample/galaxy.yml.sample

@@ -3048,6 +3048,18 @@ galaxy:
   # <config_dir>.
   #vault_config_file: vault_conf.yml
 
+  # Configuration file for URL request headers allow-list with URL
+  # pattern matching. This file defines which HTTP headers are allowed
+  # in URL fetch requests based on URL patterns, and whether they should
+  # be treated as sensitive (encrypted in the vault) or not. If no
+  # allow-list is specified, no headers will be allowed in URL requests.
+  # This provides fine-grained security control over what headers can be
+  # sent when Galaxy fetches external URLs on behalf of users, allowing
+  # different headers for different target domains or services.
+  # The value of this option will be resolved with respect to
+  # <config_dir>.
+  #url_headers_config_file: url_headers_conf.yml
+
   # Display built-in converters in the tool panel.
   #display_builtin_converters: true
 

lib/galaxy/config/sample/url_headers_conf.yml.sample

@@ -0,0 +1,115 @@
+# Allowed URL Headers Configuration
+#
+# This file defines which HTTP headers are allowed in URL fetch requests based
+# on URL patterns, and whether they should be treated as sensitive (encrypted
+# in the vault) or not.
+#
+# If no allow-list is specified or this file is empty/missing, NO headers will
+# be allowed in URL requests.
+#
+# Configuration structure:
+# patterns:
+#   - url_pattern: A regular expression pattern to match URLs
+#     headers:
+#       - name: The exact header name (case-insensitive)
+#         sensitive: Whether this header contains sensitive information that should
+#                    be encrypted when stored in the database (requires vault configuration)
+#
+# IMPORTANT:
+# ------------------------------------
+# When a URL matches MULTIPLE patterns, the union of all allowed headers is used.
+# This means you can compose permissions from multiple patterns for flexibility.
+#
+# Example: A URL matching both pattern A (allows headers X, Y) and pattern B
+# (allows headers Y, Z) will allow headers X, Y, and Z.
+#
+# Security: If ANY matching pattern marks a header as sensitive, it will be
+# treated as sensitive (secure-by-default).
+#
+# The following examples are for illustration purposes only; please use only the minimum configuration for your needs.
+# Examples:
+
+patterns:
+  # GitHub API access - allow authentication headers for GitHub URLs
+  - url_pattern: "^https://api\\.github\\.com/.*"
+    headers:
+      - name: Authorization
+        sensitive: true
+      - name: Accept
+        sensitive: false
+      - name: X-GitHub-Api-Version
+        sensitive: false
+
+  # Generic GitHub content (raw files, releases) - no auth needed
+  - url_pattern: "^https://(raw\\.githubusercontent\\.com|github\\.com/.*/releases/download)/.*"
+    headers:
+      - name: Accept
+        sensitive: false
+      - name: Accept-Encoding
+        sensitive: false
+
+  # AWS S3 buckets - allow AWS authentication headers
+  - url_pattern: "^https://.*\\.s3\\..+\\.amazonaws\\.com/.*"
+    headers:
+      - name: Authorization
+        sensitive: true
+      - name: X-Amz-Date
+        sensitive: false
+      - name: X-Amz-Content-Sha256
+        sensitive: false
+      - name: X-Amz-Security-Token
+        sensitive: true
+
+  # Generic cloud storage APIs
+  - url_pattern: "^https://.*\\.(googleapis\\.com|azure\\.com|digitaloceanspaces\\.com)/.*"
+    headers:
+      - name: Authorization
+        sensitive: true
+      - name: X-API-Key
+        sensitive: true
+      - name: Accept
+        sensitive: false
+
+  # FTP over HTTP services
+  - url_pattern: "^https?://ftp\\..*/.*"
+    headers:
+      - name: Authorization
+        sensitive: true
+      - name: Accept
+        sensitive: false
+
+  # Academic/research data repositories
+  - url_pattern: "^https://.*(zenodo\\.org|figshare\\.com|dryad\\.org|dataverse\\.org)/.*"
+    headers:
+      - name: Authorization
+        sensitive: true
+      - name: X-API-Key
+        sensitive: true
+      - name: Accept
+        sensitive: false
+
+  # HTTPS URLs - basic headers only (most restrictive for unknown sources)
+  - url_pattern: "^https://.*"
+    headers:
+      - name: Authorization
+        sensitive: true
+      - name: X-Auth-Token
+        sensitive: true
+      - name: X-API-Key
+        sensitive: true
+      - name: Accept
+        sensitive: false
+      - name: Accept-Language
+        sensitive: false
+      - name: Accept-Encoding
+        sensitive: false
+      - name: Cache-Control
+        sensitive: false
+
+# Security notes:
+# - All matching patterns contribute their allowed headers (union of permissions)
+# - If ANY pattern marks a header as sensitive, it's treated as sensitive
+# - Only add headers that are absolutely necessary for your use case
+# - When in doubt, mark headers as sensitive to ensure encryption
+# - Patterns are order-independent, making configuration more composable
+# - HTTP (non-HTTPS) URLs are generally not recommended and may be blocked

lib/galaxy/config/schemas/config_schema.yml

@@ -4172,6 +4172,20 @@ mapping:
         desc: |
           Vault config file.
 
+      url_headers_config_file:
+        type: str
+        default: url_headers_conf.yml
+        path_resolves_to: config_dir
+        required: false
+        desc: |
+          Configuration file for URL request headers allow-list with URL pattern matching.
+          This file defines which HTTP headers are allowed in URL fetch requests based
+          on URL patterns, and whether they should be treated as sensitive (encrypted
+          in the vault) or not. If no allow-list is specified, no headers will be
+          allowed in URL requests. This provides fine-grained security control over
+          what headers can be sent when Galaxy fetches external URLs on behalf of users,
+          allowing different headers for different target domains or services.
+
       display_builtin_converters:
         type: bool
         default: true