Merge pull request #9 from hdpriest-ui/add-ancillary-services

Add ancillary services

Author: hdpriest-ui

CHANGELOG.md

@@ -5,7 +5,14 @@ All notable changes to the ICRN Kernel Manager project will be documented in thi
 ## [Unreleased] - 2024-07-24
 
 ### Added
-- **Comprehensive Test Suite**: Added a complete test suite with 32 tests covering all major functionality
+- **Kernel Indexer**: New `kernel_indexer` script for indexing kernels and generating package manifests
+  - Indexes R and Python kernels and extracts package information
+  - Generates JSON manifest files for each kernel with package metadata
+  - Collates manifests into kernel-centric and package-centric indexes
+  - Supports filtering by language, kernel name, and version
+  - Extracts packages from both conda and language-specific package managers
+- **Comprehensive Test Suite**: Added a complete test suite with 52+ tests covering all major functionality
+  - `test_kernel_indexer.sh`: 20 tests covering kernel indexing and collation
 - **Test Isolation**: Each test now runs in its own isolated environment to prevent interference
 - **Mock Data**: Tests use consistent mock kernel packages and catalogs for reliable testing
 - **Error Handling Tests**: Added tests for both success and failure scenarios
@@ -28,15 +35,18 @@ All notable changes to the ICRN Kernel Manager project will be documented in thi
   - `test_update_r_libs.sh`: 6 tests covering R library management
   - `test_config.sh`: 10 tests covering configuration validation
   - `test_help.sh`: 3 tests covering help and basic commands
+  - `test_kernel_indexer.sh`: 20 tests covering kernel indexing and collation
 - **Test Environment**: Isolated environments created in `./tests/test_env/`
 - **Mock Data**: Sample R and Python kernels with proper catalog structure
+- **Mock Commands**: Tests include mock conda, Rscript, and python commands for isolated testing
 - **Error Messages**: Standardized error message format with "ERROR:" prefix
 
 ### Documentation
-- **Contributing Guide**: Added comprehensive testing section with examples
-- **Maintainer Guide**: Added testing recommendations for new kernels
+- **Contributing Guide**: Added comprehensive testing section with examples, including kernel_indexer test suite
+- **Maintainer Guide**: Added kernel indexing documentation for R and Python kernels
+- **Reference Documentation**: Added complete kernel_indexer command reference
 - **Troubleshooting**: Updated with improved error handling information
-- **README**: Added testing section with command examples
+- **README**: Added testing section with command examples, including kernel_indexer tests
 
 ## [Previous Releases]
 

README.md

@@ -144,6 +144,7 @@ The project includes a comprehensive test suite to ensure reliability and code q
 ./tests/run_tests.sh update_r_libs    # R library management  
 ./tests/run_tests.sh config           # Configuration validation
 ./tests/run_tests.sh help             # Help and basic commands
+./tests/run_tests.sh kernel_indexer   # Kernel indexing and collation
 ```
 
 **Test Features:**

copy-data-to-container.sh

@@ -0,0 +1,24 @@
+#!/bin/bash
+# Copy example data files to running Docker container
+# Run from root of git repository
+
+# Find container
+CONTAINER_ID=$(docker ps -q --filter "ancestor=icrn-web" | head -1)
+
+if [ -z "$CONTAINER_ID" ]; then
+    echo "Error: No running container found. Is the container running?"
+    exit 1
+fi
+
+echo "Found container: $CONTAINER_ID"
+echo "Copying files..."
+
+# Copy files
+docker cp web/examples/collated_manifests.json ${CONTAINER_ID}:/app/data/collated_manifests.json
+docker cp web/examples/package_index.json ${CONTAINER_ID}:/app/data/package_index.json
+
+echo "Files copied successfully!"
+echo "Triggering reload..."
+curl -X POST http://localhost:8080/api/refresh
+
+echo "Done! The website should now have data available."

documentation/source/contributing.rst

@@ -37,6 +37,7 @@ The project includes a comprehensive test suite to ensure code quality and relia
 - `jq` - JSON processor for test data validation
 - `tar` - For testing kernel packaging functionality
 - `timeout` - For testing command timeouts (usually available on Linux systems)
+- `conda` - Required by `kernel_indexer` (mocked in tests)
 
 **Running Tests:**
 
@@ -50,6 +51,7 @@ The project includes a comprehensive test suite to ensure code quality and relia
    ./tests/run_tests.sh update_r_libs    # R library management
    ./tests/run_tests.sh config           # Configuration validation
    ./tests/run_tests.sh help             # Help and basic commands
+   ./tests/run_tests.sh kernel_indexer   # Kernel indexing and collation
 
 **Test Structure:**
 - **Test Isolation**: Each test runs in its own isolated environment to prevent interference
@@ -84,6 +86,15 @@ The project includes a comprehensive test suite to ensure code quality and relia
    - Invalid command handling
    - Usage information display
 
+5. **Kernel Indexer** (`test_kernel_indexer.sh`)
+   - Command validation and argument parsing
+   - Kernel discovery and filtering
+   - R and Python kernel indexing
+   - Package extraction from conda and R
+   - Manifest generation and validation
+   - Kernel-centric and package-centric collation
+   - Error handling and edge cases
+
 **Test Environment:**
 - Tests create isolated environments in `./tests/test_env/`
 - Each test cleans up after itself

documentation/source/maintainer/python_kernels.rst

@@ -25,5 +25,39 @@ For detailed instructions on creating Python kernels, see the kernel creation do
 
    jupyter_pieces
 
+Indexing Kernels
+----------------
+
+The ``kernel_indexer`` script can be used to create package manifests and inventory all kernels in a repository. This is useful for:
+
+- Tracking which packages are included in each kernel
+- Generating package inventories for documentation
+- Creating indexes for package discovery
+
+To index all Python kernels in a repository:
+
+.. code-block:: bash
+
+   ./kernel_indexer index --kernel-root /path/to/repository --language Python
+
+This creates a ``package_manifest.json`` file in each kernel directory containing:
+- Kernel name and version
+- Language and language version
+- Complete list of packages with versions and sources
+- Indexing timestamp
+
+To create collated indexes for all kernels:
+
+.. code-block:: bash
+
+   # Create kernel-centric index (list of all kernels)
+   ./kernel_indexer collate-by-kernels --kernel-root /path/to/repository --language Python --output collated_manifests.json
+
+   # Create package-centric index (which kernels contain each package)
+   ./kernel_indexer collate-by-packages --kernel-root /path/to/repository --language Python --output package_index.json
+
+   # Or create both at once
+   ./kernel_indexer collate --kernel-root /path/to/repository --language Python --output-dir /path/to/output
+
 .. note::
    Detailed step-by-step instructions for creating Python kernels are available in the kernel creation documentation. 

documentation/source/maintainer/r_kernels.rst

@@ -31,4 +31,38 @@ Before distributing kernels, ensure they:
 - Pass validation tests
 
 .. note::
-   Detailed step-by-step instructions for creating R kernels are available in the kernel creation documentation. 
+   Detailed step-by-step instructions for creating R kernels are available in the kernel creation documentation.
+
+Indexing Kernels
+----------------
+
+The ``kernel_indexer`` script can be used to create package manifests and inventory all kernels in a repository. This is useful for:
+
+- Tracking which packages are included in each kernel
+- Generating package inventories for documentation
+- Creating indexes for package discovery
+
+To index all R kernels in a repository:
+
+.. code-block:: bash
+
+   ./kernel_indexer index --kernel-root /path/to/repository --language R
+
+This creates a ``package_manifest.json`` file in each kernel directory containing:
+- Kernel name and version
+- Language and language version
+- Complete list of packages with versions and sources
+- Indexing timestamp
+
+To create collated indexes for all kernels:
+
+.. code-block:: bash
+
+   # Create kernel-centric index (list of all kernels)
+   ./kernel_indexer collate-by-kernels --kernel-root /path/to/repository --language R --output collated_manifests.json
+
+   # Create package-centric index (which kernels contain each package)
+   ./kernel_indexer collate-by-packages --kernel-root /path/to/repository --language R --output package_index.json
+
+   # Or create both at once
+   ./kernel_indexer collate --kernel-root /path/to/repository --language R --output-dir /path/to/output 

documentation/source/reference.rst

@@ -71,8 +71,63 @@ Subcommands
 
   (Not yet implemented) Update a checked-out kernel to the latest version from the central repository.
 
+Kernel Indexer
+---------------
+
+The ``kernel_indexer`` script indexes kernels and generates package manifests for inventory and management purposes.
+
+**index**
+
+.. code-block:: bash
+
+   ./kernel_indexer index --kernel-root <PATH> [--language <LANG>] [--kernel-name <NAME>] [--kernel-version <VER>]
+
+  Index kernels and generate manifest files. Discovers all kernels in the repository and extracts package information from conda environments and language-specific package managers (e.g., R packages).
+
+  - ``--kernel-root``: Path to kernel repository root (required if not in config)
+  - ``--language``: Filter by specific language (R, Python). If omitted, processes all languages
+  - ``--kernel-name``: Index only a specific kernel (optional)
+  - ``--kernel-version``: Index only a specific version (requires --kernel-name)
+
+**collate-by-kernels**
+
+.. code-block:: bash
+
+   ./kernel_indexer collate-by-kernels --kernel-root <PATH> [--language <LANG>] [--output <PATH>]
+
+  Collate all manifest files into a kernel-centric index. Creates a JSON file listing all indexed kernels with their metadata and package counts.
+
+  - ``--kernel-root``: Path to kernel repository root (required if not in config)
+  - ``--language``: Filter by specific language (R, Python). If omitted, processes all languages
+  - ``--output``: Path for collated manifest (default: {kernel-root}/collated_manifests.json)
+
+**collate-by-packages**
+
+.. code-block:: bash
+
+   ./kernel_indexer collate-by-packages --kernel-root <PATH> [--language <LANG>] [--output <PATH>]
+
+  Collate manifests into a package-centric index. Creates a JSON file listing all packages with information about which kernels contain them.
+
+  - ``--kernel-root``: Path to kernel repository root (required if not in config)
+  - ``--language``: Filter by specific language (R, Python). If omitted, processes all languages
+  - ``--output``: Path for package-centric index (default: {kernel-root}/package_index.json)
+
+**collate**
+
+.. code-block:: bash
+
+   ./kernel_indexer collate --kernel-root <PATH> [--language <LANG>] [--output-dir <DIR>]
+
+  Run both collate-by-kernels and collate-by-packages operations. Creates both index files in the specified output directory.
+
+  - ``--kernel-root``: Path to kernel repository root (required if not in config)
+  - ``--language``: Filter by specific language (R, Python). If omitted, processes all languages
+  - ``--output-dir``: Directory for output files (default: {kernel-root})
+
 For help, run:
 
 .. code-block:: bash
 
-   ./icrn_manager help 
+   ./icrn_manager help
+   ./kernel_indexer