Skip to content

Commit c118ebf

Browse files
hdpriest-uihdpriest-ui
authored
Merge pull request #18 from hdpriest-ui/develop
Develop
2 parents 50209d4 + bb30fb5 commit c118ebf

15 files changed

Lines changed: 721 additions & 46 deletions

‎README.md‎

Lines changed: 0 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -29,16 +29,6 @@ CLI Tool & Web UI → Users discover and manage kernels
2929

3030

3131
## User Installation
32-
```sh
33-
./icrn_manager kernels init <path to central repository>
34-
```
35-
Example for development work:
36-
```sh
37-
./icrn_manager kernels init /u/hdpriest/icrn_temp_repository
38-
```
39-
<img src="documentation/demo_resources/icrn_manage_init.gif" align="center" width="600"/>
40-
41-
This command can be run again to point at a different central repository, without disrupting the user's files.
4232

4333
## Usage
4434
The ICRN Kernel Manager has all familiar operations for listing, getting, and using entites from a catalog or repository of kernel packages organized by language:

‎documentation/source/administrator/deployment.rst‎

Lines changed: 78 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,78 @@
1+
Deployment Overview
2+
===================
3+
4+
The ICRN Kernel Manager consists of three main components that must be deployed:
5+
6+
1. **CLI Tool** (``icrn_manager``): Installed on user-accessible systems
7+
2. **Kernel Indexer**: Containerized service that indexes kernel repositories
8+
3. **Web Server**: FastAPI + Nginx service providing the web interface and API
9+
10+
Architecture
11+
------------
12+
13+
.. code-block:: text
14+
15+
Kernel Repository (shared filesystem)
16+
↓ (read-write mount)
17+
Kernel Indexer (CronJob) → Generates collated_manifests.json & package_index.json
18+
↓ (JSON files)
19+
Web Server (reads JSON files) → Serves REST API
20+
↓
21+
CLI Tool & RStudio Addin → Users discover and manage kernels
22+
23+
Prerequisites
24+
-------------
25+
26+
- Shared filesystem accessible to all components
27+
- Container runtime (Docker or Apptainer/Singularity)
28+
- Kubernetes cluster (for production deployment)
29+
- Network access between components
30+
31+
Deploying the CLI Tool
32+
----------------------
33+
34+
Copy the tools to a location in the system PATH:
35+
36+
.. code-block:: bash
37+
38+
# Production paths
39+
cp ./icrn_manager /sw/icrn/prod/bin/
40+
cp ./update_r_libs.sh /sw/icrn/prod/bin/
41+
chmod +x /sw/icrn/prod/bin/icrn_manager
42+
chmod +x /sw/icrn/prod/bin/update_r_libs.sh
43+
44+
# Development paths
45+
cp ./icrn_manager /sw/icrn/dev/bin/
46+
cp ./update_r_libs.sh /sw/icrn/dev/bin/
47+
chmod +x /sw/icrn/dev/bin/icrn_manager
48+
chmod +x /sw/icrn/dev/bin/update_r_libs.sh
49+
50+
Container Requirements
51+
~~~~~~~~~~~~~~~~~~~~~~
52+
53+
Ensure user containers have ``jq`` installed, as it is required by ``icrn_manager``.
54+
55+
Environment Configuration
56+
-------------------------
57+
58+
The kernel repository path is configured in the ``icrn_manager`` script. Update the ``KERNEL_FOLDER`` variable or configure via environment:
59+
60+
.. code-block:: bash
61+
62+
# Default paths used by icrn_manager
63+
ICRN_USER_BASE=${ICRN_USER_BASE:-${HOME}/.icrn}
64+
ICRN_MANAGER_CONFIG=${ICRN_MANAGER_CONFIG:-${ICRN_USER_BASE}/manager_config.json}
65+
ICRN_USER_KERNEL_BASE=${ICRN_USER_KERNEL_BASE:-${ICRN_USER_BASE}/icrn_kernels}
66+
67+
Kubernetes Deployment
68+
---------------------
69+
70+
For Kubernetes deployment configurations, see the ``kubernetes/`` directory in the repository.
71+
72+
Key resources:
73+
74+
- Kernel Indexer CronJob
75+
- Web Server Deployment and Service
76+
- ConfigMaps for environment configuration
77+
- PersistentVolumeClaims for kernel storage
78+

‎documentation/source/administrator/index.rst‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
Administrator Guide
2+
===================
3+
4+
This section provides guidance for administrators deploying and maintaining the ICRN Kernel Manager infrastructure.
5+
6+
.. toctree::
7+
:maxdepth: 2
8+
9+
deployment
10+
kernel_indexer
11+
web_server
12+

‎documentation/source/administrator/kernel_indexer.rst‎

Lines changed: 91 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,91 @@
1+
Kernel Indexer
2+
==============
3+
4+
The kernel indexer is a containerized service that scans kernel repositories and generates JSON manifest files for the web interface and API.
5+
6+
Output Files
7+
------------
8+
9+
The indexer generates two key files:
10+
11+
``collated_manifests.json``
12+
Kernel-centric index listing all available kernels with their packages
13+
14+
``package_index.json``
15+
Package-centric index showing which kernels contain each package
16+
17+
Building the Container
18+
----------------------
19+
20+
.. code-block:: bash
21+
22+
docker build -t icrn-kernel-indexer:latest -f kernel-indexer/Dockerfile .
23+
24+
Running the Indexer
25+
-------------------
26+
27+
Docker
28+
~~~~~~
29+
30+
The indexer requires read-write access to the kernel repository:
31+
32+
.. code-block:: bash
33+
34+
docker run --rm \
35+
-v /path/to/kernel/repository:/sw/icrn/jupyter/icrn_ncsa_resources/Kernels \
36+
-e KERNEL_ROOT=/sw/icrn/jupyter/icrn_ncsa_resources/Kernels \
37+
icrn-kernel-indexer:latest
38+
39+
Apptainer/Singularity
40+
~~~~~~~~~~~~~~~~~~~~~
41+
42+
For HPC environments using Apptainer:
43+
44+
.. code-block:: bash
45+
46+
# Pull the container
47+
apptainer pull docker://hdpriest0uiuc/icrn-kernel-indexer
48+
49+
# Run with custom kernel root (development)
50+
apptainer run \
51+
--env "KERNEL_ROOT=/sw/icrn/dev/kernels" \
52+
--bind /sw/icrn/dev/kernels:/sw/icrn/dev/kernels \
53+
icrn-kernel-indexer_latest.sif
54+
55+
# Run with default paths (production)
56+
apptainer run \
57+
--bind /sw/icrn/prod/kernels:/sw/icrn/prod/kernels \
58+
icrn-kernel-indexer_latest.sif
59+
60+
Environment Variables
61+
---------------------
62+
63+
``KERNEL_ROOT``
64+
Path to the root directory containing kernel subdirectories. Defaults to ``/sw/icrn/prod/kernels``.
65+
66+
Kubernetes CronJob
67+
------------------
68+
69+
For production, deploy the indexer as a Kubernetes CronJob to automatically re-index kernels on a schedule.
70+
71+
See ``kernel-indexer/README.md`` for detailed Kubernetes deployment instructions.
72+
73+
Manual Indexing
74+
---------------
75+
76+
The ``kernel_indexer`` script can also be run directly for manual operations:
77+
78+
.. code-block:: bash
79+
80+
# Index all R kernels
81+
./kernel_indexer index --kernel-root /path/to/repository --language R
82+
83+
# Create kernel-centric index
84+
./kernel_indexer collate-by-kernels --kernel-root /path/to/repository --language R --output collated_manifests.json
85+
86+
# Create package-centric index
87+
./kernel_indexer collate-by-packages --kernel-root /path/to/repository --language R --output package_index.json
88+
89+
# Create both indexes at once
90+
./kernel_indexer collate --kernel-root /path/to/repository --language R --output-dir /path/to/output
91+

‎documentation/source/administrator/web_server.rst‎

Lines changed: 111 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,111 @@
1+
Web Server
2+
==========
3+
4+
The ICRN Kernel Manager web server provides a browser-based interface and REST API for exploring available kernels.
5+
6+
Prerequisites
7+
-------------
8+
9+
The web server requires the JSON files generated by the kernel indexer:
10+
11+
- ``collated_manifests.json``
12+
- ``package_index.json``
13+
14+
Building the Container
15+
----------------------
16+
17+
.. code-block:: bash
18+
19+
docker build -t icrn-kernel-webserver:latest -f web/Dockerfile web/
20+
21+
Running the Web Server
22+
----------------------
23+
24+
Docker with Mounted Data
25+
~~~~~~~~~~~~~~~~~~~~~~~~
26+
27+
.. code-block:: bash
28+
29+
docker run -d -p 8080:80 --name icrn-web \
30+
-v /path/to/kernel/repository:/app/data:ro \
31+
-e COLLATED_MANIFESTS_PATH=/app/data/collated_manifests.json \
32+
-e PACKAGE_INDEX_PATH=/app/data/package_index.json \
33+
icrn-kernel-webserver:latest
34+
35+
Docker with Example Data
36+
~~~~~~~~~~~~~~~~~~~~~~~~
37+
38+
.. code-block:: bash
39+
40+
docker run -d -p 8080:80 --name icrn-web icrn-kernel-webserver:latest
41+
docker cp web/examples/collated_manifests.json icrn-web:/app/data/
42+
docker cp web/examples/package_index.json icrn-web:/app/data/
43+
44+
Environment Variables
45+
---------------------
46+
47+
``COLLATED_MANIFESTS_PATH``
48+
Path to the kernel-centric manifest file. Default: ``/app/data/collated_manifests.json``
49+
50+
``PACKAGE_INDEX_PATH``
51+
Path to the package-centric index file. Default: ``/app/data/package_index.json``
52+
53+
API Endpoints
54+
-------------
55+
56+
The web server exposes the following REST API endpoints:
57+
58+
.. list-table::
59+
:header-rows: 1
60+
:widths: 30 70
61+
62+
* - Endpoint
63+
- Description
64+
* - ``GET /``
65+
- API information and status
66+
* - ``GET /health``
67+
- Health check endpoint
68+
* - ``GET /api/languages``
69+
- List of available languages
70+
* - ``GET /api/kernels/{language}``
71+
- All kernels for a specific language
72+
* - ``GET /api/kernel/{language}/{kernel_name}/{version}``
73+
- Specific kernel details
74+
* - ``GET /api/manifest/{language}/{kernel_name}/{version}``
75+
- Package manifest for a kernel
76+
* - ``GET /api/package/{package_name}``
77+
- All kernels containing a package
78+
* - ``GET /api/packages/search``
79+
- Search for packages by name
80+
* - ``POST /api/refresh``
81+
- Manually trigger data refresh
82+
83+
Kubernetes Service
84+
------------------
85+
86+
For the RStudio Addin to communicate with the web server, deploy it as a Kubernetes Service:
87+
88+
.. code-block:: yaml
89+
90+
apiVersion: v1
91+
kind: Service
92+
metadata:
93+
name: icrn-web-service
94+
namespace: kernels
95+
spec:
96+
selector:
97+
app: icrn-web
98+
ports:
99+
- port: 80
100+
targetPort: 80
101+
102+
The RStudio Addin connects to ``http://icrn-web-service.kernels:80`` by default.
103+
104+
Public URLs
105+
-----------
106+
107+
Configure ingress or load balancers to expose the web interface:
108+
109+
- **Production**: https://kernels.ncsa.illinois.edu
110+
- **Development**: https://kernels.cori-dev.ncsa.illinois.edu
111+

0 commit comments

Comments
 (0)
⚡