Skip to content

Commit 603c4f4

Browse files
committed
updating docs
1 parent b5cacbe commit 603c4f4

File tree

2 files changed

+192
-0
lines changed

2 files changed

+192
-0
lines changed

docs/guide.md

Lines changed: 188 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,188 @@
1+
# User guide
2+
3+
## Installation
4+
5+
In time, this will be on the [Python Package Index]() but for now you need to use the
6+
repository url to get it:
7+
8+
=== "uv"
9+
10+
```shell
11+
uv add https://github.com/geobeyond/pygeoapi-prefect --branch main
12+
```
13+
14+
=== "pip"
15+
16+
```shell
17+
# create a virtualenv
18+
python3 -m venv .venv
19+
20+
# activate it
21+
source .venv/bin/activate
22+
23+
# perform installation
24+
pip install https://github.com/geobeyond/pygeoapi-prefect@main
25+
```
26+
27+
28+
## Set up
29+
30+
In order to work with pygeoapi-prefect, you need to set up:
31+
32+
- a Prefect server;
33+
- pygeoapi;
34+
- at least one Prefect worker.
35+
36+
37+
### Prefect server
38+
39+
After installation, you need to have a Prefect server running and to set some environment variables
40+
so that the Prefect Manager is able to communicate with it.
41+
42+
As the most basic setup, just start a local Prefect server
43+
44+
=== "uv"
45+
46+
```shell
47+
uv run prefect server start
48+
```
49+
50+
=== "pip"
51+
52+
```shell
53+
prefect server start
54+
```
55+
56+
This will start up a local Prefect server listening on port 4200. The Prefect UI becomes available at
57+
`http://localhost:4200`
58+
59+
60+
### Pygeoapi
61+
62+
Enable the Prefect job/process manager by modifying your pygeoapi configuration file. The
63+
`server.manager.name` configuration parameter needs to be set to `pygeopai_prefect.PrefectManager`.
64+
65+
```yaml
66+
# pygeoapi configuration file
67+
server:
68+
manager:
69+
name: pygeoapi_prefect.PrefectManager
70+
```
71+
72+
73+
#### Processors
74+
75+
In addition to the manager, you must configure pygeoapi with some resources of type `process`.
76+
`PrefectManager` is able to work both with vanilla pygeoapi processors and with a custom Prefect-aware
77+
processor type.
78+
79+
80+
##### pygeoapi vanilla processes
81+
82+
Vanilla pygeoapi processes (_i.e._ those that inherit from `pygeoapi.process.base.BaseProcessor`) can be used without
83+
any modification. As usual, processes need to be specified in the pygeoapi configuration file. Example:
84+
85+
```yaml
86+
# pygeoapi configuration file
87+
resources:
88+
hello-world:
89+
type: process
90+
processor:
91+
name: HelloWorld
92+
```
93+
94+
95+
##### custom prefect-aware processes
96+
97+
TBD
98+
99+
#### Launching pygeoapi
100+
101+
When starting pygeoapi, ensure the `PREFECT_API_URL` environment variable is set. As the most basic
102+
launch of pygeoapi, you can create a `pygeoapi.env` file with these contents:
103+
104+
```shell
105+
# contents of pygeoapi.env
106+
PYGEOAPI_CONFIG=<your-pygeoapi-config.file>
107+
PYGEOAPI_OPENAPI=<your-pygeoapi-openapi.file>
108+
PREFECT_API_URL=http://127.0.0.1:4200/api
109+
PREFECT_RESULTS_PERSIST_BY_DEFAULT=true
110+
```
111+
112+
And then launch pygeoapi:
113+
114+
=== "uv"
115+
116+
```shell
117+
# set the contents of pygeoapi.env as environment variables
118+
set -o allexport; source pygeoapi.env; set +o allexport
119+
120+
uv run pygeoapi serve
121+
```
122+
123+
=== "pip"
124+
125+
```shell
126+
# set the contents of pygeoapi.env as environment variables
127+
set -o allexport; source pygeoapi.env; set +o allexport
128+
129+
pygeoapi serve
130+
```
131+
132+
This shall start the pygeoapi server, with an instance of `PrefectManager` as its process/job manager, and
133+
using the Prefect server that is specified by the `PREFECT_API_URL` environment variable. It
134+
can now be used with both vanilla pygeoapi processes and a new kind of Prefect-aware processors.
135+
136+
137+
### Prefect worker(s)
138+
139+
The Prefect server dispatches execution to workers, which means you also need to have at least a Prefect
140+
worker up and running. There are many ways to set up a Prefect worker, depending on the chosen
141+
execution model.
142+
143+
#### Run pygeoapi vanilla processes locally
144+
145+
For running pygeoapi vanilla processors locally, you can call a custom CLI command to start the worker:
146+
147+
=== "uv"
148+
149+
```shell
150+
uv run pygeoapi plugins prefect deploy-static
151+
```
152+
153+
=== "pip"
154+
155+
```shell
156+
pygeoapi plugins prefect deploy-static
157+
```
158+
159+
This will collect all vanilla processes defined in the pygeoapi configuration file, create a Prefect static
160+
deployment for each and launch a Prefect worker that spawns new processes whenever it receives a request for
161+
running a process
162+
163+
164+
#### Other execution models
165+
166+
Prefect is a very flexible platform and is able to coordinate the execution of processes in different
167+
environments, such as remote hosts, docker containers, k8s pods, etc. In order to take advantage of these
168+
other execution models you will need to both:
169+
170+
- Configure your infrastructure
171+
- Deploy your processor code using [Prefect deployments](https://docs.prefect.io/v3/concepts/deployments)
172+
- Create pygeoapi processes that inherit from the custom pygeoapi-prefect `BasePrefectProcessor` and
173+
set them up accordingly in the pygeoapi configuration
174+
175+
176+
!!! NOTE
177+
178+
The `PrefectManager` only interacts with the Prefect server and with the Prefect result storage. This means
179+
that as long as you define a Prefect Flow and are able to deploy it, then it
180+
181+
182+
183+
## Usage
184+
185+
186+
### Execute process
187+
188+
### Monitor execution via Prefect UI

mkdocs.yml

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,7 @@ edit_uri: edit/main/docs/
77

88
nav:
99
- Home: 'index.md'
10+
- New User Guide: 'guide.md'
1011
- User Guide:
1112
- Installation: 'user-guide/installation.md'
1213
- Defining processes: 'user-guide/defining-processes.md'
@@ -23,6 +24,7 @@ theme:
2324
accent: pink
2425
features:
2526
- content.action.edit
27+
- content.tabs.link
2628
- header.autohide
2729
- navigation.footer
2830
- navigation.tabs
@@ -43,5 +45,7 @@ markdown_extensions:
4345
- pymdownx.inlinehilite
4446
- pymdownx.snippets
4547
- pymdownx.superfences
48+
- pymdownx.tabbed:
49+
alternate_style: true
4650
- toc:
4751
permalink: true

0 commit comments

Comments
 (0)