|
| 1 | +# User guide |
| 2 | + |
| 3 | +## Installation |
| 4 | + |
| 5 | +In time, this will be on the [Python Package Index]() but for now you need to use the |
| 6 | +repository url to get it: |
| 7 | + |
| 8 | +=== "uv" |
| 9 | + |
| 10 | + ```shell |
| 11 | + uv add https://github.com/geobeyond/pygeoapi-prefect --branch main |
| 12 | + ``` |
| 13 | + |
| 14 | +=== "pip" |
| 15 | + |
| 16 | + ```shell |
| 17 | + # create a virtualenv |
| 18 | + python3 -m venv .venv |
| 19 | + |
| 20 | + # activate it |
| 21 | + source .venv/bin/activate |
| 22 | + |
| 23 | + # perform installation |
| 24 | + pip install https://github.com/geobeyond/pygeoapi-prefect@main |
| 25 | + ``` |
| 26 | + |
| 27 | + |
| 28 | +## Set up |
| 29 | + |
| 30 | +In order to work with pygeoapi-prefect, you need to set up: |
| 31 | + |
| 32 | +- a Prefect server; |
| 33 | +- pygeoapi; |
| 34 | +- at least one Prefect worker. |
| 35 | + |
| 36 | + |
| 37 | +### Prefect server |
| 38 | + |
| 39 | +After installation, you need to have a Prefect server running and to set some environment variables |
| 40 | +so that the Prefect Manager is able to communicate with it. |
| 41 | + |
| 42 | +As the most basic setup, just start a local Prefect server |
| 43 | + |
| 44 | +=== "uv" |
| 45 | + |
| 46 | + ```shell |
| 47 | + uv run prefect server start |
| 48 | + ``` |
| 49 | + |
| 50 | +=== "pip" |
| 51 | + |
| 52 | + ```shell |
| 53 | + prefect server start |
| 54 | + ``` |
| 55 | + |
| 56 | + This will start up a local Prefect server listening on port 4200. The Prefect UI becomes available at |
| 57 | + `http://localhost:4200` |
| 58 | + |
| 59 | + |
| 60 | +### Pygeoapi |
| 61 | + |
| 62 | +Enable the Prefect job/process manager by modifying your pygeoapi configuration file. The |
| 63 | +`server.manager.name` configuration parameter needs to be set to `pygeopai_prefect.PrefectManager`. |
| 64 | + |
| 65 | +```yaml |
| 66 | +# pygeoapi configuration file |
| 67 | +server: |
| 68 | + manager: |
| 69 | + name: pygeoapi_prefect.PrefectManager |
| 70 | +``` |
| 71 | +
|
| 72 | +
|
| 73 | +#### Processors |
| 74 | +
|
| 75 | +In addition to the manager, you must configure pygeoapi with some resources of type `process`. |
| 76 | +`PrefectManager` is able to work both with vanilla pygeoapi processors and with a custom Prefect-aware |
| 77 | +processor type. |
| 78 | + |
| 79 | + |
| 80 | +##### pygeoapi vanilla processes |
| 81 | + |
| 82 | +Vanilla pygeoapi processes (_i.e._ those that inherit from `pygeoapi.process.base.BaseProcessor`) can be used without |
| 83 | +any modification. As usual, processes need to be specified in the pygeoapi configuration file. Example: |
| 84 | + |
| 85 | +```yaml |
| 86 | +# pygeoapi configuration file |
| 87 | +resources: |
| 88 | + hello-world: |
| 89 | + type: process |
| 90 | + processor: |
| 91 | + name: HelloWorld |
| 92 | +``` |
| 93 | + |
| 94 | + |
| 95 | +##### custom prefect-aware processes |
| 96 | + |
| 97 | +TBD |
| 98 | + |
| 99 | +#### Launching pygeoapi |
| 100 | + |
| 101 | +When starting pygeoapi, ensure the `PREFECT_API_URL` environment variable is set. As the most basic |
| 102 | +launch of pygeoapi, you can create a `pygeoapi.env` file with these contents: |
| 103 | + |
| 104 | +```shell |
| 105 | +# contents of pygeoapi.env |
| 106 | +PYGEOAPI_CONFIG=<your-pygeoapi-config.file> |
| 107 | +PYGEOAPI_OPENAPI=<your-pygeoapi-openapi.file> |
| 108 | +PREFECT_API_URL=http://127.0.0.1:4200/api |
| 109 | +PREFECT_RESULTS_PERSIST_BY_DEFAULT=true |
| 110 | +``` |
| 111 | + |
| 112 | +And then launch pygeoapi: |
| 113 | + |
| 114 | +=== "uv" |
| 115 | + |
| 116 | + ```shell |
| 117 | + # set the contents of pygeoapi.env as environment variables |
| 118 | + set -o allexport; source pygeoapi.env; set +o allexport |
| 119 | +
|
| 120 | + uv run pygeoapi serve |
| 121 | + ``` |
| 122 | + |
| 123 | +=== "pip" |
| 124 | + |
| 125 | + ```shell |
| 126 | + # set the contents of pygeoapi.env as environment variables |
| 127 | + set -o allexport; source pygeoapi.env; set +o allexport |
| 128 | +
|
| 129 | + pygeoapi serve |
| 130 | + ``` |
| 131 | + |
| 132 | +This shall start the pygeoapi server, with an instance of `PrefectManager` as its process/job manager, and |
| 133 | +using the Prefect server that is specified by the `PREFECT_API_URL` environment variable. It |
| 134 | +can now be used with both vanilla pygeoapi processes and a new kind of Prefect-aware processors. |
| 135 | + |
| 136 | + |
| 137 | +### Prefect worker(s) |
| 138 | + |
| 139 | +The Prefect server dispatches execution to workers, which means you also need to have at least a Prefect |
| 140 | +worker up and running. There are many ways to set up a Prefect worker, depending on the chosen |
| 141 | +execution model. |
| 142 | + |
| 143 | +#### Run pygeoapi vanilla processes locally |
| 144 | + |
| 145 | +For running pygeoapi vanilla processors locally, you can call a custom CLI command to start the worker: |
| 146 | + |
| 147 | +=== "uv" |
| 148 | + |
| 149 | +```shell |
| 150 | +uv run pygeoapi plugins prefect deploy-static |
| 151 | +``` |
| 152 | + |
| 153 | +=== "pip" |
| 154 | + |
| 155 | +```shell |
| 156 | +pygeoapi plugins prefect deploy-static |
| 157 | +``` |
| 158 | + |
| 159 | +This will collect all vanilla processes defined in the pygeoapi configuration file, create a Prefect static |
| 160 | +deployment for each and launch a Prefect worker that spawns new processes whenever it receives a request for |
| 161 | +running a process |
| 162 | + |
| 163 | + |
| 164 | +#### Other execution models |
| 165 | + |
| 166 | +Prefect is a very flexible platform and is able to coordinate the execution of processes in different |
| 167 | +environments, such as remote hosts, docker containers, k8s pods, etc. In order to take advantage of these |
| 168 | +other execution models you will need to both: |
| 169 | + |
| 170 | +- Configure your infrastructure |
| 171 | +- Deploy your processor code using [Prefect deployments](https://docs.prefect.io/v3/concepts/deployments) |
| 172 | +- Create pygeoapi processes that inherit from the custom pygeoapi-prefect `BasePrefectProcessor` and |
| 173 | + set them up accordingly in the pygeoapi configuration |
| 174 | + |
| 175 | + |
| 176 | +!!! NOTE |
| 177 | + |
| 178 | + The `PrefectManager` only interacts with the Prefect server and with the Prefect result storage. This means |
| 179 | + that as long as you define a Prefect Flow and are able to deploy it, then it |
| 180 | + |
| 181 | + |
| 182 | + |
| 183 | +## Usage |
| 184 | + |
| 185 | + |
| 186 | +### Execute process |
| 187 | + |
| 188 | +### Monitor execution via Prefect UI |
0 commit comments