Skip to content

Commit 6f078df

Browse files
committed
default to running sync processes in-process
1 parent eab1c91 commit 6f078df

File tree

3 files changed

+24
-24
lines changed

3 files changed

+24
-24
lines changed

docs/user-guide.md

Lines changed: 19 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,7 @@ hide:
77

88
## Installation
99

10-
In time, this will be on the [Python Package Index]() but for now you need to use the
10+
In time, this will be on the [Python Package Index]() but for now you need to use the
1111
repository url to get it:
1212

1313
=== "uv"
@@ -41,7 +41,7 @@ In order to work with pygeoapi-prefect, you need to set up:
4141

4242
### Prefect server
4343

44-
After installation, you need to have a Prefect server running and to set some environment variables
44+
After installation, you need to have a Prefect server running and to set some environment variables
4545
so that the Prefect Manager is able to communicate with it.
4646

4747
As the most basic setup, just start a local Prefect server
@@ -64,7 +64,7 @@ As the most basic setup, just start a local Prefect server
6464

6565
### Pygeoapi
6666

67-
Enable the Prefect job/process manager by modifying your pygeoapi configuration file. The
67+
Enable the Prefect job/process manager by modifying your pygeoapi configuration file. The
6868
`server.manager.name` configuration parameter needs to be set to `pygeopai_prefect.PrefectManager`.
6969

7070
```yaml
@@ -79,25 +79,25 @@ server:
7979
The Prefect Manager accepts the following configuration parameters, all of which are optional.
8080
These should be specified as properties of the `server.manager` object:
8181

82-
- `use_vanilla_processor_deployments: bool = True` - Whether to call native pygeoapi processors via the
83-
Prefect deployment interface. If this is enabled (which is the default):
82+
- `use_deployment_for_sync_requests: bool = False` - Whether to call native pygeoapi processors via the
83+
Prefect deployment interface or not. If this is enabled:
8484

85-
- Jobs are executed via the Prefect worker, which you must start (as mentioned below)
85+
- Jobs are executed via the Prefect worker, which you must have started beforehand (as mentioned below)
8686
- Jobs are coordinated by the Prefect scheduler, which means that they may not start immediately
8787
- Jobs are run in a different process than pygeoapi
88-
- This means that job execution is slower, as there is a temporal overhead which is introduced by
88+
- This means that job execution is slower, as there is a temporal overhead that is introduced by
8989
the coordination that happens between the Prefect scheduler service and the Prefect worker
90-
90+
9191
If this is disabled, then jobs are executed immediately and run in the same process as pygeoapi
9292

93-
- `sync_job_execution_timeout_seconds: int = 60` - How much time to give a sync job to finish its
93+
- `sync_job_execution_timeout_seconds: int = 60` - How much time to give a sync job to finish its
9494
processing before declaring it as failed
9595

9696

9797
#### Processors
9898

99-
In addition to the manager, you must configure pygeoapi with some resources of type `process`.
100-
`PrefectManager` is able to work both with vanilla pygeoapi processors and with a custom Prefect-aware
99+
In addition to the manager, you must configure pygeoapi with some resources of type `process`.
100+
`PrefectManager` is able to work both with vanilla pygeoapi processors and with a custom Prefect-aware
101101
processor type.
102102

103103

@@ -122,7 +122,7 @@ TBD
122122

123123
#### Launching pygeoapi
124124

125-
When starting pygeoapi, ensure the `PREFECT_API_URL` environment variable is set. As the most basic
125+
When starting pygeoapi, ensure the `PREFECT_API_URL` environment variable is set. As the most basic
126126
launch of pygeoapi, you can create a `pygeoapi.env` file with these contents:
127127

128128
```shell
@@ -154,14 +154,14 @@ And then launch pygeoapi:
154154
```
155155

156156
This shall start the pygeoapi server, with an instance of `PrefectManager` as its process/job manager, and
157-
using the Prefect server that is specified by the `PREFECT_API_URL` environment variable. It
157+
using the Prefect server that is specified by the `PREFECT_API_URL` environment variable. It
158158
can now be used with both vanilla pygeoapi processes and a new kind of Prefect-aware processors.
159159

160160

161161
### Prefect worker(s)
162162

163-
The Prefect server dispatches execution to workers, which means you also need to have at least a Prefect
164-
worker up and running. There are many ways to set up a Prefect worker, depending on the chosen
163+
The Prefect server dispatches execution to workers, which means you also need to have at least a Prefect
164+
worker up and running. There are many ways to set up a Prefect worker, depending on the chosen
165165
execution model.
166166

167167
#### Run pygeoapi vanilla processes locally
@@ -187,20 +187,20 @@ running a process
187187

188188
#### Other execution models
189189

190-
Prefect is a very flexible platform and is able to coordinate the execution of processes in different
190+
Prefect is a very flexible platform and is able to coordinate the execution of processes in different
191191
environments, such as remote hosts, docker containers, k8s pods, etc. In order to take advantage of these
192192
other execution models you will need to both:
193193

194194
- Configure your infrastructure
195195
- Deploy your processor code using [Prefect deployments](https://docs.prefect.io/v3/concepts/deployments)
196-
- Create pygeoapi processes that inherit from the custom pygeoapi-prefect `BasePrefectProcessor` and
196+
- Create pygeoapi processes that inherit from the custom pygeoapi-prefect `BasePrefectProcessor` and
197197
set them up accordingly in the pygeoapi configuration
198198

199199

200200
!!! NOTE
201201

202202
The `PrefectManager` only interacts with the Prefect server and with the Prefect result storage. This means
203-
that as long as you define a Prefect Flow and are able to deploy it, then it
203+
that as long as you define a Prefect Flow and are able to deploy it, then it
204204

205205

206206

@@ -209,4 +209,4 @@ other execution models you will need to both:
209209

210210
### Execute process
211211

212-
### Monitor execution via Prefect UI
212+
### Monitor execution via Prefect UI

example-config.yml

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -19,6 +19,7 @@ server:
1919
attribution: '&copy; <a href="https://openstreetmap.org/copyright">OpenStreetMap contributors</a>'
2020
manager:
2121
name: pygeoapi_prefect.manager.PrefectManager
22+
use_deployment_for_sync_requests: true
2223
# name: TinyDB
2324
# connection: /tmp/pygeoapi-process-manager.db
2425
output_dir: /tmp/

src/pygeoapi_prefect/manager.py

Lines changed: 4 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -99,7 +99,7 @@ class PrefectManager:
9999
_processor_configurations: dict[ProcessId, dict[str, Any]]
100100

101101
is_async: bool = True
102-
use_vanilla_processor_deployments: bool
102+
use_deployment_for_sync_requests: bool
103103
supports_subscribing: bool
104104
prefect_state_map = {
105105
StateType.SCHEDULED: JobStatus.accepted,
@@ -118,9 +118,8 @@ class PrefectManager:
118118
sync_job_execution_timeout_seconds: int
119119

120120
def __init__(self, manager_def: dict[str, Any]):
121-
self.use_vanilla_processor_deployments = manager_def.get(
122-
"use_vanilla_processor_deployments", True)
123-
# self.use_vanilla_processor_deployments = False
121+
self.use_deployment_for_sync_requests = manager_def.get(
122+
"use_deployment_for_sync_requests", False)
124123
self.name = ".".join(
125124
(self.__class__.__module__, self.__class__.__qualname__)
126125
)
@@ -353,7 +352,7 @@ def execute_process(
353352
subscriber=subscriber,
354353
)
355354
if chosen_mode == ProcessExecutionMode.sync_execute:
356-
if isinstance(processor, BaseProcessor) and not self.use_vanilla_processor_deployments:
355+
if isinstance(processor, BaseProcessor) and not self.use_deployment_for_sync_requests:
357356
print(f"Executing processor {processor.metadata['id']!r} in-process...")
358357
media_type, generated_output = _execute_job_sync_without_deployment(
359358
job_id, processor, execution_request

0 commit comments

Comments
 (0)