Skip to content

Commit e34f8a0

Browse files
kneemaakneemaa
committed
feat: add Prometheus and StatsD metrics support
Add observability support with two metrics backends: - Prometheus: HTTP server exposing /metrics endpoint - StatsD: UDP push to StatsD-compatible servers Metrics tracked: - Download operations (count, bytes, duration, retries) - Photos and videos processed (separate counters) - Sync runs (count, duration, status) - API requests (count, duration, errors) - Authentication attempts and MFA requests New CLI options: - --metrics-backend (none|prometheus|statsd|both) - --prometheus-host, --prometheus-port - --statsd-host, --statsd-port, --statsd-prefix - --metrics-instance (for multi-instance deployments) Also includes: - Documentation with configuration examples - Example Grafana dashboard - Dockerfile exposes port 9090 for Prometheus
1 parent 3a97872 commit e34f8a0

21 files changed

+2822
-14
lines changed

‎CHANGELOG.md‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,8 @@
22

33
## Unreleased
44

5+
- feat: add Prometheus and StatsD metrics support (`--metrics-backend` option)
6+
57
## 1.32.2 (2025-09-01)
68

79
- fix: HTTP response content not captured for authentication and non-streaming requests [#1240](https://github.com/icloud-photos-downloader/icloud_photos_downloader/issues/1240)

‎Dockerfile‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -22,6 +22,7 @@ COPY dist/icloudpd-*.*.*-linux-musl-arm32v7 icloudpd
2222
FROM runtime_${TARGETARCH}_${TARGETVARIANT:-none} AS runtime
2323
ENV TZ=UTC
2424
EXPOSE 8080
25+
EXPOSE 9090
2526
WORKDIR /app
2627
RUN chmod +x /app/icloud /app/icloudpd
2728

‎README.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -41,6 +41,7 @@ See [Documentation](https://icloud-photos-downloader.github.io/icloud_photos_dow
4141
- One time download and an option to monitor for iCloud changes continuously (`--watch-with-interval` option)
4242
- Optimizations for incremental runs (`--until-found` and `--recent` options)
4343
- Photo metadata (EXIF) updates (`--set-exif-datetime` option)
44+
- Prometheus and StatsD metrics for monitoring (`--metrics-backend` option)
4445
- ... and many more (use `--help` option to get full list)
4546

4647
<!-- end features -->

‎docs/index.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -15,6 +15,7 @@ naming
1515
mode
1616
size
1717
raw
18+
metrics
1819
webui
1920
nas
2021
reference

‎docs/metrics.md‎

Lines changed: 196 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,196 @@
1+
# Metrics
2+
3+
```{versionadded} 1.33.0
4+
```
5+
6+
`icloudpd` supports exporting metrics for monitoring and observability. Two backends are supported: **Prometheus** and **StatsD**.
7+
8+
## Configuration
9+
10+
Enable metrics using the [`--metrics-backend`](metrics-backend-parameter) parameter:
11+
12+
```bash
13+
# Prometheus only
14+
icloudpd --username user@example.com --directory /photos --metrics-backend prometheus
15+
16+
# StatsD only
17+
icloudpd --username user@example.com --directory /photos --metrics-backend statsd
18+
19+
# Both backends simultaneously
20+
icloudpd --username user@example.com --directory /photos --metrics-backend both
21+
```
22+
23+
## Prometheus
24+
25+
When using the `prometheus` backend, an HTTP server is started that exposes metrics at the `/metrics` endpoint in Prometheus exposition format.
26+
27+
### Configuration Options
28+
29+
| Parameter | Default | Description |
30+
|-----------|---------|-------------|
31+
| [`--prometheus-host`](prometheus-host-parameter) | `0.0.0.0` | Host to bind the HTTP server |
32+
| [`--prometheus-port`](prometheus-port-parameter) | `9090` | Port for the HTTP server |
33+
34+
### Example Scrape Configuration
35+
36+
Add the following to your Prometheus `prometheus.yml`:
37+
38+
```yaml
39+
scrape_configs:
40+
- job_name: 'icloudpd'
41+
static_configs:
42+
- targets: ['localhost:9090']
43+
```
44+
45+
For Docker deployments:
46+
47+
```yaml
48+
scrape_configs:
49+
- job_name: 'icloudpd'
50+
static_configs:
51+
- targets: ['icloudpd:9090']
52+
```
53+
54+
### Docker Usage
55+
56+
When running icloudpd in Docker with Prometheus metrics, expose port 9090:
57+
58+
```bash
59+
docker run -p 9090:9090 \
60+
-v /path/to/photos:/data \
61+
-v /path/to/cookies:/cookies \
62+
icloudpd/icloudpd icloudpd \
63+
--username user@example.com \
64+
--directory /data \
65+
--cookie-directory /cookies \
66+
--metrics-backend prometheus \
67+
--prometheus-host 0.0.0.0
68+
```
69+
70+
## StatsD
71+
72+
When using the `statsd` backend, metrics are pushed via UDP to a StatsD server. This is useful for integration with Graphite, Datadog, or other StatsD-compatible systems.
73+
74+
### Configuration Options
75+
76+
| Parameter | Default | Description |
77+
|-----------|---------|-------------|
78+
| [`--statsd-host`](statsd-host-parameter) | `localhost` | StatsD server host |
79+
| [`--statsd-port`](statsd-port-parameter) | `8125` | StatsD server port |
80+
| [`--statsd-prefix`](statsd-prefix-parameter) | `icloudpd` | Prefix for all metric names |
81+
82+
### Example
83+
84+
```bash
85+
icloudpd --username user@example.com --directory /photos \
86+
--metrics-backend statsd \
87+
--statsd-host statsd.example.com \
88+
--statsd-port 8125 \
89+
--statsd-prefix myapp.icloudpd
90+
```
91+
92+
## Available Metrics
93+
94+
### Counters
95+
96+
| Metric | Labels | Description |
97+
|--------|--------|-------------|
98+
| `downloads_total` | `status`, `size` | Total number of download attempts. Status is `success` or `failure`. Size indicates the asset size variant. |
99+
| `download_bytes_total` | `size` | Total bytes downloaded successfully |
100+
| `download_retries_total` | `reason` | Number of download retries. Reason: `session`, `throttle`, or `io` |
101+
| `photos_processed_total` | `action` | Photos processed. Action: `downloaded`, `skipped`, or `deleted` |
102+
| `videos_processed_total` | `action` | Videos processed. Action: `downloaded`, `skipped`, or `deleted` |
103+
| `sync_runs_total` | `status` | Sync run completions. Status: `success`, `failure`, or `cancelled` |
104+
| `api_requests_total` | `method`, `status_code` | Total API requests to iCloud |
105+
| `api_errors_total` | `error_type` | API errors by type |
106+
| `auth_attempts_total` | `result`, `method` | Authentication attempts. Result: `started`, `success`, `failure`. Method: `password`, `2fa`, `2sa`, `2fa_web` |
107+
| `auth_mfa_requests_total` | `type` | MFA code requests. Type: `2fa` or `2sa` |
108+
109+
### Gauges
110+
111+
| Metric | Labels | Description |
112+
|--------|--------|-------------|
113+
| `up` | - | Set to 1 when icloudpd is running |
114+
| `current_progress` | - | Current sync progress percentage (0-100) |
115+
116+
### Histograms
117+
118+
| Metric | Labels | Description |
119+
|--------|--------|-------------|
120+
| `download_duration_seconds` | `size` | Time taken to download individual files |
121+
| `sync_duration_seconds` | - | Total duration of each sync run |
122+
| `api_request_duration_seconds` | `method` | Duration of individual API requests |
123+
124+
## Multi-Instance Setup
125+
126+
When running multiple `icloudpd` instances (e.g., for different iCloud accounts), use the [`--metrics-instance`](metrics-instance-parameter) parameter to distinguish them:
127+
128+
```bash
129+
# Instance for user1
130+
icloudpd --username user1@example.com --directory /photos/user1 \
131+
--metrics-backend prometheus --prometheus-port 9091 \
132+
--metrics-instance user1
133+
134+
# Instance for user2
135+
icloudpd --username user2@example.com --directory /photos/user2 \
136+
--metrics-backend prometheus --prometheus-port 9092 \
137+
--metrics-instance user2
138+
```
139+
140+
The instance label is added to all metrics, allowing you to filter and aggregate by instance in your monitoring system.
141+
142+
## Example Queries
143+
144+
### Prometheus/PromQL
145+
146+
Download success rate:
147+
```promql
148+
sum(rate(downloads_total{status="success"}[5m])) / sum(rate(downloads_total[5m]))
149+
```
150+
151+
Photos downloaded per hour:
152+
```promql
153+
sum(increase(photos_processed_total{action="downloaded"}[1h]))
154+
```
155+
156+
Videos downloaded per hour:
157+
```promql
158+
sum(increase(videos_processed_total{action="downloaded"}[1h]))
159+
```
160+
161+
Total assets (photos + videos) downloaded per hour:
162+
```promql
163+
sum(increase(photos_processed_total{action="downloaded"}[1h])) + sum(increase(videos_processed_total{action="downloaded"}[1h]))
164+
```
165+
166+
Average download duration:
167+
```promql
168+
rate(download_duration_seconds_sum[5m]) / rate(download_duration_seconds_count[5m])
169+
```
170+
171+
API error rate by type:
172+
```promql
173+
sum by (error_type) (rate(api_errors_total[5m]))
174+
```
175+
176+
Sync success rate:
177+
```promql
178+
sum(rate(sync_runs_total{status="success"}[24h])) / sum(rate(sync_runs_total[24h]))
179+
```
180+
181+
### Grafana Dashboard
182+
183+
An example Grafana dashboard is available at [`examples/grafana-dashboard.json`](https://github.com/icloud-photos-downloader/icloud_photos_downloader/blob/master/examples/grafana-dashboard.json). Import it into Grafana to get started quickly.
184+
185+
The dashboard includes:
186+
- Status overview (up/down, sync runs, avg duration)
187+
- Photos and videos processed (total and rate)
188+
- Sync activity charts
189+
- API requests and auth attempts
190+
191+
### Grafana Dashboard Tips
192+
193+
- Use `up` metric for availability alerting
194+
- Track `current_progress` for real-time sync status
195+
- Alert on `api_errors_total` increases for early warning of issues
196+
- Use `sync_duration_seconds` histograms to track performance trends

‎docs/reference.md‎

Lines changed: 68 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -374,3 +374,71 @@ This is a list of all options available for the command line interface (CLI) of
374374
```{note}
375375
The date is when the asset was created, not when it was added to iCloud.
376376
```
377+
378+
(metrics-backend-parameter)=
379+
`--metrics-backend X`
380+
381+
: Specifies the metrics backend to use for observability. Available options: `none` (default), `prometheus`, `statsd`, or `both`.
382+
383+
When set to `prometheus`, an HTTP server is started that exposes metrics at `/metrics`.
384+
When set to `statsd`, metrics are pushed via UDP to a StatsD server.
385+
When set to `both`, both backends are enabled simultaneously.
386+
387+
```{versionadded} 1.33.0
388+
```
389+
390+
```{seealso}
391+
Details on [Metrics](metrics)
392+
```
393+
394+
(prometheus-port-parameter)=
395+
`--prometheus-port X`
396+
397+
: Port for the Prometheus metrics HTTP server. Default: 9090.
398+
399+
```{versionadded} 1.33.0
400+
```
401+
402+
(prometheus-host-parameter)=
403+
`--prometheus-host X`
404+
405+
: Host to bind the Prometheus metrics HTTP server. Default: 0.0.0.0.
406+
407+
```{versionadded} 1.33.0
408+
```
409+
410+
(statsd-host-parameter)=
411+
`--statsd-host X`
412+
413+
: StatsD server host. Default: localhost.
414+
415+
```{versionadded} 1.33.0
416+
```
417+
418+
(statsd-port-parameter)=
419+
`--statsd-port X`
420+
421+
: StatsD server port. Default: 8125.
422+
423+
```{versionadded} 1.33.0
424+
```
425+
426+
(statsd-prefix-parameter)=
427+
`--statsd-prefix X`
428+
429+
: Prefix for StatsD metric names. Default: icloudpd.
430+
431+
```{versionadded} 1.33.0
432+
```
433+
434+
(metrics-instance-parameter)=
435+
`--metrics-instance X`
436+
437+
: Instance label to add to all metrics for identifying this icloudpd instance. Useful when running multiple instances (e.g., for different users).
438+
439+
```{versionadded} 1.33.0
440+
```
441+
442+
```{seealso}
443+
Details on [Metrics](metrics)
444+
```

0 commit comments

Comments
 (0)
⚡