Added OTel instrumentation and metrics export for sync client (#3954)

* Added intrastructure and integration point with OTel (#3864)

* Added intrastructure and integration point with OTel

* Added check for enabled metric groups

* Applied comments

* Added export of operation duration metric  (#3881)

* Added stadalone client metrics export

* Added support for cluster client

* Removed unused dispatchers and test

* Added export of resiliency metrics (#3888)

* Added stadalone client metrics export

* Added support for cluster client

* Added error metric export on fail commands for standalone client

* Removed unused dispatchers and test

* Added on error metrics export for cluster

* Added export of maint notification count metric

* Apply comments

* Remove redundant Union

* Added export of connection basic metrics (#3891)

* Added export of connection basic metrics

* Added new error category attribute

* Update tests/test_observability/test_recorder.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update tests/test_observability/test_recorder.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Apply comments

* Applied comments

---------

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Added pub/sub and stream lag metrics export (#3898)

* Added export of connection basic metrics

* Added new error category attribute

* Added pub/sub and stream lag metrics export

* Added metric export form XREAD

* Update redis/client.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update redis/event.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Applied comments

---------

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Added OTel benchmark test (#3903)

* Added export of connection basic metrics

* Added new error category attribute

* Added pub/sub and stream lag metrics export

* Added metric export form XREAD

* Added benchmark tests

* Added error output

* Added env variable for otel host

* Fixed endpoint

* Added iterations and results averaging

* Use unsecured gRPC

* Added --with-command-metrics argument

* Removed incorrect identation

* Update benchmarks/otel_benchmark.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update benchmarks/otel_benchmark.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update benchmarks/otel_benchmark.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update benchmarks/otel_benchmark.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update benchmarks/otel_benchmark.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

---------

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Added export of connection advanced metrics (#3910)

* Added export of connection advanced metrics

* Update tests/test_connection_pool.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update redis/observability/metrics.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Refactored kwargs instead of args

---------

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Added CSC metrics export (#3911)

* Added export of connection advanced metrics

* Added CSC metrics export

* Revert changes

* Added observable gauge registry and refactored observables metric export

* Fixed case with trackin non-cachable cache tracking

* Added memory and CPU usage (#3933)

* Added full benchmark tests + fixes

* Added Memory and CPU usage calculation

* Removed events dispatching overhead, refactored metrics attributes (#3934)

* Removed events dispatching overhead, refactored metrics attributes

* Fixed incorrect retry attempts, changed indexes to named parameters

* Fixed issue with connection attribute and metric recording

* Added async recorders and registry (#3950)

* Added async recorders and registry

* Removed async registry

* Added more coverage

* Added bucket override configuration (#3952)

* Added async recorders and registry

* Added bucket override configuration

* Removed async registry

* Added more coverage

* Fixed version constraints, removed ubused imports

* Moved module desription on top of file

* Remove async instrumentations (#3953)

* Fix tests and codestyle

* Fixed missing parameters

* Fixed tests

* Fixed tests

* Fixed tests

* Marked argument as unused

* Fixed tests

* Added missing metric export

* Fixes units and pool name attribute

Signed-off-by: Elena Kolevska <elena@kolevska.com>

* Updated test for pool name attribute

Signed-off-by: Elena Kolevska <elena@kolevska.com>

* Updates more tests

Signed-off-by: Elena Kolevska <elena@kolevska.com>

* Linter

Signed-off-by: Elena Kolevska <elena@kolevska.com>

* Added try...except blocks for recorder functions

* Added documentation

* Updated wordlist

* Update redis/cluster.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update redis/connection.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update redis/connection.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update pyproject.toml

Co-authored-by: Elena Kolevska <elena-kolevska@users.noreply.github.com>

* Codestyle fixes

* Fixed metric export

* Fixed assertions

* Added geo failover metric export

* Codestyle fixes

---------

Signed-off-by: Elena Kolevska <elena@kolevska.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Elena Kolevska <elena@kolevska.com>
Co-authored-by: Elena Kolevska <elena-kolevska@users.noreply.github.com>

Author: 3 people

.github/wordlist.txt

@@ -14,6 +14,7 @@ ClusterPubSub
 ConnectionPool
 config
 CoreCommands
+CSC
 DatabaseConfig
 DNS
 EchoHealthCheck
@@ -22,17 +23,22 @@ EVALSHA
 failover
 FQDN
 Grokzen's
+handoff
 Healthcheck
 HealthCheckPolicies
 healthcheck
 healthchecks
 INCR
+init
 IOError
 Instrumentations
 JSONCommands
 Jaeger
 Ludovico
 Magnocavallo
+MeterProvider
+MetricGroup
+millis
 MultiDbConfig
 MultiDBClient
 McCurdy
@@ -46,6 +52,12 @@ OpenCensus
 OpenTelemetry
 OpenTracing
 Otel
+otel
+OTelConfig
+otlp
+OTLPMetricExporter
+PeriodicExportingMetricReader
+proto
 PubSub
 READONLY
 RediSearch
@@ -145,6 +157,7 @@ repo
 runtime
 sedrik
 sharded
+sdk
 ssl
 str
 stunnel
@@ -160,4 +173,6 @@ unicode
 url
 virtualenv
 www
+XREAD
+XREADGROUP
 yaml

.github/workflows/integration.yaml

@@ -126,9 +126,9 @@ jobs:
     runs-on: ubuntu-latest
     needs: [ redis_version ]
     # in pipeline ofter these pypy jobs hang, so their timeout is set
-    # to just 40 minutes - close to the max time the jobs are running at the moment
+    # to just 50 minutes - close to the max time the jobs are running at the moment
     # adding more tests will make them even slower, and we will need to adjust the timeout
-    timeout-minutes: 40
+    timeout-minutes: 50
     strategy:
       max-parallel: 30
       fail-fast: false