Error collecting operator balance

[francesco-gaglione]

Description

On node startup, the json-rpc-relay logs the following error when attempting to fetch the operator balance and from that it is not accepting any request:

[2025-06-03 20:18:55.085 +0000] DEBUG (mirror-node/103 on 2b315facc0be): [Request ID: ff4b3f16-6555-4ead-886e-2f357bf93b6f] An accepted error occurred while communicating with the mirror node server: method=GET, path=accounts/0.0.2?transactions=false, status=404

[2025-06-03 20:18:55.085 +0000] ERROR (relay/103 on 2b315facc0be): Error collecting operator balance. Skipping balance set

    err: {

      "type": "TypeError",

      "message": "Cannot read properties of null (reading 'balance')",

      "stack":

          TypeError: Cannot read properties of null (reading 'balance')

              at Gauge.<anonymous> (/home/node/app/packages/relay/dist/lib/relay.js:138:62)

              at Generator.next (<anonymous>)

              at fulfilled (/home/node/app/packages/relay/dist/lib/relay.js:6:58)

              at process.processTicksAndRejections (node:internal/process/task_queues:95:5)

    }

Steps to reproduce

Just run docker-compose up -d

Additional context

No response

Hedera network

No response

Version

v0.68

Operating system

macOS

[Neurone]

Hi @francesco-gaglione v0.68 is not a hiero-local-node version, are you trying to start the json-rpc-relay or the local node?

Please be more specific in the steps to reproduce section:

• What repo are you cloning?
• Where do you get those logs after starting the containers?

In the additional context section, please describe you environment (docker compose version, macos version, node version, npm version, status of the containers after the start, etc, essentially anything that can help with the analysis) i.e.

❯ uname -a
Darwin FVFHL8TBQ05N 24.5.0 Darwin Kernel Version 24.5.0: Tue Apr 22 19:48:46 PDT 2025; root:xnu-11417.121.6~2/RELEASE_ARM64_T8103 arm64
❯ docker compose version
Docker Compose version v2.35.1-desktop.1
❯ node --version
v20.12.2
❯ npm version
{
  '@hashgraph/hedera-local': '2.37.0',
  npm: '10.7.0',
  node: '20.12.2',
  acorn: '8.11.3',
  ada: '2.7.6',
  ares: '1.27.0',
  base64: '0.5.2',
  brotli: '1.1.0',
  cjs_module_lexer: '1.2.2',
  cldr: '44.1',
  icu: '74.2',
  llhttp: '8.1.2',
  modules: '115',
  napi: '9',
  nghttp2: '1.60.0',
  nghttp3: '0.7.0',
  ngtcp2: '0.8.1',
  openssl: '3.0.13+quic',
  simdutf: '4.0.8',
  tz: '2024a',
  undici: '5.28.4',
  unicode: '15.1',
  uv: '1.46.0',
  uvwasi: '0.0.20',
  v8: '11.3.244.8-node.19',
  zlib: '1.3.0.1-motley-40e35a7'
}

Side note: in general, please use docker compose and not the legacy docker-compose executable.

[francesco-gaglione]

Hi @Neurone , sure, here’s some additional context.

I’m using hiero-local-node (same behavior also observed with docker compose, just as a side note). The issue seems to be related to the json-rpc-relay service and logs are from that container. I initially noticed it also on the WebSocket, but I’m no longer able to reproduce it there (probably because I'm not able to establish connections, but this is a side effect), only with the relay now.

For context, I simply cloned the repo and ran it. The only modification I made was in the .env file, where I updated the relay version to 0.68 (just to test if the issue was fixed in a newer release). However, I encountered the same issue with the default relay version 0.67, so it doesn’t appear to be version-specific.

Unfortunately, I don’t have clear steps to reproduce the issue. It started appearing out of nowhere — everything was working fine, but after a restart (with no code/config changes), the issue began to occur.

[Neurone]

Hi @francesco-gaglione, this seems very specific to your environment.

I'm able to run the main branch without issues, and I don't see errors in the hiero-json-rpc-relay container (but I still see the WS_CONNECTION_LIMIT = 0 parameter in the hiero-json-rpc-relay-ws container if you don't set the RELAY_WS_CONNECTION_LIMIT parameter in the .env file).

Can you please clean your docker environment docker system prune -a or test another system?

This is what I see if I run the container, just as a reference:

❯ git clone https://github.com/hiero-ledger/hiero-local-node.git
Cloning into 'hiero-local-node'...
remote: Enumerating objects: 3963, done.
remote: Counting objects: 100% (543/543), done.
remote: Compressing objects: 100% (351/351), done.
remote: Total 3963 (delta 409), reused 198 (delta 189), pack-reused 3420 (from 2)
Receiving objects: 100% (3963/3963), 7.61 MiB | 22.99 MiB/s, done.
Resolving deltas: 100% (2550/2550), done.

❯ cd hiero-local-node

❯ git rev-parse HEAD
70375f9936168fcf246dab46f69d724eb410e677

❯ docker ps
CONTAINER ID   IMAGE                                                       COMMAND                  CREATED        STATUS                          PORTS                                                                     NAMES
4f753fa91f99   gcr.io/mirrornode/hedera-mirror-monitor:0.130.0             "java org.springfram…"   18 hours ago   Up About a minute (healthy)     0.0.0.0:8082->8082/tcp                                                    mirror-node-monitor
b72ca056aa84   ghcr.io/hiero-ledger/hiero-json-rpc-relay:0.68.0            "npm run start"          18 hours ago   Up About a minute (unhealthy)   0.0.0.0:7546->7546/tcp, 8546-8547/tcp                                     json-rpc-relay
b0b8be5b25e5   ghcr.io/hiero-ledger/hiero-json-rpc-relay:0.68.0            "npm run start:ws"       18 hours ago   Up About a minute (unhealthy)   7546/tcp, 0.0.0.0:8546-8547->8546-8547/tcp                                json-rpc-relay-ws
af340901f21b   gcr.io/mirrornode/hedera-mirror-grpc:0.130.0                "java org.springfram…"   18 hours ago   Up About a minute (healthy)     0.0.0.0:5600->5600/tcp                                                    mirror-node-grpc
ef44fbd295c2   gcr.io/mirrornode/hedera-mirror-rest:0.130.0                "node --import=exten…"   18 hours ago   Up About a minute (healthy)     0.0.0.0:5551->5551/tcp                                                    mirror-node-rest
102a7e52de4d   gcr.io/mirrornode/hedera-mirror-rest-java:0.130.0           "java org.springfram…"   18 hours ago   Up About a minute (healthy)     0.0.0.0:8084->8084/tcp                                                    mirror-node-rest-java
2fa85e526a2e   gcr.io/mirrornode/hedera-mirror-web3:0.130.0                "java org.springfram…"   18 hours ago   Up About a minute (healthy)     0.0.0.0:8545->8545/tcp                                                    mirror-node-web3
cd609c4ecfba   gcr.io/hedera-registry/consensus-node:0.63.0                "/init"                  18 hours ago   Up About a minute (healthy)     0.0.0.0:9999->9999/tcp, 0.0.0.0:50211-50212->50211-50212/tcp, 50111/tcp   network-node
eddeedfbbc1f   gcr.io/mirrornode/hedera-mirror-importer:0.130.0            "java org.springfram…"   18 hours ago   Up About a minute (healthy)                                                                               mirror-node-importer
dfda84d76d94   gcr.io/hedera-registry/uploader-mirror:0.7.0                "/usr/bin/env python…"   18 hours ago   Up About a minute                                                                                         account-balances-uploader
612daf4a529b   gcr.io/hedera-registry/uploader-mirror:0.7.0                "/usr/bin/env python…"   18 hours ago   Up About a minute                                                                                         record-sidecar-uploader
3a4a96fe5801   gcr.io/hedera-registry/uploader-mirror:0.7.0                "/usr/bin/env python…"   18 hours ago   Up About a minute                                                                                         record-streams-uploader
26da3ea8321d   gcr.io/cadvisor/cadvisor:v0.47.0                            "/usr/bin/cadvisor -…"   18 hours ago   Up About a minute (healthy)     0.0.0.0:8081->8080/tcp                                                    cadvisor
d3d901e6341c   ghcr.io/hiero-ledger/hiero-block-node:0.11.0                "/bin/bash -c ${BN_W…"   18 hours ago   Up About a minute (healthy)     0.0.0.0:8080->8080/tcp, 0.0.0.0:9998->9999/tcp                            block-node
accbf55cf9ea   gcr.io/hedera-registry/hedera-mirror-node-explorer:24.4.0   "/docker-entrypoint.…"   18 hours ago   Up About a minute               80/tcp, 0.0.0.0:8090->8080/tcp                                            hedera-explorer
2f3e5cfcd3db   ghcr.io/mhga24/postgres:latest                              "docker-entrypoint.s…"   18 hours ago   Up About a minute               0.0.0.0:5433->5432/tcp                                                    mirror-node-db
e218aa6109e5   ghcr.io/mhga24/minio/minio                                  "sh -c 'mkdir -p /da…"   18 hours ago   Up About a minute               0.0.0.0:9000-9001->9000-9001/tcp                                          minio
3dc7ecf61366   ghcr.io/mhga24/grafana/grafana:8.5.16                       "/run.sh"                18 hours ago   Up About a minute               0.0.0.0:3000->3000/tcp                                                    grafana
ea96e9c0882b   ghcr.io/mhga24/prom/prometheus:v2.41.0                      "/bin/prometheus --c…"   18 hours ago   Up About a minute               0.0.0.0:9090->9090/tcp                                                    prometheus
be465221fd8b   gcr.io/hedera-registry/network-node-haveged:0.63.0          "/usr/bin/env havege…"   18 hours ago   Up About a minute                                                                                         haveged
7cc7f58fb1df   ghcr.io/mhga24/redis:6-alpine                               "docker-entrypoint.s…"   18 hours ago   Up About a minute               0.0.0.0:6379->6379/tcp                                                    relay-cache
f227aa02c3c3   ghcr.io/mhga24/envoyproxy/envoy:v1.22.0                     "/docker-entrypoint.…"   18 hours ago   Up About a minute               10000/tcp, 0.0.0.0:50213->50213/tcp                                       envoy-proxy

❯ docker logs json-rpc-relay
[...]
[2025-06-04 15:53:11.605 +0000] INFO (config-service/79 on b72ca056aa84): TEST_INITIAL_ACCOUNT_STARTING_BALANCE = 2000
[2025-06-04 15:53:11.605 +0000] INFO (config-service/79 on b72ca056aa84): TEST_TRANSACTION_RECORD_COST_TOLERANCE = 0.02
[2025-06-04 15:53:11.605 +0000] INFO (config-service/79 on b72ca056aa84): TEST_WS_SERVER = false
[2025-06-04 15:53:11.605 +0000] INFO (config-service/79 on b72ca056aa84): TIER_1_RATE_LIMIT = 100
[2025-06-04 15:53:11.605 +0000] INFO (config-service/79 on b72ca056aa84): TIER_2_RATE_LIMIT = 800
[2025-06-04 15:53:11.605 +0000] INFO (config-service/79 on b72ca056aa84): TIER_3_RATE_LIMIT = 1600
[2025-06-04 15:53:11.605 +0000] INFO (config-service/79 on b72ca056aa84): USE_ASYNC_TX_PROCESSING = true
[2025-06-04 15:53:11.605 +0000] INFO (config-service/79 on b72ca056aa84): WEB_SOCKET_HTTP_PORT = 8547
[2025-06-04 15:53:11.605 +0000] INFO (config-service/79 on b72ca056aa84): WEB_SOCKET_PORT = 8546
[2025-06-04 15:53:11.605 +0000] INFO (config-service/79 on b72ca056aa84): WS_BATCH_REQUESTS_ENABLED = true
[2025-06-04 15:53:11.605 +0000] INFO (config-service/79 on b72ca056aa84): WS_BATCH_REQUESTS_MAX_SIZE = 20
[2025-06-04 15:53:11.605 +0000] INFO (config-service/79 on b72ca056aa84): WS_CACHE_TTL = 20000
[2025-06-04 15:53:11.605 +0000] INFO (config-service/79 on b72ca056aa84): WS_CONNECTION_LIMIT = 10
[2025-06-04 15:53:11.605 +0000] INFO (config-service/79 on b72ca056aa84): WS_CONNECTION_LIMIT_PER_IP = 10
[2025-06-04 15:53:11.605 +0000] INFO (config-service/79 on b72ca056aa84): WS_MAX_INACTIVITY_TTL = 300000
[2025-06-04 15:53:11.605 +0000] INFO (config-service/79 on b72ca056aa84): WS_MULTIPLE_ADDRESSES_ENABLED = false
[2025-06-04 15:53:11.605 +0000] INFO (config-service/79 on b72ca056aa84): WS_NEW_HEADS_ENABLED = true
[2025-06-04 15:53:11.605 +0000] INFO (config-service/79 on b72ca056aa84): WS_PING_INTERVAL = 100000
[2025-06-04 15:53:11.605 +0000] INFO (config-service/79 on b72ca056aa84): WS_POLLING_INTERVAL = 500
[2025-06-04 15:53:11.605 +0000] INFO (config-service/79 on b72ca056aa84): WS_RELAY_URL = ws://127.0.0.1:8546
[2025-06-04 15:53:11.605 +0000] INFO (config-service/79 on b72ca056aa84): WS_SAME_SUB_FOR_SAME_EVENT = true
[2025-06-04 15:53:11.605 +0000] INFO (config-service/79 on b72ca056aa84): WS_SUBSCRIPTION_LIMIT = 10

[francesco-gaglione]

Hi @Neurone ,
I tried running hiero-local-node in a clean environment using docker system prune -a, but the issue still persists.

Since the error seems related to a missing account response from the mirror node, I’m sharing the relevant logs. From the relay service:

[2025-06-06 09:47:44.722 +0000] DEBUG (mirror-node/103 on e6d433d24d6b): [Request ID: cb57b1dc-e427-49ac-b005-00cddc7beec4] An accepted error occurred while communicating with the mirror node server: method=GET, path=accounts/0.0.2?transactions=false, status=404

[2025-06-06 09:47:44.722 +0000] ERROR (relay/103 on e6d433d24d6b): Error collecting operator balance. Skipping balance set

    err: {
      "type": "TypeError",
      "message": "Cannot read properties of null (reading 'balance')",
      "stack":
          TypeError: Cannot read properties of null (reading 'balance')
              at Gauge.<anonymous> (/home/node/app/packages/relay/dist/lib/relay.js:138:62)
              at Generator.next (<anonymous>)
              at fulfilled (/home/node/app/packages/relay/dist/lib/relay.js:6:58)
              at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    }

And from mirror-node-rest:

2025-06-06T09:46:27.523Z WARN Startup Deprecated 'hedera' property automatically migrated to 'hiero': HEDERA_MIRROR_REST_DB_HOST
2025-06-06T09:46:27.810Z ERROR Startup Error connecting to redis://127.0.0.1:6379: connect ECONNREFUSED 127.0.0.1:6379
...
2025-06-06T09:46:27.851Z WARN ::ffff:172.20.0.12 GET /api/v1/transactions/0.0.2-... 404 Not found NotFoundError Not found

It seems the account is not found, which causes the relay to fail when trying to access its balance.

Let me know if there’s anything else I can try to further debug this!

Side note: the strange thing is that the node was working perfectly and had been working perfectly for a long time (weeks if not months). suddenly after a docker compose down followed by a docker compose up without any changes to any files, the problem arose. On a final note, the problem seems to have arisen following some tests which consisted of deploying a contract

In the meantime I am retrieving another linux environment on which to replicate the case, I will update you.