JetStream startup seems slow for ~50M messages #8001

zig · 2026-04-02T16:46:30Z

zig
Apr 2, 2026

I have a queue of ~50M messages, events with 7 days retention. I feel it's rather slow to restart the server, even though it has plenty of CPU, the storage is CEPH, can do > 200MB/s, the data on disk take ~ 7GB.
Is that expected that it would take more than 6 minutes to startup ? Is it scanning all the data on disk on startup ?

wallyqs · 2026-04-02T16:52:37Z

wallyqs
Apr 2, 2026
Collaborator

is it a version past v2.11.11? newer versions do this in parallel

0 replies

zig · 2026-04-02T16:57:20Z

zig
Apr 2, 2026
Author

It is v2.12.5

19 replies

derekcollison Apr 4, 2026
Maintainer

Ok that does not seem normal given not that many subjects.

Could you grab a CPU profile on startup? And possibly a stacksz?

zig Apr 4, 2026
Author

Ok, stacksz I see is an endpoint, I can get this. Cpu profile, do you mean through a particular tool ?

zig Apr 4, 2026
Author

oh I think I saw, I can do nats --context sys server request profile cpu

derekcollison Apr 4, 2026
Maintainer

Same for go routines.

zig Apr 4, 2026
Author

oh no actually I can't do that while it's starting up. Meanwhile, here is the the stacksz :

goroutine 403 [running]:
github.com/nats-io/nats-server/v2/server.(*Server).HandleStacksz(0xffffb5bdaf30?, {0x933fa0, 0x40008a4000}, 0x400022e000)
	github.com/nats-io/nats-server/v2/server/monitor.go:1151 +0x68
net/http.HandlerFunc.ServeHTTP(0x40002f1080?, {0x933fa0?, 0x40008a4000?}, 0x25df38?)
	net/http/server.go:2322 +0x38
net/http.(*ServeMux).ServeHTTP(0x0?, {0x933fa0, 0x40008a4000}, 0x400022e000)
	net/http/server.go:2861 +0x190
net/http.serverHandler.ServeHTTP({0x4000d84080?}, {0x933fa0?, 0x40008a4000?}, 0x1?)
	net/http/server.go:3340 +0xb0
net/http.(*conn).serve(0x40004021b0, {0x934be8, 0x40007c1200})
	net/http/server.go:2109 +0x528
created by net/http.(*Server).Serve in goroutine 7
	net/http/server.go:3493 +0x384

goroutine 1 [sync.WaitGroup.Wait]:
sync.runtime_SemacquireWaitGroup(0x40002202a0?, 0x90?)
	runtime/sema.go:114 +0x38
sync.(*WaitGroup).Wait(0x4000622bf0)
	sync/waitgroup.go:206 +0xa8
github.com/nats-io/nats-server/v2/server.(*Account).EnableJetStream(0x4000548a88, 0x400020f200, 0x40002202a0)
	github.com/nats-io/nats-server/v2/server/jetstream.go:1665 +0x145c
github.com/nats-io/nats-server/v2/server.(*Server).configJetStream(0x40000d1008, 0x4000548a88, 0x40002202a0)
	github.com/nats-io/nats-server/v2/server/jetstream.go:791 +0x178
github.com/nats-io/nats-server/v2/server.(*Server).configAllJetStreamAccounts(0x40000d1008, 0x40002202a0)
	github.com/nats-io/nats-server/v2/server/jetstream.go:852 +0x264
github.com/nats-io/nats-server/v2/server.(*Server).enableJetStreamAccounts(0x40000d1008)
	github.com/nats-io/nats-server/v2/server/jetstream.go:710 +0x110
github.com/nats-io/nats-server/v2/server.(*Server).enableJetStream(0x40000d1008, {0x0, 0x3200000000, {0x4000705ab0, 0xf}, 0x1bf08eb000, 0x0, {0x4000366672, 0x5}, 0x0, ...})
	github.com/nats-io/nats-server/v2/server/jetstream.go:512 +0xa20
github.com/nats-io/nats-server/v2/server.(*Server).EnableJetStream(0x40000d1008, 0x4000287db0)
	github.com/nats-io/nats-server/v2/server/jetstream.go:236 +0x44c
github.com/nats-io/nats-server/v2/server.(*Server).Start(0x40000d1008)
	github.com/nats-io/nats-server/v2/server/server.go:2454 +0xe70
github.com/nats-io/nats-server/v2/server.Run(...)
	github.com/nats-io/nats-server/v2/server/service.go:21
main.main()
	github.com/nats-io/nats-server/v2/main.go:129 +0x268

goroutine 3 [syscall]:
os/signal.signal_recv()
	runtime/sigqueue.go:152 +0x30
os/signal.loop()
	os/signal/signal_unix.go:23 +0x1c
created by os/signal.Notify.func1.1 in goroutine 1
	os/signal/signal.go:152 +0x28

goroutine 81 [select]:
github.com/nats-io/nats-server/v2/server.(*Server).handleSignals.func1()
	github.com/nats-io/nats-server/v2/server/signal.go:47 +0x6c
created by github.com/nats-io/nats-server/v2/server.(*Server).handleSignals in goroutine 1
	github.com/nats-io/nats-server/v2/server/signal.go:45 +0xdc

goroutine 82 [select]:
github.com/nats-io/nats-server/v2/server.(*Server).startRateLimitLogExpiration.func1()
	github.com/nats-io/nats-server/v2/server/server.go:4713 +0xd4
github.com/nats-io/nats-server/v2/server.(*Server).startGoRoutine.func1()
	github.com/nats-io/nats-server/v2/server/server.go:4078 +0x34
created by github.com/nats-io/nats-server/v2/server.(*Server).startGoRoutine in goroutine 1
	github.com/nats-io/nats-server/v2/server/server.go:4076 +0x134

goroutine 83 [select]:
github.com/nats-io/nats-server/v2/server.(*sendq).internalLoop(0x4000716020)
	github.com/nats-io/nats-server/v2/server/sendq.go:65 +0x21c
github.com/nats-io/nats-server/v2/server.(*Server).startGoRoutine.func1()
	github.com/nats-io/nats-server/v2/server/server.go:4078 +0x34
created by github.com/nats-io/nats-server/v2/server.(*Server).startGoRoutine in goroutine 1
	github.com/nats-io/nats-server/v2/server/server.go:4076 +0x134

goroutine 4 [select]:
github.com/nats-io/nats-server/v2/server.(*Server).internalSendLoop(0x40000d1008, 0x0?)
	github.com/nats-io/nats-server/v2/server/events.go:524 +0x268
created by github.com/nats-io/nats-server/v2/server.(*Server).setSystemAccount in goroutine 1
	github.com/nats-io/nats-server/v2/server/server.go:1881 +0x31c

goroutine 5 [select]:
github.com/nats-io/nats-server/v2/server.(*Server).internalReceiveLoop(0x40000d1008, 0x4000708100)
	github.com/nats-io/nats-server/v2/server/events.go:478 +0x84
created by github.com/nats-io/nats-server/v2/server.(*Server).setSystemAccount in goroutine 1
	github.com/nats-io/nats-server/v2/server/server.go:1884 +0x364

goroutine 6 [select]:
github.com/nats-io/nats-server/v2/server.(*Server).internalReceiveLoop(0x40000d1008, 0x4000708180)
	github.com/nats-io/nats-server/v2/server/events.go:478 +0x84
created by github.com/nats-io/nats-server/v2/server.(*Server).setSystemAccount in goroutine 1
	github.com/nats-io/nats-server/v2/server/server.go:1886 +0x3ac

goroutine 7 [IO wait]:
internal/poll.runtime_pollWait(0xffffb5a15000, 0x72)
	runtime/netpoll.go:351 +0xa0
internal/poll.(*pollDesc).wait(0x40002a6000?, 0x29ca0?, 0x0)
	internal/poll/fd_poll_runtime.go:84 +0x28
internal/poll.(*pollDesc).waitRead(...)
	internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Accept(0x40002a6000)
	internal/poll/fd_unix.go:613 +0x21c
net.(*netFD).accept(0x40002a6000)
	net/fd_unix.go:161 +0x28
net.(*TCPListener).accept(0x4000729480)
	net/tcpsock_posix.go:159 +0x24
net.(*TCPListener).Accept(0x4000729480)
	net/tcpsock.go:380 +0x2c
net/http.(*Server).Serve(0x4000730200, {0x934060, 0x4000729480})
	net/http/server.go:3463 +0x24c
github.com/nats-io/nats-server/v2/server.(*Server).startMonitoring.func1()
	github.com/nats-io/nats-server/v2/server/server.go:3189 +0x38
created by github.com/nats-io/nats-server/v2/server.(*Server).startMonitoring in goroutine 1
	github.com/nats-io/nats-server/v2/server/server.go:3188 +0xfd0

goroutine 8 [select]:
github.com/nats-io/nats-server/v2/server.(*Server).startGWReplyMapExpiration.func1()
	github.com/nats-io/nats-server/v2/server/gateway.go:3381 +0xc0
github.com/nats-io/nats-server/v2/server.(*Server).startGoRoutine.func1()
	github.com/nats-io/nats-server/v2/server/server.go:4078 +0x34
created by github.com/nats-io/nats-server/v2/server.(*Server).startGoRoutine in goroutine 1
	github.com/nats-io/nats-server/v2/server/server.go:4076 +0x134

goroutine 9 [select]:
github.com/nats-io/nats-server/v2/server.(*Server).processJSAPIRoutedRequests(0x40000d1008)
	github.com/nats-io/nats-server/v2/server/jetstream_api.go:945 +0x18c
github.com/nats-io/nats-server/v2/server.(*Server).startGoRoutine.func1()
	github.com/nats-io/nats-server/v2/server/server.go:4078 +0x34
created by github.com/nats-io/nats-server/v2/server.(*Server).startGoRoutine in goroutine 1
	github.com/nats-io/nats-server/v2/server/server.go:4076 +0x134

goroutine 10 [select]:
github.com/nats-io/nats-server/v2/server.(*Server).processJSAPIRoutedRequests(0x40000d1008)
	github.com/nats-io/nats-server/v2/server/jetstream_api.go:945 +0x18c
github.com/nats-io/nats-server/v2/server.(*Server).startGoRoutine.func1()
	github.com/nats-io/nats-server/v2/server/server.go:4078 +0x34
created by github.com/nats-io/nats-server/v2/server.(*Server).startGoRoutine in goroutine 1
	github.com/nats-io/nats-server/v2/server/server.go:4076 +0x134

goroutine 11 [select]:
github.com/nats-io/nats-server/v2/server.(*Server).processJSAPIRoutedRequests(0x40000d1008)
	github.com/nats-io/nats-server/v2/server/jetstream_api.go:945 +0x18c
github.com/nats-io/nats-server/v2/server.(*Server).startGoRoutine.func1()
	github.com/nats-io/nats-server/v2/server/server.go:4078 +0x34
created by github.com/nats-io/nats-server/v2/server.(*Server).startGoRoutine in goroutine 1
	github.com/nats-io/nats-server/v2/server/server.go:4076 +0x134

goroutine 12 [select]:
github.com/nats-io/nats-server/v2/server.(*Server).processJSAPIRoutedRequests(0x40000d1008)
	github.com/nats-io/nats-server/v2/server/jetstream_api.go:945 +0x18c
github.com/nats-io/nats-server/v2/server.(*Server).startGoRoutine.func1()
	github.com/nats-io/nats-server/v2/server/server.go:4078 +0x34
created by github.com/nats-io/nats-server/v2/server.(*Server).startGoRoutine in goroutine 1
	github.com/nats-io/nats-server/v2/server/server.go:4076 +0x134

goroutine 13 [select]:
github.com/nats-io/nats-server/v2/server.(*Server).processJSAPIRoutedRequests(0x40000d1008)
	github.com/nats-io/nats-server/v2/server/jetstream_api.go:945 +0x18c
github.com/nats-io/nats-server/v2/server.(*Server).startGoRoutine.func1()
	github.com/nats-io/nats-server/v2/server/server.go:4078 +0x34
created by github.com/nats-io/nats-server/v2/server.(*Server).startGoRoutine in goroutine 1
	github.com/nats-io/nats-server/v2/server/server.go:4076 +0x134

goroutine 14 [select]:
github.com/nats-io/nats-server/v2/server.(*Server).processJSAPIRoutedRequests(0x40000d1008)
	github.com/nats-io/nats-server/v2/server/jetstream_api.go:945 +0x18c
github.com/nats-io/nats-server/v2/server.(*Server).startGoRoutine.func1()
	github.com/nats-io/nats-server/v2/server/server.go:4078 +0x34
created by github.com/nats-io/nats-server/v2/server.(*Server).startGoRoutine in goroutine 1
	github.com/nats-io/nats-server/v2/server/server.go:4076 +0x134

goroutine 15 [select]:
github.com/nats-io/nats-server/v2/server.(*Server).processJSAPIRoutedRequests(0x40000d1008)
	github.com/nats-io/nats-server/v2/server/jetstream_api.go:945 +0x18c
github.com/nats-io/nats-server/v2/server.(*Server).startGoRoutine.func1()
	github.com/nats-io/nats-server/v2/server/server.go:4078 +0x34
created by github.com/nats-io/nats-server/v2/server.(*Server).startGoRoutine in goroutine 1
	github.com/nats-io/nats-server/v2/server/server.go:4076 +0x134

goroutine 16 [select]:
github.com/nats-io/nats-server/v2/server.(*Server).processJSAPIRoutedRequests(0x40000d1008)
	github.com/nats-io/nats-server/v2/server/jetstream_api.go:945 +0x18c
github.com/nats-io/nats-server/v2/server.(*Server).startGoRoutine.func1()
	github.com/nats-io/nats-server/v2/server/server.go:4078 +0x34
created by github.com/nats-io/nats-server/v2/server.(*Server).startGoRoutine in goroutine 1
	github.com/nats-io/nats-server/v2/server/server.go:4076 +0x134

goroutine 97 [select]:
github.com/nats-io/nats-server/v2/server.(*Server).processJSAPIRoutedRequests(0x40000d1008)
	github.com/nats-io/nats-server/v2/server/jetstream_api.go:945 +0x18c
github.com/nats-io/nats-server/v2/server.(*Server).startGoRoutine.func1()
	github.com/nats-io/nats-server/v2/server/server.go:4078 +0x34
created by github.com/nats-io/nats-server/v2/server.(*Server).startGoRoutine in goroutine 1
	github.com/nats-io/nats-server/v2/server/server.go:4076 +0x134

goroutine 98 [select]:
github.com/nats-io/nats-server/v2/server.(*Server).processJSAPIRoutedRequests(0x40000d1008)
	github.com/nats-io/nats-server/v2/server/jetstream_api.go:945 +0x18c
github.com/nats-io/nats-server/v2/server.(*Server).startGoRoutine.func1()
	github.com/nats-io/nats-server/v2/server/server.go:4078 +0x34
created by github.com/nats-io/nats-server/v2/server.(*Server).startGoRoutine in goroutine 1
	github.com/nats-io/nats-server/v2/server/server.go:4076 +0x134

goroutine 99 [select]:
github.com/nats-io/nats-server/v2/server.(*Server).processJSAPIRoutedRequests(0x40000d1008)
	github.com/nats-io/nats-server/v2/server/jetstream_api.go:945 +0x18c
github.com/nats-io/nats-server/v2/server.(*Server).startGoRoutine.func1()
	github.com/nats-io/nats-server/v2/server/server.go:4078 +0x34
created by github.com/nats-io/nats-server/v2/server.(*Server).startGoRoutine in goroutine 1
	github.com/nats-io/nats-server/v2/server/server.go:4076 +0x134

goroutine 100 [select]:
github.com/nats-io/nats-server/v2/server.(*Server).processJSAPIRoutedRequests(0x40000d1008)
	github.com/nats-io/nats-server/v2/server/jetstream_api.go:945 +0x18c
github.com/nats-io/nats-server/v2/server.(*Server).startGoRoutine.func1()
	github.com/nats-io/nats-server/v2/server/server.go:4078 +0x34
created by github.com/nats-io/nats-server/v2/server.(*Server).startGoRoutine in goroutine 1
	github.com/nats-io/nats-server/v2/server/server.go:4076 +0x134

goroutine 101 [select]:
github.com/nats-io/nats-server/v2/server.(*Server).processJSAPIRoutedRequests(0x40000d1008)
	github.com/nats-io/nats-server/v2/server/jetstream_api.go:945 +0x18c
github.com/nats-io/nats-server/v2/server.(*Server).startGoRoutine.func1()
	github.com/nats-io/nats-server/v2/server/server.go:4078 +0x34
created by github.com/nats-io/nats-server/v2/server.(*Server).startGoRoutine in goroutine 1
	github.com/nats-io/nats-server/v2/server/server.go:4076 +0x134

goroutine 102 [select]:
github.com/nats-io/nats-server/v2/server.(*Server).processJSAPIRoutedRequests(0x40000d1008)
	github.com/nats-io/nats-server/v2/server/jetstream_api.go:945 +0x18c
github.com/nats-io/nats-server/v2/server.(*Server).startGoRoutine.func1()
	github.com/nats-io/nats-server/v2/server/server.go:4078 +0x34
created by github.com/nats-io/nats-server/v2/server.(*Server).startGoRoutine in goroutine 1
	github.com/nats-io/nats-server/v2/server/server.go:4076 +0x134

goroutine 103 [select]:
github.com/nats-io/nats-server/v2/server.(*Server).processJSAPIRoutedRequests(0x40000d1008)
	github.com/nats-io/nats-server/v2/server/jetstream_api.go:945 +0x18c
github.com/nats-io/nats-server/v2/server.(*Server).startGoRoutine.func1()
	github.com/nats-io/nats-server/v2/server/server.go:4078 +0x34
created by github.com/nats-io/nats-server/v2/server.(*Server).startGoRoutine in goroutine 1
	github.com/nats-io/nats-server/v2/server/server.go:4076 +0x134

goroutine 104 [select]:
github.com/nats-io/nats-server/v2/server.(*Server).processJSAPIRoutedRequests(0x40000d1008)
	github.com/nats-io/nats-server/v2/server/jetstream_api.go:945 +0x18c
github.com/nats-io/nats-server/v2/server.(*Server).startGoRoutine.func1()
	github.com/nats-io/nats-server/v2/server/server.go:4078 +0x34
created by github.com/nats-io/nats-server/v2/server.(*Server).startGoRoutine in goroutine 1
	github.com/nats-io/nats-server/v2/server/server.go:4076 +0x134

goroutine 105 [chan receive]:
github.com/nats-io/nats-server/v2/server.parallelTaskQueue.func1()
	github.com/nats-io/nats-server/v2/server/util.go:360 +0x44
created by github.com/nats-io/nats-server/v2/server.parallelTaskQueue in goroutine 1
	github.com/nats-io/nats-server/v2/server/util.go:359 +0x60

goroutine 106 [chan receive]:
github.com/nats-io/nats-server/v2/server.parallelTaskQueue.func1()
	github.com/nats-io/nats-server/v2/server/util.go:360 +0x44
created by github.com/nats-io/nats-server/v2/server.parallelTaskQueue in goroutine 1
	github.com/nats-io/nats-server/v2/server/util.go:359 +0x60

goroutine 107 [chan receive]:
github.com/nats-io/nats-server/v2/server.parallelTaskQueue.func1()
	github.com/nats-io/nats-server/v2/server/util.go:360 +0x44
created by github.com/nats-io/nats-server/v2/server.parallelTaskQueue in goroutine 1
	github.com/nats-io/nats-server/v2/server/util.go:359 +0x60

goroutine 108 [chan receive]:
github.com/nats-io/nats-server/v2/server.parallelTaskQueue.func1()
	github.com/nats-io/nats-server/v2/server/util.go:360 +0x44
created by github.com/nats-io/nats-server/v2/server.parallelTaskQueue in goroutine 1
	github.com/nats-io/nats-server/v2/server/util.go:359 +0x60

goroutine 109 [chan receive]:
github.com/nats-io/nats-server/v2/server.parallelTaskQueue.func1()
	github.com/nats-io/nats-server/v2/server/util.go:360 +0x44
created by github.com/nats-io/nats-server/v2/server.parallelTaskQueue in goroutine 1
	github.com/nats-io/nats-server/v2/server/util.go:359 +0x60

goroutine 110 [chan receive]:
github.com/nats-io/nats-server/v2/server.parallelTaskQueue.func1()
	github.com/nats-io/nats-server/v2/server/util.go:360 +0x44
created by github.com/nats-io/nats-server/v2/server.parallelTaskQueue in goroutine 1
	github.com/nats-io/nats-server/v2/server/util.go:359 +0x60

goroutine 111 [syscall]:
syscall.Syscall(0x3f, 0x7, 0x40011c2000, 0xf0420)
	syscall/syscall_linux.go:74 +0x20
syscall.read(0x400064e4e0?, {0x40011c2000?, 0x40003f46d8?, 0x1efb4?})
	syscall/zsyscall_linux_arm64.go:736 +0x40
syscall.Read(...)
	syscall/syscall_unix.go:183
internal/poll.ignoringEINTRIO(...)
	internal/poll/fd_unix.go:738
internal/poll.(*FD).Read(0x400064e4e0, {0x40011c2000, 0xf0420, 0x100000})
	internal/poll/fd_unix.go:161 +0x204
os.(*File).read(...)
	os/file_posix.go:29
os.(*File).Read(0x400055c008, {0x40011c2000?, 0x40003f4788?, 0x3f5530?})
	os/file.go:144 +0x68
io.ReadAtLeast({0x92f300, 0x400055c008}, {0x40011c2000, 0xf0420, 0x100000}, 0xf0420)
	io/io.go:335 +0x98
io.ReadFull(...)
	io/io.go:354
github.com/nats-io/nats-server/v2/server.(*msgBlock).loadBlock(0x40003d2960, {0x0, 0x13b950f8c?, 0x0})
	github.com/nats-io/nats-server/v2/server/filestore.go:7729 +0x190
github.com/nats-io/nats-server/v2/server.(*msgBlock).loadMsgsWithLock(0x40003d2960)
	github.com/nats-io/nats-server/v2/server/filestore.go:7784 +0x1b8
github.com/nats-io/nats-server/v2/server.(*msgBlock).prevMatchingMulti(0x40003d2960, 0x40008a3500, 0x3debbc, 0x400015c7e0)
	github.com/nats-io/nats-server/v2/server/filestore.go:2967 +0x8c
github.com/nats-io/nats-server/v2/server.(*fileStore).LoadPrevMsgMulti(0x40003f8008, 0x40008a3500, 0x4000cc1ae0?, 0x400015c7e0)
	github.com/nats-io/nats-server/v2/server/filestore.go:8597 +0x240
github.com/nats-io/nats-server/v2/server.(*stream).startingSequenceForSources(0x40000de488)
	github.com/nats-io/nats-server/v2/server/stream.go:4311 +0x344
github.com/nats-io/nats-server/v2/server.(*stream).setupSourceConsumers(0x40000de488)
	github.com/nats-io/nats-server/v2/server/stream.go:4358 +0x10c
github.com/nats-io/nats-server/v2/server.(*stream).subscribeToStream(0x40000de488)
	github.com/nats-io/nats-server/v2/server/stream.go:4408 +0x1a8
github.com/nats-io/nats-server/v2/server.(*stream).setLeader(0x40000de488, 0x1)
	github.com/nats-io/nats-server/v2/server/stream.go:1166 +0x15c
github.com/nats-io/nats-server/v2/server.(*Account).addStreamWithAssignment(0x4000548a88, 0x40003dc438, 0x0, 0x0, 0x0, 0x1)
	github.com/nats-io/nats-server/v2/server/stream.go:966 +0x1764
github.com/nats-io/nats-server/v2/server.(*Account).recoverStream(...)
	github.com/nats-io/nats-server/v2/server/stream.go:628
github.com/nats-io/nats-server/v2/server.(*Account).EnableJetStream.func2({0x934530, 0x4000265b80})
	github.com/nats-io/nats-server/v2/server/jetstream.go:1566 +0x18d4
github.com/nats-io/nats-server/v2/server.(*Account).EnableJetStream.func3()
	github.com/nats-io/nats-server/v2/server/jetstream.go:1661 +0x2c
github.com/nats-io/nats-server/v2/server.parallelTaskQueue.func1()
	github.com/nats-io/nats-server/v2/server/util.go:361 +0x38
created by github.com/nats-io/nats-server/v2/server.parallelTaskQueue in goroutine 1
	github.com/nats-io/nats-server/v2/server/util.go:359 +0x60

goroutine 112 [chan receive]:
github.com/nats-io/nats-server/v2/server.parallelTaskQueue.func1()
	github.com/nats-io/nats-server/v2/server/util.go:360 +0x44
created by github.com/nats-io/nats-server/v2/server.parallelTaskQueue in goroutine 1
	github.com/nats-io/nats-server/v2/server/util.go:359 +0x60

goroutine 113 [chan receive]:
github.com/nats-io/nats-server/v2/server.parallelTaskQueue.func1()
	github.com/nats-io/nats-server/v2/server/util.go:360 +0x44
created by github.com/nats-io/nats-server/v2/server.parallelTaskQueue in goroutine 1
	github.com/nats-io/nats-server/v2/server/util.go:359 +0x60

goroutine 114 [chan receive]:
github.com/nats-io/nats-server/v2/server.parallelTaskQueue.func1()
	github.com/nats-io/nats-server/v2/server/util.go:360 +0x44
created by github.com/nats-io/nats-server/v2/server.parallelTaskQueue in goroutine 1
	github.com/nats-io/nats-server/v2/server/util.go:359 +0x60

goroutine 115 [chan receive]:
github.com/nats-io/nats-server/v2/server.parallelTaskQueue.func1()
	github.com/nats-io/nats-server/v2/server/util.go:360 +0x44
created by github.com/nats-io/nats-server/v2/server.parallelTaskQueue in goroutine 1
	github.com/nats-io/nats-server/v2/server/util.go:359 +0x60

goroutine 116 [chan receive]:
github.com/nats-io/nats-server/v2/server.parallelTaskQueue.func1()
	github.com/nats-io/nats-server/v2/server/util.go:360 +0x44
created by github.com/nats-io/nats-server/v2/server.parallelTaskQueue in goroutine 1
	github.com/nats-io/nats-server/v2/server/util.go:359 +0x60

goroutine 117 [chan receive]:
github.com/nats-io/nats-server/v2/server.parallelTaskQueue.func1()
	github.com/nats-io/nats-server/v2/server/util.go:360 +0x44
created by github.com/nats-io/nats-server/v2/server.parallelTaskQueue in goroutine 1
	github.com/nats-io/nats-server/v2/server/util.go:359 +0x60

goroutine 118 [chan receive]:
github.com/nats-io/nats-server/v2/server.parallelTaskQueue.func1()
	github.com/nats-io/nats-server/v2/server/util.go:360 +0x44
created by github.com/nats-io/nats-server/v2/server.parallelTaskQueue in goroutine 1
	github.com/nats-io/nats-server/v2/server/util.go:359 +0x60

goroutine 119 [chan receive]:
github.com/nats-io/nats-server/v2/server.parallelTaskQueue.func1()
	github.com/nats-io/nats-server/v2/server/util.go:360 +0x44
created by github.com/nats-io/nats-server/v2/server.parallelTaskQueue in goroutine 1
	github.com/nats-io/nats-server/v2/server/util.go:359 +0x60

goroutine 120 [chan receive]:
github.com/nats-io/nats-server/v2/server.parallelTaskQueue.func1()
	github.com/nats-io/nats-server/v2/server/util.go:360 +0x44
created by github.com/nats-io/nats-server/v2/server.parallelTaskQueue in goroutine 1
	github.com/nats-io/nats-server/v2/server/util.go:359 +0x60

goroutine 121 [chan receive]:
github.com/nats-io/nats-server/v2/server.parallelTaskQueue.func1()
	github.com/nats-io/nats-server/v2/server/util.go:360 +0x44
created by github.com/nats-io/nats-server/v2/server.parallelTaskQueue in goroutine 1
	github.com/nats-io/nats-server/v2/server/util.go:359 +0x60

goroutine 122 [chan receive]:
github.com/nats-io/nats-server/v2/server.parallelTaskQueue.func1()
	github.com/nats-io/nats-server/v2/server/util.go:360 +0x44
created by github.com/nats-io/nats-server/v2/server.parallelTaskQueue in goroutine 1
	github.com/nats-io/nats-server/v2/server/util.go:359 +0x60

goroutine 123 [chan receive]:
github.com/nats-io/nats-server/v2/server.parallelTaskQueue.func1()
	github.com/nats-io/nats-server/v2/server/util.go:360 +0x44
created by github.com/nats-io/nats-server/v2/server.parallelTaskQueue in goroutine 1
	github.com/nats-io/nats-server/v2/server/util.go:359 +0x60

goroutine 124 [chan receive]:
github.com/nats-io/nats-server/v2/server.parallelTaskQueue.func1()
	github.com/nats-io/nats-server/v2/server/util.go:360 +0x44
created by github.com/nats-io/nats-server/v2/server.parallelTaskQueue in goroutine 1
	github.com/nats-io/nats-server/v2/server/util.go:359 +0x60

goroutine 125 [chan receive]:
github.com/nats-io/nats-server/v2/server.parallelTaskQueue.func1()
	github.com/nats-io/nats-server/v2/server/util.go:360 +0x44
created by github.com/nats-io/nats-server/v2/server.parallelTaskQueue in goroutine 1
	github.com/nats-io/nats-server/v2/server/util.go:359 +0x60

goroutine 126 [chan receive]:
github.com/nats-io/nats-server/v2/server.parallelTaskQueue.func1()
	github.com/nats-io/nats-server/v2/server/util.go:360 +0x44
created by github.com/nats-io/nats-server/v2/server.parallelTaskQueue in goroutine 1
	github.com/nats-io/nats-server/v2/server/util.go:359 +0x60

goroutine 127 [chan receive]:
github.com/nats-io/nats-server/v2/server.parallelTaskQueue.func1()
	github.com/nats-io/nats-server/v2/server/util.go:360 +0x44
created by github.com/nats-io/nats-server/v2/server.parallelTaskQueue in goroutine 1
	github.com/nats-io/nats-server/v2/server/util.go:359 +0x60

goroutine 128 [chan receive]:
github.com/nats-io/nats-server/v2/server.parallelTaskQueue.func1()
	github.com/nats-io/nats-server/v2/server/util.go:360 +0x44
created by github.com/nats-io/nats-server/v2/server.parallelTaskQueue in goroutine 1
	github.com/nats-io/nats-server/v2/server/util.go:359 +0x60

goroutine 129 [chan receive]:
github.com/nats-io/nats-server/v2/server.parallelTaskQueue.func1()
	github.com/nats-io/nats-server/v2/server/util.go:360 +0x44
created by github.com/nats-io/nats-server/v2/server.parallelTaskQueue in goroutine 1
	github.com/nats-io/nats-server/v2/server/util.go:359 +0x60

goroutine 130 [chan receive]:
github.com/nats-io/nats-server/v2/server.parallelTaskQueue.func1()
	github.com/nats-io/nats-server/v2/server/util.go:360 +0x44
created by github.com/nats-io/nats-server/v2/server.parallelTaskQueue in goroutine 1
	github.com/nats-io/nats-server/v2/server/util.go:359 +0x60

goroutine 131 [chan receive]:
github.com/nats-io/nats-server/v2/server.parallelTaskQueue.func1()
	github.com/nats-io/nats-server/v2/server/util.go:360 +0x44
created by github.com/nats-io/nats-server/v2/server.parallelTaskQueue in goroutine 1
	github.com/nats-io/nats-server/v2/server/util.go:359 +0x60

goroutine 132 [chan receive]:
github.com/nats-io/nats-server/v2/server.parallelTaskQueue.func1()
	github.com/nats-io/nats-server/v2/server/util.go:360 +0x44
created by github.com/nats-io/nats-server/v2/server.parallelTaskQueue in goroutine 1
	github.com/nats-io/nats-server/v2/server/util.go:359 +0x60

goroutine 133 [chan receive]:
github.com/nats-io/nats-server/v2/server.parallelTaskQueue.func1()
	github.com/nats-io/nats-server/v2/server/util.go:360 +0x44
created by github.com/nats-io/nats-server/v2/server.parallelTaskQueue in goroutine 1
	github.com/nats-io/nats-server/v2/server/util.go:359 +0x60

goroutine 134 [chan receive]:
github.com/nats-io/nats-server/v2/server.parallelTaskQueue.func1()
	github.com/nats-io/nats-server/v2/server/util.go:360 +0x44
created by github.com/nats-io/nats-server/v2/server.parallelTaskQueue in goroutine 1
	github.com/nats-io/nats-server/v2/server/util.go:359 +0x60

goroutine 135 [chan receive]:
github.com/nats-io/nats-server/v2/server.parallelTaskQueue.func1()
	github.com/nats-io/nats-server/v2/server/util.go:360 +0x44
created by github.com/nats-io/nats-server/v2/server.parallelTaskQueue in goroutine 1
	github.com/nats-io/nats-server/v2/server/util.go:359 +0x60

goroutine 136 [chan receive]:
github.com/nats-io/nats-server/v2/server.parallelTaskQueue.func1()
	github.com/nats-io/nats-server/v2/server/util.go:360 +0x44
created by github.com/nats-io/nats-server/v2/server.parallelTaskQueue in goroutine 1
	github.com/nats-io/nats-server/v2/server/util.go:359 +0x60

goroutine 137 [chan receive]:
github.com/nats-io/nats-server/v2/server.parallelTaskQueue.func1()
	github.com/nats-io/nats-server/v2/server/util.go:360 +0x44
created by github.com/nats-io/nats-server/v2/server.parallelTaskQueue in goroutine 1
	github.com/nats-io/nats-server/v2/server/util.go:359 +0x60

goroutine 138 [chan receive]:
github.com/nats-io/nats-server/v2/server.parallelTaskQueue.func1()
	github.com/nats-io/nats-server/v2/server/util.go:360 +0x44
created by github.com/nats-io/nats-server/v2/server.parallelTaskQueue in goroutine 1
	github.com/nats-io/nats-server/v2/server/util.go:359 +0x60

goroutine 139 [chan receive]:
github.com/nats-io/nats-server/v2/server.parallelTaskQueue.func1()
	github.com/nats-io/nats-server/v2/server/util.go:360 +0x44
created by github.com/nats-io/nats-server/v2/server.parallelTaskQueue in goroutine 1
	github.com/nats-io/nats-server/v2/server/util.go:359 +0x60

goroutine 140 [chan receive]:
github.com/nats-io/nats-server/v2/server.parallelTaskQueue.func1()
	github.com/nats-io/nats-server/v2/server/util.go:360 +0x44
created by github.com/nats-io/nats-server/v2/server.parallelTaskQueue in goroutine 1
	github.com/nats-io/nats-server/v2/server/util.go:359 +0x60

goroutine 145 [IO wait]:
internal/poll.runtime_pollWait(0xffffb5a14c00, 0x72)
	runtime/netpoll.go:351 +0xa0
internal/poll.(*pollDesc).wait(0x4000558000?, 0x40005c6000?, 0x0)
	internal/poll/fd_poll_runtime.go:84 +0x28
internal/poll.(*pollDesc).waitRead(...)
	internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0x4000558000, {0x40005c6000, 0x1000, 0x1000})
	internal/poll/fd_unix.go:165 +0x1e0
net.(*netFD).Read(0x4000558000, {0x40005c6000?, 0x4000649aa8?, 0x25e4ac?})
	net/fd_posix.go:68 +0x28
net.(*conn).Read(0x400055c000, {0x40005c6000?, 0x72?, 0x0?})
	net/net.go:196 +0x34
net/http.(*connReader).Read(0x40005b0000, {0x40005c6000, 0x1000, 0x1000})
	net/http/server.go:812 +0x230
bufio.(*Reader).fill(0x4000178000)
	bufio/bufio.go:113 +0xe0
bufio.(*Reader).Peek(0x4000178000, 0x4)
	bufio/bufio.go:152 +0x58
net/http.(*conn).serve(0x400055e000, {0x934be8, 0x40007c1200})
	net/http/server.go:2145 +0x654
created by net/http.(*Server).Serve in goroutine 7
	net/http/server.go:3493 +0x384

goroutine 225 [select]:
github.com/nats-io/nats-server/v2/server.(*consumerFileStore).flushLoop(0x40004300b0, 0x4000120d90, 0x4000120e00)
	github.com/nats-io/nats-server/v2/server/filestore.go:11430 +0xc4
created by github.com/nats-io/nats-server/v2/server.(*fileStore).ConsumerStore in goroutine 107
	github.com/nats-io/nats-server/v2/server/filestore.go:11301 +0x888

goroutine 24 [IO wait]:
internal/poll.runtime_pollWait(0xffffb5a14800, 0x72)
	runtime/netpoll.go:351 +0xa0
internal/poll.(*pollDesc).wait(0x4000378100?, 0x40003da000?, 0x0)
	internal/poll/fd_poll_runtime.go:84 +0x28
internal/poll.(*pollDesc).waitRead(...)
	internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0x4000378100, {0x40003da000, 0x1000, 0x1000})
	internal/poll/fd_unix.go:165 +0x1e0
net.(*netFD).Read(0x4000378100, {0x40003da000?, 0x40004dbaa8?, 0x25e4ac?})
	net/fd_posix.go:68 +0x28
net.(*conn).Read(0x4000718b58, {0x40003da000?, 0x72?, 0x0?})
	net/net.go:196 +0x34
net/http.(*connReader).Read(0x4000729680, {0x40003da000, 0x1000, 0x1000})
	net/http/server.go:812 +0x230
bufio.(*Reader).fill(0x400011c960)
	bufio/bufio.go:113 +0xe0
bufio.(*Reader).Peek(0x400011c960, 0x4)
	bufio/bufio.go:152 +0x58
net/http.(*conn).serve(0x40004022d0, {0x934be8, 0x40007c1200})
	net/http/server.go:2145 +0x654
created by net/http.(*Server).Serve in goroutine 7
	net/http/server.go:3493 +0x384

goroutine 141 [IO wait]:
internal/poll.runtime_pollWait(0xffffb5a14a00, 0x72)
	runtime/netpoll.go:351 +0xa0
internal/poll.(*pollDesc).wait(0x40002a6100?, 0x40001d8000?, 0x0)
	internal/poll/fd_poll_runtime.go:84 +0x28
internal/poll.(*pollDesc).waitRead(...)
	internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0x40002a6100, {0x40001d8000, 0x1000, 0x1000})
	internal/poll/fd_unix.go:165 +0x1e0
net.(*netFD).Read(0x40002a6100, {0x40001d8000?, 0x40007a9aa8?, 0x25e4ac?})
	net/fd_posix.go:68 +0x28
net.(*conn).Read(0x400037c008, {0x40001d8000?, 0x72?, 0x0?})
	net/net.go:196 +0x34
net/http.(*connReader).Read(0x4000490100, {0x40001d8000, 0x1000, 0x1000})
	net/http/server.go:812 +0x230
bufio.(*Reader).fill(0x40002515c0)
	bufio/bufio.go:113 +0xe0
bufio.(*Reader).Peek(0x40002515c0, 0x4)
	bufio/bufio.go:152 +0x58
net/http.(*conn).serve(0x400037e120, {0x934be8, 0x40007c1200})
	net/http/server.go:2145 +0x654
created by net/http.(*Server).Serve in goroutine 7
	net/http/server.go:3493 +0x384

goroutine 26 [select]:
github.com/nats-io/nats-server/v2/server.(*stream).signalConsumersLoop(0x40000de488)
	github.com/nats-io/nats-server/v2/server/stream.go:6733 +0x148
created by github.com/nats-io/nats-server/v2/server.(*Account).addStreamWithAssignment in goroutine 111
	github.com/nats-io/nats-server/v2/server/stream.go:855 +0x119c

goroutine 404 [runnable]:
syscall.Syscall(0x3f, 0xb, 0x4000d840a1, 0x1)
	syscall/syscall_linux.go:74 +0x20
syscall.read(0x4000708300?, {0x4000d840a1?, 0x0?, 0x0?})
	syscall/zsyscall_linux_arm64.go:736 +0x40
syscall.Read(...)
	syscall/syscall_unix.go:183
internal/poll.ignoringEINTRIO(...)
	internal/poll/fd_unix.go:738
internal/poll.(*FD).Read(0x4000708300, {0x4000d840a1, 0x1, 0x1})
	internal/poll/fd_unix.go:161 +0x204
net.(*netFD).Read(0x4000708300, {0x4000d840a1?, 0x4000122030?, 0x0?})
	net/fd_posix.go:68 +0x28
net.(*conn).Read(0x4000718000, {0x4000d840a1?, 0x0?, 0x0?})
	net/net.go:196 +0x34
net/http.(*connReader).backgroundRead(0x4000d84080)
	net/http/server.go:702 +0x38
created by net/http.(*connReader).startBackgroundRead in goroutine 403
	net/http/server.go:698 +0xb8

goroutine 177 [select]:
github.com/nats-io/nats-server/v2/server.(*stream).signalConsumersLoop(0x4000738008)
	github.com/nats-io/nats-server/v2/server/stream.go:6733 +0x148
created by github.com/nats-io/nats-server/v2/server.(*Account).addStreamWithAssignment in goroutine 107
	github.com/nats-io/nats-server/v2/server/stream.go:855 +0x119c

goroutine 178 [select]:
github.com/nats-io/nats-server/v2/server/ats.Register.func1()
	github.com/nats-io/nats-server/v2/server/ats/ats.go:52 +0xbc
created by github.com/nats-io/nats-server/v2/server/ats.Register in goroutine 107
	github.com/nats-io/nats-server/v2/server/ats/ats.go:48 +0xb8

goroutine 27 [select]:
github.com/nats-io/nats-server/v2/server.(*fileStore).flushStreamStateLoop(0x4000456008, 0x40001208c0, 0x40002202a0?)
	github.com/nats-io/nats-server/v2/server/filestore.go:10402 +0xcc
created by github.com/nats-io/nats-server/v2/server.newFileStoreWithCreated in goroutine 107
	github.com/nats-io/nats-server/v2/server/filestore.go:633 +0x1324

goroutine 193 [select]:
github.com/nats-io/nats-server/v2/server.(*fileStore).flushStreamStateLoop(0x40003f8008, 0x400028a770, 0x0?)
	github.com/nats-io/nats-server/v2/server/filestore.go:10402 +0xcc
created by github.com/nats-io/nats-server/v2/server.newFileStoreWithCreated in goroutine 111
	github.com/nats-io/nats-server/v2/server/filestore.go:633 +0x1324

goroutine 179 [select]:
github.com/nats-io/nats-server/v2/server.(*stream).internalLoop(0x4000738008)
	github.com/nats-io/nats-server/v2/server/stream.go:6927 +0x344
created by github.com/nats-io/nats-server/v2/server.(*stream).setupSendCapabilities in goroutine 107
	github.com/nats-io/nats-server/v2/server/stream.go:6861 +0x160

goroutine 180 [sync.RWMutex.RLock]:
sync.runtime_SemacquireRWMutexR(0x835490?, 0x20?, 0x835490?)
	runtime/sema.go:100 +0x28
sync.(*RWMutex).RLock(...)
	sync/rwmutex.go:74
github.com/nats-io/nats-server/v2/server.(*stream).internalLoop(0x40000de488)
	github.com/nats-io/nats-server/v2/server/stream.go:6892 +0x7c
created by github.com/nats-io/nats-server/v2/server.(*stream).setupSendCapabilities in goroutine 111
	github.com/nats-io/nats-server/v2/server/stream.go:6861 +0x160

goroutine 226 [select]:
github.com/nats-io/nats-server/v2/server.(*consumer).loopAndGatherMsgs(0x4000534008, 0x4000120e70)
	github.com/nats-io/nats-server/v2/server/consumer.go:5085 +0x1380
github.com/nats-io/nats-server/v2/server.(*consumer).setLeader.func2()
	github.com/nats-io/nats-server/v2/server/consumer.go:1634 +0x48
created by github.com/nats-io/nats-server/v2/server.(*consumer).setLeader in goroutine 107
	github.com/nats-io/nats-server/v2/server/consumer.go:1632 +0xda4

goroutine 227 [select]:
github.com/nats-io/nats-server/v2/server.(*consumer).processInboundAcks(0x4000534008, 0x4000120e70)
	github.com/nats-io/nats-server/v2/server/consumer.go:4765 +0x1b4
github.com/nats-io/nats-server/v2/server.(*consumer).setLeader.func3()
	github.com/nats-io/nats-server/v2/server/consumer.go:1640 +0x48
created by github.com/nats-io/nats-server/v2/server.(*consumer).setLeader in goroutine 107
	github.com/nats-io/nats-server/v2/server/consumer.go:1638 +0xdfc

goroutine 228 [select]:
github.com/nats-io/nats-server/v2/server.(*consumer).processInboundNextMsgReqs(0x4000534008, 0x4000120e70)
	github.com/nats-io/nats-server/v2/server/consumer.go:4798 +0x13c
github.com/nats-io/nats-server/v2/server.(*consumer).setLeader.func4()
	github.com/nats-io/nats-server/v2/server/consumer.go:1647 +0x48
created by github.com/nats-io/nats-server/v2/server.(*consumer).setLeader in goroutine 107
	github.com/nats-io/nats-server/v2/server/consumer.go:1645 +0xe5c

goroutine 371 [IO wait]:
internal/poll.runtime_pollWait(0xffffb5a14400, 0x72)
	runtime/netpoll.go:351 +0xa0
internal/poll.(*pollDesc).wait(0x40002a6a80?, 0x4000d86000?, 0x0)
	internal/poll/fd_poll_runtime.go:84 +0x28
internal/poll.(*pollDesc).waitRead(...)
	internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0x40002a6a80, {0x4000d86000, 0x1000, 0x1000})
	internal/poll/fd_unix.go:165 +0x1e0
net.(*netFD).Read(0x40002a6a80, {0x4000d86000?, 0x4000f1baa8?, 0x25e4ac?})
	net/fd_posix.go:68 +0x28
net.(*conn).Read(0x40001256c0, {0x4000d86000?, 0x72?, 0x0?})
	net/net.go:196 +0x34
net/http.(*connReader).Read(0x4000d84000, {0x4000d86000, 0x1000, 0x1000})
	net/http/server.go:812 +0x230
bufio.(*Reader).fill(0x40000d50e0)
	bufio/bufio.go:113 +0xe0
bufio.(*Reader).Peek(0x40000d50e0, 0x4)
	bufio/bufio.go:152 +0x58
net/http.(*conn).serve(0x400055e120, {0x934be8, 0x40007c1200})
	net/http/server.go:2145 +0x654
created by net/http.(*Server).Serve in goroutine 7
	net/http/server.go:3493 +0x384

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

JetStream startup seems slow for ~50M messages #8001

Uh oh!

{{title}}

Uh oh!

Replies: 2 comments 19 replies

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Select a reply

Uh oh!

Uh oh!

JetStream startup seems slow for ~50M messages #8001

Uh oh!

zig Apr 2, 2026

Replies: 2 comments · 19 replies

Uh oh!

wallyqs Apr 2, 2026 Collaborator

Uh oh!

zig Apr 2, 2026 Author

Uh oh!

derekcollison Apr 4, 2026 Maintainer

Uh oh!

Uh oh!

zig Apr 4, 2026 Author

Uh oh!

zig Apr 4, 2026 Author

Uh oh!

derekcollison Apr 4, 2026 Maintainer

Uh oh!

zig Apr 4, 2026 Author

zig
Apr 2, 2026

Replies: 2 comments 19 replies

wallyqs
Apr 2, 2026
Collaborator

zig
Apr 2, 2026
Author

derekcollison Apr 4, 2026
Maintainer

zig Apr 4, 2026
Author

zig Apr 4, 2026
Author

derekcollison Apr 4, 2026
Maintainer

zig Apr 4, 2026
Author