pubsub: unit tests cause emulator to use excessive CPU

[cortesi]

The pubsub emulator recently started using large amounts of CPU on OSX during our unit tests (other platforms untested). High CPU usage persists after the test suite has completed and all clients disconnected, until the process is restarted. I am able to reproduce this behaviour with simple clients and with google-cloud-go's own test suite.

Steps to reproduce:

• Start the emulator:
  gcloud beta emulators pubsub start --host-port=localhost:8822
• In the directory cloud.google.com/go/pubsub, run the test suite:
  env PUBSUB_EMULATOR_HOST=localhost:8822 go test -v

On my system, the pubsub emulator Java process CPU usage is at 45% after one run, 145% after the second run, and so forth.

This is undoubtedly a bug in the emulator, but git bisect shows that google-cloud-go only started triggering the issue with commit 927c812. This commit is included in the v0.10.0 release.

Software versions used:

• google-cloud-go master
• pubsub-emulator 2017.03.24
• OSX 10.12.5

[pongad]

Thank you for the detailed analysis. That commit increases the number of worker goroutines from 1 to 10*CPU. So this makes a lot of sense.

In your test, could you try adding sub.ReceiveSettings.NumGoroutines = 1? This should bring CPU consumption back to the level before the commit you linked.

[cortesi]

Hi there - that's the workaround that we have in place for now. We still see a slow growth in CPU usage of 1-2% with each run.

[pongad]

Ah, I'm sorry. I misread the part where you said "High CPU usage persists after the test suite has completed". I'll loop in the emulator folks.

[pongad]

The fix for this has been made to the emulator and going to be published soon.
We've also scaled back NumGoroutines, so this should alleviate the problem in the mean time.
I'll close this for now.