radiod CPU utilization increased significantly

[argilo]

It appears that commit 323ec10 has significantly increased the CPU utilization of radiod, from about 140% to 215% on my Raspberry Pi 4 running radiosonde_auto_rx. I presume the extra load relates to radiosonde_auto_rx's use of a spectrum channel. (It periodically runs powers radiosonde.local -f 403025000.0 -w 800.0 -b 7438 -i 20 -s 40302503 -c 2.)

[argilo]

I heard from @darksidelemm that tuning point_budget may help, so I'll experiment with that today.

[argilo]

Adjusting point_budget definitely makes a big difference. Here's what I measured on my Raspberry Pi 4:

Baseline radiod CPU utilization, before executing powers: ~74%
After starting powers, commit 6571ae1 (prior to the changes): ~81%
After starting powers, commit f7c227b, point_budget = 512 * 1024: ~154%
After starting powers, commit f7c227b, point_budget = 256 * 1024: ~108%
After starting powers, commit f7c227b, point_budget = 128 * 1024: ~93%
After starting powers, commit f7c227b, point_budget = 64 * 1024: ~84%
After starting powers, commit f7c227b, point_budget = 32 * 1024: ~82%

[argilo]

Since radiod no longer does any accumulation, should powers close the spectrum channel when it exits, to relieve radiod of the extra computation?

[ka9q]

I should explain how spectrum now works (you can probably suggest improvements). There's a wideband and a narrowband mode. In the narrowband mode, I set up a downconverter and store the output in a ring buffer. When (and only when) the user polls with a spectrum request, I perform a windowed periodogram on the most recent data and return it. Actually I do two and average the results to reduce the noise variance. So there's some background CPU in the narrowband mode even if you don't poll, and you should close the channel when done. The sample rate is determined from the requested number of points increased a little to let me cut off the edges with the downconverter filter skirts, and also to get a "friendly" FFT size.

In the wideband mode, nothing happens at all until the user polls. It wakes up every 20 ms to check for a command, and that's it. When I do get a poll I grab the most recent A/D data from the input queue of the fast convolver and perform a windowed FFT on it with the bin size equal to the requested RBW. The FFT has to cover the full spectrum of the A/D, and the excess bins outside the requested span are discarded. Because the FFTs get shorter (fewer, wider bins) as the requested RBW increases, I can afford to repeat the process on successively earlier chunks of the input queue and average all their results, reducing noise variance significantly on wide spans. The point_budget hack is an attempt to keep the amount of cpu invested per poll roughly constant with RBW, and it does that well. It's about 8-10 percent on my AMD Beelink when ka9q-web polls it at ~10 Hz. (N log N is more like N in practice because log N doesn't vary all that much. Cache hit ratios matter a lot more.)

Why change all this? The data processed by the FFT in the resolver into 40 Hz bins is not windowed (or rectangular windowed) so using it for spectrum analysis was questionable. It looks especially terrible on the AM broadcast band near a Mexican border blaster. The new code uses Kaiser with beta=7 by default and supports the usual alternatives (Hann, Hamming, Gaussian, Blackman, etc). Kaiser still seems best all around, especially with its tuning knob.

My old code that piggybacked on the convolver FFT slowed with increasingly large spans because of all the memory touched doing the I^2 + Q^2 power sums. The new code is much faster but at the expense of worse noise variance because fewer A/D points are processed. The noisy grass made it hard to see interference combs in wide span. But the FFTs get cheaper (rather than more expensive) with larger spans because the FFTs have bigger bins and fewer points to cover the whole spectrum. So I spent some of these savings to do more FFTs per call and average the energies to flatten the noise. But yeah, the CPU is a problem. I may still resurrect the old code as an option for VK5QI, who was using it to detect radiosondes. Despite the lack of windowing the old code seemed to work for him, though not especially quickly because he looks at 6 MHz, and my new code is too slow for his RPi 4 with the default tweaks. Like yours. Let me know which you prefer.

I implemented a crossover parameter that determines if the narrowband or wideband mode is used. Currently RBW=200 Hz and down use the narrowband mode but that's easily changed. I've only tested on one CPU, and I also set the crossover before I did the point budget stuff so it should probably increase.

So this stuff is far from perfect, and suggestions are always welcome.

[ka9q]

Hmm, maybe I should make the averaging parameter (the number of consecutive blocks FFTed and averaged) a parameter in each poll instead of my too-clever budget idea.

[argilo]

In the wideband mode, nothing happens at all until the user polls. It wakes up every 20 ms to check for a command, and that's it.

In my test above, I was running powers with -w 800.0, which I expect would put the spectrum channel in wideband mode. But I continued to see the CPU utilization figures listed, even after powers stopped executing.

[argilo]

maybe I should make the averaging parameter (the number of consecutive blocks FFTed and averaged) a parameter in each poll

That sounds like it could be useful for radiosonde_auto_rx, which wants to average over a long period (20 seconds, by default) to detect intermittent transmitters.

[argilo]

The 20-second integration time may be overkill. I expect it was inherited from rtl_power, which is forced to scan the radiosonde band in three 2-MHz chunks. But certainly an integration time of at least 1 second would be needed, to account for radiosondes that send packets at 1 Hz.

[darksidelemm]

We don't necessarily need a 1 second integration, but we do need to be able to sample at a rate fast enough to detect an intermittent sonde.

A RS41 transmits with a ~50% duty cycle, 1 second PRF, so sampling at ~100ms intervals for ~5 seconds and performing a median would likely be sufficient.

I'd need to double check this, but I think all the other sondes we support have continuous transmission of some kind.

[argilo]

Agreed that a full integration wouldn't be necessary.

Median could be problematic, since anything below 50% duty cycle would go undetected.

[argilo]

Another problem I have noticed is that powers now appears to output an inverted spectrum. (Frequencies are mirrored around the center frequency.)

[ka9q]

The inverted spectrum -- what kind of front end is this? There are three cases for front ends: complex, real, and real inverted. The RX888 in direct sample mode is real, right side up. Airspy R2 is real inverted. So is the RX888 in VHF/UHF mode, I think, though I haven't gotten it working myself. I simplified spectrum.c by flipping the spectrum of a real-inverted input so I wouldn't have to index it backward in the frequency domain. I might have screwed this up; so far I have only tested the real, right side up path (RX888).

[argilo]

I'm using an R2.

[ka9q]

I'm familiar with the RS41 and its pulsed transmissions. Yes, you can fast-poll and process them any way you like, and that might be better than averaging over a long time and getting only the average power (~50 %). But each poll should do a reasonable amount of averaging so you wouldn't have to poll at a ridiculously high rate just to get good noise statistics.

If RBW (bin size) is X Hz, then a single FFT will use only 1/X of the input stream, so the SNR in the spectrum itself (ie the variance/noise in the noise measurements from poll to poll) will be much worse than back when I continuously averaged bin powers every 20 ms frame to have them ready for the user in case he polled, usually at a much lower rate. But this burned a LOT of CPU on wide spans because I had to touch a lot of memory every 20ms, whether or not my results were used.

And I wanted proper windowing to control spectral leakage, so I came up with this new hybrid method. An RS-41 transmits FSK at 4800 baud so my intuition tells me that to detect the presence of an RS-41 strong enough to decode you need to integrate for ~1/4800 sec. So a single FFT with 4 kHz bins should be enough if you happen to catch the transmitter on. I.e., the bin size should be roughly comparable to the modulation bandwidth, which also seems intuitive.

You'll need to poll at least twice < 500 ms apart to ensure catching the transmitter on, and you should repeat that over time in case of fading. To improve detection reliability each poll should consist of several consecutive FFTs whose power spectra are then averaged. This is less demanding than making a really smooth-looking noise floor on a spectrum analyzer display.

Does my reasoning sound right?

[ka9q]

I'm going to make the number of FFTs averaged to produce one poll response a command/status parameter. It's analogous to the video bandwidth (VBW) on a spectrum analyzer. Or its reciprocal. I'll expose this parameter on the 'powers' command line, and also find some way for ka9q-web to control it.