Getting random polka dots on some mipmap levels

[SBRK]

Hi, I'm having a weird issue where I get some colored dots added to some mipmap levels of my textures. It appears to be pretty random: those textures were converted automatically by our scripts on a server, but if I try to convert them locally on my computer with the same parameters, I don't have the issue. Additionnaly, some textures have the problem and some don't (for example, the floor has been treated the same way and is fine)

[SBRK]

The problem might be linked to parallelisation of treating a lot of files. We convert them to dxt, etc, pvrtc, astc and basis all in parallel. In our logs, we can see basisu failing at the same time one of the dxt file is being treated. Just like the dots, it happens at random and only some basis convertion failed.
Maybe the memory runs out?

[mredele]

We're having similar issues. The interesting part is that we've been running the same png to dxt1 (with mipmap) workflow for several years and only very seldom had stripes on some mipmap levels. But recently we changed the machine type (to c5.18xlarge in AWS) consistently got stripes with -helperThreads 8 but I could not reproduce it on my laptop (2018 macbook pro) but with -helperThreads 16 I got similar colored dots as you.

[mredele]

The lightmap with a mipmap with dark stripes: https://storage.3d.io/ace/job/2021-01-19/12-04_9c8a5868/lm_1__1.hi-res.gz.dds
and the source png: https://storage.3d.io/ace/job/2021-01-19/12-04_9c8a5868/lm_1__1.source.png

[richgel999]

"The problem might be linked to parallelisation of treating a lot of files. We convert them to dxt, etc, pvrtc, astc and basis all in parallel. In our logs, we can see basisu failing at the same time one of the dxt file is being treated"

Is this issue occurring with basisu.exe, or crunch? The best way (from a perf perspective) to multithread either tool is to disable threading and launch them X times (i.e. running X encodes in parallel). So does the problem go away when you disable threading?

[richgel999]

Also, note that basisu's threading is implemented in a completely different way vs. crunch. crunch uses completely custom code that leverages the Win32/pthread API's, and basisu uses C++11 std helpers to implement a thread pool with simple jobs.

[mredele]

It's happening with crunch, with only the DXT1 conversion running. It's on Linux (Ubuntu 18.04). -helperThreads 0 seems to fix the problem.

[richgel999]

Strange - on Linux it uses the pthread API's. It's been many years since I've looked at the crunch codebase, but whatever is happening seems rare. We have some extremely large commercial users of the crunch lib who haven't reported it.

[richgel999]

There's a decent probability that the issue is in here or the header:
https://github.com/BinomialLLC/crunch/blob/master/crnlib/crn_threading_pthreads.cpp