What I worked on in week 4:
This week was more focused on cleanup and review response. I fixed several issues in the pytest from PR #7534 based on mentor feedback and removed the gunittest version from #7440 in favor of keeping just the pytest and a universal benchmark. The rest of the week went into cleaning up the parallel r.param.scale code and addressing review comments on the PR. Meanwhile the mentors ran their own benchmarks on their hardware, with and without pinned threads, so we could compare scaling behavior across different machines.
What I worked on in week 5:
I cleaned up the pytest from #7534 and it was merged. Right after merging it resurfaced the lu.c bug from issue #7539, since the test runs cell by cell comparisons that catch a small difference. To keep the CI from erroring out for everyone, I opened a small follow up PR that skips the affected test until the library fix lands. The skip gets removed once the bug is fixed.
Most of the week went into investigating dips in the parallel scaling at certain thread counts. The dips never happened at the same thread count consistently, so that ruled out a code bug tied to a specific configuration. My first guess was that the efficiency cores on my machine were dragging runs down. But testing the timing per thread showed they were not. At the dip, one thread simply runs up to about 2x slower than the rest. Every band waits at a barrier, so that one slow thread holds up the whole band. I ran r.neighbors through the same timing on each thread on the same machine and its threads finished evenly, even though r.param.scale splits rows the same way. I was given a suggestion it could be a frequency or power limit from the performance and efficiency core split. So I logged per cluster frequencies with powermetrics during full sweeps, with 200 second cooldowns between runs. The slow runs actually had the performance cores clocked slightly higher and the efficiency cores less engaged, which is the opposite of what a frequency limit would look like. The slow thread count also moved between runs. I ran other tests throughout the week to try and find a universal reason for why dips happen randomly across different machines. We ended up saying that there might not be a single universal cause. It looks like a minor scheduling imbalance that each machine amplifies differently. The code itself is not the issue, since the output matches serial exactly at every thread count, method, and window size.
Issue: [Bug] Race condition in libgmath · Issue #7539 · OSGeo/grass · GitHub
PR:
tests: skip r.param.scale reference test pending libgmath race fix #7539 by krcoder123 · Pull Request #7593 · OSGeo/grass · GitHub
r.param.scale: parallelize with OpenMP by krcoder123 · Pull Request #7440 · OSGeo/grass · GitHub
Thanks,
Kaushik