Hi Edouard,
Do you see a chance to backport the CI speed updates to G83? That would make my release manager life much easier as the ever lasting waiting would probably halfen … 
I could also cherry-pick that in case.
Best
Markus
I don’t know a lot about backporting, but if you’re talking about splitting of the Ubuntu workflow’s gunittest tests, I don’t see a reason why the content of the changes couldn’t be copied to a new commit for that branch too. Since they are ran less often, the tests could be split in three or even 4 jobs, where the %of time spent compiling the same code would be a bit higher, but the time for all tests to run could be smaller. The tests in the temporal folder still take half the total test time though.
If you are talking about the macOS arm runner, it might require some other changes to port. My opinion would be to leave that alone in the 8.3 as pre-arm code. But the same splitting could easily be done for 8.3, and split as wished.
In main, for macOS workflows, with only 20-25 mins spent on tests, with 3-4 min spent on compiling, I don’t see the need to split it.
Windows workflows I don’t think it’s advantageous, as more than 20 mins is spent before starting tests.
Edouard Choinière
Le 19 févr. 2024 à 19:00, Markus Neteler <neteler@osgeo.org> a écrit :
Hi Edouard,Do you see a chance to backport the CI speed updates to G83? That would make my release manager life much easier as the ever lasting waiting would probably halfen ...
I could also cherry-pick that in case.
Best
Markus
On Tue, Feb 20, 2024 at 1:59 AM Edouard Choinière <e.chs@outlook.com> wrote:
I don’t know a lot about backporting,
If "lucky", then it is just
git cherry-pick <hash>
but if you’re talking about splitting of the Ubuntu workflow’s gunittest tests, I don’t see a reason why the content of the changes couldn’t be copied to a new commit for that branch too. Since they are ran less often, the tests could be split in three or even 4 jobs, where the %of time spent compiling the same code would be a bit higher, but the time for all tests to run could be smaller. The tests in the temporal folder still take half the total test time though.
If you are talking about the macOS arm runner, it might require some other changes to port. My opinion would be to leave that alone in the 8.3 as pre-arm code. But the same splitting could easily be done for 8.3, and split as wished.
In fact the slowest CI run determines how much time I have to wait
with each release step (i.e., editing VERSION file, wait 1:30hs, do
some steps, wait 1:30hs, create tarball, wait 1:30hs, reset VERSION
file, wait 1:30hs ... which is a pain).
In main, for macOS workflows, with only 20-25 mins spent on tests, with 3-4 min spent on compiling, I don’t see the need to split it.
Okay.
Windows workflows I don’t think it’s advantageous, as more than 20 mins is spent before starting tests.
Alright.
Perhaps I have simply go through it again and hope that we abandon 8.3 soon
Best
Markus
On 20 Feb 2024, at 10:14, Markus Neteler via grass-dev <grass-dev@lists.osgeo.org> wrote:
In fact the slowest CI run determines how much time I have to wait
with each release step (i.e., editing VERSION file, wait 1:30hs, do
some steps, wait 1:30hs, create tarball, wait 1:30hs, reset VERSION
file, wait 1:30hs … which is a pain).
I agree this is a case where we have limited ourself too much, with all those
required 1.5 hrs tests, approval, etc. (not even [skip ci] would work) . What you
would need is a (ideally) direct commit access or at least “Rebase and merge”
option to merge (thus enable a number of commits to be merged at one time,
as opposed to "Squash and merge”).
We must find a solution to improve this situation for preparing releases, I
wouldn’t mind temporary lifting necessary constraints.
Releasebranches don’t require workflows to pass now so he cannot use auto-merge.
But once tests of the OSGeo4W are started, since they don’t fail the CI, you could just not wait, it won’t change anything.
Edouard Choinière
Le 20 févr. 2024 à 06:45, Nicklas Larsson n_larsson@yahoo.com a écrit :
On 20 Feb 2024, at 10:14, Markus Neteler via grass-dev <grass-dev@lists.osgeo.org> wrote:
In fact the slowest CI run determines how much time I have to wait
with each release step (i.e., editing VERSION file, wait 1:30hs, do
some steps, wait 1:30hs, create tarball, wait 1:30hs, reset VERSION
file, wait 1:30hs … which is a pain).I agree this is a case where we have limited ourself too much, with all those
required 1.5 hrs tests, approval, etc. (not even [skip ci] would work) . What you
would need is a (ideally) direct commit access or at least “Rebase and merge”
option to merge (thus enable a number of commits to be merged at one time,
as opposed to "Squash and merge”).We must find a solution to improve this situation for preparing releases, I
wouldn’t mind temporary lifting necessary constraints.
But as one of the members with the maintain role (higher than triage and write), he has access to bypass permissions. If you go look back in the PR where I explained the branch rules, I showed the checkbox that allows to bypass for example in PRs. Here is the public combined view of the rules that are active for releasebranch_8_3 https://github.com/OSGeo/grass/rules/270686?ref=refs%2Fheads%2Freleasebranch_8_3 and https://github.com/OSGeo/grass/rules?ref=refs%2Fheads%2Freleasebranch_8_3
Note how they are different than https://github.com/OSGeo/grass/rules?ref=refs%2Fheads%2Fmain
It was made such as changing workflow requirements for main don’t restrict the older branches to be merged, or that changes in the CI infrastructure (outside of our repo) make it so they don’t work anymore. So they are kind of run on a best-effort basis, instead of religiously.
Edouard Choinière
Le 20 févr. 2024 à 06:45, Nicklas Larsson n_larsson@yahoo.com a écrit :
On 20 Feb 2024, at 10:14, Markus Neteler via grass-dev <grass-dev@lists.osgeo.org> wrote:
In fact the slowest CI run determines how much time I have to wait
with each release step (i.e., editing VERSION file, wait 1:30hs, do
some steps, wait 1:30hs, create tarball, wait 1:30hs, reset VERSION
file, wait 1:30hs … which is a pain).I agree this is a case where we have limited ourself too much, with all those
required 1.5 hrs tests, approval, etc. (not even [skip ci] would work) . What you
would need is a (ideally) direct commit access or at least “Rebase and merge”
option to merge (thus enable a number of commits to be merged at one time,
as opposed to "Squash and merge”).We must find a solution to improve this situation for preparing releases, I
wouldn’t mind temporary lifting necessary constraints.
On Tue, 20 Feb 2024 at 06:45, Nicklas Larsson via grass-dev <grass-dev@lists.osgeo.org> wrote:
On 20 Feb 2024, at 10:14, Markus Neteler via grass-dev <grass-dev@lists.osgeo.org> wrote:
In fact the slowest CI run determines how much time I have to wait
with each release step (i.e., editing VERSION file, wait 1:30hs, do
some steps, wait 1:30hs, create tarball, wait 1:30hs, reset VERSION
file, wait 1:30hs … which is a pain).I agree this is a case where we have limited ourself too much, with all those
required 1.5 hrs tests, approval, etc. (not even [skip ci] would work) . What you
would need is a (ideally) direct commit access or at least “Rebase and merge”
option to merge (thus enable a number of commits to be merged at one time,
as opposed to "Squash and merge”).We must find a solution to improve this situation for preparing releases, I
wouldn’t mind temporary lifting necessary constraints.
The backports and releases are done using direct commits and pushes. That’s how the rules are set, no bypass or exception is necessary for that.
I guess the issue is not that we can’t bypass the check but that we want the checks to pass because we don’t want to base the release on a commit with failing CI.
On Tue, 20 Feb 2024 at 04:14, Markus Neteler via grass-dev <grass-dev@lists.osgeo.org> wrote:
In fact the slowest CI run determines how much time I have to wait
with each release step (i.e., editing VERSION file, wait 1:30hs, do
some steps, wait 1:30hs, create tarball, wait 1:30hs, reset VERSION
file, wait 1:30hs … which is a pain).
Isn’t the issue the release procedure itself? It has a bunch of steps which need to be done manually.
I counted 3 pushes which is what triggers the CI.
-
release VERSION file push
-
tag push
-
development VERSION file push
The release needs step 2 to be completed. We were doing step 2 only after CI for step 1 completed to make sure the CI runs on the branch at that time before the tag is made in step 2. I guess the reason to wait after step 2 before doing step 3 is to make sure that the automated part of the release procedure linked to step 2 actually went through. Is this correct?
Best,
Vaclav
On Fri, Feb 23, 2024 at 2:52 AM Vaclav Petras <wenzeslaus@gmail.com> wrote:
On Tue, 20 Feb 2024 at 04:14, Markus Neteler via grass-dev <grass-dev@lists.osgeo.org> wrote:
In fact the slowest CI run determines how much time I have to wait
with each release step (i.e., editing VERSION file, wait 1:30hs, do
some steps, wait 1:30hs, create tarball, wait 1:30hs, reset VERSION
file, wait 1:30hs ... which is a pain).Isn't the issue the release procedure itself? It has a bunch of steps which need to be done manually.
Sure. But if one has to wait 1:30 hs for the next step it overall
takes a lot of time.
Hence my (meanwhile discarded) wish to have a faster CI as it
meanwhile exists in G84.
I counted 3 pushes which is what triggers the CI.
Indeed perhaps only 3 pushes and not 5.
1) release VERSION file push
2) tag push
3) development VERSION file pushThe release needs step 2 to be completed. We were doing step 2 only after CI for step 1 completed to make sure the CI runs on the branch at that time before the tag is made in step 2.
I guess the reason to wait after step 2 before doing step 3 is to make sure that the automated part of the release procedure linked to step 2 actually went through. Is this correct?
It also includes that the complete build of artefacts is needed for
download/upload to grass.osgeo.org and the download server.
(unrelated to the CI part then also milestone cleanup, etc. follows,
so after step 3 more is to be done)
Cheers
Markus