[GRASS-dev] svn/trac -> git/github migration plan

Hi,

po 13. 5. 2019 v 13:21 odesĂ­latel Moritz Lennert
<mlennert@club.worldonline.be> napsal:

I think we just need to get the ball rolling. There will certainly be
some transition pain, but if we try to prepare everything to perfection
it will never happen :wink:

right, currently I am working on source code migration. Unfortunately
there is a new issue which is quite pain. My last attempt to create
git repo from fresh svn [1] copy (tarball) fails with

r74442 = a3741bbbdfc31118ea9ab9d98a6bd15a39b6fa83 (refs/remotes/origin/trunk)
Error from SVN, (160006): Invalid filesystem revision number: No such
revision 74542

Now retrying with svn dump. When it will be solved I will publish
links to rewritten messages log and new git repo for public review.

Ma

[1] https://trac.osgeo.org/grass/browser/grass-addons/tools/svn2git/0-migrate-fetch.sh#L17

--
Martin Landa
http://geo.fsv.cvut.cz/gwiki/Landa
http://gismentors.cz/mentors/landa

On Mon, May 13, 2019 at 1:31 PM Martin Landa <landa.martin@gmail.com> wrote:

Hi,

po 13. 5. 2019 v 13:21 odesĂ­latel Moritz Lennert
<mlennert@club.worldonline.be> napsal:
> I think we just need to get the ball rolling. There will certainly be
> some transition pain, but if we try to prepare everything to perfection
> it will never happen :wink:

right, currently I am working on source code migration. Unfortunately
there is a new issue which is quite pain. My last attempt to create
git repo from fresh svn [1] copy (tarball) fails with

r74442 = a3741bbbdfc31118ea9ab9d98a6bd15a39b6fa83 (refs/remotes/origin/trunk)
Error from SVN, (160006): Invalid filesystem revision number: No such
revision 74542

In fact, not there:

https://trac.osgeo.org/grass/changeset/74542
Error: Invalid Changeset Number
NoSuchChangeset: No changeset 74542 in the repository

... but that number is nonsense, as it was not reached yet.

--> r74478 this changeset is the highest number at time:
https://trac.osgeo.org/grass/changeset/74478

[mneteler@oboe grass77 ]$ svn up
Updating '.':
At revision 74478.
[mneteler@oboe grass77 ]$ date -u
Mon 13 May 2019 12:09:58 PM UTC

Maybe easy to solve? Or am I overlooking anything?

Markus

Hi,

po 13. 5. 2019 v 14:11 odesĂ­latel Markus Neteler <neteler@osgeo.org> napsal:

> r74442 = a3741bbbdfc31118ea9ab9d98a6bd15a39b6fa83 (refs/remotes/origin/trunk)
> Error from SVN, (160006): Invalid filesystem revision number: No such
> revision 74542

In fact, not there:

https://trac.osgeo.org/grass/changeset/74542
Error: Invalid Changeset Number
NoSuchChangeset: No changeset 74542 in the repository

... but that number is nonsense, as it was not reached yet.

yes, this error comes from `git svn fetch`. No idea why it dreams
about non existing revision. Ma

--
Martin Landa
http://geo.fsv.cvut.cz/gwiki/Landa
http://gismentors.cz/mentors/landa

Hi,

po 13. 5. 2019 v 14:58 odesĂ­latel Martin Landa <landa.martin@gmail.com> napsal:

> https://trac.osgeo.org/grass/changeset/74542
> Error: Invalid Changeset Number
> NoSuchChangeset: No changeset 74542 in the repository
>
> ... but that number is nonsense, as it was not reached yet.

yes, this error comes from `git svn fetch`. No idea why it dreams
about non existing revision. Ma

I am getting the same error when using locally dumped svn repo (till
now I was using just tarball). We have a new blocker.

Ma

--
Martin Landa
http://geo.fsv.cvut.cz/gwiki/Landa
http://gismentors.cz/mentors/landa

po 13. 5. 2019 v 15:01 odesĂ­latel Martin Landa <landa.martin@gmail.com> napsal:

r74440 = 1f1209cbe0074f1e6451b4ed40f56a22192724e7 (refs/remotes/origin/trunk)
    M raster/r.to.vect/areas_io.c
    M raster/r.to.vect/lines.c
    M raster/r.to.vect/lines_io.c
r74441 = 2fb78d92ee0ad2570a4388b30aeb3cdc63326cdb (refs/remotes/origin/trunk)
    M raster/r.thin/thin_lines.c
r74442 = a3741bbbdfc31118ea9ab9d98a6bd15a39b6fa83 (refs/remotes/origin/trunk)
Error from SVN, (160006): Invalid filesystem revision number: No such
revision 74542

it seems that it fails to grab r74442 when it starts to speak
surprisingly about non-existing. But this works:

$ svn log file:///opt/osgeo/svn/repos/grass/grass
...
------------------------------------------------------------------------
r74443 | mmetz | 2019-04-30 20:28:34 +0200 (Tue, 30 Apr 2019) | 1 line

r.thin: avoid integer overflow, remove unused variable (backport trunk r74442)
------------------------------------------------------------------------
r74442 | mmetz | 2019-04-30 20:26:10 +0200 (Tue, 30 Apr 2019) | 1 line

r.thin: avoid integer overflow, remove unused variable
------------------------------------------------------------------------
...

I run `git svn fetch` many times when doing various tests. I have no
idea what could be wrong at this moment. Ma

--
Martin Landa
http://geo.fsv.cvut.cz/gwiki/Landa
http://gismentors.cz/mentors/landa

po 13. 5. 2019 v 15:24 odesĂ­latel Martin Landa <landa.martin@gmail.com> napsal:

unfortunately attempt to run `git svn fetch` incrementally fails with
similar error:

$ git svn -r74443:75000 --authors-file=../AUTHORS.txt fetch
Error from SVN, (160006): Invalid filesystem revision number: No such
revision 74543

Ma

--
Martin Landa
http://geo.fsv.cvut.cz/gwiki/Landa
http://gismentors.cz/mentors/landa

Hi,

po 13. 5. 2019 v 15:32 odesĂ­latel Martin Landa <landa.martin@gmail.com> napsal:

unfortunately attempt to run `git svn fetch` incrementally fails with
similar error:

$ git svn -r74443:75000 --authors-file=../AUTHORS.txt fetch
Error from SVN, (160006): Invalid filesystem revision number: No such
revision 74543

I tried to run `0-migrate-fetch.sh` directly on svn osgeo server.
Fetching over `file://` fails with the same error. Strangely when
fetching over `http://` the command is somehow able to recover:

...
r74441 = 9f7eef32fec5f0e898e794854591b9015afed26d (refs/remotes/trunk)
    M raster/r.thin/thin_lines.c
r74442 = 13e9f72398b8e3db69dd5f2c567553f6d46878b5 (refs/remotes/trunk)
W: Ignoring error from SVN, path probably does not exist: (175002): RA
layer request failed: Unexpected HTTP status 500 'Internal Server
Error' on '/grass/!svn/rvr/74542'
: Additional errors:: No such revision 74542
W: Do not be alarmed at the above message git-svn is just searching
aggressively for old history.
This may take a while on large repositories
Path 'grass' was probably deleted:
RA layer request failed: Unexpected HTTP status 500 'Internal Server
Error' on '/grass/!svn/rvr/74542'
: Additional errors:: No such revision 74542
Will attempt to follow revisions r74443 .. r74542 committed before the deletion
r74443 .. r74476 OK
    M raster/r.thin/thin_lines.c
r74443 = 4e76c156c6b0b547e30a3399642074d82f5729bf
(refs/remotes/releasebranch_7_6)
    M raster/r.thin/local_proto.h
    M raster/r.thin/thin_lines.c
...

Something strange happen with svn repo after r74441. I will publish
testing repo for public review hopefully today.

Ma

--
Martin Landa
http://geo.fsv.cvut.cz/gwiki/Landa
http://gismentors.cz/mentors/landa

Hi,

st 15. 5. 2019 v 9:51 odesĂ­latel Martin Landa <landa.martin@gmail.com> napsal:

Something strange happen with svn repo after r74441. I will publish
testing repo for public review hopefully today.

OK, finally a new fresh grass repo preview available for review (last
processed commit 74475).

Please compare git content [1], branches [2] and tags [3] with svn.
Please review logs reporting changed message [4] and unchanged
messages [5]. Thanks! If no problems appears this will be the last
preview before REAL migration. Ma

[1] https://github.com/grass-svn2git/grass
[2] https://github.com/grass-svn2git/grass
[3] https://github.com/grass-svn2git/grass/releases
[4] http://geo102.fsv.cvut.cz/~landa/tmp/log_grass_touched.txt
[5] http://geo102.fsv.cvut.cz/~landa/tmp/log_grass_untouched.txt

--
Martin Landa
http://geo.fsv.cvut.cz/gwiki/Landa
http://gismentors.cz/mentors/landa

[2] https://github.com/grass-svn2git/grass

[2] https://github.com/grass-svn2git/grass/branches/all

--
Martin Landa
http://geo.fsv.cvut.cz/gwiki/Landa
http://gismentors.cz/mentors/landa

On Wed, May 15, 2019 at 9:14 AM Martin Landa <landa.martin@gmail.com> wrote:

Hi,

st 15. 5. 2019 v 9:51 odesĂ­latel Martin Landa <landa.martin@gmail.com> napsal:

Something strange happen with svn repo after r74441. I will publish
testing repo for public review hopefully today.

OK, finally a new fresh grass repo preview available for review (last
processed commit 74475).

Please compare git content [1], branches [2] and tags [3] with svn.
Please review logs reporting changed message [4] and unchanged
messages [5].

I was also looking at the list of contributors [6]. I know this was discussed, but now I’m not really sure what to expect there.

[6] https://github.com/grass-svn2git/grass/graphs/contributors

If no problems appears this will be the last
preview before REAL migration.

When comparing the content of the repositories, I see the difference in $Date$ which of course has the whole system related to it. What we are doing with that?

Another thing is the need for new contributing guidelines. Git is not Subversion and committing to master won’t work (please, let me know if you want me to show some examples). What other OSGeo projects are doing is that contributing guidelines say that you should do pull request. It seems that this is often preferred way even for core developers. From what I gathered from a small sample of people at OSGeo sprint, the core devs don’t go though fork, but they do go through a branch. In GitHub, we can set “Require pull request reviews before merging” and “Include administrators” for the master branch to enforce that. I think we should do it at least at the beginning.

Vaclav

[1] https://github.com/grass-svn2git/grass
[2] https://github.com/grass-svn2git/grass
[3] https://github.com/grass-svn2git/grass/releases
[4] http://geo102.fsv.cvut.cz/~landa/tmp/log_grass_touched.txt
[5] http://geo102.fsv.cvut.cz/~landa/tmp/log_grass_untouched.txt

–
Martin Landa
http://geo.fsv.cvut.cz/gwiki/Landa
http://gismentors.cz/mentors/landa


grass-dev mailing list
grass-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/grass-dev

Hi,

a quick comment on this one:

Vaclav Petras <wenzeslaus@gmail.com> schrieb am Mi., 15. Mai 2019, 18:20:




I was also looking at the list of contributors [6]. I know this was discussed, but now I’m not really sure what to expect there.

[6] https://github.com/grass-svn2git/grass/graphs/contributors

This are the authors who have a GitHub account known to us. There is a lookup table which maps SVN account to GitHub account (in addons/tools/svn2git somewhere).
In fact, the list is currently a tiny fraction of the real authorship!

So, if someone isn’t listed yet:
Please get a GitHub account and/or communicate it to us (name + related email).

Later on, also commit claiming is possible and easy, so nothing is lost.

Markus

On Wed, May 15, 2019 at 12:47 PM Markus Neteler <neteler@osgeo.org> wrote:

Later on, also commit claiming is possible and easy, so nothing is lost.

Sounds good. I didn’t know. We just have to be very clear about the current state not capturing everyone.

Markus Neteler wrote

So, if someone isn't listed yet:
Please get a GitHub account and/or communicate it to us (name + related
email).

svn: hellik

gh account: hellik
email: hkmyricaria@gmail.com

-----
best regards
Helmut
--
Sent from: http://osgeo-org.1560.x6.nabble.com/Grass-Dev-f3991897.html

Hi,

El mié., 15 may. 2019 13:20, Vaclav Petras <wenzeslaus@gmail.com> escribió:

On Wed, May 15, 2019 at 9:14 AM Martin Landa <landa.martin@gmail.com> wrote:

Hi,

st 15. 5. 2019 v 9:51 odesĂ­latel Martin Landa <landa.martin@gmail.com> napsal:

Something strange happen with svn repo after r74441. I will publish
testing repo for public review hopefully today.

OK, finally a new fresh grass repo preview available for review (last
processed commit 74475).

Please compare git content [1], branches [2] and tags [3] with svn.
Please review logs reporting changed message [4] and unchanged
messages [5].

I was also looking at the list of contributors [6]. I know this was discussed, but now I’m not really sure what to expect there.

[6] https://github.com/grass-svn2git/grass/graphs/contributors

If no problems appears this will be the last
preview before REAL migration.

When comparing the content of the repositories, I see the difference in $Date$ which of course has the whole system related to it. What we are doing with that?

Another thing is the need for new contributing guidelines. Git is not Subversion and committing to master won’t work (please, let me know if you want me to show some examples).

I am very interested in such examples, esp. how the workflow will be from now on for example when making a small change in manuals.

I think it is very important that those of you more familiar with git and GitHub develop or show example workflows of how to contribute, backport (if that will be still called like that), and so on.

my 0.01 cent
Vero

What other OSGeo projects are doing is that contributing guidelines say that you should do pull request. It seems that this is often preferred way even for core developers. From what I gathered from a small sample of people at OSGeo sprint, the core devs don’t go though fork, but they do go through a branch. In GitHub, we can set “Require pull request reviews before merging” and “Include administrators” for the master branch to enforce that. I think we should do it at least at the beginning.

Vaclav

[1] https://github.com/grass-svn2git/grass
[2] https://github.com/grass-svn2git/grass
[3] https://github.com/grass-svn2git/grass/releases
[4] http://geo102.fsv.cvut.cz/~landa/tmp/log_grass_touched.txt
[5] http://geo102.fsv.cvut.cz/~landa/tmp/log_grass_untouched.txt

–
Martin Landa
http://geo.fsv.cvut.cz/gwiki/Landa
http://gismentors.cz/mentors/landa


grass-dev mailing list
grass-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/grass-dev


grass-dev mailing list
grass-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/grass-dev

On Wed, May 15, 2019 at 9:27 PM Vaclav Petras <wenzeslaus@gmail.com> wrote:

On Wed, May 15, 2019 at 12:47 PM Markus Neteler <neteler@osgeo.org> wrote:

Later on, also commit claiming is possible and easy, so nothing is lost.

Sounds good. I didn't know. We just have to be very clear about the current state not capturing everyone.

So I wrote this section just now:
https://trac.osgeo.org/grass/wiki/GitMigration#Authorship:RecognizingtheGRASSGIScontributorsduringandaftermigration

Feel free to improve the wording.

Markus

--
Markus Neteler, PhD
https://www.mundialis.de - free data with free software
https://grass.osgeo.org
https://courses.neteler.org/blog

On Wed, May 15, 2019 at 9:32 PM Helmut Kudrnovsky <hellik@web.de> wrote:

Markus Neteler wrote
> So, if someone isn't listed yet:
> Please get a GitHub account and/or communicate it to us (name + related
> email).

svn: hellik

gh account: hellik

You were already in the list.

But:

email: hkmyricaria@gmail.com

Your hellik@web.de is apparently not yet registered at Github.
Solution:

Or, simply, to match your commits to your GitHub account, just add
your associated email address(es) to your account in order to claim
your contributions (https://github.com/settings/emails).

Then you should show up there after some minutes or so.

Markus

On Wed, 15 May 2019 at 21:40, Veronica Andreo <veroandreo@gmail.com> wrote:

Hi,

Hi,

Another thing is the need for new contributing guidelines. Git is not Subversion and committing to master won't work (please, let me know if you want me to show some examples).

I am very interested in such examples, esp. how the workflow will be from now on for example when making a small change in manuals.

Yes, it is important to know this. How to use branches and also how to
update local version from the master one. Should we use a private
clone and make pull request to the main repository o use the main
repository directly?

I think it is very important that those of you more familiar with git and GitHub develop or show example workflows of how to contribute, backport (if that will be still called like that), and so on.

+1

my 0.01 cent
Vero

--
ciao
Luca

www.lucadelu.org

On Wed, May 15, 2019 at 10:20 PM Luca Delucchi <lucadeluge@gmail.com> wrote:

On Wed, 15 May 2019 at 21:40, Veronica Andreo <veroandreo@gmail.com> wrote:
>> Another thing is the need for new contributing guidelines. Git is not Subversion and committing to master won't work (please, let me know if you want me to show some examples).
>
> I am very interested in such examples, esp. how the workflow will be from now on for example when making a small change in manuals.

Yes, it is important to know this. How to use branches and also how to
update local version from the master one. Should we use a private
clone and make pull request to the main repository o use the main
repository directly?

AFAIK it is recommended to open an own "feature branch" in the own
fork and then open a pull request.

Concerning backports we could check the bot magic used by other OSGeo projects.

Markus

Markus Neteler wrote

On Wed, May 15, 2019 at 9:32 PM Helmut Kudrnovsky &lt;

hellik@

&gt; wrote:

Markus Neteler wrote
> So, if someone isn't listed yet:
> Please get a GitHub account and/or communicate it to us (name + related
> email).

svn: hellik

gh account: hellik

You were already in the list.

But:

email:

hkmyricaria@

Your

hellik@

is apparently not yet registered at Github.
Solution:

Or, simply, to match your commits to your GitHub account, just add
your associated email address(es) to your account in order to claim
your contributions (https://github.com/settings/emails).

Then you should show up there after some minutes or so.

additional email in my github account added, so lets see

-----
best regards
Helmut
--
Sent from: http://osgeo-org.1560.x6.nabble.com/Grass-Dev-f3991897.html

On Wed, May 15, 2019 at 4:24 PM Markus Neteler <neteler@osgeo.org> wrote:

On Wed, May 15, 2019 at 10:20 PM Luca Delucchi <lucadeluge@gmail.com> wrote:

On Wed, 15 May 2019 at 21:40, Veronica Andreo <veroandreo@gmail.com> wrote:

Another thing is the need for new contributing guidelines. Git is not Subversion and committing to master won’t work (please, let me know if you want me to show some examples).

I am very interested in such examples, esp. how the workflow will be from now on for example when making a small change in manuals.

Yes, it is important to know this. How to use branches and also how to
update local version from the master one. Should we use a private
clone and make pull request to the main repository o use the main
repository directly?

AFAIK it is recommended to open an own “feature branch” in the own

fork and then open a pull request.

Exactly. The important part to recognize is that trying to emulate the Subversion workflow, i.e. simply committing to master, would soon turn into a repo full of automatic merge commits in the best case and big confusion in the worst case. Basically all projects (I think I checked QGIS, PROJ, GDAL, PDAL) have fork+branch+pull request as their best practice. This means that even if you are a core developer, you still should do that. The exception common for the core developers is that they skip the fork part, i.e. they branch in the main repo. However, I haven’t seen that formalized, so I think it is a tolerated practice rather than a guideline.

Hopefully, for the general case, GRASS GIS will not need any special instructions. Often projects simply point to GitHub documentation for making and maintaining a fork and making a pull request.

Although there will be extra steps for the people with direct access to source code comparing to Subversion, the benefit is that there is (will be) actually an extra step and that’s CI build and (hopefully) test as well. This should catch many issues before they get to master.