[GRASS-dev] [GRASS GIS] #3722: migrate grass svn repositories to git

GDAL · December 29, 2018, 7:52pm

#3722: migrate grass svn repositories to git
---------------------------------+-------------------------
Reporter: martinl | Owner: grass-dev@…
Type: task | Status: new
Priority: major | Milestone:
Component: Default | Version: unspecified
Keywords: svn, git, migration | CPU: Unspecified
Platform: Unspecified |
---------------------------------+-------------------------
This ticket summarizes current efforts of migrating GRASS svn repositories
(https://svn.osgeo.org/grass/) to git.

Pilot git repository (of main grass svn repo) is available for testing at

http://geo102.fsv.cvut.cz:8090/grass/grass

List of branches:

https://svn.osgeo.org/grass/grass/branches/ compare with
http://geo102.fsv.cvut.cz:8090/grass/grass/branches

List of tags:

https://svn.osgeo.org/grass/grass/tags/ compare with
http://geo102.fsv.cvut.cz:8090/grass/grass/tags

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/3722>
GRASS GIS <https://grass.osgeo.org>

GDAL · December 29, 2018, 8:00pm

#3722: migrate grass svn repositories to git
--------------------------+---------------------------------
  Reporter: martinl | Owner: grass-dev@…
      Type: task | Status: new
  Priority: major | Milestone:
Component: Default | Version: unspecified
Resolution: | Keywords: svn, git, migration
       CPU: Unspecified | Platform: Unspecified
--------------------------+---------------------------------

Comment (by martinl):

Open issues:

* choice of git repository manager (master)
1. join Github OSGeo group, https://github.com/OSGeo
2. Gitlab (or Bitbucket, ...) no OSGeo group exists
3. https://git.osgeo.org/gitea

* choice of issue tracker
1. stay with trac
2. use native git repository manager issue tracker

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/3722#comment:1>
GRASS GIS <https://grass.osgeo.org>

GDAL · December 29, 2018, 8:11pm

#3722: migrate grass svn repositories to git
--------------------------+---------------------------------
  Reporter: martinl | Owner: grass-dev@…
      Type: task | Status: new
  Priority: major | Milestone:
Component: Default | Version: unspecified
Resolution: | Keywords: svn, git, migration
       CPU: Unspecified | Platform: Unspecified
--------------------------+---------------------------------

Comment (by martinl):

Replying to [comment:1 martinl]:
> * choice of issue tracker
> 1. stay with trac

see https://trac.osgeo.org/osgeo/ticket/2233

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/3722#comment:2>
GRASS GIS <https://grass.osgeo.org>

GDAL · December 29, 2018, 10:16pm

#3722: migrate grass svn repositories to git
--------------------------+---------------------------------
  Reporter: martinl | Owner: grass-dev@…
      Type: task | Status: new
  Priority: major | Milestone:
Component: Default | Version: unspecified
Resolution: | Keywords: svn, git, migration
       CPU: Unspecified | Platform: Unspecified
--------------------------+---------------------------------

Comment (by neteler):

(maybe interesting to read:
https://trac.osgeo.org/gdal/wiki/rfc71_github_migration )

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/3722#comment:3>
GRASS GIS <https://grass.osgeo.org>

GDAL · January 1, 2019, 6:03am

#3722: migrate grass svn repositories to git
--------------------------+---------------------------------
  Reporter: martinl | Owner: grass-dev@…
      Type: task | Status: new
  Priority: major | Milestone:
Component: Default | Version: unspecified
Resolution: | Keywords: svn, git, migration
       CPU: Unspecified | Platform: Unspecified
--------------------------+---------------------------------

Comment (by mankoff):

Not a developer but interested in helping more. My preference: GitLab and
Issues in GitLab. I think issues should be local to source because these
systems all support nice integration. Pull requests and patches and
comments and bugs can all easily reference each other and files and lines
in files via # and % and ~ and etc codes.

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/3722#comment:4>
GRASS GIS <https://grass.osgeo.org>

GDAL · January 1, 2019, 10:10pm

#3722: migrate grass svn repositories to git
--------------------------+---------------------------------
  Reporter: martinl | Owner: grass-dev@…
      Type: task | Status: new
  Priority: major | Milestone:
Component: Default | Version: unspecified
Resolution: | Keywords: svn, git, migration
       CPU: Unspecified | Platform: Unspecified
--------------------------+---------------------------------

Comment (by martinl):

Replying to [comment:2 martinl]:
> Replying to [comment:1 martinl]:
> > * choice of issue tracker
> > 1. stay with trac
>
> see https://trac.osgeo.org/osgeo/ticket/2233

first test with trac and git at

http://geo102.fsv.cvut.cz:8091/grass

more specific

http://geo102.fsv.cvut.cz:8091/grass/browser/grass

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/3722#comment:5>
GRASS GIS <https://grass.osgeo.org>

GDAL · January 27, 2019, 9:28pm

#3722: migrate grass svn repositories to git
--------------------------+---------------------------------
  Reporter: martinl | Owner: grass-dev@…
      Type: task | Status: new
  Priority: major | Milestone:
Component: Default | Version: unspecified
Resolution: | Keywords: svn, git, migration
       CPU: Unspecified | Platform: Unspecified
--------------------------+---------------------------------

Comment (by martinl):

First test:

commit:

http://geo102.fsv.cvut.cz:8091/grass/changeset/c564fcb572aaa7dd4ecb8570cc477bce92677512

( or
http://geo102.fsv.cvut.cz:8090/grass/grass/commit/c564fcb572aaa7dd4ecb8570cc477bce92677512
)

related issue: http://geo102.fsv.cvut.cz:8091/grass/ticket/3724

Unfortunately issue lacks link to related commit, any idea what could be
wrong?

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/3722#comment:6>
GRASS GIS <https://grass.osgeo.org>

GDAL · January 27, 2019, 9:53pm

#3722: migrate grass svn repositories to git
--------------------------+---------------------------------
  Reporter: martinl | Owner: grass-dev@…
      Type: task | Status: new
  Priority: major | Milestone:
Component: Default | Version: unspecified
Resolution: | Keywords: svn, git, migration
       CPU: Unspecified | Platform: Unspecified
--------------------------+---------------------------------

Comment (by martinl):

Replying to [comment:6 martinl]:
> Unfortunately issue lacks link to related commit, any idea what could be
wrong?

probably something related to
https://trac.edgewall.org/wiki/CommitTicketUpdater#Checkpermissions (?)

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/3722#comment:7>
GRASS GIS <https://grass.osgeo.org>

GDAL · January 28, 2019, 10:04am

#3722: migrate grass svn repositories to git
--------------------------+---------------------------------
  Reporter: martinl | Owner: grass-dev@…
      Type: task | Status: new
  Priority: major | Milestone:
Component: Default | Version: unspecified
Resolution: | Keywords: svn, git, migration
       CPU: Unspecified | Platform: Unspecified
--------------------------+---------------------------------

Comment (by pmav99):

Hello Martin, If you don't mind a couple of questions.
1. How did you convert the svn repo to a git one?
2. The git repo you posted says that it has 36743 commits. Is that the
whole repo or just `trunk`?

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/3722#comment:8>
GRASS GIS <https://grass.osgeo.org>

GDAL · January 28, 2019, 10:08am

#3722: migrate grass svn repositories to git
--------------------------+---------------------------------
  Reporter: martinl | Owner: grass-dev@…
      Type: task | Status: new
  Priority: major | Milestone:
Component: Default | Version: unspecified
Resolution: | Keywords: svn, git, migration
       CPU: Unspecified | Platform: Unspecified
--------------------------+---------------------------------

Comment (by martinl):

Replying to [comment:8 pmav99]:

> 1. How did you convert the svn repo to a git one?

https://trac.osgeo.org/grass/browser/grass-addons/tools/svn2git

> 2. The git repo you posted says that it has 36743 commits. Is that the
whole repo or just `trunk`?

http://geo102.fsv.cvut.cz:8090/grass/grass/branches

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/3722#comment:9>
GRASS GIS <https://grass.osgeo.org>

GDAL · January 28, 2019, 11:04am

#3722: migrate grass svn repositories to git
--------------------------+---------------------------------
  Reporter: martinl | Owner: grass-dev@…
      Type: task | Status: new
  Priority: major | Milestone:
Component: Default | Version: unspecified
Resolution: | Keywords: svn, git, migration
       CPU: Unspecified | Platform: Unspecified
--------------------------+---------------------------------

Comment (by sbl):

Probably useful migration tool to look at (for gitlab):
https://github.com/tracboat/tracboat

Also this: https://docs.gitlab.com/ce/user/project/import/svn.html

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/3722#comment:10>
GRASS GIS <https://grass.osgeo.org>

GDAL · January 28, 2019, 1:06pm

#3722: migrate grass svn repositories to git
--------------------------+---------------------------------
  Reporter: martinl | Owner: grass-dev@…
      Type: task | Status: new
  Priority: major | Milestone:
Component: Default | Version: unspecified
Resolution: | Keywords: svn, git, migration
       CPU: Unspecified | Platform: Unspecified
--------------------------+---------------------------------

Comment (by pmav99):

I don't have any experience with SVN to GIT conversions, but shouldn't
there be 70k+ commits?

I did run a couple of tests with subgit. It seems a solid project. If
someone wants to try it the commands for only converting trunk is (needs 6
hours):
{{{
subgit configure https://svn.osgeo.org/grass/grass/trunk --layout
directory grass_trunk.git
subgit install grass_full.git
}}}
The result is a repo with 36800+ commits which is comparable to 36743.
That's why I asked, if that was trunk only.

I also tried to convert the whole SVN repo:
{{{
subgit configure https://svn.osgeo.org/grass/grass --layout std
grass_full.git
subgit install grass_full.git
}}}
but there is some problem with revision 68772. The logs suggest that it
has to do an old SVN bug: https://subversion.apache.org/docs/issue4129 .
This is the log:

{{{
SubGit version 3.3.5 ('Bobique') build #4042

Translating Subversion revisions to Git commits...

INSTALLATION FAILED

error: The Subversion repository is corrupted at revision r68772 because
of Subversion issue http://subversion.apache.org/docs/issue4129 .
error: To prevent this error in the futureupdate your Subversion server
version to 1.7.5 or to 1.6.18.
error: To recover your repository perform svnadmin dump/load procedure:
error: $ svnadmin dump path/to/svn/repository > repo.dump
error: $ svnadmin create path/for/recovered/svn/repository
error: $ svnadmin load path/for/recovered/svn/repository < repo.dump
error: After that please re-clone the repository.
error: svn: E204900: Checksum mismatch in
branches/releasebranch_7_2/lib/db/sqlp/sql.html: expected
7245ea62da02566d2edcab901919d4e9 but found
0061e8e51c3b551ac6123c88a66940cc
error: Checksum mismatch in
branches/releasebranch_7_2/lib/db/sqlp/sql.html: expected
7245ea62da02566d2edcab901919d4e9 but found
0061e8e51c3b551ac6123c88a66940cc
error: Unexpected error has occurred; please report along with the logs
('/home/username/Prog/svn/subgit-install-20190128-125015.zip')
error: to http://issues.tmatesoft.com/, thank you!
}}}

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/3722#comment:11>
GRASS GIS <https://grass.osgeo.org>

GDAL · January 28, 2019, 2:47pm

#3722: migrate grass svn repositories to git
--------------------------+---------------------------------
  Reporter: martinl | Owner: grass-dev@…
      Type: task | Status: new
  Priority: major | Milestone:
Component: Default | Version: unspecified
Resolution: | Keywords: svn, git, migration
       CPU: Unspecified | Platform: Unspecified
--------------------------+---------------------------------

Comment (by martinl):

Replying to [comment:11 pmav99]:
> I don't have any experience with SVN to GIT conversions, but shouldn't
there be 70k+ commits?

well, not all commits went to `trunk` in SVN.

{{{
$ git branch
* master
$ git log | grep -c ^commit
36743

$ git checkout releasebranch_7_6
Branch releasebranch_7_6 set up to track remote branch releasebranch_7_6
from origin.
Switched to a new branch 'releasebranch_7_6'
$ git log | grep -c ^commit
36620

$ git checkout releasebranch_6_4
Checking out files: 100% (9643/9643), done.
Branch releasebranch_6_4 set up to track remote branch releasebranch_6_4
from origin.
Switched to a new branch 'releasebranch_6_4'
$ git log | grep -c ^commit
22575
}}}

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/3722#comment:12>
GRASS GIS <https://grass.osgeo.org>

GDAL · January 28, 2019, 2:53pm

#3722: migrate grass svn repositories to git
--------------------------+---------------------------------
  Reporter: martinl | Owner: grass-dev@…
      Type: task | Status: new
  Priority: major | Milestone:
Component: Default | Version: unspecified
Resolution: | Keywords: svn, git, migration
       CPU: Unspecified | Platform: Unspecified
--------------------------+---------------------------------

Comment (by martinl):

Replying to [comment:12 martinl]:
> {{{
> $ git branch
> * master
> $ git log | grep -c ^commit
> 36743
> }}}

In SVN:

{{{
$ svn info
URL: https://svn.osgeo.org/grass/grass/trunk
Relative URL: ^/grass/trunk
svn log | grep -c '^r[0-9]'
36900
}}}

Number is slightly different since git experimental repo is not up-to-
date. Anyway it's not 7e4 commits.

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/3722#comment:13>
GRASS GIS <https://grass.osgeo.org>

GDAL · February 4, 2019, 2:55pm

#3722: migrate grass svn repositories to git
--------------------------+---------------------------------
  Reporter: martinl | Owner: grass-dev@…
      Type: task | Status: new
  Priority: major | Milestone:
Component: Default | Version: unspecified
Resolution: | Keywords: svn, git, migration
       CPU: Unspecified | Platform: Unspecified
--------------------------+---------------------------------

Comment (by martinl):

BTW, I did some tests with trac2github,
https://github.com/trustmaster/trac2github

Repo: https://github.com/landam/grass-gis-git-migration-test

Milestones: https://github.com/landam/grass-gis-git-migration-
test/milestones

Labels: https://github.com/landam/grass-gis-git-migration-test/labels

Issues (first 40 issues migrated): https://github.com/landam/grass-gis-
git-migration-test/issues?utf8=%E2%9C%93&q=is%3Aissue+

Known issues:

* SVN revisions should be rewritten to full URL, eg.

https://github.com/landam/grass-gis-git-migration-
test/issues/39#issuecomment-460275085

source: https://trac.osgeo.org/grass/ticket/39#comment:6

r30046 -> https://trac.osgeo.org/grass/changeset/30046

* users not transferred

Please do review, there will be probably more issues.

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/3722#comment:14>
GRASS GIS <https://grass.osgeo.org>

GDAL · February 4, 2019, 3:49pm

#3722: migrate grass svn repositories to git
--------------------------+---------------------------------
  Reporter: martinl | Owner: grass-dev@…
      Type: task | Status: new
  Priority: major | Milestone:
Component: Default | Version: unspecified
Resolution: | Keywords: svn, git, migration
       CPU: Unspecified | Platform: Unspecified
--------------------------+---------------------------------

Comment (by martinl):

Replying to [comment:14 martinl]:
> * SVN revisions should be rewritten to full URL, eg.

Solved by local modification of PHP code, see

https://github.com/landam/grass-gis-git-migration-
test/issues/49#issuecomment-460298069

source: https://trac.osgeo.org/grass/ticket/42#comment:4

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/3722#comment:15>
GRASS GIS <https://grass.osgeo.org>

GDAL · February 5, 2019, 1:06pm

#3722: migrate grass svn repositories to git
--------------------------+---------------------------------
  Reporter: martinl | Owner: grass-dev@…
      Type: task | Status: new
  Priority: major | Milestone:
Component: Default | Version: unspecified
Resolution: | Keywords: svn, git, migration
       CPU: Unspecified | Platform: Unspecified
--------------------------+---------------------------------

Comment (by neteler):

Here the PHP snippet for translating the components to a reduced set of
git labels:

{{{
    'Compiling' => 'core',
    'Database' => 'core',
    'Default' => 'core',
    'Display' => 'core',
    'Docs' => 'core',
    'Imagery' => 'core',
    'Installation' => 'core',
    'LibGIS' => 'core',
    'LibOpenGL' => 'core',
    'LibRaster' => 'core',
    'LibVector' => 'core',
    'License' => 'core',
    'Packaging' => 'core',
    'Parser' => 'core',
    'Projections/Datums' => 'core',
    'Ps.map' => 'core',
    'PyGRASS' => 'python',
    'Python' => 'python',
    'Python ctypes' => 'python',
    'Raster' => 'core',
    'Raster3D' => 'core',
    'Startup' => 'core',
    'Tcl/Tk' => 'core',
    'Tcl/Tk NVIZ' => 'core',
    'Temporal' => 'core',
    'Tests' => 'unittests',
    'Translations' => 'translations',
    'Vector' => 'core',
    'wxGUI' => 'core',
    'Addons' => NULL,
    'Datasets' => NULL,
    'Shell Scripts' => NULL,
    'Website' => NULL
}}}

hence, not needed (as addons are in a separate git repo, shell no longer
there, and datasets + CMS done differently):
{{{
# 'Addons' => NULL,
# 'Datasets' => NULL,
# 'Shell Scripts' => NULL,
# 'Website' => NULL
}}}

So, the target git labels are now:

  * core
  * python
  * translations
  * unittests

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/3722#comment:16>
GRASS GIS <https://grass.osgeo.org>

GDAL · February 6, 2019, 10:54am

#3722: migrate grass svn repositories to git
--------------------------+---------------------------------
  Reporter: martinl | Owner: grass-dev@…
      Type: task | Status: new
  Priority: major | Milestone:
Component: Default | Version: unspecified
Resolution: | Keywords: svn, git, migration
       CPU: Unspecified | Platform: Unspecified
--------------------------+---------------------------------

Comment (by martinl):

Replying to [comment:14 martinl]:
> BTW, I did some tests with trac2github,
https://github.com/trustmaster/trac2github

For record, my fork which includes GRASS related changes is available at
https://github.com/landam/trac2github/tree/grass

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/3722#comment:17>
GRASS GIS <https://grass.osgeo.org>

GDAL · March 4, 2019, 9:29pm

#3722: migrate grass svn repositories to git
--------------------------+---------------------------------
  Reporter: martinl | Owner: grass-dev@…
      Type: task | Status: new
  Priority: major | Milestone:
Component: Default | Version: unspecified
Resolution: | Keywords: svn, git, migration
       CPU: Unspecified | Platform: Unspecified
--------------------------+---------------------------------

Comment (by pmav99):

AFAI can tell, all the test git repos are currently down. Is there a link
to a working one?

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/3722#comment:18>
GRASS GIS <https://grass.osgeo.org>

GDAL · March 5, 2019, 7:04pm

#3722: migrate grass svn repositories to git
--------------------------+---------------------------------
  Reporter: martinl | Owner: grass-dev@…
      Type: task | Status: new
  Priority: major | Milestone:
Component: Default | Version: unspecified
Resolution: | Keywords: svn, git, migration
       CPU: Unspecified | Platform: Unspecified
--------------------------+---------------------------------

Comment (by martinl):

Replying to [comment:18 pmav99]:
> AFAI can tell, all the test git repos are currently down. Is there a
link to a working one?

right. Since most of questioners in GRASS git migration survey
(https://docs.google.com/forms/d/1BoTFyZRNebqVX98A3rh5GpUS2gKFfmuim78gbradDjc/viewanalytics)
prefer github as target platform I have disabled gitlab/trac service.

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/3722#comment:19>
GRASS GIS <https://grass.osgeo.org>