[GRASS-dev] PROJ 5 support in trunk

Hi all,

I have added support for the new PROJ 5+ API in trunk, finished with r72433. This means, trunk is now using exclusively the new PROJ 5+ API if available.

This new PROJ 5 API is available e.g. in Debian testing.

Please test!

During testing, I discovered some dangerous naming conflicts in GRASS and PROJ4/PROJ5: internal functions start both in GRASS and PROJ with pj_ which can lead to cryptic errors. Within GRASS, some but not all projection-related functions start with GPJ_, this should be adopted for those starting with pj_. Also the GRASS structure struct pj_info should be renamed to struct gpj_info.

Markus M

On Tue, Mar 20, 2018 at 8:04 PM, Markus Metz <markus.metz.giswork@gmail.com> wrote:

Hi all,

I have added support for the new PROJ 5+ API in trunk, finished with r72433.

Now really finished with trunk r72443.

Markus M

This means, trunk is now using exclusively the new PROJ 5+ API if available.

This new PROJ 5 API is available e.g. in Debian testing.

Please test!

During testing, I discovered some dangerous naming conflicts in GRASS and PROJ4/PROJ5: internal functions start both in GRASS and PROJ with pj_ which can lead to cryptic errors. Within GRASS, some but not all projection-related functions start with GPJ_, this should be adopted for those starting with pj_. Also the GRASS structure struct pj_info should be renamed to struct gpj_info.

Markus M

On Tue, Mar 20, 2018 at 8:04 PM, Markus Metz
<markus.metz.giswork@gmail.com> wrote:

Hi all,

I have added support for the new PROJ 5+ API in trunk, finished with r72433.

That's really great! First OSGeo project :slight_smile:

It was a lot of work for you, thanks.

This means, trunk is now using exclusively the new PROJ 5+ API if available.

This new PROJ 5 API is available e.g. in Debian testing.

Please test!

During testing, I discovered some dangerous naming conflicts in GRASS and
PROJ4/PROJ5: internal functions start both in GRASS and PROJ with pj_ which
can lead to cryptic errors. Within GRASS, some but not all
projection-related functions start with GPJ_, this should be adopted for
those starting with pj_. Also the GRASS structure struct pj_info should be
renamed to struct gpj_info.

BTW: Also GDAL now comes with support for PROJ 5, see
https://lists.osgeo.org/pipermail/gdal-dev/2018-March/048278.html

markusN

On Thu, Mar 22, 2018 at 7:16 AM, Markus Neteler <neteler@osgeo.org> wrote:

On Tue, Mar 20, 2018 at 8:04 PM, Markus Metz
<markus.metz.giswork@gmail.com> wrote:

Hi all,

I have added support for the new PROJ 5+ API in trunk, finished with r72433.

That’s really great! First OSGeo project :slight_smile:

It was a lot of work for you, thanks.

The work is not yet finished. The current implementation works but is ugly and code in modules became less readable because of lots of ifdefs for PROJ4 or PROJ5. I am now testing a new GRASS API as interface to PROJ4/5 that would allow to remove these ifdefs in modules. The old GRASS API as well as the standard PROJ4 API will still be supported. The main difference is that GRASS now uses for PROJ5 the new pipeline syntax which allows to define more complex transformations. Forward/backward transformation will also become easier by simply defining the direction. m.proj still needs to be updated to support this new pipeline syntax with any number of steps as input for proj_in.

This means, trunk is now using exclusively the new PROJ 5+ API if available.

This new PROJ 5 API is available e.g. in Debian testing.

Please test!

During testing, I discovered some dangerous naming conflicts in GRASS and
PROJ4/PROJ5: internal functions start both in GRASS and PROJ with pj_ which
can lead to cryptic errors. Within GRASS, some but not all
projection-related functions start with GPJ_, this should be adopted for
those starting with pj_. Also the GRASS structure struct pj_info should be
renamed to struct gpj_info.

BTW: Also GDAL now comes with support for PROJ 5, see
https://lists.osgeo.org/pipermail/gdal-dev/2018-March/048278.html

Nice!

Markus M

markusN

On Thu, Mar 22, 2018 at 4:38 PM, Markus Metz <markus.metz.giswork@gmail.com> wrote:

On Thu, Mar 22, 2018 at 7:16 AM, Markus Neteler <neteler@osgeo.org> wrote:

On Tue, Mar 20, 2018 at 8:04 PM, Markus Metz
<markus.metz.giswork@gmail.com> wrote:

Hi all,

I have added support for the new PROJ 5+ API in trunk, finished with r72433.

That’s really great! First OSGeo project :slight_smile:

It was a lot of work for you, thanks.

The work is not yet finished.

Now (last related commit is trunk r72522) it’s finished. I have introduced a new GRASS API that handles both PROJ 4 and PROJ 5, consisting of

GPJ_init_transform() (new, initializes coordinate transformation)
GPJ_transform() replacing pj_do_proj()
GPJ_transform_array() replacing pj_do_transform()

The old GRASS API is still there but is no longer used by core modules.

Only few ifdefs in lib/proj are needed for different PROJ versions. Modules do not need any ifdefs, they can simply call GPJ_init_transform() + GPJ_transform() without caring about the PROJ API.

As soon as PROJ 5.0.1 is out, I would like to drop support for PROJ 5.0.0 and only support PROJ 4 and PROJ 5.0.1 or higher (learning from GDAL :-)).

Markus M

The current implementation works but is ugly and code in modules became less readable because of lots of ifdefs for PROJ4 or PROJ5. I am now testing a new GRASS API as interface to PROJ4/5 that would allow to remove these ifdefs in modules. The old GRASS API as well as the standard PROJ4 API will still be supported. The main difference is that GRASS now uses for PROJ5 the new pipeline syntax which allows to define more complex transformations. Forward/backward transformation will also become easier by simply defining the direction. m.proj still needs to be updated to support this new pipeline syntax with any number of steps as input for proj_in.

This means, trunk is now using exclusively the new PROJ 5+ API if available.

This new PROJ 5 API is available e.g. in Debian testing.

Please test!

During testing, I discovered some dangerous naming conflicts in GRASS and
PROJ4/PROJ5: internal functions start both in GRASS and PROJ with pj_ which
can lead to cryptic errors. Within GRASS, some but not all
projection-related functions start with GPJ_, this should be adopted for
those starting with pj_. Also the GRASS structure struct pj_info should be
renamed to struct gpj_info.

BTW: Also GDAL now comes with support for PROJ 5, see
https://lists.osgeo.org/pipermail/gdal-dev/2018-March/048278.html

Nice!

Markus M

markusN

On 23/03/18 10:41, Markus Metz wrote:

Now (last related commit is trunk r72522) it's finished. I have introduced a new GRASS API that handles both PROJ 4 and PROJ 5, consisting of

GPJ_init_transform() (new, initializes coordinate transformation)
GPJ_transform() replacing pj_do_proj()
GPJ_transform_array() replacing pj_do_transform()

The old GRASS API is still there but is no longer used by core modules.

Only few ifdefs in lib/proj are needed for different PROJ versions. Modules do not need any ifdefs, they can simply call GPJ_init_transform() + GPJ_transform() without caring about the PROJ API.

As soon as PROJ 5.0.1 is out, I would like to drop support for PROJ 5.0.0 and only support PROJ 4 and PROJ 5.0.1 or higher (learning from GDAL :-)).

Wow ! Congratulations and thank you for all this hard work !!

Moritz

2018-03-23 10:55 GMT+01:00 Moritz Lennert <mlennert@club.worldonline.be>:

As soon as PROJ 5.0.1 is out, I would like to drop support for PROJ 5.0.0
and only support PROJ 4 and PROJ 5.0.1 or higher (learning from GDAL :-)).

Wow ! Congratulations and thank you for all this hard work !!

+1 Ma

--
Martin Landa
http://geo.fsv.cvut.cz/gwiki/Landa
http://gismentors.cz/mentors/landa

On 23 March 2018 at 11:30, Martin Landa <landa.martin@gmail.com> wrote:

2018-03-23 10:55 GMT+01:00 Moritz Lennert <mlennert@club.worldonline.be>:

As soon as PROJ 5.0.1 is out, I would like to drop support for PROJ 5.0.0
and only support PROJ 4 and PROJ 5.0.1 or higher (learning from GDAL :-)).

Wow ! Congratulations and thank you for all this hard work !!

+1 Ma

+1, thanks a lot

--
ciao
Luca

www.lucadelu.org

Thank you for this work!

Doug

···

On Fri, Mar 23, 2018 at 5:55 AM, Moritz Lennert <mlennert@club.worldonline.be> wrote:

On 23/03/18 10:41, Markus Metz wrote:

Now (last related commit is trunk r72522) it’s finished. I have introduced a new GRASS API that handles both PROJ 4 and PROJ 5, consisting of

GPJ_init_transform() (new, initializes coordinate transformation)
GPJ_transform() replacing pj_do_proj()
GPJ_transform_array() replacing pj_do_transform()

The old GRASS API is still there but is no longer used by core modules.

Only few ifdefs in lib/proj are needed for different PROJ versions. Modules do not need any ifdefs, they can simply call GPJ_init_transform() + GPJ_transform() without caring about the PROJ API.

As soon as PROJ 5.0.1 is out, I would like to drop support for PROJ 5.0.0 and only support PROJ 4 and PROJ 5.0.1 or higher (learning from GDAL :-)).

Wow ! Congratulations and thank you for all this hard work !!

Moritz


grass-dev mailing list
grass-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/grass-dev

Doug Newcomb
USFWS
551F Pylon Dr
Raleigh, NC
919-856-4520 ext. 14 doug_newcomb@fws.gov

__NOTE: This email correspondence and any attachments to and from this sender is subject to the Freedom of Information Act (FOIA) and may be disclosed to third parties.__​

Now (last related commit is trunk r72522) it's finished. I have >introduced

a new GRASS API that handles both PROJ 4 and PROJ >5, consisting of

Thanks for this Update!

Fyi See
https://lists.osgeo.org/pipermail/osgeo4w-dev/2018-March/003557.html

-------
[...]
The API from PROJ 4 lives on in PROJ 5, so GRASS 7.4 should be able to use
PROJ 5 as well. We’ve been carefull not to breaking anything with this
release.
That comes with PROJ 6 and 7. Of course there might be implementation
details
in GRASS that I am unaware of that makes using PROJ 5 impossible.
--------

-----
best regards
Helmut
--
Sent from: http://osgeo-org.1560.x6.nabble.com/Grass-Dev-f3991897.html

There is an important difference between the PROJ4 and the PROJ5+ API/syntax:

The old PROJ4 API uses latlong WGS84 as pivot datum for coordinate transformations like

projection1 → latlong WGS84 → projection2

or in ‘+to’ syntax
projection1 +to projection2

The new PROJ5+ API no longer uses a pivot datum. The advantage is that you can directly convert from one datum to another, without going through WGS84. The disadvantage is that the user/software using the new API has to make sure that either a common pivot datum is used or coordinates are correctly transformed from one datum to another, e.g. with a Helmert Transform.

Both GDAL and GRASS have implemented the new API as a simple 2-step pipeline like

  1. projection1 → some latlong
  2. some latlong → projection2

or in pipeline syntax

+proj=pipeline

+step +inv projection1

+step projection2

Even Rouault has done some tests and found sometimes subtle differences in the results between the old and new API/syntax. Further on, there is no mechanism (yet) in place to validate the pipeline, most importantly that the output of step 1 conforms to the required input for step 2.

IMHO, the implementation of the new PROJ5+ API/syntax in GDAL and GRASS should be regarded as experimental and testing by as many people as possible would be a huge benefit to PROJ/GDAL/GRASS and to all applications using these projects.

Markus M

On Sun, Mar 25, 2018 at 10:47 PM, Helmut Kudrnovsky <hellik@web.de> wrote:

Now (last related commit is trunk r72522) it’s finished. I have >introduced
a new GRASS API that handles both PROJ 4 and PROJ >5, consisting of

Thanks for this Update!

Fyi See
https://lists.osgeo.org/pipermail/osgeo4w-dev/2018-March/003557.html


[…]
The API from PROJ 4 lives on in PROJ 5, so GRASS 7.4 should be able to use
PROJ 5 as well. We’ve been carefull not to breaking anything with this
release.
That comes with PROJ 6 and 7. Of course there might be implementation
details
in GRASS that I am unaware of that makes using PROJ 5 impossible.


best regards
Helmut

Sent from: http://osgeo-org.1560.x6.nabble.com/Grass-Dev-f3991897.html


grass-dev mailing list
grass-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/grass-dev

On Mon, Mar 26, 2018 at 10:51 PM, Kristian Evers <kreve@sdfe.dk> wrote:

On 26 Mar 2018, at 21:21, Markus Metz <markus.metz.giswork@gmail.com>
wrote:

There is an important difference between the PROJ4 and the PROJ5+
API/syntax:

The old PROJ4 API uses latlong WGS84 as pivot datum for coordinate
transformations like

projection1 -> latlong WGS84 -> projection2
or in '+to' syntax
projection1 +to projection2

The new PROJ5+ API no longer uses a pivot datum. The advantage is that you
can directly convert from one datum to another, without going through
WGS84. The disadvantage is that the user/software using the new API has to
make sure that either a common pivot datum is used or coordinates are
correctly transformed from one datum to another, e.g. with a Helmert
Transform.

Both GDAL and GRASS have implemented the new API as a simple 2-step
pipeline like

1. projection1 -> some latlong
2. some latlong -> projection2

or in pipeline syntax

+proj=pipeline
+step +inv projection1
+step projection2

This is actually exactly what the proj_crs_to_crs() function does. When I
originally wrote it I had an idea that this should only be used with
init-files,
but seeing how people need a simple way to do what they’ve always done I
think it might be better to relax that requirement and allow input similar
to what goes into cs2cs and pj_transform().

The challenge is that users of other software using the new PROJ API like
GDAL and GRASS expect that reprojecting a dataset from one CRS to another
CRS "just works". Input and output CRS can be anything custom defined.
Either PROJ or other software using PROJ must have some mechanism to decide
if the requested reprojection can be done or not.

My plan for this function is that it eventually will be able to construct
a transformation between CRS’s without the WGS84 pivot. This will be guided
by the EPSG database. I have still to work out exactly how. For now
though, it works fine with already existing init files and WGS84 as a pivot
datum.
My hope is that this will land in version 6 but I won’t promise that just
yet.

Looking forward to this function! But what if a dataset comes with a CRS
definition without EPSG code? Can we expect that PROJ handles this or
should applications using the PROJ API handle this?

Markus M

Even Rouault has done some tests and found sometimes subtle differences in
the results between the old and new API/syntax. Further on, there is no
mechanism (yet) in place to validate the pipeline, most importantly that
the output of step 1 conforms to the required input for step 2.

I believe we have sorted those problems out in the coming version 5.0.1.

Regarding validation of input/output from pipeline steps. We can probably
do some basic checks, but in the end you will always be required to know
what you are doing. Mind you, we are not expecting everyday users to
construct their own pipelines all the time. It is a powertool for the
accomplished
user that knows what he is doing. Most users should have their needs met
by predefined transformations in init-files such as the epsg file. And
eventually
in a more clever way using the EPSG database directly as mentioned above.

IMHO, the implementation of the new PROJ5+ API/syntax in GDAL and GRASS
should be regarded as experimental and testing by as many people as
possible would be a huge benefit to PROJ/GDAL/GRASS and to all applications
using these projects.

Markus M

On Sun, Mar 25, 2018 at 10:47 PM, Helmut Kudrnovsky <hellik@web.de> wrote:
>
> >Now (last related commit is trunk r72522) it's finished. I have
>introduced
> a new GRASS API that handles both PROJ 4 and PROJ >5, consisting of
>
> Thanks for this Update!
>
> Fyi See
> https://lists.osgeo.org/pipermail/osgeo4w-dev/2018-March/003557.html
>
> -------
> [...]
> The API from PROJ 4 lives on in PROJ 5, so GRASS 7.4 should be able to
use
> PROJ 5 as well. We’ve been carefull not to breaking anything with this
> release.
> That comes with PROJ 6 and 7. Of course there might be implementation
> details
> in GRASS that I am unaware of that makes using PROJ 5 impossible.
> --------
>
>
>
> -----
> best regards
> Helmut
> --
> Sent from: http://osgeo-org.1560.x6.nabble.com/Grass-Dev-f3991897.html
> _______________________________________________
> grass-dev mailing list
> grass-dev@lists.osgeo.org
> https://lists.osgeo.org/mailman/listinfo/grass-dev

_______________________________________________
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

On Tue, Apr 3, 2018 at 1:57 PM, Kristian Evers <kreve@sdfe.dk> wrote:

Markus,

Sorry for the late reply – you mail got caught by the corporate spam filter.

The challenge is that users of other software using the new PROJ API like GDAL and GRASS expect that reprojecting a dataset from one CRS to another CRS “just works”. Input and output CRS can be anything custom defined. Either PROJ or other software using PROJ must have some mechanism to decide if the requested reprojection can be done or not.

And with my proposal to change proj_create_crs_to_crs() to allow input other than strings on the form “+init=file:key” you will get exactly what you have always had with the old API. You can definitely argue that this has never “just worked”, since using the WGS84 hub datum is rarely doing the correct thing, but for most users it is good enough.

For most datasets, WGS84 as pivot datum should be good enough, considering the limited spatial accuracy of many real-world datasets.

This will not be the case forever though. It is my hope that eventually we will have a system the really “just works”, even for users requiring high accuracy, but for now we only have the basic scaffolding that eventually will make this possible. Patience will be required :slight_smile:

FWIW, I added support for user-defined pipelines in GRASS in r.proj (r72598) and v.proj (r72599) with a new pipeline option.

Markus M

Looking forward to this function! But what if a dataset comes with a CRS definition without EPSG code? Can we expect that PROJ handles this or should applications using the PROJ API handle this?

Without having thought this through all the way to the end yet, I think this situation will be possible to handle. The idea being that if you specify an EPSG CRS on one side and a string like “+proj=merc” or whatever, we could default to a transformation with hub datum like we do today. Either by going from EPSG:xxxx to WGS84 (or something else that makes sense in the situation) and then transform from the hub datum to the custom CRS. It will be a best guess as to how to that transformation but better than nothing, and probably correct in most cases.

I might have overlooked something, so no promises on the final implementation yet!

/Kristian

Fra: Markus Metz [mailto:markus.metz.giswork@gmail.com]
Sendt: 27. marts 2018 21:50
Til: Kristian Evers <kreve@sdfe.dk>
Cc: GRASS developers list <grass-dev@lists.osgeo.org>; gdal-dev <gdal-dev@lists.osgeo.org>
Emne: Re: [gdal-dev] [GRASS-dev] PROJ 5 support in trunk

On Mon, Mar 26, 2018 at 10:51 PM, Kristian Evers <kreve@sdfe.dk> wrote:

On 26 Mar 2018, at 21:21, Markus Metz <markus.metz.giswork@gmail.com> wrote:

There is an important difference between the PROJ4 and the PROJ5+ API/syntax:

The old PROJ4 API uses latlong WGS84 as pivot datum for coordinate transformations like

projection1 → latlong WGS84 → projection2

or in ‘+to’ syntax
projection1 +to projection2

The new PROJ5+ API no longer uses a pivot datum. The advantage is that you can directly convert from one datum to another, without going through WGS84. The disadvantage is that the user/software using the new API has to make sure that either a common pivot datum is used or coordinates are correctly transformed from one datum to another, e.g. with a Helmert Transform.

Both GDAL and GRASS have implemented the new API as a simple 2-step pipeline like

  1. projection1 → some latlong
  2. some latlong → projection2

or in pipeline syntax

+proj=pipeline

+step +inv projection1

+step projection2

This is actually exactly what the proj_crs_to_crs() function does. When I originally wrote it I had an idea that this should only be used with init-files,

but seeing how people need a simple way to do what they’ve always done I think it might be better to relax that requirement and allow input similar

to what goes into cs2cs and pj_transform().

The challenge is that users of other software using the new PROJ API like GDAL and GRASS expect that reprojecting a dataset from one CRS to another CRS “just works”. Input and output CRS can be anything custom defined. Either PROJ or other software using PROJ must have some mechanism to decide if the requested reprojection can be done or not.

My plan for this function is that it eventually will be able to construct a transformation between CRS’s without the WGS84 pivot. This will be guided

by the EPSG database. I have still to work out exactly how. For now though, it works fine with already existing init files and WGS84 as a pivot datum.

My hope is that this will land in version 6 but I won’t promise that just yet.

Looking forward to this function! But what if a dataset comes with a CRS definition without EPSG code? Can we expect that PROJ handles this or should applications using the PROJ API handle this?

Markus M

Even Rouault has done some tests and found sometimes subtle differences in the results between the old and new API/syntax. Further on, there is no mechanism (yet) in place to validate the pipeline, most importantly that the output of step 1 conforms to the required input for step 2.

I believe we have sorted those problems out in the coming version 5.0.1.

Regarding validation of input/output from pipeline steps. We can probably do some basic checks, but in the end you will always be required to know

what you are doing. Mind you, we are not expecting everyday users to construct their own pipelines all the time. It is a powertool for the accomplished

user that knows what he is doing. Most users should have their needs met by predefined transformations in init-files such as the epsg file. And eventually

in a more clever way using the EPSG database directly as mentioned above.

IMHO, the implementation of the new PROJ5+ API/syntax in GDAL and GRASS should be regarded as experimental and testing by as many people as possible would be a huge benefit to PROJ/GDAL/GRASS and to all applications using these projects.

Markus M

On Sun, Mar 25, 2018 at 10:47 PM, Helmut Kudrnovsky <hellik@web.de> wrote:

Now (last related commit is trunk r72522) it’s finished. I have >introduced
a new GRASS API that handles both PROJ 4 and PROJ >5, consisting of

Thanks for this Update!

Fyi See
https://lists.osgeo.org/pipermail/osgeo4w-dev/2018-March/003557.html


[…]
The API from PROJ 4 lives on in PROJ 5, so GRASS 7.4 should be able to use
PROJ 5 as well. We’ve been carefull not to breaking anything with this
release.
That comes with PROJ 6 and 7. Of course there might be implementation
details
in GRASS that I am unaware of that makes using PROJ 5 impossible.


best regards
Helmut

Sent from: http://osgeo-org.1560.x6.nabble.com/Grass-Dev-f3991897.html


grass-dev mailing list
grass-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/grass-dev


gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev