[GRASS-dev] [GRASS GIS] #2732: Read lidar data as vector points using PDAL

GDAL · September 4, 2015, 3:53pm

#2732: Read lidar data as vector points using PDAL
-------------------------------------------------+-------------------------
Reporter: wenzeslaus | Owner: grass-dev@…
Type: enhancement | Status: new
Priority: major | Milestone: 7.1.0
Component: Vector | Version: svn-trunk
Keywords: lidar, las, laz, v.in.lidar, | CPU: Unspecified
v.in.pointcloud, v.in.pdal |
Platform: Unspecified |
-------------------------------------------------+-------------------------
[http://www.pdal.io/ PDAL] is a point cloud processing and format
transformation library and supersedes [http://www.liblas.org/ libLAS]
(currently used in `r.in.lidar` and `v.in.lidar`) in many ways. Currently
the most important difference is support for LAS 1.4. It also seems that
we will have to replace libLAS by PDAL at some point.

I'm attaching a patch which is adding a module `v.in.pointcloud`. It is
based on `v.in.lidar` and the difference is that it is using PDAL C++ API
for reading LAS (so it can read LAS 1.4 except for waveform). LAZ is
supported in the same way as with libLAS. When PDAL is compiled with LAZ,
then `v.in.pointcloud` supports LAZ.

Serious issues with the patch:
* the compilation check in configure has just dummy code, I don't know how
to make it compile C++

Potential improvements to patch:
* support for all PDAL supported (internally and by plugins) formats
(should be easy to add), hence the name `v.in.pointcloud`
* documentation is just copied from `v.in.lidar`
* add test (test data are needed for that as well as for documentation)

Adding PDAL processing into the import or as separate module is clearly a
possibility as well but this is out of scope of this particular patch.

General issues with PDAL for GRASS GIS:
* implemented in C++ and lacks C API (this is fixable)
* depends on C++11 (not a big issue when building modules)
* depends on Boost (hopefully will be removed in the future)
* potential limits on number of points as by default everything is
[https://lists.osgeo.org/pipermail/pdal/2015-September/000672.html in
memory and limited to 2^32 - 1 points] but there are
[https://lists.osgeo.org/pipermail/pdal/2015-September/000682.html ways
around it]

I need help with making the compilation check in configure compile C++,
otherwise I think it can go trunk.

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/2732>
GRASS GIS <https://grass.osgeo.org>

GDAL · September 4, 2015, 3:55pm

#2732: Read lidar data as vector points using PDAL
-------------------------+-------------------------------------------------
  Reporter: wenzeslaus | Owner: grass-dev@…
      Type: | Status: new
  enhancement |
  Priority: major | Milestone: 7.1.0
Component: Vector | Version: svn-trunk
Resolution: | Keywords: lidar, las, laz, v.in.lidar,
       CPU: | v.in.pointcloud, v.in.pdal
  Unspecified | Platform: Unspecified
-------------------------+-------------------------------------------------
Changes (by wenzeslaus):

* Attachment "vinpointcloud.diff" added.

derived from v.in.lidar and using standard PDAL API with LasReader only

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/2732>
GRASS GIS <https://grass.osgeo.org>

GDAL · September 4, 2015, 3:56pm

#2732: Read lidar data as vector points using PDAL
-------------------------+-------------------------------------------------
  Reporter: wenzeslaus | Owner: wenzeslaus
      Type: | Status: new
  enhancement |
  Priority: major | Milestone: 7.1.0
Component: Vector | Version: svn-trunk
Resolution: | Keywords: lidar, las, laz, v.in.lidar,
       CPU: | v.in.pointcloud, v.in.pdal
  Unspecified | Platform: Unspecified
-------------------------+-------------------------------------------------
Changes (by wenzeslaus):

* owner: grass-dev@… => wenzeslaus

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/2732#comment:1>
GRASS GIS <https://grass.osgeo.org>

GDAL · November 24, 2015, 8:02am

#2732: Read lidar data as vector points using PDAL
-------------------------+-------------------------------------------------
  Reporter: wenzeslaus | Owner: wenzeslaus
      Type: | Status: new
  enhancement |
  Priority: major | Milestone: 7.1.0
Component: Vector | Version: svn-trunk
Resolution: | Keywords: lidar, las, laz, v.in.lidar,
       CPU: | v.in.pointcloud, v.in.pdal
  Unspecified | Platform: Unspecified
-------------------------+-------------------------------------------------
Changes (by neteler):

* cc: grass-dev@… (added)

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/2732#comment:4>
GRASS GIS <https://grass.osgeo.org>

GDAL · December 30, 2015, 7:12pm

#2732: Read lidar data as vector points using PDAL
-------------------------+-------------------------------------------------
  Reporter: wenzeslaus | Owner: wenzeslaus
      Type: | Status: new
  enhancement |
  Priority: major | Milestone: 7.1.0
Component: Vector | Version: svn-trunk
Resolution: | Keywords: lidar, las, laz, v.in.lidar,
       CPU: | v.in.pointcloud, v.in.pdal
  Unspecified | Platform: Unspecified
-------------------------+-------------------------------------------------

Comment (by wenzeslaus):

First version now available in trunk. PDAL compilation support added in
r67293 and ''v.in.pdal'' in r67436. Configure GRASS with:

{{{
./configure ... --with-pdal="/path/to/pdal-config"
}}}

PDAL can be compiled with or without LAZ (LAZlib) support and PCL support,
so not all the functions may be available. If you compiled PDAL yourself
you need to do also:

{{{
export LD_LIBRARY_PATH="/path/to/pdal/install/lib:$LD_LIBRARY_PATH"
}}}

I decided that the name ''v.in.pdal'' will be best because there are some
PDAL-specific functions included. This includes also reprojection during
import. However, the name can be changed, `v.in.points` sounds good as
well.

The next steps are:

* add missing functions (implemented in GRASS) from libLAS-based
''v.in.lidar'' namely vector mask and count-based decimation
* add more [http://www.pdal.io/stages/#filters PDAL-based filters] (please
suggest!)
* implement ''r.in.pdal'' similar to ''r.in.lidar'' (also needs list of
filters to be implemented)
* implement ''!r3.in.pdal''
* implement ''v.out.pdal'' similar to ''v.out.lidar''
* move some of the common functions to `lib/lidar` (most of the common
functionality was already re-factored out from ''r.in.lidar'' and
''v.in.lidar'' and used in ''v.in.pdal'' and ''v.decimate''

Things like mask or various decimations have (or can have) GRASS
implementation and have PDAL implementation as well. It would make sense
to do these operations at the beginning and at the end of the import
process. If it would be at the beginning, only PDAL implementations can be
used.

Note also that there are still some unresolved issues related to PDAL
ability to handle large amounts of points although some solutions were
proposed on the PDAL mailing list.

Finally, note also that PDAL is a processing library, so it can be also
used in a separate module which would process vector points.

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/2732#comment:5>
GRASS GIS <https://grass.osgeo.org>

GDAL · December 30, 2015, 10:41pm

#2732: Read lidar data as vector points using PDAL
-------------------------+-------------------------------------------------
  Reporter: wenzeslaus | Owner: wenzeslaus
      Type: | Status: new
  enhancement |
  Priority: major | Milestone: 7.1.0
Component: Vector | Version: svn-trunk
Resolution: | Keywords: lidar, las, laz, v.in.lidar,
       CPU: | v.in.pointcloud, v.in.pdal
  Unspecified | Platform: Unspecified
-------------------------+-------------------------------------------------

Comment (by neteler):

Replying to [comment:5 wenzeslaus]:
> * add more [http://www.pdal.io/stages/#filters PDAL-based filters]
(please suggest!)

An idea could be to put an optional RGB point colorization here. It is a
simple PDAL filter, hence v.in.pdal could read from an additionally
specified RGB map retrieve the individual colors during import.

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/2732#comment:6>
GRASS GIS <https://grass.osgeo.org>

GDAL · December 31, 2015, 12:10am

#2732: Read lidar data as vector points using PDAL
-------------------------+-------------------------------------------------
  Reporter: wenzeslaus | Owner: wenzeslaus
      Type: | Status: new
  enhancement |
  Priority: major | Milestone: 7.1.0
Component: Vector | Version: svn-trunk
Resolution: | Keywords: lidar, las, laz, v.in.lidar,
       CPU: | v.in.pointcloud, v.in.pdal
  Unspecified | Platform: Unspecified
-------------------------+-------------------------------------------------

Comment (by wenzeslaus):

Replying to [comment:6 neteler]:
> Replying to [comment:5 wenzeslaus]:
> > * add more [http://www.pdal.io/stages/#filters PDAL-based filters]
(please suggest!)
>
> An idea could be to put an optional RGB point colorization here. It is a
simple PDAL filter, hence v.in.pdal could read from an additionally
specified RGB map retrieve the individual colors during import.

Yes, this is actually on my list already. PDAL colorization could do a
GeoTIFF while GRASS raster map would have to be done using native
colorization implemented in GRASS. I think that this is exactly where
[https://lists.osgeo.org/pipermail/grass-dev/2015-September/076552.html
using layers and categories for something else than ID and class] will be
necessary.

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/2732#comment:7>
GRASS GIS <https://grass.osgeo.org>

GDAL · March 20, 2016, 9:34pm

#2732: Read lidar data as vector points using PDAL
-------------------------+-------------------------------------------------
  Reporter: wenzeslaus | Owner: wenzeslaus
      Type: | Status: new
  enhancement |
  Priority: major | Milestone: 7.1.0
Component: Vector | Version: svn-trunk
Resolution: | Keywords: lidar, las, laz, v.in.lidar,
       CPU: | v.in.pointcloud, v.in.pdal
  Unspecified | Platform: Unspecified
-------------------------+-------------------------------------------------

Comment (by sebastic):

FYI: PDAL has been [https://tracker.debian.org/pkg/pdal packaged for
Debian] and derivatives like Ubuntu.

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/2732#comment:8>
GRASS GIS <https://grass.osgeo.org>

GDAL · March 21, 2016, 1:29am

#2732: Read lidar data as vector points using PDAL
-------------------------+-------------------------------------------------
  Reporter: wenzeslaus | Owner: wenzeslaus
      Type: | Status: new
  enhancement |
  Priority: major | Milestone: 7.1.0
Component: Vector | Version: svn-trunk
Resolution: | Keywords: lidar, las, laz, v.in.lidar,
       CPU: | v.in.pointcloud, v.in.pdal
  Unspecified | Platform: Unspecified
-------------------------+-------------------------------------------------

Comment (by wenzeslaus):

Replying to [comment:8 sebastic]:
> FYI: PDAL has been [https://tracker.debian.org/pkg/pdal packaged for
Debian] and derivatives like Ubuntu.

Thanks for the info. That's great. How do I install it? I can't find .deb,
PPA or instructions at the [http://www.pdal.io/download.html official
website].

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/2732#comment:9>
GRASS GIS <https://grass.osgeo.org>

GDAL · March 21, 2016, 5:23am

#2732: Read lidar data as vector points using PDAL
-------------------------+-------------------------------------------------
  Reporter: wenzeslaus | Owner: wenzeslaus
      Type: | Status: new
  enhancement |
  Priority: major | Milestone: 7.1.0
Component: Vector | Version: svn-trunk
Resolution: | Keywords: lidar, las, laz, v.in.lidar,
       CPU: | v.in.pointcloud, v.in.pdal
  Unspecified | Platform: Unspecified
-------------------------+-------------------------------------------------

Comment (by sebastic):

Replying to [comment:9 wenzeslaus]:
> How do I install it? I can't find .deb, PPA or instructions at the
[http://www.pdal.io/download.html official website].

Wait for the package to reach the mirrors, and then install it on a Debian
unstable system with: `apt-get install pdal libpdal-dev`.

The former gets you the `pdal` commandline application, the latter the
headers, libraries & `pdal-config`.

There is no PPA for Ubuntu, the package will be synced from Debian
unstable into the Ubuntu release after xenial.

The PDAL website doesn't mention the Debian package because it is new and
part of the distribution and not a 3rd party repository.

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/2732#comment:10>
GRASS GIS <https://grass.osgeo.org>

GDAL · July 10, 2016, 2:00pm

#2732: Read lidar data as vector points using PDAL
-------------------------+-------------------------------------------------
  Reporter: wenzeslaus | Owner: wenzeslaus
      Type: | Status: new
  enhancement |
  Priority: major | Milestone: 7.3.0
Component: Vector | Version: svn-trunk
Resolution: | Keywords: lidar, las, laz, v.in.lidar,
       CPU: | v.in.pointcloud, v.in.pdal
  Unspecified | Platform: Unspecified
-------------------------+-------------------------------------------------
Changes (by wenzeslaus):

* milestone: 7.2.0 => 7.3.0

Comment:

This is for the next release, although some code is already in place and
backports are possible in this case.

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/2732#comment:12>
GRASS GIS <https://grass.osgeo.org>

GDAL · May 19, 2017, 10:38pm

#2732: Read lidar data as vector points using PDAL
-------------------------+-------------------------------------------------
  Reporter: wenzeslaus | Owner: wenzeslaus
      Type: | Status: new
  enhancement |
  Priority: major | Milestone: 7.4.0
Component: Vector | Version: svn-trunk
Resolution: | Keywords: lidar, las, laz, v.in.lidar,
       CPU: | v.in.pointcloud, v.in.pdal
  Unspecified | Platform: Unspecified
-------------------------+-------------------------------------------------
Changes (by ptschrum):

* cc: ptschrum (added)

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/2732#comment:14>
GRASS GIS <https://grass.osgeo.org>

GDAL · May 22, 2017, 3:34am

#2732: Read lidar data as vector points using PDAL
-------------------------+-------------------------------------------------
  Reporter: wenzeslaus | Owner: wenzeslaus
      Type: | Status: new
  enhancement |
  Priority: major | Milestone: 7.4.0
Component: Vector | Version: svn-trunk
Resolution: | Keywords: lidar, las, laz, v.in.lidar,
       CPU: | v.in.pointcloud, v.in.pdal
  Unspecified | Platform: Unspecified
-------------------------+-------------------------------------------------

Comment (by hobu):

There have been a number of PDAL developments since this ticket was first
developed that might be interesting in relation to this task:

1) The PDAL
[https://github.com/PDAL/PDAL/blob/master/pdal/PipelineExecutor.hpp
PipelinExecutor] class allows you to insert PDAL Pipeline JSON and get
back PDAL PointViews quite easily. This means no more manual or laborious
pdal::Stage construction. The executor class can hide all of that for you.
Dig in the Python or Java extensions for inspiration on how to use it.

2) PDAL is now available via OSGeo4W64. We won't be committing to
maintaining the 32-bit builds, but the 64-bit builds will be generated as
part of PDAL's continuous integration and included in all major PDAL
releases.

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/2732#comment:15>
GRASS GIS <https://grass.osgeo.org>

GDAL · June 5, 2017, 3:31pm

#2732: Read lidar data as vector points using PDAL
-------------------------+-------------------------------------------------
  Reporter: wenzeslaus | Owner: ptschrum
      Type: | Status: assigned
  enhancement |
  Priority: major | Milestone: 7.4.0
Component: Vector | Version: svn-trunk
Resolution: | Keywords: lidar, las, laz, v.in.lidar,
       CPU: | v.in.pointcloud, v.in.pdal
  Unspecified | Platform: Unspecified
-------------------------+-------------------------------------------------
Changes (by ptschrum):

* owner: wenzeslaus => ptschrum
* status: new => assigned

Comment:

I (ptschrum) am working on this as a GSoC project, 2017.

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/2732#comment:16>
GRASS GIS <https://grass.osgeo.org>

GDAL · December 14, 2017, 7:29pm

#2732: Read lidar data as vector points using PDAL
-------------------------+-------------------------------------------------
  Reporter: wenzeslaus | Owner: ptschrum
      Type: | Status: assigned
  enhancement |
  Priority: major | Milestone: 7.4.0
Component: Vector | Version: svn-trunk
Resolution: | Keywords: lidar, las, laz, v.in.lidar,
       CPU: | v.in.pointcloud, v.in.pdal
  Unspecified | Platform: Unspecified
-------------------------+-------------------------------------------------

Comment (by hellik):

citing a [https://trac.osgeo.org/osgeo4w/ticket/550#comment:4 OSGeo4W
ticket]:

{{{
>no chance to fix the broken liblas/laszip stack?

There is a significant amount of software development to catch it up to
the latest LASzip release. The LASzip API that libLAS used is no longer
supported, and the new LASzip API is very different from the one that
libLAS (and previously PDAL) used.

It would be better long term to get GRASS using PDAL. There was some
effort on that topic, but I think it has stalled.
}}}

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/2732#comment:17>
GRASS GIS <https://grass.osgeo.org>

GDAL · December 14, 2017, 7:30pm

#2732: Read lidar data as vector points using PDAL
-------------------------+-------------------------------------------------
  Reporter: wenzeslaus | Owner: ptschrum
      Type: | Status: assigned
  enhancement |
  Priority: major | Milestone: 7.4.0
Component: Vector | Version: svn-trunk
Resolution: | Keywords: lidar, las, laz, v.in.lidar,
       CPU: | v.in.pointcloud, v.in.pdal
  Unspecified | Platform: Unspecified
-------------------------+-------------------------------------------------

Comment (by hellik):

wiki of the
[https://trac.osgeo.org/grass/wiki/GSoC/2017/IntegrationOfPDALintoGRASSGIS
GSoC 2017 project] to integrate PDAL into GRASS.

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/2732#comment:18>
GRASS GIS <https://grass.osgeo.org>

GDAL · February 16, 2018, 3:11am

#2732: Read lidar data as vector points using PDAL
-------------------------+-------------------------------------------------
  Reporter: wenzeslaus | Owner: ptschrum
      Type: | Status: assigned
  enhancement |
  Priority: major | Milestone: 7.4.1
Component: Vector | Version: svn-trunk
Resolution: | Keywords: lidar, las, laz, v.in.lidar,
       CPU: | v.in.pointcloud, v.in.pdal
  Unspecified | Platform: Unspecified
-------------------------+-------------------------------------------------

Comment (by wenzeslaus):

In [changeset:"72246" 72246]:
{{{
#!CommitTicketReference repository="" revision="72246"
v.in.pdal: change API calls to PDAL 1.6.0, see #3496 and #2732 (author:
felixg)
}}}

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/2732#comment:20>
GRASS GIS <https://grass.osgeo.org>

GDAL · February 16, 2018, 4:06am

#2732: Read lidar data as vector points using PDAL
-------------------------+-------------------------------------------------
  Reporter: wenzeslaus | Owner: (none)
      Type: | Status: new
  enhancement |
  Priority: major | Milestone: 7.4.1
Component: Vector | Version: svn-trunk
Resolution: | Keywords: lidar, las, laz, v.in.lidar,
       CPU: | v.in.pointcloud, v.in.pdal
  Unspecified | Platform: Unspecified
-------------------------+-------------------------------------------------
Changes (by wenzeslaus):

* owner: ptschrum => (none)
* status: assigned => new

Comment:

There is currently no owner of this issue.

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/2732#comment:21>
GRASS GIS <https://grass.osgeo.org>

GDAL · February 18, 2018, 11:09am

#2732: Read lidar data as vector points using PDAL
-------------------------+-------------------------------------------------
  Reporter: wenzeslaus | Owner: (none)
      Type: | Status: new
  enhancement |
  Priority: major | Milestone: 7.4.1
Component: Vector | Version: svn-trunk
Resolution: | Keywords: lidar, las, laz, v.in.lidar,
       CPU: | v.in.pointcloud, v.in.pdal
  Unspecified | Platform: Unspecified
-------------------------+-------------------------------------------------

Comment (by neteler):

In [changeset:"72253" 72253]:
{{{
#!CommitTicketReference repository="" revision="72253"
v.in.pdal: change API calls to PDAL 1.6.0, see #3496 and #2732 (author:
felixg, trunk r72246) + GPL header added in r72245
}}}

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/2732#comment:22>
GRASS GIS <https://grass.osgeo.org>

GDAL · March 13, 2018, 1:55pm

#2732: Read lidar data as vector points using PDAL
-------------------------+-------------------------------------------------
  Reporter: wenzeslaus | Owner: (none)
      Type: | Status: new
  enhancement |
  Priority: major | Milestone: 7.4.1
Component: Vector | Version: svn-trunk
Resolution: | Keywords: lidar, las, laz, v.in.lidar,
       CPU: | v.in.pointcloud, v.in.pdal
  Unspecified | Platform: Unspecified
-------------------------+-------------------------------------------------

Comment (by wenzeslaus):

In [changeset:"72363" 72363]:
{{{
#!CommitTicketReference repository="" revision="72363"
v.in.pdal: change API calls to PDAL 1.6.0, see #3496, #3243, #3101, #2732
(author: felixg, backport of trunk r72246) + GPL header added in r72245
}}}

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/2732#comment:23>
GRASS GIS <https://grass.osgeo.org>