[GRASS-dev] Debugging QGIS with GRASS provider on Windows

GRASS GIS GRASS GIS developer
,
1

Hi,
I need to debug QGIS GRASS provider on Windows. I have successfully
compiled GRASS 7 using MinGW [1] and QGIS using MSVC [2]. The problem
is that MSVC does not support GCC debugging symbols and vice versa. Do
you have any idea how to debug both QGIS and GRASS code at the same
time, preferably in MSVC or Qt Creator?

Is there any recent effort to compile GRASS with MSVC? I have only
found quite an old mail about MSVC port [3].

Radim

[1] https://trac.osgeo.org/grass/wiki/CompileOnWindows
[2] https://htmlpreview.github.io/?https://github.com/qgis/QGIS/blob/master/doc/INSTALL.html#toc12
[3] http://lists.osgeo.org/pipermail/grass-dev/2007-November/034108.html

2

On Tue, Jun 16, 2015 at 2:01 PM, Radim Blazek <radim.blazek@gmail.com>
wrote:

Hi,
I need to debug QGIS GRASS provider on Windows. I have successfully
compiled GRASS 7 using MinGW [1] and QGIS using MSVC [2]. The problem
is that MSVC does not support GCC debugging symbols and vice versa. Do
you have any idea how to debug both QGIS and GRASS code at the same
time, preferably in MSVC or Qt Creator?

Is there any recent effort to compile GRASS with MSVC? I have only
found quite an old mail about MSVC port [3].

There is an exploratory effort to use CMake for GRASS GIS [1] but there is
a lack of CMake experts with free time. In theory this is one of the steps
to compile GRASS with MSVC. Isn't it possible to easily compile QGIS with
MinGW thanks to CMake (since GCC is used on other platforms)?

Vaclav

[1] https://github.com/starseeker/grass

Radim

[1] https://trac.osgeo.org/grass/wiki/CompileOnWindows
[2]
https://htmlpreview.github.io/?https://github.com/qgis/QGIS/blob/master/doc/INSTALL.html#toc12
[3] http://lists.osgeo.org/pipermail/grass-dev/2007-November/034108.html
_______________________________________________
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

3

On Tue, Jun 16, 2015 at 10:48 PM, Vaclav Petras <wenzeslaus@gmail.com> wrote:

On Tue, Jun 16, 2015 at 2:01 PM, Radim Blazek <radim.blazek@gmail.com>
wrote:

Hi,
I need to debug QGIS GRASS provider on Windows. I have successfully
compiled GRASS 7 using MinGW [1] and QGIS using MSVC [2]. The problem
is that MSVC does not support GCC debugging symbols and vice versa. Do
you have any idea how to debug both QGIS and GRASS code at the same
time, preferably in MSVC or Qt Creator?

Is there any recent effort to compile GRASS with MSVC? I have only
found quite an old mail about MSVC port [3].

There is an exploratory effort to use CMake for GRASS GIS [1] but there is a
lack of CMake experts with free time.

CMake would be nice.

In theory this is one of the steps to
compile GRASS with MSVC. Isn't it possible to easily compile QGIS with MinGW
thanks to CMake (since GCC is used on other platforms)?

In theory yes, but it means moving away from final build (QGIS built
with MSVC in OSGeo4W). There are also other important libs (Qt for
example) built with MSVC.

I have somehow managed to compile limited GRASS core libs with MSVC
and GRASS Makefiles replacing gcc by cl.exe. The build system does not
seem to be the biggest problem, there are still pieces of code not
compiling under MSVC however.

Radim

Vaclav

[1] https://github.com/starseeker/grass

Radim

[1] https://trac.osgeo.org/grass/wiki/CompileOnWindows
[2]
https://htmlpreview.github.io/?https://github.com/qgis/QGIS/blob/master/doc/INSTALL.html#toc12
[3] http://lists.osgeo.org/pipermail/grass-dev/2007-November/034108.html
_______________________________________________
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

_______________________________________________
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

4

2015-06-19 10:35 GMT+03:00 Radim Blazek <radim.blazek@gmail.com>:

I have somehow managed to compile limited GRASS core libs with MSVC
and GRASS Makefiles replacing gcc by cl.exe. The build system does not
seem to be the biggest problem, there are still pieces of code not
compiling under MSVC however.

If you are ready to test changes, could you report issues with code in
trac? Then it would be possible to check if existing code can be
accommodated to compile with both GCC and MSVC.

Maris.

5

On Fri, Jun 19, 2015 at 9:44 AM, Maris Nartiss <maris.gis@gmail.com> wrote:

2015-06-19 10:35 GMT+03:00 Radim Blazek <radim.blazek@gmail.com>:

I have somehow managed to compile limited GRASS core libs with MSVC
and GRASS Makefiles replacing gcc by cl.exe. The build system does not
seem to be the biggest problem, there are still pieces of code not
compiling under MSVC however.

If you are ready to test changes, could you report issues with code in
trac? Then it would be possible to check if existing code can be
accommodated to compile with both GCC and MSVC.

I would like to, it was originally my aim to also submit patches, but
being stressed by time (QGIS release July 26), I am just putting hack
over hack to get it running and debuggable to be able to fix bugs for
the release.

Radim

6

Radim Blazek-2 wrote

On Fri, Jun 19, 2015 at 9:44 AM, Maris Nartiss &lt;

maris.gis@

&gt; wrote:

2015-06-19 10:35 GMT+03:00 Radim Blazek &lt;

radim.blazek@

&gt;:

I have somehow managed to compile limited GRASS core libs with MSVC
and GRASS Makefiles replacing gcc by cl.exe. The build system does not
seem to be the biggest problem, there are still pieces of code not
compiling under MSVC however.

If you are ready to test changes, could you report issues with code in
trac? Then it would be possible to check if existing code can be
accommodated to compile with both GCC and MSVC.

I would like to, it was originally my aim to also submit patches, but
being stressed by time (QGIS release July 26), I am just putting hack
over hack to get it running and debuggable to be able to fix bugs for
the release.

Radim

just for the record the parallel discussion on the OSGeo4W-ML

http://osgeo-org.1560.x6.nabble.com/Crash-in-thread-on-FILE-opened-by-a-lib-compiled-by-MinGW-td5211876.html

with summary:

"Finally I found it, it is similar story like with FILE.

The provider (MSVC) calls Vect__open_old with struct Map_info variable
allocated in the provider where sizeof(struct Map_info) = 1408.
Vect__open_old (MinGW) calls G_zero on that variable, where
sizeof(struct Map_info) = 1520.

It means that all structures used in GRASS libs must be also allocated
in GRASS. New functions like Vect_alloc_map have to be added to GRASS
and until it gets to GRASS and to OSGeo4w, the the structures must be
be allocated in the provider with enough space.

Strange thing that it came up only now with GRASS 7 and threads.

Thanks for your ideas Even and Jef
Radim"

-----
best regards
Helmut
--
View this message in context: http://osgeo-org.1560.x6.nabble.com/Debugging-QGIS-with-GRASS-provider-on-Windows-tp5211308p5212039.html
Sent from the Grass - Dev mailing list archive at Nabble.com.

7

Helmut Kudrnovsky wrote:

"Finally I found it, it is similar story like with FILE.

The provider (MSVC) calls Vect__open_old with struct Map_info variable
allocated in the provider where sizeof(struct Map_info) = 1408.
Vect__open_old (MinGW) calls G_zero on that variable, where
sizeof(struct Map_info) = 1520.

It means that all structures used in GRASS libs must be also allocated
in GRASS. New functions like Vect_alloc_map have to be added to GRASS
and until it gets to GRASS and to OSGeo4w, the the structures must be
be allocated in the provider with enough space.

Strange thing that it came up only now with GRASS 7 and threads.

You need to get both compilers to use the same ABI.

The "alternative" is to treat all structures as opaque, which isn't
going to happen. It isn't sufficient to have GRASS allocate and free
everything, you also can't access any fields directly, you'd need
setters and getters for everything.

Given the numbers, my first guess is that MinGW is using 8 bytes for
off_t versus 4 bytes for MSVC. Yep; the bottom of config.h.in has:

  #if defined(__MINGW32__)
  
  /* add/remove as needed */
  /* redefine off_t */
  #include <sys/types.h>
  #define off_t off64_t

And that's unaffected by --disable-largefile.

This specific issue may just need changing __MINGW32__ to WIN32 (or
_WIN32 or _WINNT, all of which are defined by MinGW) so that MSVC gets
the same treatment. Failing that, some way to disable LFS (e.g. making
MinGW builds honour --disable-largefile) is needed.

The FILE issues are presumably due to mismatches between different
versions of the MSVCRT library. malloc() and free() have the same
issue: heap data must be freed by the same version of MSVCRT which
allocated it (NVIZ originally had issues with passing Tcl_Alloc()d
pointers directly to free() instead of Tcl_Free(), which fails if Tcl
uses a different version of MSVCRT to GRASS).

I don't know whether it's possible to change the version of MSVCRT
which MinGW uses. If not, then we'll need to look at why files opened
by lib/gis functions are being read/written by user code. But it would
be preferable if FILE* could be passed around.

--
Glynn Clements <glynn@gclements.plus.com>

8

On Sat, Jun 20, 2015 at 10:32 PM, Glynn Clements <glynn@gclements.plus.com>
wrote:

This specific issue may just need changing __MINGW32__ to WIN32 (or
_WIN32 or _WINNT, all of which are defined by MinGW) so that MSVC gets
the same treatment.

This might be a good idea in general. I think I saw some __MINGW32__ #ifs
which (according to the code) meant to check for MS Windows system not
MinGW compiler.

9

On Sun, Jun 21, 2015 at 4:32 AM, Glynn Clements
<glynn@gclements.plus.com> wrote:

Helmut Kudrnovsky wrote:

"Finally I found it, it is similar story like with FILE.

The provider (MSVC) calls Vect__open_old with struct Map_info variable
allocated in the provider where sizeof(struct Map_info) = 1408.
Vect__open_old (MinGW) calls G_zero on that variable, where
sizeof(struct Map_info) = 1520.

It means that all structures used in GRASS libs must be also allocated
in GRASS. New functions like Vect_alloc_map have to be added to GRASS
and until it gets to GRASS and to OSGeo4w, the the structures must be
be allocated in the provider with enough space.

Strange thing that it came up only now with GRASS 7 and threads.

You need to get both compilers to use the same ABI.

The "alternative" is to treat all structures as opaque, which isn't
going to happen. It isn't sufficient to have GRASS allocate and free
everything, you also can't access any fields directly, you'd need
setters and getters for everything.

The problem was with Map_info structure which is accessed only through
GRASS lib. The structures accessed directly (line_pnts,line_cats) have
only int or double or pointers to them. In theory the size of int may
be different for different compilers, but in this case it is the same
and it is quite rare to have GRASS and QGIS compiled by two compilers
with different size of int.

I could not find a function for Map_info allocation, I am missing
Vect_new_map_struct and Vect_destroy_map_struct.

Given the numbers, my first guess is that MinGW is using 8 bytes for
off_t versus 4 bytes for MSVC. Yep; the bottom of config.h.in has:

        #if defined(__MINGW32__)

        /* add/remove as needed */
        /* redefine off_t */
        #include <sys/types.h>
        #define off_t off64_t

And that's unaffected by --disable-largefile.

This specific issue may just need changing __MINGW32__ to WIN32 (or
_WIN32 or _WINNT, all of which are defined by MinGW) so that MSVC gets
the same treatment. Failing that, some way to disable LFS (e.g. making
MinGW builds honour --disable-largefile) is needed.

The FILE issues are presumably due to mismatches between different
versions of the MSVCRT library.

malloc() and free() have the same
issue: heap data must be freed by the same version of MSVCRT which
allocated it (NVIZ originally had issues with passing Tcl_Alloc()d
pointers directly to free() instead of Tcl_Free(), which fails if Tcl
uses a different version of MSVCRT to GRASS).

I don't know whether it's possible to change the version of MSVCRT
which MinGW uses. If not, then we'll need to look at why files opened
by lib/gis functions are being read/written by user code. But it would
be preferable if FILE* could be passed around.

The problem with FILE (in my case - QGIS) was false alarm, it happened
when I mixed GRASS gis lib compiled by MinGW and vector lib compiled
by MSVC to be able to debug vect lib.

Radim

10

On Sun, Jun 21, 2015 at 4:42 AM, Vaclav Petras <wenzeslaus@gmail.com> wrote:

On Sat, Jun 20, 2015 at 10:32 PM, Glynn Clements <glynn@gclements.plus.com>
wrote:

This specific issue may just need changing __MINGW32__ to WIN32 (or
_WIN32 or _WINNT, all of which are defined by MinGW) so that MSVC gets
the same treatment.

This might be a good idea in general. I think I saw some __MINGW32__ #ifs
which (according to the code) meant to check for MS Windows system not MinGW
compiler.

Do you think that __MINGW32__ may be changed to WIN32 globally without
inspecting each case? There are probably really __MINGW32__ specific
places. Inspecting each __MINGW32__ individually seems to be the
largest amount of work doing port to MSVC.

Radim

11

Radim Blazek wrote:

>> This specific issue may just need changing __MINGW32__ to WIN32 (or
>> _WIN32 or _WINNT, all of which are defined by MinGW) so that MSVC gets
>> the same treatment.
>
> This might be a good idea in general. I think I saw some __MINGW32__ #ifs
> which (according to the code) meant to check for MS Windows system not MinGW
> compiler.

Do you think that __MINGW32__ may be changed to WIN32 globally without
inspecting each case? There are probably really __MINGW32__ specific
places. Inspecting each __MINGW32__ individually seems to be the
largest amount of work doing port to MSVC.

I couldn't see anything that appeared to be specific to MinGW rather
than to Windows generally.

You could try just defining __MINGW32__ for MSVC and see how far that
gets you.

--
Glynn Clements <glynn@gclements.plus.com>

12

Radim Blazek wrote:

The problem was with Map_info structure which is accessed only through
GRASS lib. The structures accessed directly (line_pnts,line_cats) have
only int or double or pointers to them. In theory the size of int may
be different for different compilers, but in this case it is the same
and it is quite rare to have GRASS and QGIS compiled by two compilers
with different size of int.

Windows has a defined C ABI, and MinGW mostly follows it (some
versions have issues with "long double", but I can only find one
occurrence of that in GRASS, and that's internal to a function).

off_t is only an issue because config.h explicitly redefines it for
MinGW.

C++ code is more problematic, as there isn't a defined C++ ABI (it
isn't even fixed for MSVC; the /vm[bgmsv] switches can change the
representation of pointer-to-member types).

--
Glynn Clements <glynn@gclements.plus.com>