[GRASS-dev] [GRASS GIS] #3841: script/core.py: create_location() fails with FileNotFoundError: [Errno 2] No such file or directory: '/tmp/grass7-root-94/tmploc/PERMANENT/MYNAME'

#3841: script/core.py: create_location() fails with FileNotFoundError: [Errno 2]
No such file or directory: '/tmp/grass7-root-94/tmploc/PERMANENT/MYNAME'
-------------------------------+-------------------------
Reporter: neteler | Owner: grass-dev@…
     Type: defect | Status: new
Priority: normal | Milestone: 7.8.0
Component: Python | Version: svn-trunk
Keywords: python3, grass.py | CPU: x86-64
Platform: Linux |
-------------------------------+-------------------------
In a docker environment (Ubuntu 18:04 based) I get this startup error with
trunk (on a pure Python 3 environment):

{{{
# note: the second line from bottom is local debug output via print()

root@6ade79b80a2b:/grassdb# grass --tmp-location EPSG:25832 --exec
g.extension -s extension=i.sentinel url=/src/i.sentinel
Starting GRASS GIS...
Creating new GRASS GIS location <tmploc>...
/tmp/grass7-root-37 tmploc EPSG:25832
ERROR: FileNotFoundError(2, 'No such file or directory')
}}}

With Python debugger (pdb) I see that it cannot open a file with
`codecs.open()`:

{{{
root@6ade79b80a2b:/grassdb# grass --tmp-location EPSG:25832 --exec
g.extension -s extension=i.sentinel url=/src/i.sentinel
Starting GRASS GIS...
Creating new GRASS GIS location <tmploc>...
/tmp/grass7-root-94 tmploc EPSG:25832
25832 None
> /usr/local/bin/grass(853)create_location()
-> gcore.create_location(gisdbase, location,
(Pdb) s
> /usr/local/bin/grass(854)create_location()
-> epsg=epsg, datum_trans=datum_trans)
(Pdb) s
--Call--
>
/usr/local/grass77/etc/python/grass/script/core.py(1518)create_location()
-> def create_location(dbase, location, epsg=None, proj4=None,
filename=None,
(Pdb) s
>
/usr/local/grass77/etc/python/grass/script/core.py(1538)create_location()
-> gisdbase = None
(Pdb) s
>
/usr/local/grass77/etc/python/grass/script/core.py(1539)create_location()
-> if epsg or proj4 or filename or wkt:
(Pdb) s
>
/usr/local/grass77/etc/python/grass/script/core.py(1541)create_location()
-> gisdbase = gisenv()['GISDBASE']
(Pdb) s
--Call--
> /usr/local/grass77/etc/python/grass/script/core.py(1075)gisenv()
-> def gisenv(env=None):
(Pdb) s
> /usr/local/grass77/etc/python/grass/script/core.py(1086)gisenv()
-> s = read_command("g.gisenv", flags='n', env=env)
(Pdb) s
--Call--
> /usr/local/grass77/etc/python/grass/script/core.py(491)read_command()
-> def read_command(*args, **kwargs):
(Pdb) s
> /usr/local/grass77/etc/python/grass/script/core.py(500)read_command()
-> encoding = 'default'
(Pdb) s
> /usr/local/grass77/etc/python/grass/script/core.py(501)read_command()
-> if 'encoding' in kwargs:
(Pdb) s
> /usr/local/grass77/etc/python/grass/script/core.py(504)read_command()
-> if _capture_stderr and 'stderr' not in kwargs.keys():
(Pdb) s
> /usr/local/grass77/etc/python/grass/script/core.py(506)read_command()
-> process = pipe_command(*args, **kwargs)

[...]

> /usr/local/grass77/etc/python/grass/script/core.py(512)read_command()
-> if _capture_stderr and returncode:
(Pdb) n
> /usr/local/grass77/etc/python/grass/script/core.py(514)read_command()
-> return handle_errors(returncode, stdout, args, kwargs)
(Pdb) n
--Return--
>
/usr/local/grass77/etc/python/grass/script/core.py(514)read_command()->'GISDBASE=/gr...T=<UNKNOWN>\n'
-> return handle_errors(returncode, stdout, args, kwargs)
(Pdb) n
> /usr/local/grass77/etc/python/grass/script/core.py(1087)gisenv()
-> return parse_key_val(s)
(Pdb) n
--Return--
>
/usr/local/grass77/etc/python/grass/script/core.py(1087)gisenv()->{'GISDBASE':
...: '<UNKNOWN>'}
-> return parse_key_val(s)
(Pdb) n
>
/usr/local/grass77/etc/python/grass/script/core.py(1542)create_location()
-> run_command('g.gisenv', set='GISDBASE=%s' % dbase)
(Pdb) n
>
/usr/local/grass77/etc/python/grass/script/core.py(1544)create_location()
-> if not os.path.exists(dbase):
(Pdb) n
>
/usr/local/grass77/etc/python/grass/script/core.py(1548)create_location()
-> if os.path.exists(os.path.join(dbase, location)):
(Pdb) n
>
/usr/local/grass77/etc/python/grass/script/core.py(1556)create_location()
-> kwargs = dict()
(Pdb) n
>
/usr/local/grass77/etc/python/grass/script/core.py(1557)create_location()
-> if datum:
(Pdb) n
>
/usr/local/grass77/etc/python/grass/script/core.py(1559)create_location()
-> if datum_trans:
(Pdb) n
>
/usr/local/grass77/etc/python/grass/script/core.py(1562)create_location()
-> if epsg:
(Pdb) n
>
/usr/local/grass77/etc/python/grass/script/core.py(1563)create_location()
-> ps = pipe_command('g.proj', quiet=True, flags='t', epsg=epsg,
(Pdb) n
>
/usr/local/grass77/etc/python/grass/script/core.py(1564)create_location()
-> location=location, stderr=PIPE, **kwargs)
(Pdb) n
>
/usr/local/grass77/etc/python/grass/script/core.py(1577)create_location()
-> if epsg or proj4 or filename or wkt:
(Pdb) n
>
/usr/local/grass77/etc/python/grass/script/core.py(1578)create_location()
-> error = ps.communicate()[1]
(Pdb) n
>
/usr/local/grass77/etc/python/grass/script/core.py(1579)create_location()
-> run_command('g.gisenv', set='GISDBASE=%s' % gisdbase)
(Pdb) n
>
/usr/local/grass77/etc/python/grass/script/core.py(1581)create_location()
-> if ps.returncode != 0 and error:
(Pdb) n
>
/usr/local/grass77/etc/python/grass/script/core.py(1584)create_location()
-> try:
(Pdb) n
>
/usr/local/grass77/etc/python/grass/script/core.py(1585)create_location()
-> fd = codecs.open(os.path.join(dbase, location, 'PERMANENT', 'MYNAME'),
(Pdb) n
>
/usr/local/grass77/etc/python/grass/script/core.py(1586)create_location()
-> encoding='utf-8', mode='w')
(Pdb) n
FileNotFoundError: [Errno 2] No such file or directory:
'/tmp/grass7-root-94/tmploc/PERMANENT/MYNAME'
>
/usr/local/grass77/etc/python/grass/script/core.py(1586)create_location()
-> encoding='utf-8', mode='w')
(Pdb) n
>
/usr/local/grass77/etc/python/grass/script/core.py(1592)create_location()
-> except OSError as e:
(Pdb) n
>
/usr/local/grass77/etc/python/grass/script/core.py(1593)create_location()
-> raise ScriptError(repr(e))
(Pdb) n
grass.exceptions.ScriptError: FileNotFoundError(2, 'No such file or
directory')
>
/usr/local/grass77/etc/python/grass/script/core.py(1593)create_location()
-> raise ScriptError(repr(e))
(Pdb) n
--Return--
>
/usr/local/grass77/etc/python/grass/script/core.py(1593)create_location()->None
-> raise ScriptError(repr(e))
(Pdb) n
grass.exceptions.ScriptError: FileNotFoundError(2, 'No such file or
directory')
> /usr/local/bin/grass(854)create_location()
-> epsg=epsg, datum_trans=datum_trans)
(Pdb) n
> /usr/local/bin/grass(864)create_location()
-> except gcore.ScriptError as err:
(Pdb) n
> /usr/local/bin/grass(865)create_location()
-> fatal(err.value.strip('"').strip("'").replace('\\n', os.linesep))
(Pdb) n
ERROR: FileNotFoundError(2, 'No such file or directory')
SystemExit: Exiting...
> /usr/local/bin/grass(865)create_location()
-> fatal(err.value.strip('"').strip("'").replace('\\n', os.linesep))
(Pdb) n
--Return--
> /usr/local/bin/grass(865)create_location()->None
-> fatal(err.value.strip('"').strip("'").replace('\\n', os.linesep))
}}}

The code in question:

https://trac.osgeo.org/grass/browser/grass/trunk/lib/python/script/core.py#L1573

{{{
1573 try:
1574 fd = codecs.open(os.path.join(dbase, location,
'PERMANENT', 'MYNAME'),
1575 encoding='utf-8', mode='w')
1576 if desc:
1577 fd.write(desc + os.linesep)
1578 else:
1579 fd.write(os.linesep)
1580 fd.close()
1581 except OSError as e:
1582 raise ScriptError(repr(e))
}}}

Version used:

{{{
lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 18.04.2 LTS
Release: 18.04
Codename: bionic

python3 --version
Python 3.6.7
}}}

Any ideas here?

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/3841&gt;
GRASS GIS <https://grass.osgeo.org>

#3841: script/core.py: create_location() fails with FileNotFoundError: [Errno 2]
No such file or directory: '/tmp/grass7-root-94/tmploc/PERMANENT/MYNAME'
----------------------+-------------------------------
  Reporter: neteler | Owner: grass-dev@…
      Type: defect | Status: new
  Priority: normal | Milestone: 7.8.0
Component: Python | Version: svn-trunk
Resolution: | Keywords: python3, grass.py
       CPU: x86-64 | Platform: Linux
----------------------+-------------------------------

Comment (by pmav99):

I think the error is that the PERMANENT directory does not exist.

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/3841#comment:1&gt;
GRASS GIS <https://grass.osgeo.org>

#3841: script/core.py: create_location() fails with FileNotFoundError: [Errno 2]
No such file or directory: '/tmp/grass7-root-94/tmploc/PERMANENT/MYNAME'
----------------------+-------------------------------
  Reporter: neteler | Owner: grass-dev@…
      Type: defect | Status: new
  Priority: normal | Milestone: 7.8.0
Component: Python | Version: svn-trunk
Resolution: | Keywords: python3, grass.py
       CPU: x86-64 | Platform: Linux
----------------------+-------------------------------

Comment (by neteler):

Confirmed, that PERMANENT isn't there (would be cool if core.py could say
so...).

After further debugging: The reason is indeed a crash of g.proj:

{{{
root@cea3e6ed4c0c:/grassdb# grass --tmp-location EPSG:25832 --exec
g.version -gb
Starting GRASS GIS...
Creating new GRASS GIS location <tmploc>...
>
/usr/local/grass77/etc/python/grass/script/core.py(1584)create_location()
-> try:
(Pdb) s
>
/usr/local/grass77/etc/python/grass/script/core.py(1585)create_location()
-> fd = open(os.path.join(dbase, location, 'PERMANENT', 'MYNAME'),
(Pdb)
[1]+ Stopped grass --tmp-location EPSG:25832 --exec
g.version -gb

root@cea3e6ed4c0c:/grassdb# l /tmp/
grass7-root-50/ simple.laz

root@cea3e6ed4c0c:/grassdb# l /tmp/grass7-root-50/
gisrc
}}}

Test with an existing location:

{{{
GRASS 7.7.svn (nc_basic_spm_grass7):/grassdb > g.proj -t epsg=25832
location=myloc
Illegal instruction (core dumped)
}}}

PROJ Version in this Ubuntu docker:

{{{
cs2cs -v
Using from definition:
Rel. 4.9.3, 15 August 2016

GRASS 7.7.svn (nc_basic_spm_grass7):/grassdb > ldd
/usr/local/grass77/bin/g.proj | grep proj
         libgrass_gproj.7.7.so =>
/usr/local/grass77/lib/libgrass_gproj.7.7.so (0x00007f4b04011000)
         libproj.so.12 => /usr/lib/x86_64-linux-gnu/libproj.so.12
(0x00007f4b0294a000)
}}}

GNU Debugger:

{{{
GRASS 7.7.svn (nc_basic_spm_grass7):/grassdb > gdb g.proj
GNU gdb (Ubuntu 8.1-0ubuntu3) 8.1.0.20180409-git

(gdb) r -t epsg=25832 location=myloc
Starting program: /usr/local/grass77/bin/g.proj -t epsg=25832
location=myloc
warning: Error disabling address space randomization: Operation not
permitted
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
[New Thread 0x7f5b60c7c700 (LWP 193)]
[New Thread 0x7f5b6047b700 (LWP 194)]
[New Thread 0x7f5b5dc7a700 (LWP 195)]
[New Thread 0x7f5b59479700 (LWP 196)]
[New Thread 0x7f5b56c78700 (LWP 197)]
[New Thread 0x7f5b54477700 (LWP 198)]
[New Thread 0x7f5b53c76700 (LWP 199)]

Thread 1 "g.proj" received signal SIGILL, Illegal instruction.
0x00007f5b77d18b7e in read_datum_table () from
/usr/local/grass77/lib/libgrass_gproj.7.7.so
(gdb) bt full
#0 0x00007f5b77d18b7e in read_datum_table () from
/usr/local/grass77/lib/libgrass_gproj.7.7.so
No symbol table info available.
#1 0x00007f5b77d17d0b in GPJ_osr_to_grass () from
/usr/local/grass77/lib/libgrass_gproj.7.7.so
No symbol table info available.
#2 0x00005621585e8ceb in input_epsg ()
No symbol table info available.
#3 0x00005621585e7de3 in main ()
No symbol table info available.
}}}

Installed PROJ related packages:
{{{
dpkg-query -l | grep proj
ii libproj-dev:amd64 4.9.3-2
amd64 Cartographic projection library (development files)
ii libproj12:amd64 4.9.3-2
amd64 Cartographic projection library
ii proj-data 4.9.3-2
all Cartographic projection filter and library (datum package)

# the datum files are apparently installed:

apt-file list proj-data
proj-data: /usr/share/doc/proj-data/AUTHORS
proj-data: /usr/share/doc/proj-data/NEWS.Debian.gz
proj-data: /usr/share/doc/proj-data/NEWS.gz
[...]
proj-data: /usr/share/proj/alaska
proj-data: /usr/share/proj/conus
proj-data: /usr/share/proj/epsg
[...]
}}}

I have no clue why g.proj crashes here.

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/3841#comment:2&gt;
GRASS GIS <https://grass.osgeo.org>

#3841: script/core.py: create_location() fails with FileNotFoundError: [Errno 2]
No such file or directory: '/tmp/grass7-root-94/tmploc/PERMANENT/MYNAME'
----------------------+-------------------------------
  Reporter: neteler | Owner: grass-dev@…
      Type: defect | Status: new
  Priority: normal | Milestone: 7.8.0
Component: Python | Version: svn-trunk
Resolution: | Keywords: python3, grass.py
       CPU: x86-64 | Platform: Linux
----------------------+-------------------------------

Comment (by neteler):

Replying to [comment:2 neteler]:
> Confirmed, that PERMANENT isn't there (would be cool if core.py could
say so...).

... to have a more reasonabe error message here would be greatly
appreciated.

Back to the problem:

> After further debugging: The reason is indeed a crash of g.proj:
> ...
> GNU Debugger:
>
> {{{
> GRASS 7.7.svn (nc_basic_spm_grass7):/grassdb > gdb g.proj
> GNU gdb (Ubuntu 8.1-0ubuntu3) 8.1.0.20180409-git
>
> (gdb) r -t epsg=25832 location=myloc
> Starting program: /usr/local/grass77/bin/g.proj -t epsg=25832
location=myloc
> ...
> Thread 1 "g.proj" received signal SIGILL, Illegal instruction.
>
> ...
> }}}
>
> ...
>
> I have no clue why g.proj crashes here.

Likely found it with support by Markus Metz:
https://github.com/mundialis/grass-
py3-pdal/commit/b7b4058a9e52a0a08dca5b819831b54583dcbacc

Take home message: do not use `-march=native` for gcc in a Dockerfile.

(docker image rebuilding is ongoing).

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/3841#comment:3&gt;
GRASS GIS <https://grass.osgeo.org>

#3841: script/core.py: create_location() fails with FileNotFoundError: [Errno 2]
No such file or directory: '/tmp/grass7-root-94/tmploc/PERMANENT/MYNAME'
----------------------+-------------------------------
  Reporter: neteler | Owner: grass-dev@…
      Type: defect | Status: closed
  Priority: normal | Milestone: 7.8.0
Component: Python | Version: svn-trunk
Resolution: invalid | Keywords: python3, grass.py
       CPU: x86-64 | Platform: Linux
----------------------+-------------------------------
Changes (by neteler):

* status: new => closed
* resolution: => invalid

Comment:

Solved by avoiding `-march=native` for gcc in a Dockerfile.

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/3841#comment:4&gt;
GRASS GIS <https://grass.osgeo.org>