[GRASS-dev] [GRASS GIS] #7: Location wizard: should predefine DB connection for new location

#7: Location wizard: should predefine DB connection for new location
---------------------+------------------------------------------------------
Reporter: neteler | Owner: grass-dev@lists.osgeo.org
     Type: defect | Status: new
Priority: major | Milestone: 6.3.0
Component: Python | Version: unspecified
Keywords: |
---------------------+------------------------------------------------------
The Location wizard does not predefine the DB connection for new location
which leads to problems with several modules (needs to generate MAPSET/VAR
file). Probably we should predefine SQLite here.

--
Ticket URL: <http://trac.osgeo.org/grass/ticket/7&gt;
GRASS GIS <http://grass.osgeo.org>
Commonly referred to as GRASS, this is a Geographic Information System (GIS) used for geospatial data management and analysis, image processing, graphics/maps production, spatial modeling, and visualization. GRASS is currently used in academic and commercial settings around the world, as well as by many governmental agencies and environmental consulting companies.

#7: Location wizard: should predefine DB connection for new location
----------------------+-----------------------------------------------------
  Reporter: neteler | Owner: grass-dev@lists.osgeo.org
      Type: defect | Status: new
  Priority: major | Milestone: 6.3.0
Component: Python | Version: unspecified
Resolution: | Keywords:
----------------------+-----------------------------------------------------
Comment (by cmbarton):

Replying to [ticket:7 neteler]:
> The Location wizard does not predefine the DB connection for new
location which leads to problems with several modules (needs to generate
MAPSET/VAR file). Probably we should predefine SQLite here.

I agree. However, except for xy locations, all are created using g.proj.
So this is an issue for g.proj. That is, the location wizard does not
operate 'inside' a location where it could use GRASS commands to do this.

Michael

--
Ticket URL: <http://trac.osgeo.org/grass/ticket/7#comment:1&gt;
GRASS-trac <http://grass.osgeo.org>
Commonly referred to as GRASS, this is a Geographic Information System (GIS) used for geospatial data management and analysis, image processing, graphics/maps production, spatial modeling, and visualization. GRASS is currently used in academic and commercial settings around the world, as well as by many governmental agencies and environmental consulting companies.

#7: Location wizard: should predefine DB connection for new location
----------------------+-----------------------------------------------------
  Reporter: neteler | Owner: grass-dev@lists.osgeo.org
      Type: defect | Status: new
  Priority: major | Milestone: 6.3.0
Component: Python | Version: svn-trunk
Resolution: | Keywords:
----------------------+-----------------------------------------------------
Changes (by neteler):

* cc: pkelly, cmbarton (added)
  * version: unspecified => svn-trunk

Comment:

Paul,

could you please take a look? It affects g.proj which should call
functionality from Vect_default_field_info() to define the VAR
file for the DBMI connection in a newly created location
(see lib/vector/Vlib/field.c). I guess that implementation in
g.proj takes 10 minutes max... :slight_smile:

Markus

--
Ticket URL: <http://trac.osgeo.org/grass/ticket/7#comment:2&gt;
GRASS-trac <http://grass.osgeo.org>
Commonly referred to as GRASS, this is a Geographic Information System (GIS) used for geospatial data management and analysis, image processing, graphics/maps production, spatial modeling, and visualization. GRASS is currently used in academic and commercial settings around the world, as well as by many governmental agencies and environmental consulting companies.

#7: Location wizard: should predefine DB connection for new location
----------------------+-----------------------------------------------------
  Reporter: neteler | Owner: grass-dev@lists.osgeo.org
      Type: defect | Status: new
  Priority: major | Milestone: 6.4.0
Component: Python | Version: svn-trunk
Resolution: | Keywords:
----------------------+-----------------------------------------------------
Changes (by neteler):

  * milestone: 6.3.0 => 6.4.0

--
Ticket URL: <http://trac.osgeo.org/grass/ticket/7#comment:3&gt;
GRASS-trac <http://grass.osgeo.org>
Commonly referred to as GRASS, this is a Geographic Information System (GIS) used for geospatial data management and analysis, image processing, graphics/maps production, spatial modeling, and visualization. GRASS is currently used in academic and commercial settings around the world, as well as by many governmental agencies and environmental consulting companies.

#7: Location wizard: should predefine DB connection for new location
----------------------+-----------------------------------------------------
  Reporter: neteler | Owner: grass-dev@lists.osgeo.org
      Type: defect | Status: new
  Priority: major | Milestone: 6.4.0
Component: Python | Version: svn-trunk
Resolution: | Keywords:
----------------------+-----------------------------------------------------
Comment (by hamish):

As mentioned before, I do not think that the VAR file should be a part of
the default MAPSET folder specification.

see http://thread.gmane.org/gmane.comp.gis.grass.devel/22676

e.g. for users who mainly work with rasters this is unneeded filesystem
noise.

C modules which require a DB to be set should be exiting with an error if
none is found (suggesting to run db.connect). Scripts like v.db.add* can
run that automatically if needed. Any other module bugs can be fixed,
exiting with an error is not much work if that doesn't happen already,
just check a return code.

Since I commented out the auto-set to DBF in 6.3cvs's init.sh 3 months ago
there have been no complaints about the change. This is not done in 6.2
either, no complaints there.

Hamish

--
Ticket URL: <http://trac.osgeo.org/grass/ticket/7#comment:4&gt;
GRASS-trac <http://grass.osgeo.org>
Commonly referred to as GRASS, this is a Geographic Information System (GIS) used for geospatial data management and analysis, image processing, graphics/maps production, spatial modeling, and visualization. GRASS is currently used in academic and commercial settings around the world, as well as by many governmental agencies and environmental consulting companies.

#7: Location wizard: should predefine DB connection for new location
----------------------+-----------------------------------------------------
  Reporter: neteler | Owner: grass-dev@lists.osgeo.org
      Type: defect | Status: new
  Priority: major | Milestone: 6.3.0
Component: default | Version: svn-trunk
Resolution: | Keywords:
----------------------+-----------------------------------------------------
Changes (by neteler):

  * component: Python => default
  * milestone: 6.4.0 => 6.3.0

Comment:

I disagree and think that new locations should be valid locations - or -
*all* modules should be able to work with locations lacking the VAR file.
I see no point in having this trivial thing lacking and then make the user
stumble about several commands not functioning.

The "file system noise" is in average 70 bytes (!) per MAPSET which can be
just ignored as a problem.

If we have a default DBMI driver (which is yet DBF), we also have to set
default settings. No complaints in 6.2 probably because use cases are
different.

Scripts like v.db.add* cannot run that automatically because we don't want
to hardcode DBF (or whatever driver) many times scattered all over the
source code but only in a single place (which was the init script and I
still complain that this was removed leading to broken DBMI settings - see
the archive for this).

Markus

--
Ticket URL: <http://trac.osgeo.org/grass/ticket/7#comment:5&gt;
GRASS-trac <http://grass.osgeo.org>
Commonly referred to as GRASS, this is a Geographic Information System (GIS) used for geospatial data management and analysis, image processing, graphics/maps production, spatial modeling, and visualization. GRASS is currently used in academic and commercial settings around the world, as well as by many governmental agencies and environmental consulting companies.

#7: Location wizard: should predefine DB connection for new location
----------------------+-----------------------------------------------------
  Reporter: neteler | Owner: grass-dev@lists.osgeo.org
      Type: defect | Status: new
  Priority: major | Milestone: 6.3.0
Component: default | Version: svn-trunk
Resolution: | Keywords:
----------------------+-----------------------------------------------------
Comment (by cmbarton):

Replying to [comment:5 neteler]:
> I disagree and think that new locations should be valid locations - or -
*all* modules should be able to work with locations lacking the VAR file.
I see no point in having this trivial thing lacking and then make the user
stumble about several commands not functioning.
>
> The "file system noise" is in average 70 bytes (!) per MAPSET which can
be just ignored as a problem.
>
> If we have a default DBMI driver (which is yet DBF), we also have to set
default settings. No complaints in 6.2 probably because use cases are
different.
>
> Scripts like v.db.add* cannot run that automatically because we don't
want to hardcode DBF (or whatever driver) many times scattered all over
the source code but only in a single place (which was the init script and
I still complain that this was removed leading to broken DBMI settings -
see the archive for this).
>
> Markus

I agree with Markus. All locations should be equal. This extra file is
irrelevant to those working only with raster, essentially invisible, and
takes up no space. The alternative is fine also (i.e., the lack of VAR
causes no problem with any module; it is simply created when needed). But
we should not be creating locations that have hidden gotchas.

Michael

--
Ticket URL: <http://trac.osgeo.org/grass/ticket/7#comment:6&gt;
GRASS-trac <http://grass.osgeo.org>
Commonly referred to as GRASS, this is a Geographic Information System (GIS) used for geospatial data management and analysis, image processing, graphics/maps production, spatial modeling, and visualization. GRASS is currently used in academic and commercial settings around the world, as well as by many governmental agencies and environmental consulting companies.

#7: Location wizard: should predefine DB connection for new location
----------------------+-----------------------------------------------------
  Reporter: neteler | Owner: grass-dev@lists.osgeo.org
      Type: defect | Status: new
  Priority: major | Milestone: 6.3.0
Component: default | Version: svn-trunk
Resolution: | Keywords:
----------------------+-----------------------------------------------------
Comment (by hamish):

re. create VAR file for each mapset if it doesn't exist
(see also default DB -> SQLite)

Markus:
> I disagree and think that new locations should be valid locations -
..

Michael:
> But we should not be creating locations that have hidden gotchas.

In that case, wouldn't all locations / datasets created with earlier
versions of grass be considered invalid?
(if so we need to re-add auto-creation of VAR in init.sh)

Markus:
> or - *all* modules should be able to work with locations lacking
> the VAR file.

This is the solution I would like to pursue. Can anyone give examples of
which modules break? AFAIK they have all been fixed (v.db.add*).
Do any remain?

> No complaints in 6.2 probably because use cases are different.

How so? I don't understand what changed. Changes to/new v.db.add*??

For fixing C modules it should be as hard as testing a return value and
calling G_fatal_error() with "DB unset. Please run db.select".
For scripts it should be as simple as either calling db.select in the
script if unset or just letting the C module's G_fatal_error() spit out
the appropriate error.

> I see no point in having this trivial thing lacking and then make
> the user stumble about several commands not functioning.
..
> If we have a default DBMI driver (which is yet DBF), we also have
> to set default settings.
..
> Scripts like v.db.add* cannot run that automatically because we
> don't want to hardcode DBF (or whatever driver) many times
> scattered all over the source code but only in a single place

I agree it should be set in a single place and it is annoying to go
through the step manually.

v.in.ascii (.c) creates dbf/ and sets it as default if it doesn't exist:
[new mapset]
G6.3svn> echo "1 2 foo bar" | v.in.ascii out=foomap fs=space x=1 y=2
Scanning input for column types...
Maximum input row length: 12
Maximum number of columns: 4
Minimum number of columns: 4
WARNING: Default driver / database set to:
          driver: dbf
          database: $GISDBASE/$LOCATION_NAME/$MAPSET/dbf/
Importing points...
...

this happens in.c (aka v.in.ascii's main.c),
  Fi = Vect_default_field_info(&Map, 1, NULL, GV_1TABLE);

which is from lib/vector/Vlib/field.c Vect_default_field_info()
  (hardcoded to DBF)

so any C module which creates a new table should use/check that?
AFAICT that seems to be the case for the C modules.
grass63/vector$ grep -rI Vect_default_field_info * | \
   grep -v \.svn | cut -f1 -d/

If db.select + default values is not sufficient for a single place there
could be a maintenance script which tests and sets the default DB when
needed. ? v.db.add* scripts could call that. I am happy to author that if
it is decided to go that way.

> (which was the init script and I still complain that this was
> removed leading to broken DBMI settings - see the archive for
> this).

- I hadn't understood the reason for putting it in init.sh was that was
supposed to be the central default DB set point. If that is to be the case
it's better to put that in a support script than in init.sh

- Initially removed due to nasty scripting bug which overwrote VAR
unconditionally, so if anyone decides to revert the removal, please
take care not to reintroduce that.

> The "file system noise" is in average 70 bytes (!) per MAPSET
> which can be just ignored as a problem.

I don't know if the joke translates well from English, but there is an old
saying "cleanliness is next to impossible." But still a worthy goal.

Vect_default_field_info() limits it to scripts which don't create tables
without using other C modules, and we can code a support script module to
handle those cases with only 1 line a module.

Hamish

--
Ticket URL: <http://trac.osgeo.org/grass/ticket/7#comment:7&gt;
GRASS-trac <http://grass.osgeo.org>
GRASS Geographic Information System (GRASS GIS) - http://grass.osgeo.org/

On 21/01/08 04:47, GRASS-trac wrote:

#7: Location wizard: should predefine DB connection for new location
----------------------+-----------------------------------------------------
  Reporter: neteler | Owner: grass-dev@lists.osgeo.org
      Type: defect | Status: new Priority: major | Milestone: 6.3.0 Component: default | Version: svn-trunk Resolution: | Keywords: ----------------------+-----------------------------------------------------
Comment (by hamish):

For fixing C modules it should be as hard as testing a return value and
calling G_fatal_error() with "DB unset. Please run db.select".
For scripts it should be as simple as either calling db.select in the
script if unset or just letting the C module's G_fatal_error() spit out
the appropriate error.

I imagine you mean db.connect instead of db.select...

Moritz

#7: Location wizard: should predefine DB connection for new location
----------------------+-----------------------------------------------------
  Reporter: neteler | Owner: grass-dev@lists.osgeo.org
      Type: defect | Status: new
  Priority: major | Milestone: 6.3.0
Component: default | Version: svn-trunk
Resolution: | Keywords:
----------------------+-----------------------------------------------------
Comment (by neteler):

Comment (by hamish):
> In that case, wouldn't all locations / datasets created with earlier
> versions of grass be considered invalid?
> (if so we need to re-add auto-creation of VAR in init.sh)

No, because it was there unless you took it out...

> Markus:
> > or - *all* modules should be able to work with locations lacking
> > the VAR file.
>
> This is the solution I would like to pursue. Can anyone give examples
of
> which modules break? AFAIK they have all been fixed (v.db.add*).
> Do any remain?

AFAIK all db.* modules and scripts at least. Also I suspect the v.*
scripts
to face a problem.

...
> v.in.ascii (.c) creates dbf/ and sets it as default if it doesn't
exist:

g.proj does not for example.
Likewise possibly v.in.ogr and r.in.gdal (and other modules which
generate locations)

> so any C module which creates a new table should use/check that?
> AFAICT that seems to be the case for the C modules.
> grass63/vector$ grep -rI Vect_default_field_info * | \
> grep -v \.svn | cut -f1 -d/

Please also check all pure db.* modules.

...
> > The "file system noise" is in average 70 bytes (!) per MAPSET
> > which can be just ignored as a problem.
>
> I don't know if the joke translates well from English, but there is an
old
> saying "cleanliness is next to impossible." But still a worthy goal.

I think we have quite some more severe other problems... think about all
the todo's concerning buffer sizes.

>
> Vect_default_field_info() limits it to scripts which don't create
tables
> without using other C modules, and we can code a support script module
to
> handle those cases with only 1 line a module.

Still someone has to check all modules/scripts...
And as mentioned: the db.* modules do not call Vect_*().

Markus

--
Ticket URL: <http://trac.osgeo.org/grass/ticket/7#comment:8&gt;
GRASS-trac <http://grass.osgeo.org>
GRASS Geographic Information System (GRASS GIS) - http://grass.osgeo.org/

#7: Location wizard: should predefine DB connection for new location
----------------------+-----------------------------------------------------
  Reporter: neteler | Owner: grass-dev@lists.osgeo.org
      Type: defect | Status: new
  Priority: major | Milestone: 6.3.0
Component: default | Version: svn-trunk
Resolution: | Keywords:
----------------------+-----------------------------------------------------
Comment (by neteler):

Hi,

any news here? With the upcoming release we'll get tons of
requests if we don't solve this problem...

I still don't understand why Hamish insists to do it the
complicated way.

Markus

--
Ticket URL: <http://trac.osgeo.org/grass/ticket/7#comment:9&gt;
GRASS GIS <http://grass.osgeo.org>
GRASS Geographic Information System (GRASS GIS) - http://grass.osgeo.org/

Sorry, just noticed there was a call to action by me in that bug report which I missed. Anyway, I think the key issue here is whether or not the database connection file is considered an essential part of a valid GRASS location.
1. If it *is*, then creation of it should be added to the places that crate locations: lib/init/mke_mapset.c and lib/gis/make_mapset.c.
2. If it is *not*, then the vector/db library functions that need to access it should be updated to either gracefully fail or create default settings on finding it absent, and this behaviour should be made consistent in all places.

Whatever way it is, I think adding a check to it in Init.sh is an ugly hack that just works around the problem rather than fixing it.

BUT: It seems option 1 is already partially implemented!
lib/init/mke_mapset.c (called if you create a new location using g.setproj, or the text-based startup) does indeed create the VAR file. So it looks to me that the anomaly is that this has been left out of lib/gis/make_mapset.c (called if you create a new location using g.proj, r.in.gdal or v.in.ogr or the GUI startup).

If we add creation of the VAR file to lib/gis/make_mapset.c, ideally we should also do an audit of the vector modules and remove the functionality that creates a default VAR file if not present, as it will no longer be needed. What do you think of this solution, Hamish?

Paul

On Feb 18, 2008, at 4:00 AM, GRASS GIS wrote:

#7: Location wizard: should predefine DB connection for new location
----------------------+-----------------------------------------------------
  Reporter: neteler | Owner: grass-dev@lists.osgeo.org
      Type: defect | Status: new
  Priority: major | Milestone: 6.3.0
Component: default | Version: svn-trunk
Resolution: | Keywords:
----------------------+-----------------------------------------------------
Comment (by neteler):

Hi,

any news here? With the upcoming release we'll get tons of
requests if we don't solve this problem...

I still don't understand why Hamish insists to do it the
complicated way.

Markus

--
Ticket URL: <#7 (Location wizard: should predefine DB connection for new location) – GRASS GIS;
GRASS GIS <http://grass.osgeo.org>
GRASS Geographic Information System (GRASS GIS) - http://grass.osgeo.org/

Markus,

This really needs to be done in g.proj, although I could hack it into the location wizard if needed. From what I was following of the discussion, there were some *philosophical* reasons for not doing it, but the cost for making a var file is essentially nil and the cost of not having one potentially high.

Michael

#7: Location wizard: should predefine DB connection for new location
----------------------+-----------------------------------------------------
  Reporter: neteler | Owner: hamish
      Type: defect | Status: assigned
  Priority: major | Milestone: 6.4.0
Component: default | Version: svn-trunk
Resolution: | Keywords:
----------------------+-----------------------------------------------------
Changes (by hamish):

  * status: new => assigned
  * owner: grass-dev@lists.osgeo.org => hamish
* cc: grass-dev@lists.osgeo.org (added)
  * milestone: 6.3.0 => 6.4.0

Comment:

for 6.3.0 I have re-enabled automatic VAR creation if it doesn't exist.

more soon.....

Hamish

--
Ticket URL: <http://trac.osgeo.org/grass/ticket/7#comment:10&gt;
GRASS GIS <http://grass.osgeo.org>
GRASS Geographic Information System (GRASS GIS) - http://grass.osgeo.org/

On Thu, 13 Mar 2008, GRASS GIS wrote:

for 6.3.0 I have re-enabled automatic VAR creation if it doesn't exist.

more soon.....

Attached is a patch (completely untested) that illustrates my idea for solving the problem. I think there should also be something in lib/gis/make_mapset.c (used by g.mapsets) but there are some complications there in relation to inheriting the DB settings; see my earlier mail.

Paul

(attachments)

make_loc.diff (1.08 KB)

#7: Location wizard: should predefine DB connection for new location
----------------------+-----------------------------------------------------
  Reporter: neteler | Owner: hamish
      Type: defect | Status: assigned
  Priority: major | Milestone: 6.4.0
Component: default | Version: svn-trunk
Resolution: | Keywords:
----------------------+-----------------------------------------------------
Comment (by hamish):

Eric wrote:
> I created a new location using today's svn source, and noticed that
> db.connect doesn't have its database parameter pre-populated with
> $GISDBASE/$LOCATION_NAME/$MAPSET/dbf as it used to; it is now blank.
> Possible bug? Would it possible to get the module to at least populate
> $GISDBASE/$LOCATION_NAME/$MAPSET for this parameter?

db.connect takes its default driver, database, schema, and group settings
from the $MAPSET/VAR file by way of the db_get_default_*_name() fns.
(lib/db/dbmi_base/default_name.c) This is a circular chicken-or-egg
situation. (the default is to set it back to what it already is, which is
a redundant task)

Martin:
> yes, see
> http://trac.osgeo.org/grass/ticket/7
>
> It seems to me that the VAR file should defined for newly created
> mapset by default to avoid possible problems.

that approach deals with the symptoms, but doesn't address the underlying
problem: what to do when the default DB settings are unknown. To be
robust it needs to be dealt with at module run-time and it needs to fail
in a safe way if the VAR file is somehow damaged.

A secondary issue is abstracting the default DB settings back to one place
so that when we make the switch to sqlite-as-default, we don't have to
change ad hoc VAR file & dbf/ directory creation code in a dozen places.

- For C code this was already condensed into Vect_default_field_info() but
could be abstracted further*,

- For shell scripts we can add a new flag to db.connect* to check if the
DB is set, and if not then set it. This would then be called from all
scripts which create vector tables outside of v.* modules, and could be
used in init.sh too. (although if we have done everything correctly then
we won't need to put it in init.sh or any other mapset creation code;
thus I suggest we keep it out of init.sh)

[*] now in SVN/trunk as DB_DEFAULT_DRIVER, db_set_default_connection(),
and 'db.connect -c'. (r30545) Init.sh and various scripts are now updated
to use that.

Eric:
> Thanks for the pointer. I'll hardcode my database paths for now.

see the db.connect help page example for DBF (I usually just cut and
paste that to the command line when needed). It accepts variable
LOCATION, MAPSET, you don't need to hardcode it. (Be sure to keep the
path with $VARIABLES in single quotes!)

Martin:
> anyway if you run some command which calls Vect_default_field_info()
> from Vlib, VAR file is created and default driver set up to dbf.
>
> e.g. g.copy vect=

i.e. any C vector module which goes down that street should do the trick.
The vector lib will set the default database if it goes to find it and
finds it unset.

Paul:
> Also I posted a suggested solution to the dev list a while ago, but
> there were no comments.

could you post a link to that in the bug report?

> > It seems to me that the VAR file should defined for newly created
> > mapset by default to avoid possible problems.
>
> For every mapset, or just for each location? If for every mapset, should
> it be inherited from whatever the setting for PERMANENT is, or just
> blindly set to dbf every time a new mapset is created? If it does have
> to be created for every mapset and inherited from PERMANENT, presumably
> the state of the DB connection for PERMANENT can't be determined
> without using Vectlib functions? But all the mapset/location-handling
> functions are in GISlib, and it would really not be a good idea to have
> GIS dependent on Vectlib functions (I think). So this is really quite
> complicated. Perhaps the functions that deal with the default database
> settings could be moved out of Vectlib and into GISlib.

whoa, keep it simple. if it is requested and not set, then set it. as
long as C vector modules are funneled through a single vector lib
function which checks & sets, and scripts doing low level stuff don't
make assumptions, we should be ok.

Michael:
> I'm with Marcus and Martin on this. The potential for problems is high
> if this is not done; the cost to do it is very small. Can we just go
> ahead and make this change to g.proj?

But it's not high. I commented out the auto VAR file creation sometime
before Christmas, and this is the first actual-usage complaint I've heard.
And it's not even a complaint, Eric just noticed that it wasn't set and
was wondering why.

[now I've made it live again in init.sh, but I still don't really think it
should be there; at least the 'mkdir dbf/' stuff is out of init.sh now]

AFAIK we are not (primarily) talking about g.proj, we are talking about
Init.sh. If mapset/location creation is a problem all we have to is keep
the 'db.connect -c' in init.sh and it will get the VAR file the first time
the mapset is accessed.

Paul:
> Attached is a patch (completely untested) that illustrates my idea for
> solving the problem.

now obviated?

> I think there should also be something in lib/gis/make_mapset.c (used by
> g.mapsets) but there are some complications there in relation to
> inheriting the DB settings; see my earlier mail.

(*g.mapset* not *g.mapsets*)

.... I still don't think this belongs in any mapset creation code and
don't think a VAR file should be included in the minimum requirements for
the definition of a valid mapset (ie 'dir/WIND').

Hamish

--
Ticket URL: <http://trac.osgeo.org/grass/ticket/7#comment:11&gt;
GRASS GIS <http://grass.osgeo.org>
GRASS Geographic Information System (GRASS GIS) - http://grass.osgeo.org/

Eric:
>> Thanks for the pointer. I'll hardcode my database paths for now.

Hamish:

see the db.connect help page example for DBF (I usually just cut and
paste that to the command line when needed). It accepts variable
LOCATION, MAPSET, you don't need to hardcode it. (Be sure to keep the
path with $VARIABLES in single quotes!)

Should each of $GISDABSE, $LOCATION_NAME, and $MAPSET have populated values
on location startup? These variables always remain blank for each location that
I try:

~ >echo $GISDBASE

~ >echo $LOCATION_NAME

~ >echo $MAPSET

~ >

are these variables only available from the shell if I export `g.gisenv` or eval `g.gisenv`
first?

~ Eric.

Eric:
>> Thanks for the pointer. I'll hardcode my database paths for now.

Hamish:

see the db.connect help page example for DBF (I usually just cut and
paste that to the command line when needed). It accepts variable
LOCATION, MAPSET, you don't need to hardcode it. (Be sure to keep the
path with $VARIABLES in single quotes!)

Should each of $GISDABSE, $LOCATION_NAME, and $MAPSET have populated values
on location startup? These variables always remain blank for each location that
I try:

~ >echo $GISDBASE

~ >echo $LOCATION_NAME

~ >echo $MAPSET

~ >

are these variables only available from the shell if I export `g.gisenv` or eval `g.gisenv`
first?

~ Eric.

#7: Location wizard: should predefine DB connection for new location
----------------------+-----------------------------------------------------
  Reporter: neteler | Owner: hamish
      Type: defect | Status: assigned
  Priority: major | Milestone: 6.4.0
Component: default | Version: svn-trunk
Resolution: | Keywords:
----------------------+-----------------------------------------------------
Comment (by hamish):

Eric wrote:

I created a new location using today's svn source, and noticed that
db.connect doesn't have its database parameter pre-populated with
$GISDBASE/$LOCATION_NAME/$MAPSET/dbf as it used to; it is now blank.
Possible bug? Would it possible to get the module to at least populate
$GISDBASE/$LOCATION_NAME/$MAPSET for this parameter?

db.connect takes its default driver, database, schema, and group settings
from the $MAPSET/VAR file by way of the db_get_default_*_name() fns.
(lib/db/dbmi_base/default_name.c) This is a circular chicken-or-egg
situation. (the default is to set it back to what it already is, which is
a redundant task)

Martin:

yes, see
#7 (Location wizard: should predefine DB connection for new location) – GRASS GIS

It seems to me that the VAR file should defined for newly created
mapset by default to avoid possible problems.

that approach deals with the symptoms, but doesn't address the underlying
problem: what to do when the default DB settings are unknown. To be
robust it needs to be dealt with at module run-time and it needs to fail
in a safe way if the VAR file is somehow damaged.

A secondary issue is abstracting the default DB settings back to one place
so that when we make the switch to sqlite-as-default, we don't have to
change ad hoc VAR file & dbf/ directory creation code in a dozen places.

- For C code this was already condensed into Vect_default_field_info() but
could be abstracted further*,

- For shell scripts we can add a new flag to db.connect* to check if the
DB is set, and if not then set it. This would then be called from all
scripts which create vector tables outside of v.* modules, and could be
used in init.sh too. (although if we have done everything correctly then
we won't need to put it in init.sh or any other mapset creation code;
thus I suggest we keep it out of init.sh)

[*] now in SVN/trunk as DB_DEFAULT_DRIVER, db_set_default_connection(),
and 'db.connect -c'. (r30545) Init.sh and various scripts are now updated
to use that.

Eric:

Thanks for the pointer. I'll hardcode my database paths for now.

see the db.connect help page example for DBF (I usually just cut and
paste that to the command line when needed). It accepts variable
LOCATION, MAPSET, you don't need to hardcode it. (Be sure to keep the
path with $VARIABLES in single quotes!)

Martin:

anyway if you run some command which calls Vect_default_field_info()
from Vlib, VAR file is created and default driver set up to dbf.

e.g. g.copy vect=

i.e. any C vector module which goes down that street should do the trick.
The vector lib will set the default database if it goes to find it and
finds it unset.

Paul:

Also I posted a suggested solution to the dev list a while ago, but
there were no comments.

could you post a link to that in the bug report?

It seems to me that the VAR file should defined for newly created
mapset by default to avoid possible problems.

For every mapset, or just for each location? If for every mapset, should
it be inherited from whatever the setting for PERMANENT is, or just
blindly set to dbf every time a new mapset is created? If it does have
to be created for every mapset and inherited from PERMANENT, presumably
the state of the DB connection for PERMANENT can't be determined
without using Vectlib functions? But all the mapset/location-handling
functions are in GISlib, and it would really not be a good idea to have
GIS dependent on Vectlib functions (I think). So this is really quite
complicated. Perhaps the functions that deal with the default database
settings could be moved out of Vectlib and into GISlib.

whoa, keep it simple. if it is requested and not set, then set it. as
long as C vector modules are funneled through a single vector lib
function which checks & sets, and scripts doing low level stuff don't
make assumptions, we should be ok.

Michael:

I'm with Marcus and Martin on this. The potential for problems is high
if this is not done; the cost to do it is very small. Can we just go
ahead and make this change to g.proj?

But it's not high. I commented out the auto VAR file creation sometime
before Christmas, and this is the first actual-usage complaint I've heard.
And it's not even a complaint, Eric just noticed that it wasn't set and
was wondering why.

[now I've made it live again in init.sh, but I still don't really think it
should be there; at least the 'mkdir dbf/' stuff is out of init.sh now]

AFAIK we are not (primarily) talking about g.proj, we are talking about
Init.sh. If mapset/location creation is a problem all we have to is keep
the 'db.connect -c' in init.sh and it will get the VAR file the first time
the mapset is accessed.

Paul:

Attached is a patch (completely untested) that illustrates my idea for
solving the problem.

now obviated?

I think there should also be something in lib/gis/make_mapset.c (used by
g.mapsets) but there are some complications there in relation to
inheriting the DB settings; see my earlier mail.

(*g.mapset* not *g.mapsets*)

.... I still don't think this belongs in any mapset creation code and
don't think a VAR file should be included in the minimum requirements for
the definition of a valid mapset (ie 'dir/WIND').

Hamish

If we want this to be easily changeable in the future, it should be dealt with in a central place (g.proj, init.sh [or its replacement], etc) so that changing it once will switch from a dbf to sqlite default.

This has come up periodically over the past year, as lack of a VAR file seems to cause different scattered modules to fail, and is usually puzzling to users as to what is going wrong.

It sounds like this can be fixed 2 ways: change g.proj to create a VAR file when it creates a location (this is how locations are created in the location wizard) or change modules to gracefully deal with the lack of a VAR file. The ongoing discussion about which of these two ways to fix it has left this unfixed for months. I'm agnostic about which way is best, except that I don't think that we should hack it into the location wizard wxPython code.

Michael

Eric:
>> Thanks for the pointer. I'll hardcode my database paths for now.

Hamish:
> see the db.connect help page example for DBF (I usually just cut
> and paste that to the command line when needed). It accepts
> variable LOCATION, MAPSET, you don't need to hardcode it. (Be sure
> to keep the path with $VARIABLES in single quotes!)

Eric:

Should each of $GISDABSE, $LOCATION_NAME, and $MAPSET have populated
values on location startup? These variables always remain blank for
each location that I try:

~ >echo $GISDBASE
~ >echo $LOCATION_NAME
~ >echo $MAPSET
~ >

are these variables only available from the shell if I export
`g.gisenv` or eval `g.gisenv` first?

If you want to use them as variables from the shell you should use
  eval `g.gisenv`
first. Back in the GRASS 5 days they used to be set all the time but
this caused problems if you had the GUI running and then from the
terminal window you changed e.g. the mapset with g.mapset. The already
launched GUI would not and could not know about the change. So g.gisenv
was created to hold those variables.

But this has nothing (or at least little) to do with the task at hand.
db.connect will parse those variables itself, which is why you have to
enclose them in single quotes when given on the command line- to
prevent the shell from attempting to expand them (into nothing).

so just cut and paste:
db.connect driver=dbf database='$GISDBASE/$LOCATION_NAME/$MAPSET/dbf/'

and it works.

Hamish

      ____________________________________________________________________________________
Be a better friend, newshound, and
know-it-all with Yahoo! Mobile. Try it now. http://mobile.yahoo.com/;_ylt=Ahu06i62sR8HDtDypao8Wcj9tAcJ