[GRASS-dev] g.mlist as C implementation?

Hamish wrote:

> I had overlooked the part where loop which adds the separator was
> replaced with a call to "tr".
>
> Technically, this corresponds to a reduction in functionality, as the
> previous version would accept any string as a separator, but using
> "tr" restricts it to a single character (which is probably all that's
> required).

is it possible to use sed instead of tr?
  sed -e "s/\n/$sep/" # doesn't work

Yes, but you can't do it with a single "s" command. You would need to
collect all of the input into the pattern space with the N command.
This could be a problem for non-GNU versions of sed which may have a
limit on the maximum length of the pattern space.

OTOH, tr acts as a filter, writing out data as it processes it, so
there is no limit on the amount of data.

I will have a look at merging Martin, Markus and Glynn's suggestions for
a flat output g.list flag. As the feature is needed for parsing, I'd
rather do it as a flag vs. a ncols= option which retains the text and
fancy format mapset underlines. I've never really known what the "g"
stands for, but as we use it a lot for parsable output, the flag will
be '-g' here too.

so it would look like:

G63> g.list -g rast
map1@user1
map2@user1
map3@user1
map_a@PERMANENT
map_b@PERMANENT
map_c@PERMANENT

G63> g.list -gf rast
map1@user1:
map2@user1:This map has a title
map3@user1:
map_a@PERMANENT:Raw x,y,z data binned into a raster grid by cell min
map_b@PERMANENT:Land Use
map_c@PERMANENT:

According to G_legal_filename(), ":" is legal in map names (that won't
work on Windows):

  if (*s == '/' || *s == '"' || *s == '\'' || *s <= ' ' ||
            *s == '@' || *s == ',' || *s == '=' || *s == '*' || *s > 0176) {
    fprintf(stderr, _("Illegal filename. Character <%c> not allowed.\n"), *s);

I'd suggest using a comma for the field separator.

--
Glynn Clements <glynn@gclements.plus.com>

Hi,
back to old discussions?
http://grass.itc.it/pipermail/grass-dev/2007-January/028510.html

Just a idea - would it be possible to #IFDEF OS specific illegal chars
for this function? Only problem for such approach could be tar'ing
location/gisdbase and moving to other system (move *nix files with ":"
to windows).
Or - for GRASS 7 - get rid of map_name==file_name at all. Assign some
random file name (gr_0982193) and store map name in it's header
(my!@*&^*!#(*^!*@&raster). Such approach could eliminate illegal map
name problem at all. Drawback - no easy way how to tell which map is
stored in which file w/o GRASS (it can be solved by providing some
"table of contents" file).

Just trying to confuse everyone,
Maris.

2007/9/13, Glynn Clements <glynn@gclements.plus.com>:

According to G_legal_filename(), ":" is legal in map names (that won't
work on Windows):

  if (*s == '/' || *s == '"' || *s == '\'' || *s <= ' ' ||
            *s == '@' || *s == ',' || *s == '=' || *s == '*' || *s > 0176) {
    fprintf(stderr, _("Illegal filename. Character <%c> not allowed.\n"), *s);

I'd suggest using a comma for the field separator.

--
Glynn Clements <glynn@gclements.plus.com>

_______________________________________________
grass-dev mailing list
grass-dev@grass.itc.it
http://grass.itc.it/mailman/listinfo/grass-dev

Hi,

I modified my previous patch, now with -g flag...

$ g.list rast map=overlay
----------------------------------------------
raster files available in mapset <overlay>:
r r1

$ g.list rast map=overlay -f
----------------------------------------------
raster files available in mapset <overlay>:
r1: test
r:

$ g.list rast map=overlay -g
r1@overlay
r@overlay

$ g.list rast map=overlay -gf
r1@overlay:test
r@overlay:

BTW the patch is really not nice, maybe it would be better to modify
G_list() to add mapset to the element name automatically and lister()
fn for titles...

Martin

2007/9/12, Markus Neteler <neteler@itc.it>:

Hamish wrote on 09/12/2007 12:47 PM:
> what should a the C flat 'g.list -g' output look like? how about:
> [user1]
> map1
> map2
> map3
> [PERMANENT]
> map_a
> map_b
> map_c
>
For parsing reasons, it should be
[user1] map1
[user1] map2
[user1] map3
[PERMANENT] map1
[PERMANENT] map2
[PERMANENT] map3

or even
user1:map1
user1:map2
user1:map3
PERMANENT:map1
PERMANENT:map2
PERMANENT:map3

and 'g.list -gf':
user1:map1:title1
user1:map2:title2
PERMANENT:map1:title1

Markus

------------------
ITC -> dall'1 marzo 2007 Fondazione Bruno Kessler
ITC -> since 1 March 2007 Fondazione Bruno Kessler
------------------

_______________________________________________
grass-dev mailing list
grass-dev@grass.itc.it
http://grass.itc.it/mailman/listinfo/grass-dev

--
Martin Landa <landa.martin@gmail.com> * http://gama.fsv.cvut.cz/~landa *

(attachments)

g-list-fg.diff (7.49 KB)

Hamish wrote:
> I will have a look at merging Martin, Markus and Glynn's suggestions for
> a flat output g.list flag. As the feature is needed for parsing, I'd
> rather do it as a flag vs. a ncols= option which retains the text and
> fancy format mapset underlines. I've never really known what the "g"
> stands for, but as we use it a lot for parsable output, the flag will
> be '-g' here too.
>
> so it would look like:

...

> G63> g.list -gf rast
> map1@user1:
> map2@user1:This map has a title
> map3@user1:
> map_a@PERMANENT:Raw x,y,z data binned into a raster grid by cell min
> map_b@PERMANENT:Land Use
> map_c@PERMANENT:

According to G_legal_filename(), ":" is legal in map names (that won't
work on Windows):

  if (*s == '/' || *s == '"' || *s == '\'' || *s <= ' ' ||
            *s == '@' || *s == ',' || *s == '=' || *s == '*' || *s > 0176) {
    fprintf(stderr, _("Illegal filename. Character <%c> not allowed.\n"), *s);

I'd suggest using a comma for the field separator.

I'm not crazy about commas as field separators as there's a good chance that
the map title has some in it and a poorly written script would easily do
something like "cut -d, -f2" and chop it in half. Same goes for using a single
' ' space.

My two preferred choices would be : or |.
":" mimics what we use for deg:min:sec separators and makes some contextual

   sense as a sep.
"|" mimics what we use for vector attribute column separators.

(trying to reuse ideas wherever possible for consistency's sake)

We could add : and | to the G_legal_filename() blacklist, even though that
could hurt backwards compatibility if the fn is used to test ,old_map names.
Esp. that, as you point out, calling a map "C:" is problematic on Windows.
(my vote it is to do it after making sure that g.rename doesn't call
G_legal_filename() for the old map name)

Using "=" for the sep seems inappropriate as
- it looks funny in this context, and
- "map@mapset=Some Title" wouldn't be reasonably used as eval `shell variable`.

If all options are deemed too ugly I guess we could use the old \t standby.

Having now looked at the g.list code all I can say is yuk. There's about 7
convoluted layers of abstraction and G_spawn()ing to executables with MINGW32
#ifdefs and pipe checking and "more" paging thrown in when AFAICT there really
only needs to be 2 or 3 layers of clean C functions.

general/manage/cmd/list.c
general/manage/lister/{cell|vector} -> $GISBASE/etc/lister/*.exe
general/manage/lib/do_list.c
lib/gis/list.c
lib/gis/ls.c

seem like they could all be combined into just
  general/manage/cmd/list.c + lib/gis/ls.c

My idea was to do a new simple G_list_element_flat() fn in lib/gis/list.c
which just creates a G__ls() list then loops through and fprintf(out,)s the
name, optionally the mapset, and optionally the sep+map title*, then closing
'\n'.

[*] title only makes sense if element is cell or vector, silently set to ""
otherwise.

I'm not really sure, Martin's patch is fine but I don't see any way to extent
it to append "@mapset" and "[sep]Map Title" as G_ls_format() goes right to the
stream. My idea above kind of suffers from the same problem- It seems a bit
wrong to have the same function be so low level that it calls {G_}ls and so
high level that it evals the result to query the map title with. It's nice to
have a deep lib fn do the list but the dump to stdout has to happen after the
title query and appending.

I would like to avoid having the new fn rely on any of the lister stuff or
pagers if at all possible. Just send result to FILE*out. (maybe I'm just not
good enough in C to write my own custom function to use for the lister arg and
the solution really isn't that hard, just follow the 'g.list -f' example?)

So perhaps we could use this opportunity to simplify this part of the code?
  (about here's where I cry oceanographer not software engineer :wink:

Implementing 'g.list -g' SHOULD just be a 5 minute job..... but it ain't.

Hamish

____________________________________________________________________________________
Got a little couch potato?
Check out fun summer activities for kids.
http://search.yahoo.com/search?fr=oni_on_mail&p=summer+activities+for+kids&cs=bz

Martin Landa wrote:

Hi,

I modified my previous patch, now with -g flag...

..

$ g.list rast map=overlay -g
r1@overlay
r@overlay

$ g.list rast map=overlay -gf
r1@overlay:test
r@overlay:

great, thanks!

BTW the patch is really not nice, maybe it would be better to modify
G_list() to add mapset to the element name automatically and lister()
fn for titles...

Agreed, see my last post. I tried to say the same thing, but was less clear.

(we need to be careful about modifying very old lib fn interfaces, probably
better to write a new G_list_full() or something as a wrapper so we don't break
old/3rd party code) And lets dump lister/*.exe if we can- just unneeded
complication IMO.

Hamish

____________________________________________________________________________________
Be a better Globetrotter. Get better travel answers from someone who knows. Yahoo! Answers - Check it out.
http://answers.yahoo.com/dir/?link=list&sid=396545469

As soon as this goes into the cvs, I'll modify the wxPython GIS element
selection control to use the new g.list.

Michael

On 9/13/07 2:12 AM, "Hamish" <hamish_nospam@yahoo.com> wrote:

Martin Landa wrote:

Hi,

I modified my previous patch, now with -g flag...

..

$ g.list rast map=overlay -g
r1@overlay
r@overlay

$ g.list rast map=overlay -gf
r1@overlay:test
r@overlay:

great, thanks!

BTW the patch is really not nice, maybe it would be better to modify
G_list() to add mapset to the element name automatically and lister()
fn for titles...

Agreed, see my last post. I tried to say the same thing, but was less clear.

(we need to be careful about modifying very old lib fn interfaces, probably
better to write a new G_list_full() or something as a wrapper so we don't
break
old/3rd party code) And lets dump lister/*.exe if we can- just unneeded
complication IMO.

Hamish

______________________________________________________________________________
______
Be a better Globetrotter. Get better travel answers from someone who knows.
Yahoo! Answers - Check it out.
http://answers.yahoo.com/dir/?link=list&sid=396545469

__________________________________________
Michael Barton, Professor of Anthropology
Director of Graduate Studies
School of Human Evolution & Social Change
Center for Social Dynamics & Complexity
Arizona State University

phone: 480-965-6213
fax: 480-965-7671
www: http://www.public.asu.edu/~cmbarton

Martin Landa wrote:

BTW the patch is really not nice, maybe it would be better to modify
G_list() to add mapset to the element name automatically and lister()
fn for titles...

What (if anything) actually uses G_list()? And what's with the
G_ELEMENT_* constants rather than just using the element name?

--
Glynn Clements <glynn@gclements.plus.com>

Maris Nartiss wrote:

back to old discussions?
http://grass.itc.it/pipermail/grass-dev/2007-January/028510.html

Just a idea - would it be possible to #IFDEF OS specific illegal chars
for this function? Only problem for such approach could be tar'ing
location/gisdbase and moving to other system (move *nix files with ":"
to windows).

It would make more sense to prohibit the characters which are illegal
on Windows for all platforms.

Or - for GRASS 7 - get rid of map_name==file_name at all. Assign some
random file name (gr_0982193) and store map name in it's header
(my!@*&^*!#(*^!*@&raster). Such approach could eliminate illegal map
name problem at all. Drawback - no easy way how to tell which map is
stored in which file w/o GRASS (it can be solved by providing some
"table of contents" file).

I don't think that it's worth going that far. If we were going to do
that, we would have done it for vectors, which are limited to
alphanumerics and underscore due to SQL.

--
Glynn Clements <glynn@gclements.plus.com>

Hamish wrote:

> > I will have a look at merging Martin, Markus and Glynn's suggestions for
> > a flat output g.list flag. As the feature is needed for parsing, I'd
> > rather do it as a flag vs. a ncols= option which retains the text and
> > fancy format mapset underlines. I've never really known what the "g"
> > stands for, but as we use it a lot for parsable output, the flag will
> > be '-g' here too.
> >
> > so it would look like:
...
> > G63> g.list -gf rast
> > map1@user1:
> > map2@user1:This map has a title
> > map3@user1:
> > map_a@PERMANENT:Raw x,y,z data binned into a raster grid by cell min
> > map_b@PERMANENT:Land Use
> > map_c@PERMANENT:
>
> According to G_legal_filename(), ":" is legal in map names (that won't
> work on Windows):
>
> if (*s == '/' || *s == '"' || *s == '\'' || *s <= ' ' ||
> *s == '@' || *s == ',' || *s == '=' || *s == '*' || *s > 0176) {
> fprintf(stderr, _("Illegal filename. Character <%c> not allowed.\n"), *s);
>
> I'd suggest using a comma for the field separator.

I'm not crazy about commas as field separators as there's a good chance that
the map title has some in it and a poorly written script would easily do
something like "cut -d, -f2" and chop it in half. Same goes for using a single
' ' space.

Anything can appear in the map title. Scripts which process "full"
output will have to be written correctly (i.e. break at the first
occurrence) whichever separator is used.

My two preferred choices would be : or |.
":" mimics what we use for deg:min:sec separators and makes some contextual

   sense as a sep.
"|" mimics what we use for vector attribute column separators.

(trying to reuse ideas wherever possible for consistency's sake)

Of those two, | is less likely to appear in titles. Personally, I'd
choose the most common character (probably comma) as a separator so
that bugs in scripts show up sooner rather than later.

We could add : and | to the G_legal_filename() blacklist, even though that
could hurt backwards compatibility if the fn is used to test ,old_map names.
Esp. that, as you point out, calling a map "C:" is problematic on Windows.
(my vote it is to do it after making sure that g.rename doesn't call
G_legal_filename() for the old map name)

I suggest only using G_legal_filename for new files/mapsets/etc. For
existing files, just try to find the named object; if the name was
never legal, then it won't exist.

Also, : isn't valid anywhere in a filename on Windows (or on Windows
filesystems generally; Linux can use FAT and SMB filesystems).

Using "=" for the sep seems inappropriate as
- it looks funny in this context, and
- "map@mapset=Some Title" wouldn't be reasonably used as eval `shell variable`.

If all options are deemed too ugly I guess we could use the old \t standby.

Again, \t is valid in titles.

Actually, I'll change my preference for using a comma as the separator
to a preference for using a space. That's almost certain to occur in
titles (but cannot occur in names), so that will almost guarantee that
any scripts handle the separator correctly.

Having now looked at the g.list code all I can say is yuk. There's about 7
convoluted layers of abstraction and G_spawn()ing to executables with MINGW32
#ifdefs and pipe checking and "more" paging thrown in when AFAICT there really
only needs to be 2 or 3 layers of clean C functions.

general/manage/cmd/list.c
general/manage/lister/{cell|vector} -> $GISBASE/etc/lister/*.exe
general/manage/lib/do_list.c
lib/gis/list.c
lib/gis/ls.c

seem like they could all be combined into just
  general/manage/cmd/list.c + lib/gis/ls.c

AFAICT, the "lister" idea is meant to ensure that neither g.list nor
libgis needs to be updated if a new type is added to etc/element_list.
IOW, the set of possible element types isn't hard-coded anywhere.

But, yes, the whole setup generally is a mess.

My idea was to do a new simple G_list_element_flat() fn in lib/gis/list.c
which just creates a G__ls() list then loops through and fprintf(out,)s the
name, optionally the mapset, and optionally the sep+map title*, then closing
'\n'.

[*] title only makes sense if element is cell or vector, silently set to ""
otherwise.

libgis doesn't (and cannot) depend upon the vector libraries, so a
libgis function can't get the title for a vector map.

I'm not really sure, Martin's patch is fine but I don't see any way to extent
it to append "@mapset" and "[sep]Map Title" as G_ls_format() goes right to the
stream. My idea above kind of suffers from the same problem- It seems a bit
wrong to have the same function be so low level that it calls {G_}ls and so
high level that it evals the result to query the map title with. It's nice to
have a deep lib fn do the list but the dump to stdout has to happen after the
title query and appending.

I would like to avoid having the new fn rely on any of the lister stuff or
pagers if at all possible. Just send result to FILE*out. (maybe I'm just not
good enough in C to write my own custom function to use for the lister arg and
the solution really isn't that hard, just follow the 'g.list -f' example?)

I'd be inclined to suggest that most of G_list_element belongs in
g.list rather than in libgis. However, a number of modules actually
use it, so we're probably stuck with keeping it in libgis until 7.x.

So perhaps we could use this opportunity to simplify this part of the code?
  (about here's where I cry oceanographer not software engineer :wink:

Implementing 'g.list -g' SHOULD just be a 5 minute job..... but it ain't.

"g.list -g" probably is a 5-minute job. "g.list -gf" is likely to be
rather more work, due to the need to accomodate new types in
etc/element_list without re-compilation.

--
Glynn Clements <glynn@gclements.plus.com>

Maris Nartiss wrote:

Just a idea - would it be possible to #IFDEF OS specific illegal chars
for this function? Only problem for such approach could be tar'ing
location/gisdbase and moving to other system (move *nix files with ":"
to windows).
Or - for GRASS 7 - get rid of map_name==file_name at all. Assign some
random file name (gr_0982193) and store map name in it's header
(my!@*&^*!#(*^!*@&raster). Such approach could eliminate illegal map
name problem at all. Drawback - no easy way how to tell which map is
stored in which file w/o GRASS (it can be solved by providing some
"table of contents" file).

For me, I use "locate" and "find" all the time to find what mapset I put
something in. To have to use find+grep would be a pain.

2c
Hamish

Glynn Clements wrote:

> We could add : and | to the G_legal_filename() blacklist, even though
> that could hurt backwards compatibility if the fn is used to
> test ,old_map names. Esp. that, as you point out, calling a map "C:"
> is problematic on Windows. (my vote it is to do it after making sure
> that g.rename doesn't call G_legal_filename() for the old map name)

I suggest only using G_legal_filename for new files/mapsets/etc. For
existing files, just try to find the named object; if the name was
never legal, then it won't exist.

FWIW I just tested this by renaming a vector/ map dir in the file system
to put a space in the name, g.list shows it but g.rename gives an error:

G:vector > g.rename vect="map 1","map:1"
Illegal filename. Character < > not allowed.
WARNING: vector <map 1> not found

[g.list -gf]
Hamish:

> > map2@user1:This map has a title
My two preferred choices would be : or |.
":" mimics what we use for deg:min:sec separators and makes some
    contextual sense as a sep.
"|" mimics what we use for vector attribute column separators.

(trying to reuse ideas wherever possible for consistency's sake)

Glynn:

Actually, I'll change my preference for using a comma as the separator
to a preference for using a space. That's almost certain to occur in
titles (but cannot occur in names), so that will almost guarantee that
any scripts handle the separator correctly.

a space is fine with me...

Hamish

Glynn Clements wrote on 09/12/2007 03:54 PM:

Markus Neteler wrote:
  

It appears that Hamish's implementation based on Glynn's former notes
is much faster:
time sh g.mlist_glynn type=rast map=neteler pat="map*"
real 0m1.200s
user 0m0.749s
sys 0m0.464s

time sh g.mlist_hamish type=rast map=neteler pat="map*"
real 0m0.338s
user 0m0.190s
sys 0m0.052s

Is that the contribution to portability? :slight_smile:
    
...

I've committed an update which has no loops other than the top-level
"for mapset in $mapsets" loop.
  

I can report a significant speed-up up (CVS version from Glynn):

real 0m0.305s
user 0m0.164s
sys 0m0.062s

So "latest Glynn" now corresponds to "Hamish" version.

Thanks,
Markus

PS: Any objections to backport to 6.2 (for 6.2.3)?

------------------
ITC -> dall'1 marzo 2007 Fondazione Bruno Kessler
ITC -> since 1 March 2007 Fondazione Bruno Kessler
------------------