[GRASS-dev] [GRASS GIS] #181: tac command missing in OSX

GRASS GRASS developer
1

#181: tac command missing in OSX
-----------------------+----------------------------------------------------
Reporter: kyngchaos | Owner: grass-dev@lists.osgeo.org
     Type: defect | Status: new
Priority: minor | Milestone: 6.3.1
Component: default | Version: 6.3.0
Keywords: |
-----------------------+----------------------------------------------------
v.in.garmin and v.in.mapgen use the 'tac' utility command. This is not
included in Mac OS X, and possibly some other *nix flavors.

It looks like 'tail -r' achieves the same purpose. Included are patches
for v.in.garmin and v.in.mapgen. It tested OK with v.in.mapgen on a
mapgen file from NOAA. I have no matlab file to test it on, or garmin
file.

Feedback and testing would be helpful.

Also, the BSD manpage I have for tail says the -r flag is an extension to
the POSIX spec for tail, so it's possible it's not completely portable.
Maybe a further test for the flag is needed.

--
Ticket URL: <http://trac.osgeo.org/grass/ticket/181&gt;
GRASS GIS <http://grass.osgeo.org>

2

#181: tac command missing in OSX
------------------------+---------------------------------------------------
  Reporter: kyngchaos | Owner: grass-dev@lists.osgeo.org
      Type: defect | Status: new
  Priority: minor | Milestone: 6.3.1
Component: default | Version: 6.3.0
Resolution: | Keywords:
------------------------+---------------------------------------------------
Comment (by 1gray):

Indeed, `tail -r' isn't portable, as per
[http://www.opengroup.org/onlinepubs/000095399/utilities/tail.html POSIX].
Neither it's allowed by `tail' included with
[http://www.gnu.org/software/coreutils/ GNU Coreutils].

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/181#comment:1&gt;
GRASS GIS <http://grass.osgeo.org>

3

#181: tac command missing in OSX
------------------------+---------------------------------------------------
  Reporter: kyngchaos | Owner: grass-dev@lists.osgeo.org
      Type: defect | Status: new
  Priority: minor | Milestone: 6.3.1
Component: default | Version: 6.3.0
Resolution: | Keywords: v.in.garmin, v.in.mapgen
------------------------+---------------------------------------------------
Changes (by hamish):

  * keywords: => v.in.garmin, v.in.mapgen

Comment:

No "tail -r" in Debian's version of it.

It is a shame that OSX doesn't have tac, it is an elegant solution to the
problem which I am hesitant to replace.

Can the Mac build provide a copy of tac somehow? (or just automatically
apply the 'tail -r' patches with your build..)

Does tac exist in Cygwin / Msys?

I think the NOAA coastline extractor webpage gives you an option to
download in "matlab" format. (??) anyway it is very simple, just 2 or 3
columns of numbers with arc-lines separated by 2 or 3 NaNs.

e.g. 2D:
{{{
NaN NaN
12.34 56.78
23.45 67.89
NaN NaN
}}}

For v.in.garmin I can provide gpstrans and gardump download text files if
needed. (I randomly just noticed that GpsDrive provides a Garmin download
tool called "garble" as well. hmph.)

Hamish

--
Ticket URL: <http://trac.osgeo.org/grass/ticket/181#comment:2&gt;
GRASS GIS <http://grass.osgeo.org>

4

#181: tac command missing in OSX
------------------------+---------------------------------------------------
  Reporter: kyngchaos | Owner: grass-dev@lists.osgeo.org
      Type: defect | Status: new
  Priority: minor | Milestone: 6.3.1
Component: default | Version: 6.3.0
Resolution: | Keywords: v.in.garmin, v.in.mapgen
------------------------+---------------------------------------------------
Comment (by kyngchaos):

Replying to [comment:2 hamish]:
> Can the Mac build provide a copy of tac somehow? (or just automatically
apply the 'tail -r' patches with your build..)

I ''could'' provide a copy or patch it in my binary, but that's just a
packager's solution, and those who build from source would still have to
deal with it somehow. tac is annoying in that it's part of a GNU bundle,
core-utilities, I couldn't find only tac source. And my first
quick-n-dirty attempt to compile it failed.

How about testing for the -r flag, as I mentioned? A non-existent option
on OSX returns `tail: illegal option xxx` plus a usage string with the
available flags. What about other versions of tail? Or can you thing of
a good way to test for the flag? If anything, I could test for
uname=Darwin (tho that's awfully specific).

--
Ticket URL: <http://trac.osgeo.org/grass/ticket/181#comment:3&gt;
GRASS GIS <http://grass.osgeo.org>

5

#181: tac command missing in OSX
------------------------+---------------------------------------------------
  Reporter: kyngchaos | Owner: grass-dev@lists.osgeo.org
      Type: defect | Status: new
  Priority: minor | Milestone: 6.3.1
Component: default | Version: 6.3.0
Resolution: | Keywords: v.in.garmin, v.in.mapgen
------------------------+---------------------------------------------------
Comment (by 1gray):

Replying to [comment:3 kyngchaos]:
> Replying to [comment:2 hamish]:
> > Can the Mac build provide a copy of tac somehow? (or just
automatically apply the 'tail -r' patches with your build..)

> I ''could'' provide a copy or patch it in my binary, but that's just a
packager's solution, and those who build from source would still have to
deal with it somehow. tac is annoying in that it's part of a GNU bundle,
core-utilities, I couldn't find only tac source. And my first
quick-n-dirty attempt to compile it failed.

That's how most bundles work. One couldn't, e. g., easily compile
`r.what` without compiling the rest of GRASS as well.

> How about testing for the -r flag, as I mentioned? A non-existent
option on OSX returns `tail: illegal option xxx` plus a usage string with
the available flags. What about other versions of tail? Or can you thing
of a good way to test for the flag? If anything, I could test for
uname=Darwin (tho that's awfully specific).

I'd recommend to check for the existence of `tac` instead, like
(untested):
{{{
search_for_executable () {
     echo "$PATH" | tr : \\n | while read _search_for_executable_dir ; do
         if [ -x "$_search_for_executable_dir/$1" ]; then
             echo "$_search_for_executable_dir/$1"
             break
         fi
     done
}

if [ -n "$TAC" ]; then
     TAC=tac
else
     TAC=$(search_for_executable "tac")
fi

if [ -z "$TAC" ]; then
     ## no `tac' found, fall back to `tail -r' instead
     TAC="tail -r"
fi

...

$TAC arguments...
}}}

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/181#comment:4&gt;
GRASS GIS <http://grass.osgeo.org>

6

#181: tac command missing in OSX
------------------------+---------------------------------------------------
  Reporter: kyngchaos | Owner: grass-dev@lists.osgeo.org
      Type: defect | Status: new
  Priority: minor | Milestone: 6.3.1
Component: default | Version: 6.3.0
Resolution: | Keywords: v.in.garmin, v.in.mapgen
------------------------+---------------------------------------------------
Comment (by glynn):

> > How about testing for the -r flag, as I mentioned?
>
> I'd recommend to check for the existence of `tac` instead,

Agreed.

> like (untested):

I suggest either using "type", or just running e.g. "tac < /dev/null" and
checking for an error. Scanning the path is problematic; it doesn't allow
for shell built-ins or aliases, may not allow for extensions on Windows,
etc.

That still doesn't solve the issue of what to do if the system has neither
tac nor "tail -r", neither of which are mandated by POSIX/XPG/SUS.

Off the top of my head, the only thing that I can think of is to prefix
each line with the line number using e.g. awk, use "sort -r", then strip
the line numbers.

--
Ticket URL: <http://trac.osgeo.org/grass/ticket/181#comment:5&gt;
GRASS GIS <http://grass.osgeo.org>

7

#181: tac command missing in OSX
------------------------+---------------------------------------------------
  Reporter: kyngchaos | Owner: grass-dev@lists.osgeo.org
      Type: defect | Status: new
  Priority: minor | Milestone: 6.3.1
Component: default | Version: 6.3.0
Resolution: | Keywords: v.in.garmin, v.in.mapgen
------------------------+---------------------------------------------------
Comment (by kyngchaos):

Replying to [comment:4 1gray]:
> I'd recommend to check for the existence of `tac` instead, like
(untested):

My patch does this with ''which tac'', a simple technique used used in
many GRASS scripts. What I meant was, if tac isn't found, and tail is
found, somehow check if tail has the ''-r'' flag.

But, this is really just an immediate fix to take care of the OSX case
(and any other *nix without tac) where tail -r does exist. As Glynn
points out, neither tac nor tail -r are in POSIX/SUS standards. A more
portable fix would be nice in the long term. I wonder if some sort of
requirement in GRASS 7 development would be a good idea to stick to
POSIX/SUS commands?

--
Ticket URL: <http://trac.osgeo.org/grass/ticket/181#comment:6&gt;
GRASS GIS <http://grass.osgeo.org>

8

#181: tac command missing in OSX
------------------------+---------------------------------------------------
  Reporter: kyngchaos | Owner: grass-dev@lists.osgeo.org
      Type: defect | Status: new
  Priority: minor | Milestone: 6.3.1
Component: default | Version: 6.3.0
Resolution: | Keywords: v.in.garmin, v.in.mapgen
------------------------+---------------------------------------------------
Comment (by glynn):

Replying to [comment:6 kyngchaos]:

> > I'd recommend to check for the existence of `tac` instead, like
(untested):
>
> My patch does this with ''which tac'', a simple technique used used in
many GRASS scripts.

FWIW, "which" isn't mandated by POSIX, while "type" is. Also, "which"
doesn't work with built-ins or aliases (not that tac or tail are likely to
be either).

--
Ticket URL: <http://trac.osgeo.org/grass/ticket/181#comment:7&gt;
GRASS GIS <http://grass.osgeo.org>

9

GRASS GIS wrote:

I wonder if some sort of
requirement in GRASS 7 development would be a good idea to stick to
POSIX/SUS commands?

Well, we normally try to follow that rule already, but we don't all
have an encyclopaedic knowledge of the standards, so the use of
non-portable features often occurs, and may go unnoticed for some
time. We don't have the resource to run extensive audits.

Also, some scripts may require utilities beyond anything which is
specified by POSIX. E.g. there isn't any POSIX-mandated equivalent of
wget or curl.

In any case, for 7.x we should be moving to Python for scripts. That
should largely eliminate the need to call external utilities. Simple
text processing, as is normally performed with cat/sort/grep/sed etc,
can be implemented in a few lines of code using only core functions.
More complex utilities often have equivalent modules (e.g. httplib
instead of wget/curl).

--
Glynn Clements <glynn@gclements.plus.com>

10

#181: tac command missing in OSX
------------------------+---------------------------------------------------
  Reporter: kyngchaos | Owner: grass-dev@lists.osgeo.org
      Type: defect | Status: new
  Priority: minor | Milestone: 6.3.1
Component: default | Version: 6.3.0
Resolution: | Keywords: v.in.garmin, v.in.mapgen
------------------------+---------------------------------------------------
Comment (by kyngchaos):

Ah. Looks like type is a shell builtin on OSX (and a script in
/usr/bin/type to execute the builtin).

So, `type -t tac` and `type -t tail` to check for existence looks like the
equivalent to `which` (ie no output if it's not found).

For checking for the -r flag for tail (for the immediate problem in OSX)
-- maybe, similar to your tac example, `tail -r < /dev/null`? Tests OK
here (missing tail and an illegal flag both return errors).

--
Ticket URL: <http://trac.osgeo.org/grass/ticket/181#comment:8&gt;
GRASS GIS <http://grass.osgeo.org>

11

#181: tac command missing in OSX
------------------------+---------------------------------------------------
  Reporter: kyngchaos | Owner: grass-dev@lists.osgeo.org
      Type: defect | Status: new
  Priority: minor | Milestone: 6.3.1
Component: default | Version: 6.3.0
Resolution: | Keywords: v.in.garmin, v.in.mapgen
------------------------+---------------------------------------------------
Comment (by hamish):

William:
> tac is annoying in that it's part of a GNU bundle, core-utilities, I
> couldn't find only tac source. And my first quick-n-dirty attempt to
> compile it failed.

1gray:
> That's how most bundles work. One couldn't, e. g., easily compile r.what
> without compiling the rest of GRASS as well.

it's not like tac has many dependencies:
{{{
$ ldd `which tac`
         linux-gate.so.1 => (0xffffe000)
         libc.so.6 => /lib/tls/i686/cmov/libc.so.6 (0xb7e0b000)
         /lib/ld-linux.so.2 (0xb7f5a000)
}}}

I assume it's a trivial C program. What's the complile error?

While not POSIX, I think tac will be widely available. I would still like
to hear from Cygwin/MSys/other platforms.. to me the fact that it hasn't
been moved out of Debian's hardcore minimalist coreutils is a good
anecdotal endorsement for wide availability.

fwiw, on Debian/etch:
{{{
$ tail -r foo
tail: invalid option -- r
Try `tail --help' for more information.
$ echo $?
1
}}}

A not so nice for the user idea is to abort with an error if tac is not
found.

Glynn:
> Off the top of my head, the only thing that I can think of is to prefix
> each line with the line number using e.g. awk, use "sort -r", then strip
> the line numbers.

awk would be a great solution as this is fed directly into that. But I
have no idea how to write that.

Here is where it is needed- create lines for 'v.in.ascii format=standard':
{{{
     # add vertex counts
     cat "${TMP}.gpst" | sed -e '1d' | tac | awk 'BEGIN { FS="\t" ; R=0 } \
         $1=="T" { printf(" %.7f %.7f\n", $4, $3) ; ++R } ; \
         $1=="" { printf("L %d 1\n", R) ; R=0 } END {;}' | tac >
"${TMP}.base"
}}}

another thing for the future is to remove the vertex count requirement in
the vector ascii file. I suppose it adds some level of error checking, but
the computer can count.

William:
> I wonder if some sort of requirement in GRASS 7 development would be a
good
> idea to stick to POSIX/SUS commands?

that is the current goal, but I see little reason to go nuts trying to be
very strict about that, e.g. for something like `which` that may not be
POSIX but is, literally, everywhere. ie the overwhelming common use
exemption.

for GRASS 7 the goal AFAIU is to replace shell scripts with more portable
Python scripts.

Hamish

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/181#comment:9&gt;
GRASS GIS <http://grass.osgeo.org>

12

#181: tac command missing in OSX
------------------------+---------------------------------------------------
  Reporter: kyngchaos | Owner: grass-dev@lists.osgeo.org
      Type: defect | Status: new
  Priority: minor | Milestone: 6.3.1
Component: default | Version: 6.3.0
Resolution: | Keywords: v.in.garmin, v.in.mapgen
------------------------+---------------------------------------------------
Comment (by glynn):

Replying to [comment:9 hamish]:

> While not POSIX, I think tac will be widely available. I would still
like to hear from Cygwin/MSys/other platforms.

Cygwin has it. My MSys installation doesn't, but it's quite out of date
(it has separate textutils and fileutils; AFAICT, newer versions have
coreutils, which probably has it). I suspect that non-GNU systems (e.g.
Solaris) probably won't have it.

> Glynn:
> > Off the top of my head, the only thing that I can think of is to
prefix
> > each line with the line number using e.g. awk, use "sort -r", then
strip
> > the line numbers.
>
> awk would be a great solution as this is fed directly into that. But I
have no idea how to write that.

Replace:
{{{
... | tac | ...
}}}
with:
{{{
... | awk '{print NR,$0}' | sort -nr | sed 's/^[0-9]* //' | ...
}}}

> > I wonder if some sort of requirement in GRASS 7 development would be a
good
> > idea to stick to POSIX/SUS commands?
>
> that is the current goal, but I see little reason to go nuts trying to
be very strict about that, e.g. for something like `which` that may not be
POSIX but is, literally, everywhere. ie the overwhelming common use
exemption.

I don't see any reason to use "which" when "type" is mandated by POSIX.

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/181#comment:10&gt;
GRASS GIS <http://grass.osgeo.org>

13

#181: tac command missing in OSX
------------------------+---------------------------------------------------
  Reporter: kyngchaos | Owner: grass-dev@lists.osgeo.org
      Type: defect | Status: new
  Priority: minor | Milestone: 6.3.1
Component: default | Version: 6.3.0
Resolution: | Keywords: v.in.garmin, v.in.mapgen
------------------------+---------------------------------------------------
Comment (by 1gray):

Replying to [comment:10 glynn]:
> Replying to [comment:9 hamish]:

![...]

> > awk would be a great solution as this is fed directly into that. But
> > I have no idea how to write that.

> Replace:
> `... | tac | ...`
> with:
> `... | awk '{print NR,$0}' | sort -nr | sed 's/^[0-9]* //' | ...`

I see no reason to use Awk to number lines, since there's
[http://www.opengroup.org/onlinepubs/000095399/utilities/nl.html nl].
Neither I see a reason to use `sort`, since Awk has arrays:
{{{
awk_tac () {
     awk '1 { last = NR; line[last] = $0; } END { for (i = last; i > 0;
i--) { print line[i]; } }'
}

if [ -n "$TAC" ] ; then
     ## do nothing
elif type tac ; then
     TAC=tac
else
     TAC=awk_tac
fi

... | $TAC | ...
}}}

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/181#comment:11&gt;
GRASS GIS <http://grass.osgeo.org>

14

#181: tac command missing in OSX
------------------------+---------------------------------------------------
  Reporter: kyngchaos | Owner: grass-dev@lists.osgeo.org
      Type: defect | Status: new
  Priority: minor | Milestone: 6.3.1
Component: default | Version: 6.3.0
Resolution: | Keywords: v.in.garmin, v.in.mapgen
------------------------+---------------------------------------------------
Comment (by hamish):

Now we are getting somewhere. One little bug:

{{{
if [ -n "$TAC" ] ; then
     ## do nothing
elif type tac ; then
     TAC=tac
else
     TAC=awk_tac
fi
}}}

`##do nothing` with no command before it is a syntax error. you would need
to run a noop like "true". but it is better to rewrite the logic of the
test so that there is no empty case.

also output of `type tac` would need to be redirected to /dev/null.

Hamish

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/181#comment:12&gt;
GRASS GIS <http://grass.osgeo.org>

15

#181: tac command missing in OSX
------------------------+---------------------------------------------------
  Reporter: kyngchaos | Owner: grass-dev@lists.osgeo.org
      Type: defect | Status: new
  Priority: minor | Milestone: 6.3.1
Component: default | Version: 6.3.0
Resolution: | Keywords: v.in.garmin, v.in.mapgen
------------------------+---------------------------------------------------
Comment (by 1gray):

Replying to [comment:12 hamish]:

> Now we are getting somewhere. One little bug:

![...]

> `##do nothing` with no command before it is a syntax error. you would
> need to run a noop like "true".

Indeed.

> but it is better to rewrite the logic of the test so that there is no
> empty case.

> also output of `type tac` would need to be redirected to /dev/null.

Yes. And `-t` should be given as well.

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/181#comment:13&gt;
GRASS GIS <http://grass.osgeo.org>

16

#181: tac command missing in OSX
------------------------+---------------------------------------------------
  Reporter: kyngchaos | Owner: grass-dev@lists.osgeo.org
      Type: defect | Status: new
  Priority: minor | Milestone: 6.3.1
Component: default | Version: 6.3.0
Resolution: | Keywords: v.in.garmin, v.in.mapgen
------------------------+---------------------------------------------------
Comment (by 1gray):

Replying to [comment:9 hamish]:

![...]

>> That's how most bundles work. One couldn't, e. g., easily compile
>> `r.what` without compiling the rest of GRASS as well.

> it's not like tac has many dependencies:
{{{
> $ ldd `which tac`
> linux-gate.so.1 => (0xffffe000)
> libc.so.6 => /lib/tls/i686/cmov/libc.so.6 (0xb7e0b000)
> /lib/ld-linux.so.2 (0xb7f5a000)
}}}

> I assume it's a trivial C program.

Unfortunately, the above doesn't show the static libraries linked in
(and Coreutils probably comes with one.) Furthermore, IIRC, Coreutils
uses [http://www.gnu.org/software/gnulib/ Gnulib], which is ''not'' a
library.

I'm not saying that making GRASS have a copy of `tac` is impossible.
However, the amount of work needed to ''maintain'' it seems to be
unreasonable.

> What's the complile error?

![...]

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/181#comment:14&gt;
GRASS GIS <http://grass.osgeo.org>

17

#181: tac command missing in OSX
------------------------+---------------------------------------------------
  Reporter: kyngchaos | Owner: grass-dev@lists.osgeo.org
      Type: defect | Status: new
  Priority: minor | Milestone: 6.3.1
Component: default | Version: 6.3.0
Resolution: | Keywords: v.in.garmin, v.in.mapgen
  Platform: MacOSX | Cpu: Unspecified
------------------------+---------------------------------------------------
Changes (by neteler):

  * platform: => MacOSX
  * cpu: => Unspecified

--
Ticket URL: <http://trac.osgeo.org/grass/ticket/181#comment:15&gt;
GRASS GIS <http://grass.osgeo.org>

18

#181: tac command missing in OSX
------------------------+---------------------------------------------------
  Reporter: kyngchaos | Owner: grass-dev@lists.osgeo.org
      Type: defect | Status: new
  Priority: minor | Milestone: 6.4.0
Component: default | Version: 6.3.0
Resolution: | Keywords: v.in.garmin, v.in.mapgen
  Platform: MacOSX | Cpu: Unspecified
------------------------+---------------------------------------------------
Changes (by hamish):

  * milestone: 6.3.1 => 6.4.0

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/181#comment:16&gt;
GRASS GIS <http://grass.osgeo.org>

19

#181: tac command missing in OSX
--------------------------------------+-------------------------------------
Reporter: kyngchaos | Owner: grass-dev@…
     Type: defect | Status: new
Priority: minor | Milestone: 6.4.0
Component: default | Version: 6.3.0
Keywords: v.in.garmin, v.in.mapgen | Platform: MacOSX
      Cpu: Unspecified |
--------------------------------------+-------------------------------------

Comment(by hamish):

I'd suggest adding a g.tac script using the above awk method in $(ETC),
along the same lines as $(ETC)/echo, `grocat`, and `run`; but looking at
the purpose of $(ETC)/grocat (grass65/lib/gtcltk/grocat.c), perhaps a -r
flag could simply be added there for this?

{{{
  * MODULE: grocat
  * AUTHOR(S): Paul Kelly
  * PURPOSE: Copies stdin to stdout in line-buffered mode until end
  * of file is received.
}}}

(see also `lib/init/echo.c`, which "replaces the standard UNIX echo which
varies from machine to machine")

I assume for grass 7 this is a non-issue, as v.in.mapgen (eg) methods have
already been rewritten to do all this in python.

Hamish

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/181#comment:17&gt;
GRASS GIS <http://grass.osgeo.org>

20

#181: tac command missing in OSX
--------------------------------------------------+-------------------------
Reporter: kyngchaos | Owner: hamish
     Type: defect | Status: assigned
Priority: minor | Milestone: 6.4.1
Component: Default | Version: 6.3.0
Keywords: v.in.garmin, v.in.mapgen, v.in.lines | Platform: MacOSX
      Cpu: Unspecified |
--------------------------------------------------+-------------------------
Changes (by hamish):

* cc: grass-dev@… (added)
  * keywords: v.in.garmin, v.in.mapgen => v.in.garmin, v.in.mapgen,
               v.in.lines
  * status: new => assigned
  * owner: grass-dev@… => hamish
  * milestone: 6.4.0 => 6.4.1

Comment:

As this is limited to a couple of shell scripts, I'll take the easy way
out and use the awk replacement solution.

Hamish

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/181#comment:18&gt;
GRASS GIS <http://grass.osgeo.org>