[GRASS5] Opinion wanted - null data

GRASS GRASS developer
1

Hello all

As you know, I have been looking at the issue of removing the feature of
"0 is no data", and the situation is currently at a point where I'm not
sure what is the best thing to do.

First, there have been 8 out of 9 users request that the "0 is no data"
feature disappear in 5.0. Originally I was planning to hide the null
file from the users for 5.0 since we are close to a release and then
remove it for 5.1. However, the more I look at the code and think about
it, the more I feel that there will not be an elegant way to eliminate
the null file for 5.1. The best I can think of is some way to detect
whether a data file is pre 5.1 and then automatically convert it when
grass reads it. Even though this sounds simple, we would have to keep
the code that interprets the null file around for quite a while in order
to convert these files.

The best solution would be to make the change now. That way the users
are happy and the code for reading the old data is kept in a script, or
at least it would be separate from Grass, since the users would run the
script themselves. Of course, there is a big problem with this option.
This change does not seem to be trivial (especially since it changes the
gis library), and would probably leave the code in an unstable state for
some period of time. Thus, there would be a slight delay in the release
of Grass 5.0.

On top of all this is the fact that I have run out of time. Next week I
am moving to Canada and therefore, I will be off-line for an unknown
amount of time. It could be a couple of weeks, or it could be over a
month. I just don't know. Thus, it is unlikely I will be able to
contribute anything to this change in the near future.

One bit of good news is that the change, in theory, is relatively simple
if we also eliminate the multiple byte lengths for storing CELL data. We
simply save the data as it is in memory and remove references to the
null file and multiple byte length CELL data. However, I'm not sure if
it will be a simple change practically. I have not had time to do a full
investigation of the code, but so far, I have not found anything that
would indicate a major problem (which of course means nothing :slight_smile: ).

Thus the situation is:

1. We have a request from our users to eliminate the null file for 5.0

2. The change would involve library code changes but we are close to
   a release

3. The developer that would most likely make the changes (me) is
   unavailable for an unknown time

The options I see that we have are (in no particular order):

1. Release 5.0 stable without any null changes and indicate to users
   that an investigation of the code revealed that a major change was
   required and thus the change is delayed until 5.1 rather than
   risking the stability of 5.0

2. Make the change and prolong the release by the amount of time for me
   to get back on-line and for coding and testing

3. Another developer volunteers to make the changes and the release is
   delayed for only a minimum amount of time (coding and testing)

4. Another developer volunteers to hide the null file and we do an
   automatic change for 5.1

Frankly, I'm at a loss as to which option is best. I think I prefer
option 3 since it would please the users, but I hate passing off work to
others.

Anyway, I just thought that I should raise this issue with the
developers. What do people think of these options? Is there anybody who
wants to volunteer for 3 or 4?

Thanks for listening.

--
Sincerely,

Jazzman (a.k.a. Justin Hickey) e-mail: jhickey@hpcc.nectec.or.th
High Performance Computing Center
National Electronics and Computer Technology Center (NECTEC)
Bangkok, Thailand

People who think they know everything are very irritating to those
of us who do. ---Anonymous

Jazz and Trek Rule!!!

From neteler Tue Jul 3 12:35:54 2001
Return-Path: <neteler>
Received: by hgeo02.geog.uni-hannover.de (SMI-8.6/SMI-SVR4)
  id MAA13408; Tue, 3 Jul 2001 12:35:54 +0100
Date: Tue, 3 Jul 2001 12:35:36 +0100
From: Markus Neteler <neteler@geog.uni-hannover.de>
To: grass5 developers list <grass5@geog.uni-hannover.de>
Message-ID: <20010703123535.G7244@hgeo02.geog.uni-hannover.de>
Mail-Followup-To: grass5 developers list <grass5@geog.uni-hannover.de>
References: <20010624095703.A10119@hgeo02.geog.uni-hannover.de> <15157.50441.593830.596566@cerise.nosuchdomain.co.uk> <20010624174922.A12235@hgeo02.geog.uni-hannover.de> <15158.8823.650627.515067@cerise.nosuchdomain.co.uk> <20010625083219.J29754@hgeo02.geog.uni-hannover.de> <15159.19652.187050.234424@cerise.nosuchdomain.co.uk> <20010702225024.A9260@hgeo02.geog.uni-hannover.de> <15169.21789.291611.895541@cerise.nosuchdomain.co.uk> <20010703100656.D7244@hgeo02.geog.uni-hannover.de> <15169.36229.59095.722307@cerise.nosuchdomain.co.uk>
Mime-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
User-Agent: Mutt/1.2.5i
In-Reply-To: <15169.36229.59095.722307@cerise.nosuchdomain.co.uk>; from glynn.clements@virgin.net on Tue, Jul 03, 2001 at 10:16:53AM +0100
Subject: [GRASS5] LAPACK: Numercial stuff in GRASS
Sender: grass5-admin@geog.uni-hannover.de
Errors-To: grass5-admin@geog.uni-hannover.de
X-BeenThere: grass5@geog.uni-hannover.de
X-Mailman-Version: 2.0.5
Precedence: bulk
List-Help: <mailto:grass5-request@geog.uni-hannover.de?subject=help>
List-Post: <mailto:grass5@geog.uni-hannover.de>
List-Subscribe: <http://www.geog.uni-hannover.de/mailman/listinfo/grass5&gt;,
  <mailto:grass5-request@geog.uni-hannover.de?subject=subscribe>
List-Id: GRASS 5 Developers mailing list <grass5.geog.uni-hannover.de>
List-Unsubscribe: <http://www.geog.uni-hannover.de/mailman/listinfo/grass5&gt;,
  <mailto:grass5-request@geog.uni-hannover.de?subject=unsubscribe>
List-Archive: <http://www.geog.uni-hannover.de/pipermail/grass5/&gt;
Status: O
Content-Length: 329
Lines: 13

Hi,

for those being interested in LAPACK/gmath programming, I
have make a PDF file from the online LAPACK documentation:
http://www.geog.uni-hannover.de/users/neteler/tmp/lapack/
LAPACK Users' Guide Third Edition 1999

The GRAS "gmath" lib is a growing wrapper to LAPACK to provide
stable numerical functions.

Cheers

Markus

From neteler Tue Jul 3 16:58:51 2001
Return-Path: <neteler>
Received: by hgeo02.geog.uni-hannover.de (SMI-8.6/SMI-SVR4)
  id QAA15056; Tue, 3 Jul 2001 16:58:50 +0100
Date: Tue, 3 Jul 2001 16:58:50 +0100
From: Markus Neteler <neteler@geog.uni-hannover.de>
To: John Huddleston <jhudd@lamar.colostate.edu>
Cc: Andreas.Lange@Rhein-Main.de, WinGRASS <wingrass@geog.uni-hannover.de>,
        grass5 developers list <grass5@geog.uni-hannover.de>,
        GRASS Verein <Grass-Verein@intevation.de>, grasslist@baylor.edu
Message-ID: <20010703165850.B13718@hgeo02.geog.uni-hannover.de>
Mail-Followup-To: John Huddleston <jhudd@lamar.colostate.edu>,
  Andreas.Lange@Rhein-Main.de,
  WinGRASS <wingrass@geog.uni-hannover.de>,
  grass5 developers list <grass5@geog.uni-hannover.de>,
  GRASS Verein <Grass-Verein@intevation.de>, grasslist@baylor.edu
References: <3B40AC24.B29ABBC@Rhein-Main.de> <001301c103c4$7794d340$b4345281@kfgwflow>
Mime-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
User-Agent: Mutt/1.2.5i
In-Reply-To: <001301c103c4$7794d340$b4345281@kfgwflow>; from jhudd@lamar.colostate.edu on Tue, Jul 03, 2001 at 07:31:27AM -0600
Subject: [GRASS5] Re: [winGRASS] cygwin and winGRASS status update
Sender: grass5-admin@geog.uni-hannover.de
Errors-To: grass5-admin@geog.uni-hannover.de
X-BeenThere: grass5@geog.uni-hannover.de
X-Mailman-Version: 2.0.5
Precedence: bulk
List-Help: <mailto:grass5-request@geog.uni-hannover.de?subject=help>
List-Post: <mailto:grass5@geog.uni-hannover.de>
List-Subscribe: <http://www.geog.uni-hannover.de/mailman/listinfo/grass5&gt;,
  <mailto:grass5-request@geog.uni-hannover.de?subject=subscribe>
List-Id: GRASS 5 Developers mailing list <grass5.geog.uni-hannover.de>
List-Unsubscribe: <http://www.geog.uni-hannover.de/mailman/listinfo/grass5&gt;,
  <mailto:grass5-request@geog.uni-hannover.de?subject=unsubscribe>
List-Archive: <http://www.geog.uni-hannover.de/pipermail/grass5/&gt;
Status: O
Content-Length: 1514
Lines: 38

Hi John, hi all interested in winGRASS

(cc to grass5, gav)

[Note: the current problem of winGRASS is the lack of a generic
GRASS windows driver. As using a Windows-Xserver at time, the installation
is pretty complicated. If a generic GRASS windows driver would be available
GRASS installation would become a snap]

On Tue, Jul 03, 2001 at 07:31:27AM -0600, John Huddleston wrote:
[...]

Once upon a time, I wrote to Markus about getting a small
team together for 3-5 days to hammer out the issues together.
I still think this is a good idea. If there are funds, or if we could
get a sponsor for the WinGRASS version, this would help.
We could cut a WinGRASS CD, charge $79 (US) and help
defray the costs.

I still like this idea! Even using existing GPL'ed software to
extract their driver and modify for GRASS might be a way (gnuplot or
similar).

Ideas to support a winGRASS developers meeting are:

- find a generous sponsor (or more) to fund a GRASS monitor driver for
   windows
- start to presale a WinGRASS CDROM. I am sure that *many* people would
   purchase such a CDROM, but I am not sure how many could be sold in advance
   to collect money for a winGRASS developers meeting.

The German "GRASS Anwender-Vereinigung e.V." (GAV) is a forum to collect
fundings and to donate to such projects. But at time the GAV is far away
to sponsor such a meeting.

Probably there is a company or institution to spend a part of their "ESRI
tax" or whatever to push this idea?

Markus Neteler

2

On Tue, 3 Jul 2001, Justin Hickey wrote:

As you know, I have been looking at the issue of removing the feature of
"0 is no data", and the situation is currently at a point where I'm not
sure what is the best thing to do.

1. Release 5.0 stable without any null changes and indicate to users
   that an investigation of the code revealed that a major change was
   required and thus the change is delayed until 5.1 rather than
   risking the stability of 5.0

I go for 1. After looking recently at the r.in.ascii issue, and reading
other code (s.kcv), I would be very apprehensive about changing something
that works.

Roger

--
Roger Bivand
Economic Geography Section, Department of Economics, Norwegian School of
Economics and Business Administration, Breiviksveien 40, N-5045 Bergen,
Norway. voice: +47 55 95 93 55; fax +47 55 95 93 93
e-mail: Roger.Bivand@nhh.no
and: Department of Geography and Regional Development, University of
Gdansk, al. Mar. J. Pilsudskiego 46, PL-81 378 Gdynia, Poland.

From neteler Fri Jul 6 15:22:30 2001
Return-Path: <neteler>
Received: by hgeo02.geog.uni-hannover.de (SMI-8.6/SMI-SVR4)
  id PAA01941; Fri, 6 Jul 2001 15:22:30 +0100
Date: Fri, 6 Jul 2001 15:22:30 +0100
From: Markus Neteler <neteler@geog.uni-hannover.de>
To: grass5@geog.uni-hannover.de
Subject: Re: [GRASS5] Re: [GRASSLIST:2065] Re: Compiling grass5.0.0pre1 under Solaris 8???
Message-ID: <20010706152230.B1798@hgeo02.geog.uni-hannover.de>
Mail-Followup-To: grass5@geog.uni-hannover.de
References: <15171.51518.626689.812573@cerise.nosuchdomain.co.uk> <Pine.LNX.4.33.0107050750540.2341-100000@boyd.geog.mcgill.ca> <15173.16982.799595.751427@cerise.nosuchdomain.co.uk>
Mime-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
User-Agent: Mutt/1.2.5i
In-Reply-To: <15173.16982.799595.751427@cerise.nosuchdomain.co.uk>; from glynn.clements@virgin.net on Fri, Jul 06, 2001 at 05:45:10AM +0100
Sender: grass5-admin@geog.uni-hannover.de
Errors-To: grass5-admin@geog.uni-hannover.de
X-BeenThere: grass5@geog.uni-hannover.de
X-Mailman-Version: 2.0.5
Precedence: bulk
List-Help: <mailto:grass5-request@geog.uni-hannover.de?subject=help>
List-Post: <mailto:grass5@geog.uni-hannover.de>
List-Subscribe: <http://www.geog.uni-hannover.de/mailman/listinfo/grass5&gt;,
  <mailto:grass5-request@geog.uni-hannover.de?subject=subscribe>
List-Id: GRASS 5 Developers mailing list <grass5.geog.uni-hannover.de>
List-Unsubscribe: <http://www.geog.uni-hannover.de/mailman/listinfo/grass5&gt;,
  <mailto:grass5-request@geog.uni-hannover.de?subject=unsubscribe>
List-Archive: <http://www.geog.uni-hannover.de/pipermail/grass5/&gt;
Status: O
Content-Length: 3364
Lines: 72

On Fri, Jul 06, 2001 at 05:45:10AM +0100, Glynn Clements wrote:

[This started on GRASSLIST, but should be brought to the attention of
the developers.]

Lawrence Houston wrote:

> > > Having troubles compiling grass5.0.0pre1 under Solaris 8 (Intel) using
> > > SUN's Companion (04/01). LIBGIS fails within error.c:
> > >
> > > --------------------------------------------------------------------------------
> > > gcc -g -O2 -I/opt/sfw/include -I/opt/grass5/grass5.0.0pre1/src/include -I/usr/include -c error.c -o OBJ.i386-pc-solaris2.8/error.o
> > > error.c: In function `G_fatal_error':
> > > error.c:72: `__builtin_va_alist' undeclared (first use in this function)
> > > error.c:72: (Each undeclared identifier is reported only once
> > > error.c:72: for each function it appears in.)
> > > error.c: In function `G_warning':
> > > error.c:87: `__builtin_va_alist' undeclared (first use in this function)
> > > *** Error code 1
> > > make: Fatal error: Command failed for target `OBJ.i386-pc-solaris2.8/error.o'
> > > --------------------------------------------------------------------------------
> >
> > Apparently this can occur when gcc tries to use vendor-supplied
> > headers in preference to its own (typically in the directory
> > /usr/lib/gcc-lib/<platform>/<version>/include).
> >
> > Try removing all occurrences of -I/usr/include from the file
> > src/CMD/head/head.<platform>
>
> Glynn:
>
> Thanks very much since removing all occurrences of -I/usr/include from
> head.i386-pc-solaris2.8 has done the trick, I now have grass5.0.0pre1
> built under Solaris 8 (Intel)! There were also warnings about conflicts
> between the System TERMLIB and NCURSES, although I those may NOT be
> critical since full-screen cursor movement appears to "work"?
>
> I thought using the gcc (2.95.2) included in SUN's Companion would be an
> quick way to build GRASS under Solaris 8, did NOT anticipate conflicts
> with the existing System "stuff"???

It appears that the configure script needs to avoid using
-I/usr/include when checking for header file locations.

AFAIK, the compiler ought to search that directory automatically (and
if it doesn't, the user should be able to use the "--with-includes="
switch). It seems that explicitly adding -I/usr/include prevents
compiler-specific headers (e.g. stdarg.h) from being used correctly.

Currently, the header file checks loop over a list of directories
(including /usr/include), performing each test with "-I$dir". Any such
test needs to first be performed without any "-I" switch, rather than
with "-I/usr/include".

NB: my original advice was based upon comments made on the xemacs-beta
list. That related to someone who explicitly added /usr/include via
"--site-includes="; XEmacs' configure script (unlike GRASS') doesn't
add "-I/usr/include" automatically.

Looking at some configure.in scripts from other packages, most of them
don't go to great lengths to find include directories automatically.
Mostly they require the user to explicitly specify non-standard
include directories.

Glynn,

I suggest to change the configure[.in] as suggested and give it a try.
I can test 3 machines here (Solaris, CRAY, Linux).

Will you going to change if there are no objections?

Markus

From neteler Fri Jul 6 15:48:47 2001
Return-Path: <neteler>
Received: by hgeo02.geog.uni-hannover.de (SMI-8.6/SMI-SVR4)
  id PAA02106; Fri, 6 Jul 2001 15:48:47 +0100
Date: Fri, 6 Jul 2001 15:48:46 +0100
From: Markus Neteler <neteler@geog.uni-hannover.de>
To: grass5 developers list <grass5@geog.uni-hannover.de>
Subject: Re: [GRASS5] shift bug in r.in.ascii/r.out.ascii
Message-ID: <20010706154846.F1798@hgeo02.geog.uni-hannover.de>
Mail-Followup-To: grass5 developers list <grass5@geog.uni-hannover.de>
References: <20010629151210.L16481@hgeo02.geog.uni-hannover.de> <Pine.LNX.4.21.0106291928330.2921-100000@reclus.nhh.no> <15164.50234.521459.627900@cerise.nosuchdomain.co.uk> <20010702101622.N16481@hgeo02.geog.uni-hannover.de> <15168.58547.699622.256027@cerise.nosuchdomain.co.uk> <3B44ECB4.FF5EF776@unity.ncsu.edu>
Mime-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
User-Agent: Mutt/1.2.5i
In-Reply-To: <3B44ECB4.FF5EF776@unity.ncsu.edu>; from hmitaso@unity.ncsu.edu on Thu, Jul 05, 2001 at 05:39:48PM -0500
Sender: grass5-admin@geog.uni-hannover.de
Errors-To: grass5-admin@geog.uni-hannover.de
X-BeenThere: grass5@geog.uni-hannover.de
X-Mailman-Version: 2.0.5
Precedence: bulk
List-Help: <mailto:grass5-request@geog.uni-hannover.de?subject=help>
List-Post: <mailto:grass5@geog.uni-hannover.de>
List-Subscribe: <http://www.geog.uni-hannover.de/mailman/listinfo/grass5&gt;,
  <mailto:grass5-request@geog.uni-hannover.de?subject=subscribe>
List-Id: GRASS 5 Developers mailing list <grass5.geog.uni-hannover.de>
List-Unsubscribe: <http://www.geog.uni-hannover.de/mailman/listinfo/grass5&gt;,
  <mailto:grass5-request@geog.uni-hannover.de?subject=unsubscribe>
List-Archive: <http://www.geog.uni-hannover.de/pipermail/grass5/&gt;
Status: O
Content-Length: 3628
Lines: 98

On Thu, Jul 05, 2001 at 05:39:48PM -0500, Helena wrote:

Has anybody tested whether r.in.arc and r.out.arc does not have the same problem ?
At least one version of r.out.arc was written based on r.out.ascii so the bug might
have
been transfered there too.

Helena

I have just run the same test - it seems to be working alright.

But... even here we have the precision problem:

ncols 140
nrows 144
xllcorner 3568200
yllcorner 5765462.5
cellsize 12.500000
NODATA_value -9999
126.870003 126.550003 126.25 125.93 125.599998
       ^^^ ^^^^ ^^^^^

Same story as with r.stats, r.to.sites, r.in./out.ascii, r.in./out.arc.

While the NULL thing is a severe change probably interesting for 5.1, the
precision issue might be solved now.

Regards

Markus

Glynn Clements wrote:

> Markus Neteler wrote:
>
> > > > Looks to me as if fseek is moving one byte too far back in the buffer on
> > > > line 142 in r.in.ascii/cmd/gethead.c, putting the last digit of the number
> > > > of columns as the first cell value read. Since it is followed by a
> > > > newline, it is treated as value.
> > > > fseek(fd, -(len+1), SEEK_CUR);
> > > >
> > > > For me, fseek(fd, -(len), SEEK_CUR); fixes it, but I haven't tried
> > > > it with other header fields or command line options.
> > >
> > > Suggestion: call ftell() before G_getl() and use fseek(SEEK_SET) to
> > > restore the position. Alternatively, you could use fgetpos() and
> > > fsetpos(); these are more portable, but that probably isn't an issue
> > > here.
> >
> > Thanks for your help! I've tried Roger's suggestion and tested in a
> > Transverse Mercator and a lat/long location - it seems to work.
> >
> > This version is uploaded to CVS now.
> >
> > Glynn, do you think your idea is to be preferred?
>
> Yes.
>
> Basically, restoring the position to one which was previously recorded
> with ftell() or fgetpos() is more robust than trying to compute the
> required offset from the current file position.
>
> In this case, the code is assuming that the number of bytes by which
> G_getl() advances the stream's position is equal to the length of the
> string which G_getl() stores in the buffer.
>
> Whilst this may happen to be the case, I can't see anything which
> actually states that it is so (G_getl() doesn't appear to have an
> entry in progman50.pdf).
>
> Furthermore, the existing implementation of G_getl() suggests that it
> isn't the case as G_getl() removes the trailing newline. This implies
> that the original version (offset of "-(len+1)") should be correct;
> maybe there is another (equal-but-opposite) off-by-one bug elsewhere?
>
> There is another issue, which isn't applicable here but may be worth
> bearing in mind generally, which is that the behaviour of fseek() on a
> text (as opposed to binary) stream is undefined unless the offset is
> either zero, or the value returned from a previous call to ftell() (in
> the case of SEEK_SET).
>
> --
> Glynn Clements <glynn.clements@virgin.net>
> _______________________________________________
> grass5 mailing list
> grass5@geog.uni-hannover.de
> http://www.geog.uni-hannover.de/mailman/listinfo/grass5

_______________________________________________
grass5 mailing list
grass5@geog.uni-hannover.de
http://www.geog.uni-hannover.de/mailman/listinfo/grass5

--
Markus Neteler * University of Hannover
Institute of Physical Geography and Landscape Ecology
Schneiderberg 50 * D-30167 Hannover * Germany
Tel: ++49-(0)511-762-4494 Fax: -3984