[GRASS5] 0 != no data

GRASS GIS GRASS GIS developer
1

Hi all

Recently we experienced a problem with using aerial photographs as
raster maps. The photographs are grayscale and thus range in value from
0 to 255. Here 0 is a valid CELL value, however, Grass 5 treated it as a
"no data" value. So, displaying our maps, these white areas appeared
(where they should have been black) that were displayed as "(Null) no
data" in d.what.rast. Initially, we thought that the data had been
corrupted somehow, but it was only Grass interpreting 0 as "no data".

For me, this is a show stopper for Grass5.0. Since Grass now has well
defined values for Null, 0 should never be considered as "no data". This
only leads to confusion to users who have no idea that in previous grass
versions, 0 was associated with "no data". Like us, they will assume
that their files have been corrupted somehow.

The fix for this was to run r.support and create a null file. This leads
me to another problem, namely, r.support has the option to consider 0 as
"no data". I think this option should be removed. Also, I don't see a
reason for having a null file. The value in the raster file should
indicate whether it is null or not. Thus, I think that the option to
r.support to set 0 as valid data by creating a null file, should also be
removed. Basically, Grass should simply read the raster file and if any
values are equal to our defined Null values they are Null, straight
forward and simple. Note that if some users have a reason for wanting 0
to be "no data" then they can simply use r.mapcalc to convert 0 to Null.

Finally, d.what.rast currently reports the value 0 as "no data" even
after the null file is created. I think this needs to be changed as
well.

Unfortunately, I have no idea how to fix this problem. The ideal
solution is to find the code that interprets 0 as "no data" and
eliminate it. However, I don't even know how to search for this type of
code. Does anyone have any ideas on how to solve this?

Thanks for listening.

--
Sincerely,

Jazzman (a.k.a. Justin Hickey) e-mail: jhickey@hpcc.nectec.or.th
High Performance Computing Center
National Electronics and Computer Technology Center (NECTEC)
Bangkok, Thailand

People who think they know everything are very irritating to those
of us who do. ---Anonymous

Jazz and Trek Rule!!!

2

Justin Hickey wrote:

Hi all

Recently we experienced a problem with using aerial photographs as
raster maps. The photographs are grayscale and thus range in value from
0 to 255. Here 0 is a valid CELL value, however, Grass 5 treated it as a
"no data" value. So, displaying our maps, these white areas appeared
(where they should have been black) that were displayed as "(Null) no
data" in d.what.rast. Initially, we thought that the data had been
corrupted somehow, but it was only Grass interpreting 0 as "no data".

For me, this is a show stopper for Grass5.0. Since Grass now has well
defined values for Null, 0 should never be considered as "no data". This
only leads to confusion to users who have no idea that in previous grass
versions, 0 was associated with "no data". Like us, they will assume
that their files have been corrupted somehow.

The fix for this was to run r.support and create a null file. This leads
me to another problem, namely, r.support has the option to consider 0 as
"no data". I think this option should be removed. Also, I don't see a
reason for having a null file. The value in the raster file should
indicate whether it is null or not. Thus, I think that the option to
r.support to set 0 as valid data by creating a null file, should also be
removed. Basically, Grass should simply read the raster file and if any
values are equal to our defined Null values they are Null, straight
forward and simple. Note that if some users have a reason for wanting 0
to be "no data" then they can simply use r.mapcalc to convert 0 to Null.

Finally, d.what.rast currently reports the value 0 as "no data" even
after the null file is created. I think this needs to be changed as
well.

Unfortunately, I have no idea how to fix this problem. The ideal
solution is to find the code that interprets 0 as "no data" and
eliminate it. However, I don't even know how to search for this type of
code. Does anyone have any ideas on how to solve this?

Thanks for listening.

--
Sincerely,

Jazzman (a.k.a. Justin Hickey) e-mail: jhickey@hpcc.nectec.or.th
High Performance Computing Center
National Electronics and Computer Technology Center (NECTEC)
Bangkok, Thailand

People who think they know everything are very irritating to those
of us who do. ---Anonymous

Jazz and Trek Rule!!!

Justin,

Zero should (hopefully) be imported as zero. I wonder if the problem is
actually with the import program. What program did you use to import the
images? Maybe as a test, try converting an image to another supported
format (eg. TIFF, SunRaster) and re-importing to see how the zeroes are
handled.

Hope this helps.

--
Bob Covill

Tekmap Consulting
P.O. Box 2016 Fall River, N.S.
B2T 1K6
Canada

E-Mail: bcovill@tekmap.ns.ca
Phone: 902-860-1496
Fax: 902-860-1498

3

Justin Hickey wrote:

Unfortunately, I have no idea how to fix this problem. The ideal
solution is to find the code that interprets 0 as "no data" and
eliminate it. However, I don't even know how to search for this type of
code. Does anyone have any ideas on how to solve this?

Until recently, the most likely reason would have been the use of
G_put_map_row(), which treats zero as null unless G_zeros_r_nulls(0)
is called first.

However, Markus recently replaced all of uses of this function with
G_put_raster_row() (or similar), which doesn't treat zero as null.

There is one reference to G_put_map_row_random() in
src/imagery/i.ortho.photo/photo.rectify/write.c; this function also
uses the zeros_r_nulls setting. Are you using i.ortho.photo?

Apart from that, something would have to be explicitly setting zero
cells to null using G__put_null_value_row() (or even lower level
functions).

--
Glynn Clements <glynn.clements@virgin.net>

4

Justin,

treating 0 as NULL by default for the maps which do not have a null file is a
historical issue
and was requested by overhelming majority of old GRASS users who used 0 for
NULL before
NULL was implemented. I don't like it either but I lost - too many people
wanted it.
The import program which you have used to import your data should have
created the null file and allowed you to treat 0 as 0 and not NULL.
I would like to see this behavior changed, however the users have to be
consulted because
they may still prefer it.

Helena

Bob Covill wrote:

Justin Hickey wrote:
>
> Hi all
>
> Recently we experienced a problem with using aerial photographs as
> raster maps. The photographs are grayscale and thus range in value from
> 0 to 255. Here 0 is a valid CELL value, however, Grass 5 treated it as a
> "no data" value. So, displaying our maps, these white areas appeared
> (where they should have been black) that were displayed as "(Null) no
> data" in d.what.rast. Initially, we thought that the data had been
> corrupted somehow, but it was only Grass interpreting 0 as "no data".
>
> For me, this is a show stopper for Grass5.0. Since Grass now has well
> defined values for Null, 0 should never be considered as "no data". This
> only leads to confusion to users who have no idea that in previous grass
> versions, 0 was associated with "no data". Like us, they will assume
> that their files have been corrupted somehow.
>
> The fix for this was to run r.support and create a null file. This leads
> me to another problem, namely, r.support has the option to consider 0 as
> "no data". I think this option should be removed. Also, I don't see a
> reason for having a null file. The value in the raster file should
> indicate whether it is null or not. Thus, I think that the option to
> r.support to set 0 as valid data by creating a null file, should also be
> removed. Basically, Grass should simply read the raster file and if any
> values are equal to our defined Null values they are Null, straight
> forward and simple. Note that if some users have a reason for wanting 0
> to be "no data" then they can simply use r.mapcalc to convert 0 to Null.
>
> Finally, d.what.rast currently reports the value 0 as "no data" even
> after the null file is created. I think this needs to be changed as
> well.
>
> Unfortunately, I have no idea how to fix this problem. The ideal
> solution is to find the code that interprets 0 as "no data" and
> eliminate it. However, I don't even know how to search for this type of
> code. Does anyone have any ideas on how to solve this?
>
> Thanks for listening.
>
> --
> Sincerely,
>
> Jazzman (a.k.a. Justin Hickey) e-mail: jhickey@hpcc.nectec.or.th
> High Performance Computing Center
> National Electronics and Computer Technology Center (NECTEC)
> Bangkok, Thailand
> ==================================================================
> People who think they know everything are very irritating to those
> of us who do. ---Anonymous
>
> Jazz and Trek Rule!!!
> ==================================================================

Justin,

Zero should (hopefully) be imported as zero. I wonder if the problem is
actually with the import program. What program did you use to import the
images? Maybe as a test, try converting an image to another supported
format (eg. TIFF, SunRaster) and re-importing to see how the zeroes are
handled.

Hope this helps.

--
Bob Covill

Tekmap Consulting
P.O. Box 2016 Fall River, N.S.
B2T 1K6
Canada

E-Mail: bcovill@tekmap.ns.ca
Phone: 902-860-1496
Fax: 902-860-1498
_______________________________________________
grass5 mailing list
grass5@geog.uni-hannover.de
http://www.geog.uni-hannover.de/mailman/listinfo/grass5

5

Hi Bob

Bob Covill wrote:

Justin Hickey wrote:
>
> Hi all
>
> Recently we experienced a problem with using aerial photographs as
> raster maps. The photographs are grayscale and thus range in value
> from 0 to 255. Here 0 is a valid CELL value, however, Grass 5
> treated it as a "no data" value. So, displaying our maps, these
> white areas appeared (where they should have been black) that were
> displayed as "(Null) no data" in d.what.rast. Initially, we thought
> that the data had been corrupted somehow, but it was only Grass
> interpreting 0 as "no data".
>

Zero should (hopefully) be imported as zero. I wonder if the problem
is actually with the import program. What program did you use to
import the images? Maybe as a test, try converting an image to
another supported format (eg. TIFF, SunRaster) and re-importing to
see how the zeroes are handled.

The map was originally imported using r.in.tif.

--
Sincerely,

Jazzman (a.k.a. Justin Hickey) e-mail: jhickey@hpcc.nectec.or.th
High Performance Computing Center
National Electronics and Computer Technology Center (NECTEC)
Bangkok, Thailand

People who think they know everything are very irritating to those
of us who do. ---Anonymous

Jazz and Trek Rule!!!

6

Hi Glynn and everyone else

Glynn Clements wrote:

Until recently, the most likely reason would have been the use of
G_put_map_row(), which treats zero as null unless G_zeros_r_nulls(0)
is called first.

However, Markus recently replaced all of uses of this function with
G_put_raster_row() (or similar), which doesn't treat zero as null.

There is one reference to G_put_map_row_random() in
src/imagery/i.ortho.photo/photo.rectify/write.c; this function also
uses the zeros_r_nulls setting. Are you using i.ortho.photo?

Apart from that, something would have to be explicitly setting zero
cells to null using G__put_null_value_row() (or even lower level
functions).

I looked into this further and it appears that the original map was
imported using Grass 4.3 and then copied to a Grass 5 database.
Obviously there would not be a null file in this case. If the file is
imported using r.in.tiff in Grass5, the null file is automatically
created. What threw me was that the first time I ran d.rast, the file
displayed normally. The second time it was run, the null values
appeared. As mentioned in my previous post, the solution is to run
r.support to create a null file. Thus, I no longer think that this is a
show stopper.

However, I would still propose that we eliminate this null file and
never treat 0 as "no data". It has been pointed out that this was
originally implemented for Grass 4.x users who still wanted 0 to mean
"no data" in Grass 5. But with a simple r.mapcalc command, these users
can transform the 0 data to proper NULL values. Thus, there is no need
for the null file.

So, if there are no objections, I will post a proposal to the Grasslist
mailing list to eliminate the null file. If that meets with a positive
response, I will add this to the TODO file.

By the way, I think we should change the output of d.what.rast for 5.0,
since regardless of whether 0 is interpreted as "no data" or not, the
program lists it as no data. If there are no objections I'll make this
change.

Thank you to those who made suggestions, it helped me track this thing
down.

--
Sincerely,

Jazzman (a.k.a. Justin Hickey) e-mail: jhickey@hpcc.nectec.or.th
High Performance Computing Center
National Electronics and Computer Technology Center (NECTEC)
Bangkok, Thailand

People who think they know everything are very irritating to those
of us who do. ---Anonymous

Jazz and Trek Rule!!!

From neteler Mon Jun 11 14:05:50 2001
Return-Path: <neteler>
Received: by hgeo02.geog.uni-hannover.de (SMI-8.6/SMI-SVR4)
  id OAA23159; Mon, 11 Jun 2001 14:05:50 +0100
Date: Mon, 11 Jun 2001 14:05:50 +0100
From: Markus Neteler <neteler@geog.uni-hannover.de>
To: Glynn Clements <glynn.clements@virgin.net>
Cc: grass5 developers list <grass5@geog.uni-hannover.de>
Subject: Re: [GRASS5] Driver Update
Message-ID: <20010611140550.Q15422@hgeo02.geog.uni-hannover.de>
Mail-Followup-To: Glynn Clements <glynn.clements@virgin.net>,
  grass5 developers list <grass5@geog.uni-hannover.de>
References: <15083.644.576215.930324@cerise.nosuchdomain.co.uk> <20010510164735.C20265@hgeo02.geog.uni-hannover.de> <15100.591.495664.493609@cerise.nosuchdomain.co.uk> <20010511172543.A29624@hgeo02.geog.uni-hannover.de> <15100.33453.372858.117251@cerise.nosuchdomain.co.uk> <20010527104025.I12157@hgeo02.geog.uni-hannover.de> <15121.14708.979191.289876@cerise.nosuchdomain.co.uk> <20010601153714.G12814@hgeo02.geog.uni-hannover.de> <15128.6607.537624.62145@cerise.nosuchdomain.co.uk> <15131.24138.606527.722069@cerise.nosuchdomain.co.uk>
Mime-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
User-Agent: Mutt/1.2.5i
In-Reply-To: <15131.24138.606527.722069@cerise.nosuchdomain.co.uk>; from glynn.clements@virgin.net on Mon, Jun 04, 2001 at 11:09:14AM +0100
Sender: grass5-admin@geog.uni-hannover.de
Errors-To: grass5-admin@geog.uni-hannover.de
X-BeenThere: grass5@geog.uni-hannover.de
X-Mailman-Version: 2.0.5
Precedence: bulk
List-Help: <mailto:grass5-request@geog.uni-hannover.de?subject=help>
List-Post: <mailto:grass5@geog.uni-hannover.de>
List-Subscribe: <http://www.geog.uni-hannover.de/mailman/listinfo/grass5&gt;,
  <mailto:grass5-request@geog.uni-hannover.de?subject=subscribe>
List-Id: GRASS 5 Developers mailing list <grass5.geog.uni-hannover.de>
List-Unsubscribe: <http://www.geog.uni-hannover.de/mailman/listinfo/grass5&gt;,
  <mailto:grass5-request@geog.uni-hannover.de?subject=unsubscribe>
List-Archive: <http://www.geog.uni-hannover.de/pipermail/grass5/&gt;
Status: O
Content-Length: 2164
Lines: 53

Hi Glynn,

sorry for the delay, was abroad.

On Mon, Jun 04, 2001 at 11:09:14AM +0100, Glynn Clements wrote:

Glynn Clements wrote:

> > while working with the current Xdriver I found a side effect
> > on KDE: If you are displaying a larger raster map and try to
> > resize, the Xdriver redraws immediately, even if the mouse
> > button is not released yet. This leads to a sort
> > of impossibility to resize (sorry, in the past I was always
> > voting for the immediate redraw...).
> >
> > As I remember you cannot find out if the mouse button is released
> > or not. So the only solution might be a little sleep before
> > redrawing to catch the full ongoing resize event. At time on
> > KDE the resize event is interrupted by the redraw.
>
> Two possible solutions, either or both of which could be implemented:
>
> 1. Delay the resize until after a given interval has elapsed with no
> further redraw events being received (I presume that this is what you
> are proposing). How long should the interval be?
>
> 2. Don't disable resize while a redraw is in progress. Redraw commands
> shouldn't be interactive (or, put another way, interactive commands
> shouldn't be passed to D_add_to_list()).
>
> Option 2 may make option 1 unnecessary. Option 1 has the drawback that
> there will always be a delay before redraw occurs; I'm reluctant to
> make non-KDE users suffer for KDE's flaws.

I've implemented option 2; let me know if you have any problems with
it.

now the monitor resize seems to work perfect on KDE.

I also added an explicit check to the Get_location_with_* functions
just in case they are called during a redraw. However, this won't help
if a program which waits for keyboard input gets added to the list.

Ideally we should check any programs which call D_add_to_list() to
ensure that they never add themselves to the redraw list if they are
run in a way which will require user interaction.

I will continue to test and report any problems. It seems your improvements
catch most (all?) cases. My students will find the others :slight_smile:

Thanks for the fix,

Markus

From neteler Mon Jun 11 14:12:25 2001
Return-Path: <neteler>
Received: by hgeo02.geog.uni-hannover.de (SMI-8.6/SMI-SVR4)
  id OAA23243; Mon, 11 Jun 2001 14:12:19 +0100
Date: Mon, 11 Jun 2001 14:12:18 +0100
From: Markus Neteler <neteler@geog.uni-hannover.de>
To: ziyadin <cakirz@itu.edu.tr>
Cc: grass5 developers list <grass5@geog.uni-hannover.de>
Subject: Re: [GRASS5] compiling grass5.0.0pre1
Message-ID: <20010611141218.A23183@hgeo02.geog.uni-hannover.de>
Mail-Followup-To: ziyadin <cakirz@itu.edu.tr>,
  grass5 developers list <grass5@geog.uni-hannover.de>
References: <3B14FF96.220EF7E9@itu.edu.tr> <3B15848A.BD51236D@nb.sympatico.ca> <3B15ED16.25718017@itu.edu.tr> <20010531122307.K26010@hgeo02.geog.uni-hannover.de> <3B164363.A26ED951@itu.edu.tr> <20010531155839.U26010@hgeo02.geog.uni-hannover.de> <3B1B7CD1.DFE5FEF1@itu.edu.tr>
Mime-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
User-Agent: Mutt/1.2.5i
In-Reply-To: <3B1B7CD1.DFE5FEF1@itu.edu.tr>; from cakirz@itu.edu.tr on Mon, Jun 04, 2001 at 03:19:29PM +0300
Sender: grass5-admin@geog.uni-hannover.de
Errors-To: grass5-admin@geog.uni-hannover.de
X-BeenThere: grass5@geog.uni-hannover.de
X-Mailman-Version: 2.0.5
Precedence: bulk
List-Help: <mailto:grass5-request@geog.uni-hannover.de?subject=help>
List-Post: <mailto:grass5@geog.uni-hannover.de>
List-Subscribe: <http://www.geog.uni-hannover.de/mailman/listinfo/grass5&gt;,
  <mailto:grass5-request@geog.uni-hannover.de?subject=subscribe>
List-Id: GRASS 5 Developers mailing list <grass5.geog.uni-hannover.de>
List-Unsubscribe: <http://www.geog.uni-hannover.de/mailman/listinfo/grass5&gt;,
  <mailto:grass5-request@geog.uni-hannover.de?subject=unsubscribe>
List-Archive: <http://www.geog.uni-hannover.de/pipermail/grass5/&gt;
Status: O
Content-Length: 1479
Lines: 46

Dear Ziya,

On Mon, Jun 04, 2001 at 03:19:29PM +0300, ziyadin wrote:

Dear Markus,
I have finally manage to compile all the modules that come with
grass5.0.0pre1. There was only one left (src/raster/wildfire) but
I compiled it manually. The problem with this was that Gmake and
some other files (e.g. src) of this module did not exist.

mhh, that's surprising. Here I have a Gmakefile in
src/raster/wildfire/

you should be able to compile from there.

The problem with the other modules (apart from my missing
libraries) that I could not compile was that there is a problem either
with my tar program or with grass5.0.0pre1.tar.gz file;
when I gunzip and untar the file tree directories are missing. These are
../src/sites
../src/scripts
src.contrib

Are you sure? I have just checked the tarball, the directories are in.
Perhaps you have been looking in the wrong directory.

I later obtained these from snap_shots source file on your server.
I have also made a test and compiled this version. It works.

o.k.

I have one problem now; when I invoke nviz I get this error mesage

Mapset <PERMANENT> in Location <spearfish>
GRASS 5.0.0pre1 > nviz
/usr/local/grass5/bin/nviz: tclsh: not found

I would be happy if you could give some suggestions about this.

Check for the file tclsh8.0 (or tclsh8.1 or tclsh8.2...) and set
a link to tclsh:
ln -s tclsh8.1 tclsh

Then it should start.

Good luck,

Markus

From neteler Mon Jun 11 14:24:09 2001
Return-Path: <neteler>
Received: by hgeo02.geog.uni-hannover.de (SMI-8.6/SMI-SVR4)
  id OAA23376; Mon, 11 Jun 2001 14:24:09 +0100
Date: Mon, 11 Jun 2001 14:24:09 +0100
From: Markus Neteler <neteler@geog.uni-hannover.de>
To: GRASS <grass@geni.cemtlo.com>
Cc: grass5 developers list <grass5@geog.uni-hannover.de>
Message-ID: <20010611142409.C23183@hgeo02.geog.uni-hannover.de>
Mail-Followup-To: GRASS <grass@geni.cemtlo.com>,
  grass5 developers list <grass5@geog.uni-hannover.de>
References: <20010529132445.G18447@hgeo02.geog.uni-hannover.de> <200106051039.f55AdGZ28715@geni.cemtlo.com>
Mime-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
User-Agent: Mutt/1.2.5i
In-Reply-To: <200106051039.f55AdGZ28715@geni.cemtlo.com>; from grass@geni.cemtlo.com on Tue, Jun 05, 2001 at 07:39:16PM +0900
Subject: [GRASS5] Re: r.sunmask - memory leakage in GRASS lib
Sender: grass5-admin@geog.uni-hannover.de
Errors-To: grass5-admin@geog.uni-hannover.de
X-BeenThere: grass5@geog.uni-hannover.de
X-Mailman-Version: 2.0.5
Precedence: bulk
List-Help: <mailto:grass5-request@geog.uni-hannover.de?subject=help>
List-Post: <mailto:grass5@geog.uni-hannover.de>
List-Subscribe: <http://www.geog.uni-hannover.de/mailman/listinfo/grass5&gt;,
  <mailto:grass5-request@geog.uni-hannover.de?subject=subscribe>
List-Id: GRASS 5 Developers mailing list <grass5.geog.uni-hannover.de>
List-Unsubscribe: <http://www.geog.uni-hannover.de/mailman/listinfo/grass5&gt;,
  <mailto:grass5-request@geog.uni-hannover.de?subject=unsubscribe>
List-Archive: <http://www.geog.uni-hannover.de/pipermail/grass5/&gt;
Status: O
Content-Length: 4810
Lines: 156

Hi Huidae,
hi all, (as this might be interesting for all)

thanks for looking into the memory leakage in
  r.sunmask -
  src/libes/gis/get_row.c

On Tue, Jun 05, 2001 at 07:39:16PM +0900, GRASS wrote:

Hi Markus,

Sorry for my late response.

From: Markus Neteler <neteler@geog.uni-hannover.de>
> Hi Huidae, hi Justin,
>
> Justin: many thanks for below comments! I try to involve Huidae :slight_smile:
>
> Huidae: the r.sunmask still contains the memory leakage which appears
> with bigger DEMs - it leads to a partially system crash. Justin has
> been investigating, but didn't find the leakage yet. May I ask you
> for assistance? I am willing to learn how to debug such problems,
> but I am out of references/tutorials.
> Perhaps this problem indicates a memory leakage in GRASS GIS library
> which would also affect other modules.
>
> Here what Justin found out:
> On Tue, May 29, 2001 at 05:32:10PM +0700, Justin Hickey wrote:
> > Hi Markus
> >
> > The latest update is not too good but I have a little bit of
> > information. A debugger I have on our SGI can do a memory leak trace,
> > unfortunately it dumps core trying to run r.sunmask. I don't know why.
> > The error message doesn't make any sense to me.
> >
> > So I played around a bit and determined the following. The memory is
> > being allocated every so often inside the
> >
> > while (col1<window.cols)
> >
> > loop (line 385). The "top" program indicates that it increments in 32K
> > chunks every 40 - 50 iterations through that loop. The number of
> > iterations is not accurate due to timing differences between top and the
> > debugger. The only place I found an allocation was in a call to
> > G_zlib_read() from src/libes/gis/flate.c The call stack is
> >
> > G_get_raster_row()
> > get_map_row_nomask()
> > read_data()
> > read_data_fp_compressed()
> > G_zlib_read()
> >
> > However, it appears that this memory is freed properly and thus should
> > not be the cause of the leak. :frowning:
> >
> > This doesn't mean that it is the only allocation function, in fact the
> > evidence indicates that there must be another one somewhere. However,
> > I'm not sure where it is and you would have to trace through the
> > possible calls to find it.
> >
> > One thing to keep in mind is that the figures I mentioned above could be
> > dependent on the data. That is, the chunk size and number of iterations
> > could be very different for different data.
> >
> > I'm sorry, but I can't spend any more time on this. My suggestion would
> > be to start with the G_get_raster_row() function since it seems to be
> > the only thing in that loop which might do some memory allocation. Then
> > you trace through each function call inside G_get_raster_row() and try
> > to find memory allocated but not freed. You might want to look into
> > Linux debugging tools that could help you with this.
> >
> > Anyway, I hope this helps a little bit and good luck with the hunt!
> > Sorry I couldn't be of much help.
> >
> > --
> > Sincerely,
> >
> > Jazzman (a.k.a. Justin Hickey) e-mail: jhickey@hpcc.nectec.or.th
> [...]
>
> As I need the module for a project, I have to get it running. Therefore
> I am bothering you... my apologies!
>
> If you run the module with on a 2000x2000 DEM (or more), you can see
> the memory consumption.
>

I've tested r.sunmask.

The memory leakage appears in

  col = data_buf;
  
  pair = readamount / (n+1);
  while (pair-- > 0) {
    repeat = *cmp++;
    while(repeat--)
      for (k = 0; k < n; k++)
        *col++ = cmp[k]; /**********/
    cmp += n;
  }

in read_data_compressed()
in src/libes/gis/get_row.c

I've experienced the following case with 2000*2000 map:

  COMPRESSED = 1
  readamount = 18 // size of cmp when reading
  n = 1
  DATA_NCOLS = 2000

so

  if (COMPRESSED < 0 || readamount < n * DATA_NCOLS)

is true,

  cmp structure is..

  pair1: REPEAT | repeat1 | repeat2 | ... | repeatREPEAT
  pair2: REPEAT | repeat1 | repeat2 | ... | repeatREPEAT

    ...

  pairP: REPEAT | repeat1 | repeat2 | ... | repeatREPEAT

  repeatn structure is..
  
    cmp1 | cmp2 | ... | cmpn one byte per each section

...

Anyway, I met a memory leakage only if

  *col++ = cmp[k];

is active.

I think a compressed data structure is somewhat different from an
uncompressing algorithm. I'm not sure :-).

I don't have much idea on a compression algorithm.

I'll continue to investigate.

Yours,
Huidae Cho

Perhaps Eric Miller can suggest as he worked on the compression?
It seems we are close the the problem now.

Thanks for investigating, Huidae,

Markus

From neteler Mon Jun 11 14:44:00 2001
Return-Path: <neteler>
Received: by hgeo02.geog.uni-hannover.de (SMI-8.6/SMI-SVR4)
  id OAA23549; Mon, 11 Jun 2001 14:44:00 +0100
Date: Mon, 11 Jun 2001 14:44:00 +0100
From: Markus Neteler <neteler@geog.uni-hannover.de>
To: grass5@geog.uni-hannover.de
Subject: Re: [GRASS5] Darwin Pre1 NVIZ
Message-ID: <20010611144400.B23503@hgeo02.geog.uni-hannover.de>
Mail-Followup-To: grass5@geog.uni-hannover.de
References: <F97OONwIidWZe2BS7QT0000e753@hotmail.com> <15133.33019.929881.777086@cerise.nosuchdomain.co.uk> <20010605194015.D23813@calico.local> <15133.41062.452043.244362@cerise.nosuchdomain.co.uk>
Mime-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
User-Agent: Mutt/1.2.5i
In-Reply-To: <15133.41062.452043.244362@cerise.nosuchdomain.co.uk>; from glynn.clements@virgin.net on Wed, Jun 06, 2001 at 04:15:50AM +0100
Sender: grass5-admin@geog.uni-hannover.de
Errors-To: grass5-admin@geog.uni-hannover.de
X-BeenThere: grass5@geog.uni-hannover.de
X-Mailman-Version: 2.0.5
Precedence: bulk
List-Help: <mailto:grass5-request@geog.uni-hannover.de?subject=help>
List-Post: <mailto:grass5@geog.uni-hannover.de>
List-Subscribe: <http://www.geog.uni-hannover.de/mailman/listinfo/grass5&gt;,
  <mailto:grass5-request@geog.uni-hannover.de?subject=subscribe>
List-Id: GRASS 5 Developers mailing list <grass5.geog.uni-hannover.de>
List-Unsubscribe: <http://www.geog.uni-hannover.de/mailman/listinfo/grass5&gt;,
  <mailto:grass5-request@geog.uni-hannover.de?subject=unsubscribe>
List-Archive: <http://www.geog.uni-hannover.de/pipermail/grass5/&gt;
Status: O
Content-Length: 772
Lines: 21

On Wed, Jun 06, 2001 at 04:15:50AM +0100, Glynn Clements wrote:
[...]

Anyway, all of this is moot unless g3d finds its way back into the
source tree.

So, should I get rid of the GLw checks altogether? Or is g3d's absence
likely to be only temporary?

Glynn,

this g3d absence *is* only temporary unless the bugs are fixed. Jaro
Hofierka offered to look into them, probably he will find the time.
Or, Helena, could you provide some help?

Basic bug is the 3D coordinates system confusion: The 3D coordinates system
should be similar to GRASS 2D, just with a 3rd dimension. At time some
modules read bottom-up, other top-down etc. Maybe much of the problems
can be cleaned out without too much work, only someone has to fully
understand the g3d concept.

Markus

From neteler Mon Jun 11 15:44:43 2001
Return-Path: <neteler>
Received: by hgeo02.geog.uni-hannover.de (SMI-8.6/SMI-SVR4)
  id PAA24129; Mon, 11 Jun 2001 15:44:43 +0100
Date: Mon, 11 Jun 2001 15:44:43 +0100
From: Markus Neteler <neteler@geog.uni-hannover.de>
To: grass5@geog.uni-hannover.de
Subject: Re: [GRASS5] Darwin Pre1 gtty stty errors
Message-ID: <20010611154443.E23925@hgeo02.geog.uni-hannover.de>
Mail-Followup-To: grass5@geog.uni-hannover.de
References: <15132.13310.868959.945983@cerise.nosuchdomain.co.uk> <200106050522.WAA07011@avocet.mail.pas.earthlink.net> <15132.30517.624912.72271@cerise.nosuchdomain.co.uk> <15133.44162.682393.984594@cerise.nosuchdomain.co.uk> <20010605213631.E28461@calico.local> <15133.49359.961007.76658@cerise.nosuchdomain.co.uk> <20010605235706.A29570@calico.local> <15133.58130.485743.341112@cerise.nosuchdomain.co.uk>
Mime-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
User-Agent: Mutt/1.2.5i
In-Reply-To: <15133.58130.485743.341112@cerise.nosuchdomain.co.uk>; from glynn.clements@virgin.net on Wed, Jun 06, 2001 at 09:00:18AM +0100
Sender: grass5-admin@geog.uni-hannover.de
Errors-To: grass5-admin@geog.uni-hannover.de
X-BeenThere: grass5@geog.uni-hannover.de
X-Mailman-Version: 2.0.5
Precedence: bulk
List-Help: <mailto:grass5-request@geog.uni-hannover.de?subject=help>
List-Post: <mailto:grass5@geog.uni-hannover.de>
List-Subscribe: <http://www.geog.uni-hannover.de/mailman/listinfo/grass5&gt;,
  <mailto:grass5-request@geog.uni-hannover.de?subject=subscribe>
List-Id: GRASS 5 Developers mailing list <grass5.geog.uni-hannover.de>
List-Unsubscribe: <http://www.geog.uni-hannover.de/mailman/listinfo/grass5&gt;,
  <mailto:grass5-request@geog.uni-hannover.de?subject=unsubscribe>
List-Archive: <http://www.geog.uni-hannover.de/pipermail/grass5/&gt;
Status: O
Content-Length: 1044
Lines: 27

On Wed, Jun 06, 2001 at 09:00:18AM +0100, Glynn Clements wrote:
[...]

> > BTW, this doesn't really apply to d.colors, which is almost entirely
> > interface.
>
> Probably d.colors will disappear eventually. 24bit displays are pretty
> common...

That doesn't really relate to the issue of assigning colours to raster
maps. A raster cell stores either a discrete category or scalar value,
not a colour. Regardless of display hardware, colour tables
("palettes") are still an essential part of the process, so a means to
edit them is still necessary.

It seems more likely that d.colors would simply be replaced with a GUI
version. It doesn't seem likely that anyone would want to
interactively edit colour tables without being able to see the results
(which implies running XDRIVER, which implies being able to run a GUI
tool).

It seems that a GUI (tcl/tk) version is already present:
  src/tcltkgrass/todo/r.colortool
It offers RGB sliders etc. Only it needs a bit of update... Volunteers?

Cheers

Markus

7

Justin

as I understand it, this issue has already been resolved (I believe I was
wrong in my
answer when I thought that it was not). The null file should stay - that
is
the file which ensures that NULL is different from 0. What had to be
changed
was reading of files which did not have NULL file - originally, 0 was
treated as NULL
for such files. As I understand it (correct me if I am wrong) this was
changed
to read 0 as a 0-value if NULL file is missing . When NULL file is there,
there is no
problem, the only issue were old files which did not have NULL-file, which
was your case.
With NULL file present 0 should not be treated as NULL unless you define
it as such.

Helena

Hickey wrote:

Hi Glynn and everyone else

Glynn Clements wrote:
> Until recently, the most likely reason would have been the use of
> G_put_map_row(), which treats zero as null unless G_zeros_r_nulls(0)
> is called first.
>
> However, Markus recently replaced all of uses of this function with
> G_put_raster_row() (or similar), which doesn't treat zero as null.
>
> There is one reference to G_put_map_row_random() in
> src/imagery/i.ortho.photo/photo.rectify/write.c; this function also
> uses the zeros_r_nulls setting. Are you using i.ortho.photo?
>
> Apart from that, something would have to be explicitly setting zero
> cells to null using G__put_null_value_row() (or even lower level
> functions).

I looked into this further and it appears that the original map was
imported using Grass 4.3 and then copied to a Grass 5 database.
Obviously there would not be a null file in this case. If the file is
imported using r.in.tiff in Grass5, the null file is automatically
created. What threw me was that the first time I ran d.rast, the file
displayed normally. The second time it was run, the null values
appeared. As mentioned in my previous post, the solution is to run
r.support to create a null file. Thus, I no longer think that this is a
show stopper.

However, I would still propose that we eliminate this null file and
never treat 0 as "no data". It has been pointed out that this was
originally implemented for Grass 4.x users who still wanted 0 to mean
"no data" in Grass 5. But with a simple r.mapcalc command, these users
can transform the 0 data to proper NULL values. Thus, there is no need
for the null file.

So, if there are no objections, I will post a proposal to the Grasslist
mailing list to eliminate the null file. If that meets with a positive
response, I will add this to the TODO file.

By the way, I think we should change the output of d.what.rast for 5.0,
since regardless of whether 0 is interpreted as "no data" or not, the
program lists it as no data. If there are no objections I'll make this
change.

Thank you to those who made suggestions, it helped me track this thing
down.

--
Sincerely,

Jazzman (a.k.a. Justin Hickey) e-mail: jhickey@hpcc.nectec.or.th
High Performance Computing Center
National Electronics and Computer Technology Center (NECTEC)
Bangkok, Thailand

People who think they know everything are very irritating to those
of us who do. ---Anonymous

Jazz and Trek Rule!!!

_______________________________________________
grass5 mailing list
grass5@geog.uni-hannover.de
http://www.geog.uni-hannover.de/mailman/listinfo/grass5

From neteler Mon Jun 11 16:35:59 2001
Return-Path: <neteler>
Received: by hgeo02.geog.uni-hannover.de (SMI-8.6/SMI-SVR4)
  id QAA24511; Mon, 11 Jun 2001 16:35:59 +0100
Date: Mon, 11 Jun 2001 16:35:59 +0100
From: Markus Neteler <neteler@geog.uni-hannover.de>
To: grass5 developers list <grass5@geog.uni-hannover.de>
Cc: grasslist@baylor.edu
Message-ID: <20010611163559.D24187@hgeo02.geog.uni-hannover.de>
Mail-Followup-To: grass5 developers list <grass5@geog.uni-hannover.de>,
  grasslist@baylor.edu
Mime-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
User-Agent: Mutt/1.2.5i
Subject: [GRASS5] GRASS 5 environment variables online
Sender: grass5-admin@geog.uni-hannover.de
Errors-To: grass5-admin@geog.uni-hannover.de
X-BeenThere: grass5@geog.uni-hannover.de
X-Mailman-Version: 2.0.5
Precedence: bulk
List-Help: <mailto:grass5-request@geog.uni-hannover.de?subject=help>
List-Post: <mailto:grass5@geog.uni-hannover.de>
List-Subscribe: <http://www.geog.uni-hannover.de/mailman/listinfo/grass5&gt;,
  <mailto:grass5-request@geog.uni-hannover.de?subject=subscribe>
List-Id: GRASS 5 Developers mailing list <grass5.geog.uni-hannover.de>
List-Unsubscribe: <http://www.geog.uni-hannover.de/mailman/listinfo/grass5&gt;,
  <mailto:grass5-request@geog.uni-hannover.de?subject=unsubscribe>
List-Archive: <http://www.geog.uni-hannover.de/pipermail/grass5/&gt;
Status: O
Content-Length: 420
Lines: 19

Hi all,

as the GRASS environment variables are of wide interest, I have
put them online:

http://www.geog.uni-hannover.de/grass/gdp/html_grass5/index.html
-> GRASS 5 Environment variables

This file is still under construction. Developers may add missing
explanations in source code at:
html/env_vars.html

(was documents/envVars.txt before)

The web file is updated overnight automagically.

Regards

Markus Neteler

From neteler Mon Jun 11 16:49:10 2001
Return-Path: <neteler>
Received: by hgeo02.geog.uni-hannover.de (SMI-8.6/SMI-SVR4)
  id QAA24622; Mon, 11 Jun 2001 16:49:10 +0100
Date: Mon, 11 Jun 2001 16:49:10 +0100
From: Markus Neteler <neteler@geog.uni-hannover.de>
To: Request Tracker <grass-bugs@intevation.de>
Cc: grass5@geog.uni-hannover.de
Subject: Re: [GRASS5] [bug #750] (grass) g.manual problem
Message-ID: <20010611164910.G24187@hgeo02.geog.uni-hannover.de>
Mail-Followup-To: Request Tracker <grass-bugs@intevation.de>,
  grass5@geog.uni-hannover.de
References: <20010608100155.45F4D139F7@mailman.intevation.de>
Mime-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
User-Agent: Mutt/1.2.5i
In-Reply-To: <20010608100155.45F4D139F7@mailman.intevation.de>; from grass-bugs@intevation.de on Fri, Jun 08, 2001 at 12:01:55PM +0200
Sender: grass5-admin@geog.uni-hannover.de
Errors-To: grass5-admin@geog.uni-hannover.de
X-BeenThere: grass5@geog.uni-hannover.de
X-Mailman-Version: 2.0.5
Precedence: bulk
List-Help: <mailto:grass5-request@geog.uni-hannover.de?subject=help>
List-Post: <mailto:grass5@geog.uni-hannover.de>
List-Subscribe: <http://www.geog.uni-hannover.de/mailman/listinfo/grass5&gt;,
  <mailto:grass5-request@geog.uni-hannover.de?subject=subscribe>
List-Id: GRASS 5 Developers mailing list <grass5.geog.uni-hannover.de>
List-Unsubscribe: <http://www.geog.uni-hannover.de/mailman/listinfo/grass5&gt;,
  <mailto:grass5-request@geog.uni-hannover.de?subject=unsubscribe>
List-Archive: <http://www.geog.uni-hannover.de/pipermail/grass5/&gt;
Status: O
Content-Length: 940
Lines: 35

On Fri, Jun 08, 2001 at 12:01:55PM +0200, Request Tracker wrote:

this bug's URL: http://intevation.de/rt/webrt?serial_num=750
-------------------------------------------------------------------------

Subject: g.manual problem

Platform: SGI/Irix
Xwindows version: IRIX
Xwindows manager: IRIX
TclTk version: tcl/tk 8.3
grass downloaded at: CVS Server, Germany
grass binary for platform: I compiled the sources myself
grass sources source: yes, I am using the latest GRASS from CVS
c compiler name: cc

The "g.manual -1" and "g.manual -a" commands do nothing but "g.manual --help" and the man page indicate they should list all manual entries.

Justin Hickey

Hi Justin,

here it works fine. Be sure that you have eliminated the (old) binary
version of g.manual. Check

which g.manual
/usr/local/grass5/scripts/g.manual

that you use the script version. What does

echo $GRASS_PAGER
more

tell you?

Markus

8

Hi Helena

Helena Mitasova wrote:

as I understand it, this issue has already been resolved (I believe I
was wrong in my answer when I thought that it was not). The null file
should stay - that is the file which ensures that NULL is different
from 0.

Please forgive my ignorance here, but I still do not see the purpose of
the null file. The gis library has a null value defined for each type of
map; CELL, FCELL, and DCELL. Namely, in hex these are (on a 32 bit
machine):

CELL => 80000000 which is INT_MIN from float.h
FCELL => FFFFFFFF defined as NaN
DCELL => FFFFFFFFFFFFFFFF defined as NaN

None of these values are 0, so why do we need a null file to ensure NULL
is different from 0? The library should handle that automatically.

What had to be changed was reading of files which did not have NULL
file - originally, 0 was treated as NULL for such files. As I
understand it (correct me if I am wrong) this was changed to read 0
as a 0-value if NULL file is missing.

Unfortunately, this wasn't changed. If the null file does not exist,
then 0 values are interpreted as NULL. But I agree with you, it should
be changed and 0 should never be interpreted as NULL.

When NULL file is there, there is no problem,

Ahh, but if 0 is not interpreted as NULL, the problem doesn't exist in
the first place and thus, the null file is not necessary.

the only issue were old files which did not have
NULL-file, which was your case. With NULL file present 0 should not
be treated as NULL unless you define it as such.

With old files, there would be no problem if 0 was not interpreted as
NULL, that is, 0 would be 0. If a user wanted 0 to be interpreted as
NULL he/she simply runs the following command in r.mapcalc

map=if(map,map,null())

Now the map file has proper null values defined (or at least it should
since I don't know if this info would be stored in a null file with the
current source code) and 0 can still be used as 0 if need be in the
future.

Therefore, if we eliminate the code that interprets 0 as NULL, then we
can also eliminate the null file. And then we can clean up programs like
r.support by eliminating references to this confusing (in my opinion)
concept of 0 as NULL.

This seems to me to be a clear and clean method. The only NULL values
are those defined in the library. Using a null file to switch between a
feature that is not really necessary seems like a hack to me.

Unfortunately, if we decide to eliminate the null file, it would have
been nice to have done this for 5.0. But this type of change is too
major at this late point. Perhaps we can make minor changes to r.support
and d.what.rast to hide the fact that there is a null file to minimize
the effect of eliminating it for 5.1.

I don't know, maybe I'm way off base here and am missing something. I
just can't see a reason for the null file.

--
Sincerely,

Jazzman (a.k.a. Justin Hickey) e-mail: jhickey@hpcc.nectec.or.th
High Performance Computing Center
National Electronics and Computer Technology Center (NECTEC)
Bangkok, Thailand

People who think they know everything are very irritating to those
of us who do. ---Anonymous

Jazz and Trek Rule!!!

9

On Tue, 12 Jun 2001, Justin Hickey wrote:

Hi Helena

Helena Mitasova wrote:
> as I understand it, this issue has already been resolved (I believe I
> was wrong in my answer when I thought that it was not). The null file
> should stay - that is the file which ensures that NULL is different
> from 0.

Please forgive my ignorance here, but I still do not see the purpose of
the null file. The gis library has a null value defined for each type of
map; CELL, FCELL, and DCELL. Namely, in hex these are (on a 32 bit
machine):

CELL => 80000000 which is INT_MIN from float.h
FCELL => FFFFFFFF defined as NaN
DCELL => FFFFFFFFFFFFFFFF defined as NaN

None of these values are 0, so why do we need a null file to ensure NULL
is different from 0? The library should handle that automatically.

> What had to be changed was reading of files which did not have NULL
> file - originally, 0 was treated as NULL for such files. As I
> understand it (correct me if I am wrong) this was changed to read 0
> as a 0-value if NULL file is missing.

Unfortunately, this wasn't changed. If the null file does not exist,
then 0 values are interpreted as NULL. But I agree with you, it should
be changed and 0 should never be interpreted as NULL.

> When NULL file is there, there is no problem,

Ahh, but if 0 is not interpreted as NULL, the problem doesn't exist in
the first place and thus, the null file is not necessary.

> the only issue were old files which did not have
> NULL-file, which was your case. With NULL file present 0 should not
> be treated as NULL unless you define it as such.

With old files, there would be no problem if 0 was not interpreted as
NULL, that is, 0 would be 0. If a user wanted 0 to be interpreted as
NULL he/she simply runs the following command in r.mapcalc

map=if(map,map,null())

Now the map file has proper null values defined (or at least it should
since I don't know if this info would be stored in a null file with the
current source code) and 0 can still be used as 0 if need be in the
future.

Therefore, if we eliminate the code that interprets 0 as NULL, then we
can also eliminate the null file. And then we can clean up programs like
r.support by eliminating references to this confusing (in my opinion)
concept of 0 as NULL.

This seems to me to be a clear and clean method. The only NULL values
are those defined in the library. Using a null file to switch between a
feature that is not really necessary seems like a hack to me.

Unfortunately, if we decide to eliminate the null file, it would have
been nice to have done this for 5.0. But this type of change is too
major at this late point. Perhaps we can make minor changes to r.support
and d.what.rast to hide the fact that there is a null file to minimize
the effect of eliminating it for 5.1.

I don't know, maybe I'm way off base here and am missing something. I
just can't see a reason for the null file.

No, not way off, but perhaps more in 5.1 territory? The chosen NULL
solution was dictated by the fact that all pre-NULL programs (many really
for categorical rather than numeric layers) used 0 as 0 and NULL (see
traces in r.slope.aspect adding 1 to zero slopes etc. and now commented
out. Moving straight to the representation you describe was thought to
raise too large transition problems for the many legacy users, so their
needs had to be incorporated. I don't know how the user base splits now
between 4.* and 5.* releases, but guess that 4.* still has a loyal
following. Maybe first we should get them over to 5.0 using the existing
transitional mechanisms, and only next time round migrate them away from 0
as NULL (a bit like lzw2z)?

Roger

--
Roger Bivand
Economic Geography Section, Department of Economics, Norwegian School of
Economics and Business Administration, Breiviksveien 40, N-5045 Bergen,
Norway. voice: +47 55 95 93 55; fax +47 55 95 93 93
e-mail: Roger.Bivand@nhh.no
and: Department of Geography and Regional Development, University of
Gdansk, al. Mar. J. Pilsudskiego 46, PL-81 378 Gdynia, Poland.

From neteler Tue Jun 12 09:17:14 2001
Return-Path: <neteler>
Received: by hgeo02.geog.uni-hannover.de (SMI-8.6/SMI-SVR4)
  id JAA29137; Tue, 12 Jun 2001 09:17:14 +0100
Date: Tue, 12 Jun 2001 09:17:14 +0100
From: Markus Neteler <neteler@geog.uni-hannover.de>
To: grass5@geog.uni-hannover.de
Subject: Re: [GRASS5] Curiuos problem
Message-ID: <20010612091714.L24187@hgeo02.geog.uni-hannover.de>
Mail-Followup-To: grass5@geog.uni-hannover.de
References: <200106110631.XAA13889@gull.mail.pas.earthlink.net> <20010611085658.A15422@hgeo02.geog.uni-hannover.de> <3B24F5B5.D14AD26D@grr.ulaval.ca>
Mime-Version: 1.0
Content-Type: text/plain; charset=iso-8859-1
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
User-Agent: Mutt/1.2.5i
In-Reply-To: <3B24F5B5.D14AD26D@grr.ulaval.ca>; from lagace@grr.ulaval.ca on Mon, Jun 11, 2001 at 12:45:41PM -0400
Sender: grass5-admin@geog.uni-hannover.de
Errors-To: grass5-admin@geog.uni-hannover.de
X-BeenThere: grass5@geog.uni-hannover.de
X-Mailman-Version: 2.0.5
Precedence: bulk
List-Help: <mailto:grass5-request@geog.uni-hannover.de?subject=help>
List-Post: <mailto:grass5@geog.uni-hannover.de>
List-Subscribe: <http://www.geog.uni-hannover.de/mailman/listinfo/grass5&gt;,
  <mailto:grass5-request@geog.uni-hannover.de?subject=subscribe>
List-Id: GRASS 5 Developers mailing list <grass5.geog.uni-hannover.de>
List-Unsubscribe: <http://www.geog.uni-hannover.de/mailman/listinfo/grass5&gt;,
  <mailto:grass5-request@geog.uni-hannover.de?subject=unsubscribe>
List-Archive: <http://www.geog.uni-hannover.de/pipermail/grass5/&gt;
Status: O
Content-Length: 1105
Lines: 28

Robert,

On Mon, Jun 11, 2001 at 12:45:41PM -0400, Robert Lagacé wrote:

This morning, we started experimenting with v.digit through tcltkgrass.
We created two new maps "dtest_1" and "digit_test" with few lines, one area
and two sites. We converted them with v.ascii and made topology with
v.support.

The problem is that we cannot find the files anywhere in the mapset
using file commands (ls) and (find). Grass modules (v.display, v.ascii,
v.support) in tcltk grass see them. The ls and find command see all others
maps that GRASS see except the ones created with v.digit. These files
should be in "dig" and "dig_ascii" and "dig_plus" directory.
Where could be saved??

this is quite strange, here it runs well. Perhaps you are working
in different mapsets?

V.digit show avery small problem : window terminal does not repaint when
we resize it. It repaint without problem when exposing it.

This was already fixed in CVS by Glynn: v.digit now freezes the monitor size
to avoid such confusions. After leaving v.digit it can be resized again.

Regards

Markus

10

Hi Roger

Roger Bivand wrote:

On Tue, 12 Jun 2001, Justin Hickey wrote:
> Unfortunately, if we decide to eliminate the null file, it would
> have been nice to have done this for 5.0. But this type of change
> is too major at this late point. Perhaps we can make minor changes
> to r.support and d.what.rast to hide the fact that there is a null
> file to minimize the effect of eliminating it for 5.1.
>
> I don't know, maybe I'm way off base here and am missing something.
> I just can't see a reason for the null file.
>

No, not way off, but perhaps more in 5.1 territory?

For the actual change, yes.

The chosen NULL
solution was dictated by the fact that all pre-NULL programs (many
really for categorical rather than numeric layers) used 0 as 0 and
NULL (see traces in r.slope.aspect adding 1 to zero slopes etc. and
now commented out. Moving straight to the representation you describe
was thought to raise too large transition problems for the many
legacy users, so their needs had to be incorporated. I don't know how
the user base splits now between 4.* and 5.* releases, but guess that
4.* still has a loyal following. Maybe first we should get them over
to 5.0 using the existing transitional mechanisms, and only next time
round migrate them away from 0 as NULL (a bit like lzw2z)?

I tend to disagree on this one. In your proposal, the users will have to
go through two transitions: one for using new floating point and NULL
code, and then a second one for removing the 0 = NULL feature. To me,
making a single transition is better in this case. Also, I don't think
the transition problem in this case will be large. The 4.x users are
used to 0 being NULL. All we do is inform them that they need to run
r.mapcalc (or we can provide a script for them) to update their data to
GRASS 5 if they want to convert their 0 values to NULL values, and that
from now on 0 is a valid value. They would probably expect this anyway
since they know the data format has changed.

In order to implement this, we simply hide the fact that a null file is
required. As far as I can tell, most import programs generate the null
file. The only case we need to worry about is when users copy 4.x data
to 5 databases. In this case, we tell them to run our script to upgrade
and we run r.mapcalc and generate the null file for them.

The only remaining case I can think of is if a user wants 0 to represent
NULL for newly created maps. In this caes I think we should say "Sorry,
but Grass 5 requires 0 to be a valid data value". Although backward
compatibility is good in most cases, sometimes it is bad, and I believe
that this is one of those cases. I think it is pretty safe to say that
most users (if not all) consider "0 as an invalid data value" a bug.
Afterall, that's the reason the NULL value concept was introduced, to
fix this bug.

So now we get users accustomed to upgrading their data with our upgrade
script, and when we eliminate the null file for 5.1, there is no
transition for them. I think they would be happier this way because they
are expecting a big change in their operation methods this time round.
But they probably will not expect a big change in their operations for
5.1.

That's just my 2 cents worth.

--
Sincerely,

Jazzman (a.k.a. Justin Hickey) e-mail: jhickey@hpcc.nectec.or.th
High Performance Computing Center
National Electronics and Computer Technology Center (NECTEC)
Bangkok, Thailand

People who think they know everything are very irritating to those
of us who do. ---Anonymous

Jazz and Trek Rule!!!

11

Hi Justin

On Tue, 12 Jun 2001, Justin Hickey wrote:

Roger Bivand wrote:
>
> The chosen NULL
> solution was dictated by the fact that all pre-NULL programs (many
> really for categorical rather than numeric layers) used 0 as 0 and
> NULL (see traces in r.slope.aspect adding 1 to zero slopes etc. and
> now commented out. Moving straight to the representation you describe
> was thought to raise too large transition problems for the many
> legacy users, so their needs had to be incorporated. I don't know how
> the user base splits now between 4.* and 5.* releases, but guess that
> 4.* still has a loyal following. Maybe first we should get them over
> to 5.0 using the existing transitional mechanisms, and only next time
> round migrate them away from 0 as NULL (a bit like lzw2z)?

I tend to disagree on this one. In your proposal, the users will have to
go through two transitions: one for using new floating point and NULL
code, and then a second one for removing the 0 = NULL feature. To me,
making a single transition is better in this case. Also, I don't think
the transition problem in this case will be large. The 4.x users are
used to 0 being NULL. All we do is inform them that they need to run
r.mapcalc (or we can provide a script for them) to update their data to
GRASS 5 if they want to convert their 0 values to NULL values, and that
from now on 0 is a valid value. They would probably expect this anyway
since they know the data format has changed.

In order to implement this, we simply hide the fact that a null file is
required. As far as I can tell, most import programs generate the null
file. The only case we need to worry about is when users copy 4.x data
to 5 databases. In this case, we tell them to run our script to upgrade
and we run r.mapcalc and generate the null file for them.

The only remaining case I can think of is if a user wants 0 to represent
NULL for newly created maps. In this caes I think we should say "Sorry,
but Grass 5 requires 0 to be a valid data value". Although backward
compatibility is good in most cases, sometimes it is bad, and I believe
that this is one of those cases. I think it is pretty safe to say that
most users (if not all) consider "0 as an invalid data value" a bug.
Afterall, that's the reason the NULL value concept was introduced, to
fix this bug.

So now we get users accustomed to upgrading their data with our upgrade
script, and when we eliminate the null file for 5.1, there is no
transition for them. I think they would be happier this way because they
are expecting a big change in their operation methods this time round.
But they probably will not expect a big change in their operations for
5.1.

That seems well judged. My explanation was really about the history -
essentially the same points that Helena made earlier in this thread. I
still feel that 5.0 should be as open as possible to people using 4.*, but
agree that from 5.1 the solutions chosen should be as clean and
maintainable as possible. While the pipe-sockets transition doesn't affect
legacy data, NULL might, so your script solution would work well. I know
that the legacy NW Leicestershire data set is 4.* style - I suppose
updating the demonstration data sets would help too?

Roger

--
Roger Bivand
Economic Geography Section, Department of Economics, Norwegian School of
Economics and Business Administration, Breiviksveien 40, N-5045 Bergen,
Norway. voice: +47 55 95 93 55; fax +47 55 95 93 93
e-mail: Roger.Bivand@nhh.no
and: Department of Geography and Regional Development, University of
Gdansk, al. Mar. J. Pilsudskiego 46, PL-81 378 Gdynia, Poland.

12

Roger explained the issue correctly - let me just add a little bit more.
When NULL was being implemented there were 2 groups of users:
1. Those who worked mostly with classes and had large number of files
   with 0 set as NULL - that was the standard when I came to CERL.
2. Those who started to use raster maps as representation of continuous
fields
    and 0 was a data value. Both lack of FP support and NULL was a big
problem.

It was clear that one of those 2 groups will have to update their databases
when NULL is introduced and because the first one was overhelming majority,
we ended up with what we have now.
As I said, I don't like it because I am the user in the no. 2 group, but I
strongly
suggest to send a message to the users group to see what the reaction would
be.
There are people who have 10 years of GRASS4.* data accumulated in their
databases
and my experience is that even those old data are quite often used.
The removal of the NULL file may require a lot of work and it would be good
to evaluate how does it compare with other priorities.

The issue with LZW compression was different because there weren't very many
people who had lots of floating point data because the FP GRASS was never
officially released. So it was relatively easy and fair to ask them to
update.

So Markus, let us know what do you think and if you believe that it is worth
considering such a change please write a message to the users list to see
what the reaction would be,

thanks

Helena

Roger Bivand wrote:

Hi Justin

On Tue, 12 Jun 2001, Justin Hickey wrote:

> Roger Bivand wrote:
> >
> > The chosen NULL
> > solution was dictated by the fact that all pre-NULL programs (many
> > really for categorical rather than numeric layers) used 0 as 0 and
> > NULL (see traces in r.slope.aspect adding 1 to zero slopes etc. and
> > now commented out. Moving straight to the representation you describe
> > was thought to raise too large transition problems for the many
> > legacy users, so their needs had to be incorporated. I don't know how
> > the user base splits now between 4.* and 5.* releases, but guess that
> > 4.* still has a loyal following. Maybe first we should get them over
> > to 5.0 using the existing transitional mechanisms, and only next time
> > round migrate them away from 0 as NULL (a bit like lzw2z)?
>
> I tend to disagree on this one. In your proposal, the users will have to
> go through two transitions: one for using new floating point and NULL
> code, and then a second one for removing the 0 = NULL feature. To me,
> making a single transition is better in this case. Also, I don't think
> the transition problem in this case will be large. The 4.x users are
> used to 0 being NULL. All we do is inform them that they need to run
> r.mapcalc (or we can provide a script for them) to update their data to
> GRASS 5 if they want to convert their 0 values to NULL values, and that
> from now on 0 is a valid value. They would probably expect this anyway
> since they know the data format has changed.
>
> In order to implement this, we simply hide the fact that a null file is
> required. As far as I can tell, most import programs generate the null
> file. The only case we need to worry about is when users copy 4.x data
> to 5 databases. In this case, we tell them to run our script to upgrade
> and we run r.mapcalc and generate the null file for them.
>
> The only remaining case I can think of is if a user wants 0 to represent
> NULL for newly created maps. In this caes I think we should say "Sorry,
> but Grass 5 requires 0 to be a valid data value". Although backward
> compatibility is good in most cases, sometimes it is bad, and I believe
> that this is one of those cases. I think it is pretty safe to say that
> most users (if not all) consider "0 as an invalid data value" a bug.
> Afterall, that's the reason the NULL value concept was introduced, to
> fix this bug.
>
> So now we get users accustomed to upgrading their data with our upgrade
> script, and when we eliminate the null file for 5.1, there is no
> transition for them. I think they would be happier this way because they
> are expecting a big change in their operation methods this time round.
> But they probably will not expect a big change in their operations for
> 5.1.
>
That seems well judged. My explanation was really about the history -
essentially the same points that Helena made earlier in this thread. I
still feel that 5.0 should be as open as possible to people using 4.*, but
agree that from 5.1 the solutions chosen should be as clean and
maintainable as possible. While the pipe-sockets transition doesn't affect
legacy data, NULL might, so your script solution would work well. I know
that the legacy NW Leicestershire data set is 4.* style - I suppose
updating the demonstration data sets would help too?

Roger

--
Roger Bivand
Economic Geography Section, Department of Economics, Norwegian School of
Economics and Business Administration, Breiviksveien 40, N-5045 Bergen,
Norway. voice: +47 55 95 93 55; fax +47 55 95 93 93
e-mail: Roger.Bivand@nhh.no
and: Department of Geography and Regional Development, University of
Gdansk, al. Mar. J. Pilsudskiego 46, PL-81 378 Gdynia, Poland.

_______________________________________________
grass5 mailing list
grass5@geog.uni-hannover.de
http://www.geog.uni-hannover.de/mailman/listinfo/grass5

From neteler Tue Jun 12 17:11:48 2001
Return-Path: <neteler>
Received: by hgeo02.geog.uni-hannover.de (SMI-8.6/SMI-SVR4)
  id RAA02368; Tue, 12 Jun 2001 17:11:48 +0100
Date: Tue, 12 Jun 2001 17:11:47 +0100
From: Markus Neteler <neteler@geog.uni-hannover.de>
To: grasslist <grasslist@baylor.edu>
Cc: grass5 developers list <grass5@geog.uni-hannover.de>
Message-ID: <20010612171147.C24187@hgeo02.geog.uni-hannover.de>
Mail-Followup-To: grasslist <grasslist@baylor.edu>,
  grass5 developers list <grass5@geog.uni-hannover.de>
References: <01061119523600.07399@carex.home.net>
Mime-Version: 1.0
Content-Type: text/plain; charset=iso-8859-1
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
User-Agent: Mutt/1.2.5i
In-Reply-To: <01061119523600.07399@carex.home.net>; from Morten.Sickel@newmedia.no on Mon, Jun 11, 2001 at 07:52:36PM +0200
Subject: [GRASS5] Re: [GRASSLIST:1942] m.proj
Sender: grass5-admin@geog.uni-hannover.de
Errors-To: grass5-admin@geog.uni-hannover.de
X-BeenThere: grass5@geog.uni-hannover.de
X-Mailman-Version: 2.0.5
Precedence: bulk
List-Help: <mailto:grass5-request@geog.uni-hannover.de?subject=help>
List-Post: <mailto:grass5@geog.uni-hannover.de>
List-Subscribe: <http://www.geog.uni-hannover.de/mailman/listinfo/grass5&gt;,
  <mailto:grass5-request@geog.uni-hannover.de?subject=subscribe>
List-Id: GRASS 5 Developers mailing list <grass5.geog.uni-hannover.de>
List-Unsubscribe: <http://www.geog.uni-hannover.de/mailman/listinfo/grass5&gt;,
  <mailto:grass5-request@geog.uni-hannover.de?subject=unsubscribe>
List-Archive: <http://www.geog.uni-hannover.de/pipermail/grass5/&gt;
Status: O
Content-Length: 1915
Lines: 49

Hi Morten,

(cc to "grass5" list)

On Mon, Jun 11, 2001 at 07:52:36PM +0200, Morten Sickel wrote:

Hi,

I have been using m.proj for a while to project GPS data from lat/lon to a
local coordinate system. As far as I can see, the calculations are working,
but the user interface is not the best one. I have been thinking about making
a better UI for that module, that is:
- Reading input and output files from the command line
- Making a --help text
- Reading projection information from the command line, maybe using a file
   similiar to arc/info projection files (The default ouput projection should
   be the location's projection)
In addition, it should interface better to other grass modules. i.e the
(default) output file should be directly readable by s.in.ascii.

that's great ideas! We would welcome this contribution. Maybe (later) the
projection description file could be generated from a tcl/tk window to
have a full GUI interface to m.proj.

Before I start doing any serius work on this, I would like to know, Does
anybody else do anyting similiar? (not as far as I can see in ONGOING) Or is
anything coming that will replace m.proj?

At time m.proj is untouched and waiting for improvements. Related updates
will be the integration of the latest PROJ4 library including datum
transforms. But this doesn't affect m.proj too much (at least not the
proposed improvements).

(Yes I know that I could make a lat/lon location and then use s.proj to pull
the points over, but even with the current limitations on m.proj, I find this
easier)

Morten Sickel
GIS consultant
Drøbak, Norway

Perhaps you consider to switch over to "grass5" list for further details (if
required).

Regards and thanks in advance,

--
Markus Neteler * University of Hannover
Institute of Physical Geography and Landscape Ecology
Schneiderberg 50 * D-30167 Hannover * Germany

From neteler Tue Jun 12 17:36:37 2001
Return-Path: <neteler>
Received: by hgeo02.geog.uni-hannover.de (SMI-8.6/SMI-SVR4)
  id RAA02510; Tue, 12 Jun 2001 17:36:37 +0100
Date: Tue, 12 Jun 2001 17:36:36 +0100
From: Markus Neteler <neteler@geog.uni-hannover.de>
To: grass5@geog.uni-hannover.de
Subject: Re: [GRASS5] 0 != no data
Message-ID: <20010612173636.E24187@hgeo02.geog.uni-hannover.de>
Mail-Followup-To: grass5@geog.uni-hannover.de
References: <Pine.LNX.4.21.0106121504440.13574-100000@reclus.nhh.no> <3B260843.F963A104@unity.ncsu.edu>
Mime-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
User-Agent: Mutt/1.2.5i
In-Reply-To: <3B260843.F963A104@unity.ncsu.edu>; from hmitaso@unity.ncsu.edu on Tue, Jun 12, 2001 at 07:17:07AM -0500
Sender: grass5-admin@geog.uni-hannover.de
Errors-To: grass5-admin@geog.uni-hannover.de
X-BeenThere: grass5@geog.uni-hannover.de
X-Mailman-Version: 2.0.5
Precedence: bulk
List-Help: <mailto:grass5-request@geog.uni-hannover.de?subject=help>
List-Post: <mailto:grass5@geog.uni-hannover.de>
List-Subscribe: <http://www.geog.uni-hannover.de/mailman/listinfo/grass5&gt;,
  <mailto:grass5-request@geog.uni-hannover.de?subject=subscribe>
List-Id: GRASS 5 Developers mailing list <grass5.geog.uni-hannover.de>
List-Unsubscribe: <http://www.geog.uni-hannover.de/mailman/listinfo/grass5&gt;,
  <mailto:grass5-request@geog.uni-hannover.de?subject=unsubscribe>
List-Archive: <http://www.geog.uni-hannover.de/pipermail/grass5/&gt;
Status: O
Content-Length: 8005
Lines: 159

On Tue, Jun 12, 2001 at 07:17:07AM -0500, Helena wrote:

Roger explained the issue correctly - let me just add a little bit more.
When NULL was being implemented there were 2 groups of users:
1. Those who worked mostly with classes and had large number of files
   with 0 set as NULL - that was the standard when I came to CERL.
2. Those who started to use raster maps as representation of continuous
  fields and 0 was a data value. Both lack of FP support and NULL was a big
  problem.

It was clear that one of those 2 groups will have to update their databases
when NULL is introduced and because the first one was overhelming majority,
we ended up with what we have now.
As I said, I don't like it because I am the user in the no. 2 group, but I
strongly suggest to send a message to the users group to see what the
reaction would be.

There are people who have 10 years of GRASS4.* data accumulated in their
databases and my experience is that even those old data are quite often
used. The removal of the NULL file may require a lot of work and it would
be good to evaluate how does it compare with other priorities.

The issue with LZW compression was different because there weren't very many
people who had lots of floating point data because the FP GRASS was never
officially released. So it was relatively easy and fair to ask them to
update.

So Markus, let us know what do you think and if you believe that it is worth
considering such a change please write a message to the users list to see
what the reaction would be,

If you ask me: I agree with Justin that the additional NULL file is somewhat
confusing and obviously not necessary any more.

In my opinion major changes should be done for 5.0 and not for 5.1 as there
are many 4.x users still waiting for the stable release. When published,
they will switch over to 5.0. To have much extra work again in 5.1 to
convert raster data will not be very satisfying.

At time I am not clear about the percentage of GRASS users working already
with 5.x. Can't we implement a sort of autodetection which checkes the data
at GRASS startup and going to modify the NULL stuff if required? A sort of
automated raster data upgrade (similar to r.lzw2z, but fully automated).

Even if it delays the 5.0 again (luckily pre1 is out), the raster part
should be stable for 5.0 as we have a new vector management for 5.1. GRASS
5.0 is announced to provide a new raster engine, so it should be completed.

Concerning the 4.x existing data: I have already put a 4.x and a 5.x version
online of SPEARFISH and friends. It was a simple run of r.support (-r now).
I didn't dig too deeply into the NULL problems yet, but it doesn't look
unfeasible to me as it should be mostly a library issue.

Generally we should minimize the *number* of transitions due to the large
number of datasets around. I would prefer a transition more time consuming
for the individual conversion rather than invoking new transitions when
upgrading to 5.1 (if I come from 4.x). Multiply the number of transitions
with number of datasets which doesn't equals user's happiness...

Shall I start a poll on number of 4.x/5.x users? There are some free poll
sites around.

We should't carry GRASS history around for all versions (backward
compatibility). I actually don't see any reason to fall back to 4.x,
hope that most 5.0.0pre1 users agree.
Think of the Windows/DOS story... :slight_smile:

Just my 0.02 Euro,

Markus

Roger Bivand wrote:

> Hi Justin
>
> On Tue, 12 Jun 2001, Justin Hickey wrote:
>
> > Roger Bivand wrote:
> > >
> > > The chosen NULL
> > > solution was dictated by the fact that all pre-NULL programs (many
> > > really for categorical rather than numeric layers) used 0 as 0 and
> > > NULL (see traces in r.slope.aspect adding 1 to zero slopes etc. and
> > > now commented out. Moving straight to the representation you describe
> > > was thought to raise too large transition problems for the many
> > > legacy users, so their needs had to be incorporated. I don't know how
> > > the user base splits now between 4.* and 5.* releases, but guess that
> > > 4.* still has a loyal following. Maybe first we should get them over
> > > to 5.0 using the existing transitional mechanisms, and only next time
> > > round migrate them away from 0 as NULL (a bit like lzw2z)?
> >
> > I tend to disagree on this one. In your proposal, the users will have to
> > go through two transitions: one for using new floating point and NULL
> > code, and then a second one for removing the 0 = NULL feature. To me,
> > making a single transition is better in this case. Also, I don't think
> > the transition problem in this case will be large. The 4.x users are
> > used to 0 being NULL. All we do is inform them that they need to run
> > r.mapcalc (or we can provide a script for them) to update their data to
> > GRASS 5 if they want to convert their 0 values to NULL values, and that
> > from now on 0 is a valid value. They would probably expect this anyway
> > since they know the data format has changed.
> >
> > In order to implement this, we simply hide the fact that a null file is
> > required. As far as I can tell, most import programs generate the null
> > file. The only case we need to worry about is when users copy 4.x data
> > to 5 databases. In this case, we tell them to run our script to upgrade
> > and we run r.mapcalc and generate the null file for them.
> >
> > The only remaining case I can think of is if a user wants 0 to represent
> > NULL for newly created maps. In this caes I think we should say "Sorry,
> > but Grass 5 requires 0 to be a valid data value". Although backward
> > compatibility is good in most cases, sometimes it is bad, and I believe
> > that this is one of those cases. I think it is pretty safe to say that
> > most users (if not all) consider "0 as an invalid data value" a bug.
> > Afterall, that's the reason the NULL value concept was introduced, to
> > fix this bug.
> >
> > So now we get users accustomed to upgrading their data with our upgrade
> > script, and when we eliminate the null file for 5.1, there is no
> > transition for them. I think they would be happier this way because they
> > are expecting a big change in their operation methods this time round.
> > But they probably will not expect a big change in their operations for
> > 5.1.
> >
> That seems well judged. My explanation was really about the history -
> essentially the same points that Helena made earlier in this thread. I
> still feel that 5.0 should be as open as possible to people using 4.*, but
> agree that from 5.1 the solutions chosen should be as clean and
> maintainable as possible. While the pipe-sockets transition doesn't affect
> legacy data, NULL might, so your script solution would work well. I know
> that the legacy NW Leicestershire data set is 4.* style - I suppose
> updating the demonstration data sets would help too?
>
> Roger
>
> --
> Roger Bivand
> Economic Geography Section, Department of Economics, Norwegian School of
> Economics and Business Administration, Breiviksveien 40, N-5045 Bergen,
> Norway. voice: +47 55 95 93 55; fax +47 55 95 93 93
> e-mail: Roger.Bivand@nhh.no
> and: Department of Geography and Regional Development, University of
> Gdansk, al. Mar. J. Pilsudskiego 46, PL-81 378 Gdynia, Poland.
>
> _______________________________________________
> grass5 mailing list
> grass5@geog.uni-hannover.de
> http://www.geog.uni-hannover.de/mailman/listinfo/grass5

_______________________________________________
grass5 mailing list
grass5@geog.uni-hannover.de
http://www.geog.uni-hannover.de/mailman/listinfo/grass5

--
Markus Neteler * University of Hannover
Institute of Physical Geography and Landscape Ecology
Schneiderberg 50 * D-30167 Hannover * Germany
Tel: ++49-(0)511-762-4494 Fax: -3984

From neteler Tue Jun 12 17:52:56 2001
Return-Path: <neteler>
Received: by hgeo02.geog.uni-hannover.de (SMI-8.6/SMI-SVR4)
  id RAA02649; Tue, 12 Jun 2001 17:52:56 +0100
Date: Tue, 12 Jun 2001 17:52:55 +0100
From: Markus Neteler <neteler@geog.uni-hannover.de>
To: grass5@geog.uni-hannover.de
Subject: Re: [GRASS5] nviz crashing in pre1
Message-ID: <20010612175255.I24187@hgeo02.geog.uni-hannover.de>
Mail-Followup-To: grass5@geog.uni-hannover.de
References: <3B1D2AA2.D0EDB689@unity.ncsu.edu>
Mime-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
User-Agent: Mutt/1.2.5i
In-Reply-To: <3B1D2AA2.D0EDB689@unity.ncsu.edu>; from hmitaso@unity.ncsu.edu on Tue, Jun 05, 2001 at 02:53:22PM -0400
Sender: grass5-admin@geog.uni-hannover.de
Errors-To: grass5-admin@geog.uni-hannover.de
X-BeenThere: grass5@geog.uni-hannover.de
X-Mailman-Version: 2.0.5
Precedence: bulk
List-Help: <mailto:grass5-request@geog.uni-hannover.de?subject=help>
List-Post: <mailto:grass5@geog.uni-hannover.de>
List-Subscribe: <http://www.geog.uni-hannover.de/mailman/listinfo/grass5&gt;,
  <mailto:grass5-request@geog.uni-hannover.de?subject=subscribe>
List-Id: GRASS 5 Developers mailing list <grass5.geog.uni-hannover.de>
List-Unsubscribe: <http://www.geog.uni-hannover.de/mailman/listinfo/grass5&gt;,
  <mailto:grass5-request@geog.uni-hannover.de?subject=unsubscribe>
List-Archive: <http://www.geog.uni-hannover.de/pipermail/grass5/&gt;
Status: O
Content-Length: 904
Lines: 25

On Tue, Jun 05, 2001 at 02:53:22PM -0400, Helena Mitasova wrote:

Has anybody tested nviz in 5.0pre1? It has a strange behavior - if I
start it with -x flag it runs fine. If I start it with -q or a given
elevation file it crashes everything and logs me out. However, -q and
elevation
runs OK if nviz was run with -x flag first. So it seems that -x option
sets something which is missing
in the other options. Or am I missing something here? I am running it
with RedHat 7.1
I am not sure whether this is a bug or I don't have something set up
properly.

Helena,

here it crashes, too. AARGH!! Recently it was o.k.

- If I start without maps, it opens.
- If I start with a raster map, it crashes (seg.fault).
- If I start with a raster map and "-q" it opens but doesn't display
  anything
- If I start with a raster map and "-x" it opens and works.

Bob, do you have any ideas?

Markus

13

Hello all

Markus Neteler wrote:

On Tue, Jun 12, 2001 at 07:17:07AM -0500, Helena wrote:
> Roger explained the issue correctly - let me just add a little bit
> more. When NULL was being implemented there were 2 groups of users:
> 1. Those who worked mostly with classes and had large number of
> files with 0 set as NULL - that was the standard when I came to
> CERL.
> 2. Those who started to use raster maps as representation of
> continuous fields and 0 was a data value. Both lack of FP
> support and NULL was a big problem.
>
> It was clear that one of those 2 groups will have to update their
> databases when NULL is introduced and because the first one was
> overhelming majority, we ended up with what we have now.
> As I said, I don't like it because I am the user in the no. 2
> group, but I strongly suggest to send a message to the users group
> to see what the reaction would be.

Yes, it was my intention to question the users list before doing
anything.

> There are people who have 10 years of GRASS4.* data accumulated in
> their databases and my experience is that even those old data are
> quite often used. The removal of the NULL file may require a lot of
> work and it would be good to evaluate how does it compare with
> other priorities.

The actual major changes would be done after 5.0 is released. I only
suggest users switch now when they expect a major change.

> The issue with LZW compression was different because there weren't
> very many people who had lots of floating point data because the FP
> GRASS was never officially released. So it was relatively easy and
> fair to ask them to update.
>
> So Markus, let us know what do you think and if you believe that it
> is worth considering such a change please write a message to the
> users list to see what the reaction would be,

If you ask me: I agree with Justin that the additional NULL file is
somewhat confusing and obviously not necessary any more.

In my opinion major changes should be done for 5.0 and not for 5.1 as
there are many 4.x users still waiting for the stable release. When
published, they will switch over to 5.0. To have much extra work
again in 5.1 to convert raster data will not be very satisfying.

At time I am not clear about the percentage of GRASS users working
already with 5.x. Can't we implement a sort of autodetection which
checkes the data at GRASS startup and going to modify the NULL stuff
if required? A sort of automated raster data upgrade (similar to
r.lzw2z, but fully automated).

No, we cannot automate the conversion. The deciding factor in whether to
convert a given file or not is user preference (do they want 0 to be
interpreted as NULL). If we automatically convert 4.x files so that 0 is
turned to NULL, then users who want 0 = 0 will have to convert back
again. However, I believe I can write a script that, given a list of
databases, can go through a user's data and perform the conversion. I
haven't tested it, but I have an idea of how to do it. This of course
would depend on whether users have maintained their data properly. That
is, all directories under a database are locations and all directories
under a location are mapsets.

Even if it delays the 5.0 again (luckily pre1 is out), the raster
part should be stable for 5.0 as we have a new vector management for
5.1. GRASS 5.0 is announced to provide a new raster engine, so it
should be completed.

Concerning the 4.x existing data: I have already put a 4.x and a 5.x
version online of SPEARFISH and friends. It was a simple run of
r.support (-r now). I didn't dig too deeply into the NULL problems
yet, but it doesn't look unfeasible to me as it should be mostly a
library issue.

Generally we should minimize the *number* of transitions due to the
large number of datasets around. I would prefer a transition more
time consuming for the individual conversion rather than invoking new
transitions when upgrading to 5.1 (if I come from 4.x). Multiply the
number of transitions with number of datasets which doesn't equals
user's happiness...

Shall I start a poll on number of 4.x/5.x users? There are some free
poll sites around.

Yes, start a poll but not on the number of grass 4 and 5 users. The
question should be related to whether we keep the feature or not. See
below for more on this.

We should't carry GRASS history around for all versions (backward
compatibility). I actually don't see any reason to fall back to 4.x,
hope that most 5.0.0pre1 users agree.
Think of the Windows/DOS story... :slight_smile:

Since I will be posting this issue to the user list, I would like to
suggest the wording for the post here and have all concerned developers
change, or add to it, to make sure I don't leave anything out. Helena, I
hope you don't mind me borrowing part of your post.

======================== Start of post ===============================

Subject: We need your opinion

Hello all users

Recently on the developers mailing list, there has been a discussion
concerning the current implementation of NULL. When the NULL concept was
first introduced and implemented there were basically 2 groups of users:

1. Those who worked mostly with classes and had a large number of
   files with 0 set as NULL

2. Those who started to use raster maps as representation of
   continuous fields and 0 was a data value

The majority of the users were in group 1 and thus, the decision was
made to keep the feature that 0 could be interpreted as NULL.

In the discussion on the developers list, some developers have indicated
that since GRASS 5.0 has the concept of NULL (which is represented by
non-zero values), it is confusing to also have the feature that 0 can be
interpreted as NULL, especially to new users of GRASS. This feature has
also caused some problems for some users who had greyscale aerial
photographs stored as raster maps in GRASS 4.x, and then transferred
them to GRASS 5.0. The problem was that the color black (value 0) was
being interpreted as NULL causing them to believe that their data had
been corrupted somehow.

In order to eliminate this confusion, a proposal was put forth on the
list that 0 should never be considered as NULL in GRASS 5.0. Also, in
order to facilitate those users who wish to use their GRASS 4.x files
with the new GRASS 5.0, we would provide a script to assist them in
converting the 0 values of their data to appropriate NULL values in
GRASS 5.0.

This script has not been written yet but the proposed usage would be
that a user would run the script and provide a list of GRASS 4.x
databases which contain raster files in which 0 should be interpreted as
NULL. The script will automatically go through all locations and mapsets
in each database and perform the conversion for all raster files. The
only requirement for this script would be that the GRASS 4.x data would
have to be maintained in proper GRASS databases. That is, all
directories under a database will be locations, and all directories
under a location will be mapsets. If a user has files that he/she does
not want converted then either the files can be moved to a different
database, or the user can provide a specific location, or mapset, or
raster file to the script.

What we would like you to do is provide us with your opinion on whether
we should keep the feature of 0 being interpreted as NULL. To make
things a little easier, we have set up a poll at

<URL>

for you to submit your opinion. We would greatly appreciate it if you
could take a few moments to cast a vote.

Thank you for your time in this matter.

========================= End of post ================================

So, please feel free to edit the above message in any way. For the poll,
perhaps we could say something like.

The feature of 0 being interpreted as NULL should be

    a) kept in GRASS 5.0
    b) eliminated from GRASS 5.0, but a script will be provided to
       convert GRASS 4.x files to GRASS 5.0 format for those who wish
       0 to be NULL

Again, please feel free to edit this statement.

Thanks for listening.

--
Sincerely,

Jazzman (a.k.a. Justin Hickey) e-mail: jhickey@hpcc.nectec.or.th
High Performance Computing Center
National Electronics and Computer Technology Center (NECTEC)
Bangkok, Thailand

People who think they know everything are very irritating to those
of us who do. ---Anonymous

Jazz and Trek Rule!!!

14

Justin Hickey wrote:

Interesting, but I think you may be thinking of instances where Grass
interprets CELL files as unsigned 1 byte values (ie all values are in
the range of 0 to 255). However, Grass cannot change the size of type
CELL, so CELL files can only be 1 byte data if all CELL values are 1
byte. In the file src/include/gis.h we have the following line:

typedef int CELL;

Hmmm... Does this mean that the "format" field in the header file
is not used ? (Cf progmangrass50, section 5.3.1). I have also some
(uncompressed) raster files clearly in bytes (image size = row x column)

Thus, all CELL values would be 1 byte if type int was 1 byte. Since most
machines define int as 4 bytes, then I think what is happening is that
whatever modules you were using interpreted CELL values as unsigned 1
byte values, but the actual size would still be 4 bytes. Thus, the NULL
value would still be 80000000.

[...]

So, if type int is a single byte, then cellNullPattern would be 80 in
hex.

The critical assumption is that CELL will always be defined as a
"signed" integer (of any size) so that the NULL value is the lowest
number in the possible range of values. I think this is a safe
assumption since most people would expect valid CELL data to include
negative values. Of course, if it is unsigned, then my code will choose
a value in the middle of the range, for example 128 for a 1 byte
unsigned int, which is not an end point of the range of values.

This is the reason that make me worry about the changes : a classic sheme
is to use positive number for categories and 0 for "unknown" or NULL.
On the other hand, i have found that Grass null files are often bigger
than their raster counterpart. Maybe just adding a field "null: value"
in the header file have been enough... this also take a value from the
range of possible values (which is only annoying for byte encoded raster),
but you can choose it (and even consider that the null value is 0 when
the field is missing, ensuring compatibility with grass4.0 rasters)

--
Michel WURTZ - DIG - Maison de la télédétection
               500, rue J.F. Breton
               34093 MONTPELLIER Cedex 5

15

Hi Michel

Michel Wurtz wrote:

Justin Hickey wrote:
>
> Interesting, but I think you may be thinking of instances where
> Grass interprets CELL files as unsigned 1 byte values (ie all
> values are in the range of 0 to 255). However, Grass cannot change
> the size of type CELL, so CELL files can only be 1 byte data if all
> CELL values are 1 byte. In the file src/include/gis.h we have the
> following line:
>
> typedef int CELL;
>

Hmmm... Does this mean that the "format" field in the header file
is not used ? (Cf progmangrass50, section 5.3.1).

Quite possibly. I'm not sure if this section has been updated for
floating point code (there are several sections that need to be
updated), but I see no way of Grass being able to change the type of
CELL data. CELL is defined as an int, thus all CELL data will be of type
int.

I have also some
(uncompressed) raster files clearly in bytes (image size = row x
column)

Hmmm. Did these files originate with Grass 4.x? Perhaps they are still
in Grass 4.x format (ie you never saved these maps using Grass5). In
this case, Grass would represent your data with 4 bytes internally (ie
when Grass reads your file into memory) but the file would be unchanged.
If these files originate with Grass 5 then I would say that whatever
module saved the file does not save CELL data but uses the header
"format" field instead. This would indicate (to me anyway) that the
module in question was not converted to floating point code properly,
since the resulting file would not be CELL data.

Anyone with more knowledge about the "format" field please clarify how
it works in Grass 5.0 (if at all).

> Thus, all CELL values would be 1 byte if type int was 1 byte. Since
> most machines define int as 4 bytes, then I think what is happening
> is that whatever modules you were using interpreted CELL values as
> unsigned 1 byte values, but the actual size would still be 4 bytes.
> Thus, the NULL value would still be 80000000.
>
[...]
> So, if type int is a single byte, then cellNullPattern would be 80
> in hex.
>
> The critical assumption is that CELL will always be defined as a
> "signed" integer (of any size) so that the NULL value is the lowest
> number in the possible range of values. I think this is a safe
> assumption since most people would expect valid CELL data to
> include negative values. Of course, if it is unsigned, then my code
> will choose a value in the middle of the range, for example 128 for
> a 1 byte unsigned int, which is not an end point of the range of
> values.

This is the reason that make me worry about the changes : a classic
sheme is to use positive number for categories and 0 for "unknown" or
NULL. On the other hand, i have found that Grass null files are often
bigger than their raster counterpart. Maybe just adding a field
"null: value" in the header file have been enough... this also take a
value from the range of possible values (which is only annoying for
byte encoded raster), but you can choose it (and even consider that
the null value is 0 when the field is missing, ensuring compatibility
with grass4.0 rasters)

Well, using a "null" field in the header would be convenient, but it
would be difficult to maintain the code. For floating point data, we
have perfect values for NULL, all 1 bits, which is defined as "not a
number". That is, the value is illegal. We should not let user's choose
floating point NULL values since floating point representation differ's
per machine. Thus, you would have two mechanisms for NULL (fixed for
floating point, and user choice for CELL) which is confusing.

Also, for CELL data all integer values are valid and the NULL library
routines need to know which value should be considered NULL. Bit
patterns were chosen for the NULL value since we could ensure that for
any given machine the bit pattern would be the lowest signed int. This
is not the case if user's choose their own value, which can lead to
different patterns for NULL on different machines. It also goes against
good programming practice. NULL would be dependent on the data instead
of the system.

Anyway, I have to go. Talk to you later.

--
Sincerely,

Jazzman (a.k.a. Justin Hickey) e-mail: jhickey@hpcc.nectec.or.th
High Performance Computing Center
National Electronics and Computer Technology Center (NECTEC)
Bangkok, Thailand

People who think they know everything are very irritating to those
of us who do. ---Anonymous

Jazz and Trek Rule!!!

16

On Tue, Jun 26, 2001 at 06:17:16PM +0700, Justin Hickey wrote:

Hi Michel

Michel Wurtz wrote:
>
> Justin Hickey wrote:
> >
> > Interesting, but I think you may be thinking of instances where
> > Grass interprets CELL files as unsigned 1 byte values (ie all
> > values are in the range of 0 to 255). However, Grass cannot change
> > the size of type CELL, so CELL files can only be 1 byte data if all
> > CELL values are 1 byte. In the file src/include/gis.h we have the
> > following line:
> >
> > typedef int CELL;
> >
>
> Hmmm... Does this mean that the "format" field in the header file
> is not used ? (Cf progmangrass50, section 5.3.1).

Quite possibly. I'm not sure if this section has been updated for
floating point code (there are several sections that need to be
updated), but I see no way of Grass being able to change the type of
CELL data. CELL is defined as an int, thus all CELL data will be of type
int.

No. We must distinguish between the file format and the memory
representation. Most programs I've worked on never do anything
to check how many bytes should be used for a CELL when writing a new
file. It can be specified, however, via the G_set_cell_format()
function. CELL's are always represented as an int in memory (unless
they're being read as an FCELL or DCELL, the contrary is also true).

The G_[set/get]_cell_format() functions apparently are not documented.

So, regards NULL's in CELL files with a decreased byte count, do you say
-128 is NULL for single byte, etc... ? All integers must be treated as
signed, I think. If you had an unsigned byte CELL, would 0 == NULL?
Sounds like what the whole NULL file concept was trying to avoid.

--
Eric G. Miller <egm2@jps.net>

17

Hi Eric and all

"Eric G. Miller" wrote:

On Tue, Jun 26, 2001 at 06:17:16PM +0700, Justin Hickey wrote:
> Hi Michel
>
> Michel Wurtz wrote:
> > Hmmm... Does this mean that the "format" field in the header file
> > is not used ? (Cf progmangrass50, section 5.3.1).
>
> Quite possibly. I'm not sure if this section has been updated for
> floating point code (there are several sections that need to be
> updated), but I see no way of Grass being able to change the type
> of CELL data. CELL is defined as an int, thus all CELL data will be
> of type int.

No. We must distinguish between the file format and the memory
representation. Most programs I've worked on never do anything
to check how many bytes should be used for a CELL when writing a new
file. It can be specified, however, via the G_set_cell_format()
function. CELL's are always represented as an int in memory (unless
they're being read as an FCELL or DCELL, the contrary is also true).

Ahh, I didn't know Grass had the option to specify saving files with
different byte lengths. I had assumed that CELL data was saved as is.

The G_[set/get]_cell_format() functions apparently are not
documented.

So, regards NULL's in CELL files with a decreased byte count, do you
say -128 is NULL for single byte, etc... ? All integers must be
treated as signed, I think. If you had an unsigned byte CELL, would
0 == NULL? Sounds like what the whole NULL file concept was trying to
avoid.

Thanks for pointing me to the proper place in the code. Thinking about
your last paragraph got me wondering why do we support saving CELL data
in different byte sizes. Does anyone know why Grass allows storage of
CELL data in smaller byte sizes. The only reason I can think of is to
save disk space. But this does not seem like a good reason to me,
especially if we eliminate the null file. Seeing as the null file can
sometimes take up more space than the original cell data file, storing
CELL data as type CELL (ie 4 bytes) can use the space freed by the null
file. So, I don't see why we cannot store CELL data as 4 bytes.

Then I looked through the code and found that if we eliminate both the
null file and the multiple byte size CELL files, then the file handling
code becomes greatly simplified. I noticed that the floating point code
uses XDR, thus, if we save 4 byte CELL data only, then we can use XDR
for CELL files as well. What do people think?

A couple of things I saw in my brief look:

In the file src/libes/gis/G.h XDR_FLOAT_NBYTES and XDR_DOUBLE_NBYTES are
both hardcoded to 4 and 8. Shouldn't these be based on sizeof(FCELL) and
sizeof(DCELL)?

In the same file, the G__ structure defines both "fp_nbytes" and inside
the fileinfo struct "nbytes". But it seems that the code (at least
G__open_cell_old() in src/libes/gis/opencell.c) uses the fileinfo nbytes
value, so does anyone know what the fp_nbytes is used for?

Finally, out of the 8 people that gave an opinion concerning removing
the "0 = no data" feature, 7 of them indicated they would like it
removed. Furthermore, some would like to see the null file removed in
5.0 as well. I originally planned to actually leave the null file in
place and only change references to it in the code that users would see.
The reason for this was that eliminating the null file requires major
changes to the gis library and I think changing it is not a good idea
when we are trying to finalize a release. However, with this new input
from the users, perhaps we need to think about the right course of
action. I still think we should keep the null file for now but maybe
others have different opinions.

Thanks for listening.

--
Sincerely,

Jazzman (a.k.a. Justin Hickey) e-mail: jhickey@hpcc.nectec.or.th
High Performance Computing Center
National Electronics and Computer Technology Center (NECTEC)
Bangkok, Thailand

People who think they know everything are very irritating to those
of us who do. ---Anonymous

Jazz and Trek Rule!!!

18

On Thu, Jun 28, 2001 at 06:37:40PM +0700, Justin Hickey wrote:

Thanks for pointing me to the proper place in the code. Thinking about
your last paragraph got me wondering why do we support saving CELL data
in different byte sizes. Does anyone know why Grass allows storage of
CELL data in smaller byte sizes. The only reason I can think of is to
save disk space. But this does not seem like a good reason to me,
especially if we eliminate the null file. Seeing as the null file can
sometimes take up more space than the original cell data file, storing
CELL data as type CELL (ie 4 bytes) can use the space freed by the null
file. So, I don't see why we cannot store CELL data as 4 bytes.

The NULL file probably should never be bigger than the CELL data, but it
could be because it is not compressed.

In the file src/libes/gis/G.h XDR_FLOAT_NBYTES and XDR_DOUBLE_NBYTES are
both hardcoded to 4 and 8. Shouldn't these be based on sizeof(FCELL) and
sizeof(DCELL)?

No, I think the 4 and 8 bytes *are* the definitons for the xdr_float,
and xdr_double. Granted, on machines with a different idea of floats
and doubles there will be a loss of precision, but I think XDR has a
type corresponding to long double as well.

In the same file, the G__ structure defines both "fp_nbytes" and inside
the fileinfo struct "nbytes". But it seems that the code (at least
G__open_cell_old() in src/libes/gis/opencell.c) uses the fileinfo nbytes
value, so does anyone know what the fp_nbytes is used for?

There's more than one place where the bytes are looked at. I believe
the "nbytes" is taken from cellhd (which is -1 for any floating point),
and then the fp_nbytes is taken from the cell_misc/<cell>/f_format file.
I'd have to look to be sure, but I think that's what is going on. The
representation for CELL's is totally different from FCELL/DCELL.

Finally, out of the 8 people that gave an opinion concerning removing
the "0 = no data" feature, 7 of them indicated they would like it
removed. Furthermore, some would like to see the null file removed in
5.0 as well. I originally planned to actually leave the null file in
place and only change references to it in the code that users would see.
The reason for this was that eliminating the null file requires major
changes to the gis library and I think changing it is not a good idea
when we are trying to finalize a release. However, with this new input
from the users, perhaps we need to think about the right course of
action. I still think we should keep the null file for now but maybe
others have different opinions.

It's not a trivial task. It would probably add instability to the code
base for a while...

--
Eric G. Miller <egm2@jps.net>