[GRASS5] timeseries: null file, r3

GRASS GIS GRASS GIS developer
1

Hi there,

I need to store many thousands of maps for different model simulation runs as each has at least 25 maps in a timeseries.

Currently running 100 replications of a 25 year model on a 2000x2000 region results in about 2.1 gig.
This on its own is okay, but I need to run sensitivity analyses on a range of parameters, meaning each value I test for a parameter gives a location of 2.1 gig!

Checking out the actual data files however shows that the cell files are only about 60k each. The problem is the uncompressed null bitmasks that are 700k each. Now, I can explicitly remove the null bitmask using "r.null -r", but I was wondering if the null mask will just get automatically created next time I run a GRASS command? Also, I'm under the impression that if there is no null bitmask then values equal to 0 in the cell file are interpreted as nulls, is this right? Is there anyway to specify NOT creating a null mask when creating a map?

A while back I suggested using 3d rasters to store timeseries, but I got the impression at the time it wasn't very far developed and I knew little about GRASS then. Does anyone who knows more about the r3 design know if storing a timeseries in this way would be more space efficient? The timeseries is a sequence of snapshots of a growing region, so for the most part a region in a map at time t+1 will encompass the region in a map at time t.

Thanks,
Joel

--
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Joel Pitt, Room B534
Bio-Protection & Ecology Division
PO Box 84
Lincoln University 8150
New Zealand

2

Joel Peter William Pitt wrote:

I need to store many thousands of maps for different model simulation
runs as each has at least 25 maps in a timeseries.

Currently running 100 replications of a 25 year model on a 2000x2000
region results in about 2.1 gig.
This on its own is okay, but I need to run sensitivity analyses on a
range of parameters, meaning each value I test for a parameter gives a
location of 2.1 gig!

Checking out the actual data files however shows that the cell files are
only about 60k each. The problem is the uncompressed null bitmasks that
are 700k each. Now, I can explicitly remove the null bitmask using
"r.null -r", but I was wondering if the null mask will just get
automatically created next time I run a GRASS command?

The null file is created when the map is created, or via r.null.

Also, I'm under the impression that if there is no null bitmask then
values equal to 0 in the cell file are interpreted as nulls, is this
right?

For integer maps, yes. If a FP map doesn't have a null bitmap, all
cells are valid.

This is for compatibility with 4.3, which didn't have null bitmaps. It
didn't have FP maps either, so compatibility isn't an issue there.

This behaviour could easily be removed; see the following code in
get_null_value_row_nomask() in lib/gis/get_row.c:

      if (read_null_bits(null_fd, fcb->null_work_buf,
             i+fcb->min_null_row, fcb->cellhd.cols, fd) < 0)
      {
    if (fcb->map_type == CELL_TYPE)
    {
        /*
          If can't read null row, assume that all map 0's are nulls
          use allocated G__.mask_buf to read map row */
        get_map_row_nomask(fd, (void *) G__.mask_buf, i+fcb->min_null_row,
               CELL_TYPE);
        for (j = 0; j < G__.window.cols; j++)
        {
      if (G__.mask_buf[j] == 0)
          flags[j] = 1;
      else
          flags[j] = 0;
        }
    }
    else /* fp map */
    {
        /* if can't read null row, assume that all data is valid */
        G_zero(flags, sizeof(char) * G__.window.cols);
        /* the flags row is ready now */
    }
      } /*if no null file */

To disable the behaviour, replace the above block with:

      if (read_null_bits(null_fd, fcb->null_work_buf,
             i+fcb->min_null_row, fcb->cellhd.cols, fd) < 0)
      {
    /* if can't read null row, assume that all data is valid */
    G_zero(flags, sizeof(char) * G__.window.cols);
    /* the flags row is ready now */
      } /*if no null file */

4.3 is old enough that we could probably just remove that behaviour
altogether. It just requires anyone using maps which were created
using 4.3 to add a null bitmap with r.null.

Is there anyway to specify NOT creating a null mask when creating a
map?

No.

--
Glynn Clements <glynn@gclements.plus.com>