In the event my experience working with rather large GRASS rasters may be useful, I thought I would share it. The following notes were compiled running the GRASS 7.0.0beta4-141230 package on Mac OS X 10.10.1. The system is a 6-core Xeon with 32GB of RAM, running in full 64-bit mode.
1. Using an r.external virtual mosaic as the input proved impossible trying to run r.resample on it on trying to produce a nearly teracell raster (I ended up breaking sub-tiles). I closely followed the instructions on the Wiki.
2. On smaller sub-tiles of said image, r.resample was insanely slow even with the external imagery. At first I thought it was a limitation of being overwhelmed with small sector requests and purchased a terabyte SSD, copied the external imagery to the SSD, but that didn’t noticeably help. Throughput was way less than 1mb/sec.
3. g.remove is unusable for long lists. Actually applies to I think any utility that takes a list for input. I wish I could just put it to a file so it could parse it line by line (As I understand the issue it is a limitation of the insanely long shell arguments).
If anyone wants me to elaborate on anything, or has pointers how to do things better - please let me know!
On Sat, Jan 17, 2015 at 7:00 PM, Jeshua Lacock <jeshua@3dtopo.com> wrote:
Greetings,
In the event my experience working with rather large GRASS rasters may be useful, I thought I would share it. The following notes were compiled running the GRASS 7.0.0beta4-141230 package on Mac OS X 10.10.1. The system is a 6-core Xeon with 32GB of RAM, running in full 64-bit mode.
Using an r.external virtual mosaic as the input proved impossible trying to run r.resample on it on trying to produce a nearly teracell raster (I ended up breaking sub-tiles). I closely followed the instructions on the Wiki.
On smaller sub-tiles of said image, r.resample was insanely slow even with the external imagery. At first I thought it was a limitation of being overwhelmed with small sector requests and purchased a terabyte SSD, copied the external imagery to the SSD, but that didn’t noticeably help. Throughput was way less than 1mb/sec.
g.remove is unusable for long lists. Actually applies to I think any utility that takes a list for input. I wish I could just put it to a file so it could parse it line by line (As I understand the issue it is a limitation of the insanely long shell arguments).
maybe silly question regarding g.remove, have you used the pattern option instead of the name option?
Best,
Anna
If anyone wants me to elaborate on anything, or has pointers how to do things better - please let me know!
On Jan 17, 2015, at 8:00 PM, Anna Petrášová <kratochanna@gmail.com> wrote:
maybe silly question regarding g.remove, have you used the pattern option instead of the name option?
Hi,
Thank you for your suggestion Anna!
Well I didn’t want to use wildcards because one little mistake and —
The list was verified first by eye.
Besides that, this same potential issue applies to many modules that may not have pattern matching that I might be more comfortable with then essentially "rm -R".
On Jan 17, 2015, at 8:04 PM, Jeshua Lacock <jeshua@3DTOPO.com> wrote:
maybe silly question regarding g.remove, have you used the pattern option instead of the name option?
Hi,
Thank you for your suggestion Anna!
Well I didn’t want to use wildcards because one little mistake and —
The list was verified first by eye.
Besides that, this same potential issue applies to many modules that may not have pattern matching that I might be more comfortable with then essentially "rm -R".
Actually, I misspoke, the wildcard will not work with long lists either as it will crash with some kind argument too long message as well.
On Sun, Jan 18, 2015 at 1:00 AM, Jeshua Lacock <jeshua@3dtopo.com> wrote:
Greetings,
In the event my experience working with rather large GRASS rasters may be useful, I thought I would share it. The following notes were compiled running the GRASS 7.0.0beta4-141230 package on Mac OS X 10.10.1. The system is a 6-core Xeon with 32GB of RAM, running in full 64-bit mode.
1. Using an r.external virtual mosaic as the input proved impossible trying to run r.resample on it on trying to produce a nearly teracell raster (I ended up breaking sub-tiles). I closely followed the instructions on the Wiki.
2. On smaller sub-tiles of said image, r.resample was insanely slow even with the external imagery.
You may check/benchmark the other resampling modules, some of them may
be more modern (ideally faster):
On Sun, Jan 18, 2015 at 1:03 PM, Jeshua Lacock <jeshua@3dtopo.com> wrote:
Actually, I misspoke, the wildcard will not work with long lists either as
it will crash with some kind argument too long message as well.
This is strange, the wild card is interpreted inside the module, so the
argument list shouldn't be too long. If you don't quite the pattern the
shell can do the expansion but this is not what you want.
On Thu, Jan 22, 2015 at 9:10 PM, Jeshua Lacock <jeshua@3dtopo.com> wrote:
Since that is indeed the case, it would be super nice to add a pattern
flag to r.patch and v.patch.
It would. I'm not sure if there is a ticket for it. Note that some other
modules accept file with names which gives different possibilities than
pattern option, e.g. usage with temporal framework.
1575828.22366994|740444.684641808
1575808.07032398|745857.66226285
1577200.35143553|749013.036058446
1577244.2755904|750323.681238862
cat: |: No such file or directory
cat: r.profile: No such file or directory
cat: input=DEM: No such file or directory
cat: output=profilepoints.csv: No such file or directory
since my command line skills are virtually none existent, can anyone suggest what is wrong (my line mimics that on the man page - or so I thought)
On Sun, Jun 7, 2015 at 11:13 AM, Stuart Edwards <sedwards2@cinci.rr.com>
wrote:
Following from my last post, I have tried to run r.profile from the
command line in the gui and get the following result:
Try to run it in terminal, not the GUI command line. GUI command line works
mainly for running grass commands, but combining with shell commands like
this doesn't work very well.
1575828.22366994|740444.684641808
1575808.07032398|745857.66226285
1577200.35143553|749013.036058446
1577244.2755904|750323.681238862
cat: |: No such file or directory
cat: r.profile: No such file or directory
cat: input=DEM: No such file or directory
cat: output=profilepoints.csv: No such file or directory
since my command line skills are virtually none existent, can anyone
suggest what is wrong (my line mimics that on the man page - or so I
thought)