[GRASS-user] wget (corine) "landcover_tiles_v2" for greece works, but not for europe(?)

Nikos_Alexandris1 · January 12, 2009, 3:19pm

# get GR_100K.shp which is GREEA18328I.zip [1] from [2]
# under the column "CellCode" are listed all tiles (in this case all
tiles which cover Greece) with their exact name.

#v.db.connect -p
## if requiredδημιουργία της σύνδεσης αν δεν υφίσταται
# db.connect driver=sqlite
database=/geo/grassdb/ellas/laea/PERMANENT/sqlite.d

# import GR_100K.shp
v.in.ogr dsn=/geo/geodata/europe/GREEA18328I/GR_100K.shp
out=eea_gr_100k --o

# check newly imported table
db.select eea_gr_100k

cat|CellCode|EofOrigin|NofOrigin
1|100kmE51N18|5100000|1800000
2|100kmE51N19|5100000|1900000
3|100kmE52N16|5200000|1600000
4|100kmE52N17|5200000|1700000
5|100kmE52N18|5200000|1800000
6|100kmE52N19|5200000|1900000
7|100kmE52N20|5200000|2000000
8|100kmE53N15|5300000|1500000
9|100kmE53N16|5300000|1600000
10|100kmE53N17|5300000|1700000
[...]

# create file with tiles-list
db.select eea_gr_100k -c | cut -d"|" -f2 > corine_tiles_ellas

100kmE51N18
100kmE51N19
100kmE52N16
100kmE52N17
100kmE52N18
100kmE52N19
[...]

# wget within a for loop
for x in `cat corine_tiles_ellas`; do wget
http://dataservice.eea.europa.eu/staticfiles/landcover_tiles_v2/$x.zip;
done

--2009-01-12 15:12:26--
http://dataservice.eea.europa.eu/staticfiles/landcover_tiles_v2/100kmE51N18.zip
Resolving dataservice.eea.europa.eu... 217.74.209.209
Connecting to dataservice.eea.europa.eu|217.74.209.209|:80...
connected.
HTTP request sent, awaiting response... 200 OK
Length: 408346 (399K) [application/x-zip-compressed]
Saving to: `100kmE51N18.zip'
100%
[============================>] 408,346 1.12M/s in 0.3s
2009-01-12 15:12:27 (1.12 MB/s) - `100kmE51N18.zip' saved
[408346/408346]
[...]

Now, if I try to repeat this process for the whole of europe (based on
the file Grid_ETRS89_LAEA5210_100K), I only get 31KB files which are
corrupted (?) and anyway, they are to small to carry any information.
What could be the reason?

Thanks, Nikos

[1] http://dataservice.eea.europa.eu/download.asp?id=18328&filetype=.zip
[2] http://dataservice.eea.europa.eu/dataservice/metadetails.asp?id=760
[3] http://dataservice.eea.europa.eu/download.asp?id=16267&filetype=.zip

Nikos_Alexandris1 · January 12, 2009, 11:23pm

On Mon, 2009-01-12 at 16:19 +0100, Nikos Alexandris wrote:

# get GR_100K.shp which is GREEA18328I.zip [1] from [2]
# under the column "CellCode" are listed all tiles (in this case all
tiles which cover Greece) with their exact name.

#v.db.connect -p
## if requiredδημιουργία της σύνδεσης αν δεν υφίσταται
# db.connect driver=sqlite
database=/geo/grassdb/ellas/laea/PERMANENT/sqlite.d

# import GR_100K.shp
v.in.ogr dsn=/geo/geodata/europe/GREEA18328I/GR_100K.shp
out=eea_gr_100k --o

# check newly imported table
db.select eea_gr_100k

cat|CellCode|EofOrigin|NofOrigin
1|100kmE51N18|5100000|1800000
2|100kmE51N19|5100000|1900000
3|100kmE52N16|5200000|1600000
4|100kmE52N17|5200000|1700000
5|100kmE52N18|5200000|1800000
6|100kmE52N19|5200000|1900000
7|100kmE52N20|5200000|2000000
8|100kmE53N15|5300000|1500000
9|100kmE53N16|5300000|1600000
10|100kmE53N17|5300000|1700000
[...]

# create file with tiles-list
db.select eea_gr_100k -c | cut -d"|" -f2 > corine_tiles_ellas

100kmE51N18
100kmE51N19
100kmE52N16
100kmE52N17
100kmE52N18
100kmE52N19
[...]

# wget within a for loop
for x in `cat corine_tiles_ellas`; do wget
http://dataservice.eea.europa.eu/staticfiles/landcover_tiles_v2/$x.zip;
done

--2009-01-12 15:12:26--
http://dataservice.eea.europa.eu/staticfiles/landcover_tiles_v2/100kmE51N18.zip
Resolving dataservice.eea.europa.eu... 217.74.209.209
Connecting to dataservice.eea.europa.eu|217.74.209.209|:80...
connected.
HTTP request sent, awaiting response... 200 OK
Length: 408346 (399K) [application/x-zip-compressed]
Saving to: `100kmE51N18.zip'
100%
[============================>] 408,346 1.12M/s in 0.3s
2009-01-12 15:12:27 (1.12 MB/s) - `100kmE51N18.zip' saved
[408346/408346]
[...]

Now, if I try to repeat this process for the whole of europe (based on
the file Grid_ETRS89_LAEA5210_100K[3]), I only get 31KB files which are
corrupted (?) and anyway, they are to small to carry any information.
What could be the reason?

(Answerting to myself

It's simple. The attribute table of Grid_ETRS89_LAEA5210_100K holds lots
of "empty cells" since it covers a huge rectangular (a total of more
than 8000 boxes if I remember well).

I downloaded each country's 100k-eea-reference-grid shapefile, imported
all in grass and exported a "european non-empty" list (file attached -
around 1300 boxes). I re-run the wget-for loop and everything works
fine

[1] http://dataservice.eea.europa.eu/download.asp?id=18328&filetype=.zip
[2] http://dataservice.eea.europa.eu/dataservice/metadetails.asp?id=760
[3] http://dataservice.eea.europa.eu/download.asp?id=16267&filetype=.zip

(attachments)

corine_v2_tiles_list_europe (15.9 KB)

Nikos_Alexandris1 · January 13, 2009, 1:25am

On Tue, 2009-01-13 at 00:23 +0100, Nikos Alexandris wrote:

On Mon, 2009-01-12 at 16:19 +0100, Nikos Alexandris wrote:

[...]

The attribute table of Grid_ETRS89_LAEA5210_100K holds lots
of "empty cells" since it covers a huge rectangular (a total of more
than 8000 boxes if I remember well).

I downloaded each country's 100k-eea-reference-grid shapefile, imported
all in grass and exported a "european non-empty" list (file attached -
around 1300 boxes). I re-run the wget-for loop and everything works
fine

> [3] http://dataservice.eea.europa.eu/download.asp?id=16267&filetype=.zip

Something is wrong... While my list has no double entries (I
cross-checked with "sort -u" the output of "db.tables -p"), wget
downloaded 262 tiles two times, another 56 tiles three times and another
8 tiles four times!!

Why is that? Connection problems? Most of the double files have the same
size.

Here how I go about it (for all available reference-grid shapefiles):
# check per-country 100k reference-grid-cells
db.tables -p | grep -v grid

AL_100k
AT_100k
BA_100k
BE_100k
BG_100k
CH_100k
CY_100k
CZ_100k
DE_100k
DK_100k
EE_100k
ES_100k
FI_100k
FR_100k
GR_100k
HR_100k
HU_100k
IE_100k
IS_100k
IT_100k
LT_100k
LV_100k
ME_100k
MK_100k
NL_100k
NO_100k
PL_100k
PT_100k
RO_100k
RS_100k
SE_100k
SI_100k
SK_100k
TR_100k
UK_100k

# count tiles from attribute tables
for x in `db.tables -p | grep -v grid`; do db.select -c $x | cut -d"|"
-f2; done | wc -l

1362

1362

# create a tiles-list
for x in `db.tables -p | grep -v grid`; do db.select -c $x | cut -d"|"
-f2; done > corine_v2_tiles_list_europe

# get tiles
for x in `cat corine_v2_tiles_list_europe`; do wget
http://dataservice.eea.europa.eu/staticfiles/landcover_tiles_v2/$x.zip
-a wget_corine_v2_tiles_europe.log && echo "Tile $x downloaded"; done
## unfortunately I used the log in the wrong way initially: instead of
"-a" I used "-o" which overwrites for each loop an existing log

# check results
# total number of downloaded tiles agrees with the above mentioned count
# but... there a lot of double- and triple-downloaded tiles.

Any clue what's messing up this attempt?

Nikos_Alexandris1 · January 13, 2009, 3:00am

On Mon, 2009-01-12 at 16:19 +0100, Nikos Alexandris wrote:

# get GR_100K.shp which is GREEA18328I.zip [1] from [2]
# under the column "CellCode" are listed all tiles (in this case all
tiles which cover Greece) with their exact name.

[...]

[1] http://dataservice.eea.europa.eu/download.asp?id=18328&filetype=.zip
[2] http://dataservice.eea.europa.eu/dataservice/metadetails.asp?id=760
[3] http://dataservice.eea.europa.eu/download.asp?id=16267&filetype=.zip

Ah, something important I forgot: many thanks to Markus for pointing out
to "wget" and Dimitris Zachariadis for pointing to the
http://dataservice.eea.europa.eu/staticfiles/landcover_tiles_v2/…
address.

Nikos