[Geoserver-users] GeoServer getting data NOT from database

GeoServer GeoServer User
1

Hi,

I have the following scenario which I hope someone could give me some
advice on how I should approach it. My application reads data where the
interval between reading each record is about 10 secs, and I need to store
these data in the PostgreSQL database so that the map will reflect these
latest data updates. However, I do not wish to write to the database once
a record is read as such frequent writing would be very expensive. Instead
my intention is to do batch processing of the records after a longer
duration (say, 5mins). But my problem now is that if I do batch
processing, the information shown on my map will not be the most updated.

Is there a way to resolve this problem such that while the records of data
are being accumulated to be batch-processed to the database, these data
can at the same time be stored elsewhere so that the map can read from
there and draw the latest information? In other words, is it possible that
a request sent from my application to GeoServer performs an update with
the data stored in elsewhere? If possible, where would it be, how can
these data be read and how does it work with MapBuilder?

My apologies for the long description and many questions. Would greatly
appreciate help on this. Thanks a lot.

Regards,
Lee Wai See

[This e-mail is confidential and may be priviledged. If you are not the
intended recipient, please kindly notify us immediately and delete the message
from your system; please do not copy or use it for any purpose, nor disclose
its contents to any other person. Thank you.]
---ST Electronics Group---

2

Hi Lee,

Just a comment. Are you sure that inserting data every 10 seconds is
so expensive for your database? PostgreSQL is a very robust database
and should not have problems dealing with this. The idea that you are
proposing, apart of difficult to integrate I believe in Geoserver,
does not sound very efficient neither. I would play much more with the
database before starting figuring out a special solution.

So you have to analyze the whole process. You are going to be updating
the database every 10 sec., but how often are you gonna read from it?
Is it going to be that you have users accessing the information every
15 sec? If you dont expect no so many users in your web application
then your server is not going to be very busy inserting records in the
database.

An now, just a crazy idea that I think will not perform so well, but
still... If what you want to present in MapBuilder is a WMS layer then
you could have a dynamic SLD associated with it so that in the
intervals between updates in the database you insert there in-line
features. Geoserver will draw them together with whatever comes from
the database. Every time you do a batch processing you delete the
inline features of the SLD and the process start again.
Again, I dont think this will perform well, I am pretty sure that it
will perform better if you just insert in the database every 10sec.
But in the situation where you want to avoid having inserts every 10
sec. and you dont have so many users viewing the data but when they do
you want them to be totally up to date, that might work.
Every 10sec you could be writing to the database and to the SLD file
or you can have a temporary table where you insert this intermediate
records and the SLD is generated using PHP or whatever and take the
temporary records from this little table.

But again, try before just using the database because you might be
surprised on how well they perform. Think that inserts can take maybe
0.02sec, if it is every 10 sec, then the inserts in the database will
only represent 0,2% of your server processing time!

I hope it helps you a bit.

Javier.

On 8/7/06, leewse@anonymised.com <leewse@anonymised.com> wrote:

Hi,

I have the following scenario which I hope someone could give me some
advice on how I should approach it. My application reads data where the
interval between reading each record is about 10 secs, and I need to store
these data in the PostgreSQL database so that the map will reflect these
latest data updates. However, I do not wish to write to the database once
a record is read as such frequent writing would be very expensive. Instead
my intention is to do batch processing of the records after a longer
duration (say, 5mins). But my problem now is that if I do batch
processing, the information shown on my map will not be the most updated.

Is there a way to resolve this problem such that while the records of data
are being accumulated to be batch-processed to the database, these data
can at the same time be stored elsewhere so that the map can read from
there and draw the latest information? In other words, is it possible that
a request sent from my application to GeoServer performs an update with
the data stored in elsewhere? If possible, where would it be, how can
these data be read and how does it work with MapBuilder?

My apologies for the long description and many questions. Would greatly
appreciate help on this. Thanks a lot.

Regards,
Lee Wai See

[This e-mail is confidential and may be priviledged. If you are not the
intended recipient, please kindly notify us immediately and delete the message
from your system; please do not copy or use it for any purpose, nor disclose
its contents to any other person. Thank you.]
---ST Electronics Group---

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys -- and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
Geoserver-users mailing list
Geoserver-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-users

3

Well, you could write to a shapefile. But that would take just as much time as writing to the database. If the interval is 10 seconds, that is plenty of time to store the new feature before the next one comes in. So I would give that a try unless it does prove to take too long.
Another way would be write a little program that would write out the data to a temporary file, and those features would be included as inline features in an SLD document to be rendered with the map.
But I think it should be fast enough to insert the feature every 10 seconds.

Brent Owens
(The Open Planning Project)

leewse@anonymised.com wrote:

Hi,

I have the following scenario which I hope someone could give me some
advice on how I should approach it. My application reads data where the
interval between reading each record is about 10 secs, and I need to store
these data in the PostgreSQL database so that the map will reflect these
latest data updates. However, I do not wish to write to the database once
a record is read as such frequent writing would be very expensive. Instead
my intention is to do batch processing of the records after a longer
duration (say, 5mins). But my problem now is that if I do batch
processing, the information shown on my map will not be the most updated.

Is there a way to resolve this problem such that while the records of data
are being accumulated to be batch-processed to the database, these data
can at the same time be stored elsewhere so that the map can read from
there and draw the latest information? In other words, is it possible that
a request sent from my application to GeoServer performs an update with
the data stored in elsewhere? If possible, where would it be, how can
these data be read and how does it work with MapBuilder?

My apologies for the long description and many questions. Would greatly
appreciate help on this. Thanks a lot.

Regards,
Lee Wai See

[This e-mail is confidential and may be priviledged. If you are not the
intended recipient, please kindly notify us immediately and delete the message
from your system; please do not copy or use it for any purpose, nor disclose
its contents to any other person. Thank you.]
---ST Electronics Group---

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys -- and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
Geoserver-users mailing list
Geoserver-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-users

4

You also could write to a memory datastore. That would be fast. Would take up a bit of memory, obviously, but shouldn't be too bad. Though yes, database writing shouldn't be too expensive. GeoServer's transaction reading code isn't as efficient as it could be, but you'd have to use it anyways if you're writing to a memory datastore.

Chris

Brent Owens wrote:

Well, you could write to a shapefile. But that would take just as much time as writing to the database. If the interval is 10 seconds, that is plenty of time to store the new feature before the next one comes in. So I would give that a try unless it does prove to take too long.
Another way would be write a little program that would write out the data to a temporary file, and those features would be included as inline features in an SLD document to be rendered with the map.
But I think it should be fast enough to insert the feature every 10 seconds.

Brent Owens
(The Open Planning Project)

leewse@anonymised.com wrote:

Hi,

I have the following scenario which I hope someone could give me some
advice on how I should approach it. My application reads data where the
interval between reading each record is about 10 secs, and I need to store
these data in the PostgreSQL database so that the map will reflect these
latest data updates. However, I do not wish to write to the database once
a record is read as such frequent writing would be very expensive. Instead
my intention is to do batch processing of the records after a longer
duration (say, 5mins). But my problem now is that if I do batch
processing, the information shown on my map will not be the most updated.

Is there a way to resolve this problem such that while the records of data
are being accumulated to be batch-processed to the database, these data
can at the same time be stored elsewhere so that the map can read from
there and draw the latest information? In other words, is it possible that
a request sent from my application to GeoServer performs an update with
the data stored in elsewhere? If possible, where would it be, how can
these data be read and how does it work with MapBuilder?

My apologies for the long description and many questions. Would greatly
appreciate help on this. Thanks a lot.

Regards,
Lee Wai See

[This e-mail is confidential and may be priviledged. If you are not the
intended recipient, please kindly notify us immediately and delete the message
from your system; please do not copy or use it for any purpose, nor disclose
its contents to any other person. Thank you.]
---ST Electronics Group---

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys -- and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
Geoserver-users mailing list
Geoserver-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-users

-------------------------------------------------------------------------
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
_______________________________________________
Geoserver-users mailing list
Geoserver-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-users

!DSPAM:1003,44d8b71c306551804284693!

--
Chris Holmes
The Open Planning Project
http://topp.openplans.org

5

Hi Brent,

Thanks so much for your reply.

Please bear with me for a few more questions:

1. Will writing to a shapefile as you have suggested be faster, slower or
perhaps about the same time as doing a database insert followed by a read?

2. Then what about the memory between the two options?

Regards,
Wai See

Well, you could write to a shapefile. But that would take just as much
time as writing to the database. If the interval is 10 seconds, that is
plenty of time to store the new feature before the next one comes in. So
I would give that a try unless it does prove to take too long.
Another way would be write a little program that would write out the
data to a temporary file, and those features would be included as inline
features in an SLD document to be rendered with the map.
But I think it should be fast enough to insert the feature every 10
seconds.

Brent Owens
(The Open Planning Project)

leewse@anonymised.com wrote:

Hi,

I have the following scenario which I hope someone could give me some
advice on how I should approach it. My application reads data where the
interval between reading each record is about 10 secs, and I need to
store
these data in the PostgreSQL database so that the map will reflect these
latest data updates. However, I do not wish to write to the database
once
a record is read as such frequent writing would be very expensive.
Instead
my intention is to do batch processing of the records after a longer
duration (say, 5mins). But my problem now is that if I do batch
processing, the information shown on my map will not be the most
updated.

Is there a way to resolve this problem such that while the records of
data
are being accumulated to be batch-processed to the database, these data
can at the same time be stored elsewhere so that the map can read from
there and draw the latest information? In other words, is it possible
that
a request sent from my application to GeoServer performs an update with
the data stored in elsewhere? If possible, where would it be, how can
these data be read and how does it work with MapBuilder?

My apologies for the long description and many questions. Would greatly
appreciate help on this. Thanks a lot.

Regards,
Lee Wai See

[This e-mail is confidential and may be priviledged. If you are not the
intended recipient, please kindly notify us immediately and delete the
message
from your system; please do not copy or use it for any purpose, nor
disclose
its contents to any other person. Thank you.]
---ST Electronics Group---

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share
your
opinions on IT & business topics through brief surveys -- and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
Geoserver-users mailing list
Geoserver-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-users

6

My apologies but one more question: When you mentioned that "it will
perform better if you just insert in the database", do you mean in terms
of both time and memory? Thanks.

Hi Javier,

Thanks so much for the suggestions.

So you have to analyze the whole process. You are going to be updating
the database every 10 sec., but how often are you gonna read from it?
Is it going to be that you have users accessing the information every
15 sec? If you dont expect no so many users in your web application
then your server is not going to be very busy inserting records in the
database.

Please do bear with me for another question: My application works such
that once there is a write to the database, a read will be triggered and
the data will be sent to the users' end. But I suppose it is still
acceptable as it would not be too much more expensive for the database,
based on what I understand from your analysis?

Regards.

Hi Lee,

Just a comment. Are you sure that inserting data every 10 seconds is
so expensive for your database? PostgreSQL is a very robust database
and should not have problems dealing with this. The idea that you are
proposing, apart of difficult to integrate I believe in Geoserver,
does not sound very efficient neither. I would play much more with the
database before starting figuring out a special solution.

So you have to analyze the whole process. You are going to be updating
the database every 10 sec., but how often are you gonna read from it?
Is it going to be that you have users accessing the information every
15 sec? If you dont expect no so many users in your web application
then your server is not going to be very busy inserting records in the
database.

An now, just a crazy idea that I think will not perform so well, but
still... If what you want to present in MapBuilder is a WMS layer then
you could have a dynamic SLD associated with it so that in the
intervals between updates in the database you insert there in-line
features. Geoserver will draw them together with whatever comes from
the database. Every time you do a batch processing you delete the
inline features of the SLD and the process start again.
Again, I dont think this will perform well, I am pretty sure that it
will perform better if you just insert in the database every 10sec.
But in the situation where you want to avoid having inserts every 10
sec. and you dont have so many users viewing the data but when they do
you want them to be totally up to date, that might work.
Every 10sec you could be writing to the database and to the SLD file
or you can have a temporary table where you insert this intermediate
records and the SLD is generated using PHP or whatever and take the
temporary records from this little table.

But again, try before just using the database because you might be
surprised on how well they perform. Think that inserts can take maybe
0.02sec, if it is every 10 sec, then the inserts in the database will
only represent 0,2% of your server processing time!

I hope it helps you a bit.

Javier.

On 8/7/06, leewse@anonymised.com <leewse@anonymised.com> wrote:

Hi,

I have the following scenario which I hope someone could give me some
advice on how I should approach it. My application reads data where the
interval between reading each record is about 10 secs, and I need to
store
these data in the PostgreSQL database so that the map will reflect
these
latest data updates. However, I do not wish to write to the database
once
a record is read as such frequent writing would be very expensive.
Instead
my intention is to do batch processing of the records after a longer
duration (say, 5mins). But my problem now is that if I do batch
processing, the information shown on my map will not be the most
updated.

Is there a way to resolve this problem such that while the records of
data
are being accumulated to be batch-processed to the database, these data
can at the same time be stored elsewhere so that the map can read from
there and draw the latest information? In other words, is it possible
that
a request sent from my application to GeoServer performs an update with
the data stored in elsewhere? If possible, where would it be, how can
these data be read and how does it work with MapBuilder?

My apologies for the long description and many questions. Would greatly
appreciate help on this. Thanks a lot.

Regards,
Lee Wai See

[This e-mail is confidential and may be priviledged. If you are not the
intended recipient, please kindly notify us immediately and delete the
message
from your system; please do not copy or use it for any purpose, nor
disclose
its contents to any other person. Thank you.]
---ST Electronics Group---

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share
your
opinions on IT & business topics through brief surveys -- and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
Geoserver-users mailing list
Geoserver-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-users

-------------------------------------------------------------------------
Using Tomcat but need to do more? Need to support web services,
security?
Get stuff done quickly with pre-integrated technology to make your job
easier
Download IBM WebSphere Application Server v.1.0.1 based on Apache
Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
_______________________________________________
Geoserver-users mailing list
Geoserver-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-users

7

1) The speed should be the same, unless your database is a long ways a way, physically. But if it is local it should be just as fast especially if you are only inserting a feature every 10 seconds.

2) The memory between all Geotools datastores is the same (minus the memory-datastore). Geotools is very smart in how it grabs data as it will only grab a little at a time, no matter what datastore it is using (shapefile, postgis, oracle...). We have yet to break the datastores by throwing too much data at them.

good luck,

Brent Owens
(The Open Planning Project)

leewse@anonymised.com wrote:

Hi Brent,

Thanks so much for your reply.

Please bear with me for a few more questions:

1. Will writing to a shapefile as you have suggested be faster, slower or
perhaps about the same time as doing a database insert followed by a read?

2. Then what about the memory between the two options?

Regards,
Wai See

Well, you could write to a shapefile. But that would take just as much
time as writing to the database. If the interval is 10 seconds, that is
plenty of time to store the new feature before the next one comes in. So
I would give that a try unless it does prove to take too long.
Another way would be write a little program that would write out the
data to a temporary file, and those features would be included as inline
features in an SLD document to be rendered with the map.
But I think it should be fast enough to insert the feature every 10
seconds.

Brent Owens
(The Open Planning Project)

leewse@anonymised.com wrote:
    

Hi,

I have the following scenario which I hope someone could give me some
advice on how I should approach it. My application reads data where the
interval between reading each record is about 10 secs, and I need to
store
these data in the PostgreSQL database so that the map will reflect these
latest data updates. However, I do not wish to write to the database
once
a record is read as such frequent writing would be very expensive.
Instead
my intention is to do batch processing of the records after a longer
duration (say, 5mins). But my problem now is that if I do batch
processing, the information shown on my map will not be the most
updated.

Is there a way to resolve this problem such that while the records of
data
are being accumulated to be batch-processed to the database, these data
can at the same time be stored elsewhere so that the map can read from
there and draw the latest information? In other words, is it possible
that
a request sent from my application to GeoServer performs an update with
the data stored in elsewhere? If possible, where would it be, how can
these data be read and how does it work with MapBuilder?

My apologies for the long description and many questions. Would greatly
appreciate help on this. Thanks a lot.

Regards,
Lee Wai See

[This e-mail is confidential and may be priviledged. If you are not the
intended recipient, please kindly notify us immediately and delete the
message
from your system; please do not copy or use it for any purpose, nor
disclose
its contents to any other person. Thank you.]
---ST Electronics Group---

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share
your
opinions on IT & business topics through brief surveys -- and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
Geoserver-users mailing list
Geoserver-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-users

8

Uhmm..
Inserting into a database is not very memory consuming so I suppose yes. What is memory and time consuming is Geoserver producing the maps from the database, but this is something you can not avoid.

Javier.

On 09/08/2006, at 16:30, leewse@anonymised.com wrote:

My apologies but one more question: When you mentioned that "it will
perform better if you just insert in the database", do you mean in terms
of both time and memory? Thanks.

Hi Javier,

Thanks so much for the suggestions.

So you have to analyze the whole process. You are going to be updating
the database every 10 sec., but how often are you gonna read from it?
Is it going to be that you have users accessing the information every
15 sec? If you dont expect no so many users in your web application
then your server is not going to be very busy inserting records in the
database.

Please do bear with me for another question: My application works such
that once there is a write to the database, a read will be triggered and
the data will be sent to the users' end. But I suppose it is still
acceptable as it would not be too much more expensive for the database,
based on what I understand from your analysis?

Regards.

Hi Lee,

Just a comment. Are you sure that inserting data every 10 seconds is
so expensive for your database? PostgreSQL is a very robust database
and should not have problems dealing with this. The idea that you are
proposing, apart of difficult to integrate I believe in Geoserver,
does not sound very efficient neither. I would play much more with the
database before starting figuring out a special solution.

So you have to analyze the whole process. You are going to be updating
the database every 10 sec., but how often are you gonna read from it?
Is it going to be that you have users accessing the information every
15 sec? If you dont expect no so many users in your web application
then your server is not going to be very busy inserting records in the
database.

An now, just a crazy idea that I think will not perform so well, but
still... If what you want to present in MapBuilder is a WMS layer then
you could have a dynamic SLD associated with it so that in the
intervals between updates in the database you insert there in-line
features. Geoserver will draw them together with whatever comes from
the database. Every time you do a batch processing you delete the
inline features of the SLD and the process start again.
Again, I dont think this will perform well, I am pretty sure that it
will perform better if you just insert in the database every 10sec.
But in the situation where you want to avoid having inserts every 10
sec. and you dont have so many users viewing the data but when they do
you want them to be totally up to date, that might work.
Every 10sec you could be writing to the database and to the SLD file
or you can have a temporary table where you insert this intermediate
records and the SLD is generated using PHP or whatever and take the
temporary records from this little table.

But again, try before just using the database because you might be
surprised on how well they perform. Think that inserts can take maybe
0.02sec, if it is every 10 sec, then the inserts in the database will
only represent 0,2% of your server processing time!

I hope it helps you a bit.

Javier.

On 8/7/06, leewse@anonymised.com <leewse@anonymised.com> wrote:

Hi,

I have the following scenario which I hope someone could give me some
advice on how I should approach it. My application reads data where the
interval between reading each record is about 10 secs, and I need to
store
these data in the PostgreSQL database so that the map will reflect
these
latest data updates. However, I do not wish to write to the database
once
a record is read as such frequent writing would be very expensive.
Instead
my intention is to do batch processing of the records after a longer
duration (say, 5mins). But my problem now is that if I do batch
processing, the information shown on my map will not be the most
updated.

Is there a way to resolve this problem such that while the records of
data
are being accumulated to be batch-processed to the database, these data
can at the same time be stored elsewhere so that the map can read from
there and draw the latest information? In other words, is it possible
that
a request sent from my application to GeoServer performs an update with
the data stored in elsewhere? If possible, where would it be, how can
these data be read and how does it work with MapBuilder?

My apologies for the long description and many questions. Would greatly
appreciate help on this. Thanks a lot.

Regards,
Lee Wai See

[This e-mail is confidential and may be priviledged. If you are not the
intended recipient, please kindly notify us immediately and delete the
message
from your system; please do not copy or use it for any purpose, nor
disclose
its contents to any other person. Thank you.]
---ST Electronics Group---

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share
your
opinions on IT & business topics through brief surveys -- and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
Geoserver-users mailing list
Geoserver-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-users

-------------------------------------------------------------------------
Using Tomcat but need to do more? Need to support web services,
security?
Get stuff done quickly with pre-integrated technology to make your job
easier
Download IBM WebSphere Application Server v.1.0.1 based on Apache
Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
_______________________________________________
Geoserver-users mailing list
Geoserver-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-users