Hi,
is there any progress in memory management? [1]
I create 5000 times a distance matrix (pgr_kdijkstra) with ~300 points and run oom. (>20 GB)
I avoid that by just split the data and open multiple connections (one after another) but it is a bit complicated.
[1] https://github.com/pgRouting/pgrouting/issues?q=is%3Aissue+is%3Aopen+memory
--
Cheers
yobiSource
On 12/22/2014 8:06 PM, yobiSource wrote:
Hi,
is there any progress in memory management? [1]
I create 5000 times a distance matrix (pgr_kdijkstra) with ~300 points
and run oom. (>20 GB)I avoid that by just split the data and open multiple connections (one
after another) but it is a bit complicated.[1]
https://github.com/pgRouting/pgrouting/issues?q=is%3Aissue+is%3Aopen+memory
Hi yobi,
I can appreciate how frustrating issues are like this. They are for the developers also. Maybe you should open a ticket for this issue and decribe what you are doing in some detail. Ideally getting a small database dump and a query that reproduces problem would be very helpful.
While there are other commands that have memory issues, they are most likely not related to your OOM issue. Information you should post in the ticket includes:
OS: Windows, Linux
Hardware: 32/64 bit, Memory, amount of swapfile space
How many shared_buffers in your postgresql.conf file?
How many edges in your graph?
select count(*) from edge_table;
edge and node ids:
select min(source) as smin, max(source) as smaxm, min(target) as tmin, max(target) as tmax, min(gid) as gmin, max(gid) as gmax
from edge_table;
What is your query that is running out on memory?
Describe the steps to reproduce the problem.
Then you can refer to the ticket number when asking about the issue and all the data is in one place.
As far as progress, we have been tied up on other projects because we get very little funding for pgrouting, but with some more detailed information we might be able to help you work around the issue.
Also you might want to look at the tools in github branch develop in src/common/sql/pgrouting_conversion_tools.sql which has a lot of tools for building distance matrices.
Also src/kdijkstra/src/k_targets_sp.c which has some new stuff that is not in the tagged 2.0.0 release.
-Steve
Thanks for the response.
I have collected all the data in a Gist:
https://gist.github.com/yobiSource/ab024968aeb390452825
in short:
Ubuntu 12.04 x64
Mem/Swap: 16GB/16GB
PostgreSQL 9.3.5 (PostGis 2.1.4)
shared_buffers = 3GB
edge_table (OSM data) count=1733763 size=246MB
smin=1 smax=1606592 tmin=1 tmax=1606590 gmin=1 gmax=1733763
The query [1] uses the (optimized) function pgr_vidstodmatrix from dev. After 5:20 hours, the most memory and swap is in use.
The memory is released only when the connection is closed, even when the execution is terminated!
I'm creating a ticket after Christmas.
I noticed in k_targets_boost_wrapper line 36-49 possibly a solution but i can't compile on the production-server.
[1] https://gist.github.com/yobiSource/ab024968aeb390452825#file-sql
[2] https://github.com/pgRouting/pgrouting/blob/develop/src/kdijkstra/src/k_targets_boost_wrapper.cpp
Am 23.12.2014 um 02:31 schrieb Stephen Woodbridge:
On 12/22/2014 8:06 PM, yobiSource wrote:
Hi,
is there any progress in memory management? [1]
I create 5000 times a distance matrix (pgr_kdijkstra) with ~300 points
and run oom. (>20 GB)I avoid that by just split the data and open multiple connections (one
after another) but it is a bit complicated.[1]
https://github.com/pgRouting/pgrouting/issues?q=is%3Aissue+is%3Aopen+memoryHi yobi,
I can appreciate how frustrating issues are like this. They are for the
developers also. Maybe you should open a ticket for this issue and
decribe what you are doing in some detail. Ideally getting a small
database dump and a query that reproduces problem would be very helpful.While there are other commands that have memory issues, they are most
likely not related to your OOM issue. Information you should post in the
ticket includes:OS: Windows, Linux
Hardware: 32/64 bit, Memory, amount of swapfile space
How many shared_buffers in your postgresql.conf file?
How many edges in your graph?
select count(*) from edge_table;edge and node ids:
select min(source) as smin, max(source) as smaxm, min(target) as
tmin, max(target) as tmax, min(gid) as gmin, max(gid) as gmax
from edge_table;What is your query that is running out on memory?
Describe the steps to reproduce the problem.
Then you can refer to the ticket number when asking about the issue and
all the data is in one place.As far as progress, we have been tied up on other projects because we
get very little funding for pgrouting, but with some more detailed
information we might be able to help you work around the issue.Also you might want to look at the tools in github branch develop in
src/common/sql/pgrouting_conversion_tools.sql which has a lot of tools
for building distance matrices.Also src/kdijkstra/src/k_targets_sp.c which has some new stuff that is
not in the tagged 2.0.0 release.-Steve
_______________________________________________
pgrouting-dev mailing list
pgrouting-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/pgrouting-dev
--
Cheers
yobiSource
Ok, this is great detail so please post a ticket when you have a chance. It clearly sounds like this is a serious memory leak, I took a quick look at the source but didn't see anything obvious. Unfortunately, I will not have any time in the next month to look at this.
You can't do this on a production machine because it needs to shut down the server, but this script runs the backend process under valgrind which is a very handy memory checker.
change the path to the postgresql stuff to match your system.
#!/bin/bash
sudo service postgresql stop
echo "SELECT pgr_vidstodmatrix(vertex_ids, 'edge_table', bbox) as dm FROM dm_groups" | sudo -u postgres valgrind --leak-check=yes -v --undef-value-errors=yes --track-origins=yes --log-file=valgrindlog /usr/lib/postgresql/9.3/bin/postgres --single -D /etc/postgresql/9.3/main/ your_database_name
sudo service postgresql start
-Steve
On 12/24/2014 1:25 PM, yobiSource wrote:
Thanks for the response.
I have collected all the data in a Gist:
https://gist.github.com/yobiSource/ab024968aeb390452825in short:
Ubuntu 12.04 x64
Mem/Swap: 16GB/16GB
PostgreSQL 9.3.5 (PostGis 2.1.4)
shared_buffers = 3GB
edge_table (OSM data) count=1733763 size=246MB
smin=1 smax=1606592 tmin=1 tmax=1606590 gmin=1 gmax=1733763The query [1] uses the (optimized) function pgr_vidstodmatrix from dev.
After 5:20 hours, the most memory and swap is in use.
The memory is released only when the connection is closed, even when the
execution is terminated!I'm creating a ticket after Christmas.
I noticed in k_targets_boost_wrapper line 36-49 possibly a solution but
i can't compile on the production-server.[1] https://gist.github.com/yobiSource/ab024968aeb390452825#file-sql
[2]
https://github.com/pgRouting/pgrouting/blob/develop/src/kdijkstra/src/k_targets_boost_wrapper.cppAm 23.12.2014 um 02:31 schrieb Stephen Woodbridge:
On 12/22/2014 8:06 PM, yobiSource wrote:
Hi,
is there any progress in memory management? [1]
I create 5000 times a distance matrix (pgr_kdijkstra) with ~300 points
and run oom. (>20 GB)I avoid that by just split the data and open multiple connections (one
after another) but it is a bit complicated.[1]
https://github.com/pgRouting/pgrouting/issues?q=is%3Aissue+is%3Aopen+memoryHi yobi,
I can appreciate how frustrating issues are like this. They are for the
developers also. Maybe you should open a ticket for this issue and
decribe what you are doing in some detail. Ideally getting a small
database dump and a query that reproduces problem would be very helpful.While there are other commands that have memory issues, they are most
likely not related to your OOM issue. Information you should post in the
ticket includes:OS: Windows, Linux
Hardware: 32/64 bit, Memory, amount of swapfile space
How many shared_buffers in your postgresql.conf file?
How many edges in your graph?
select count(*) from edge_table;edge and node ids:
select min(source) as smin, max(source) as smaxm, min(target) as
tmin, max(target) as tmax, min(gid) as gmin, max(gid) as gmax
from edge_table;What is your query that is running out on memory?
Describe the steps to reproduce the problem.
Then you can refer to the ticket number when asking about the issue and
all the data is in one place.As far as progress, we have been tied up on other projects because we
get very little funding for pgrouting, but with some more detailed
information we might be able to help you work around the issue.Also you might want to look at the tools in github branch develop in
src/common/sql/pgrouting_conversion_tools.sql which has a lot of tools
for building distance matrices.Also src/kdijkstra/src/k_targets_sp.c which has some new stuff that is
not in the tagged 2.0.0 release.-Steve
_______________________________________________
pgrouting-dev mailing list
pgrouting-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/pgrouting-dev
I was able to run over the holidays that script.
But I had to abort after 66:30 hours.
log output: https://gist.github.com/yobiSource/ab024968aeb390452825#file-valgrind
Am 25.12.2014 um 05:09 schrieb Stephen Woodbridge:
Ok, this is great detail so please post a ticket when you have a chance.
It clearly sounds like this is a serious memory leak, I took a quick
look at the source but didn't see anything obvious. Unfortunately, I
will not have any time in the next month to look at this.You can't do this on a production machine because it needs to shut down
the server, but this script runs the backend process under valgrind
which is a very handy memory checker.change the path to the postgresql stuff to match your system.
#!/bin/bash
sudo service postgresql stop
echo "SELECT pgr_vidstodmatrix(vertex_ids, 'edge_table', bbox) as dm
FROM dm_groups" | sudo -u postgres valgrind --leak-check=yes -v
--undef-value-errors=yes --track-origins=yes --log-file=valgrindlog
/usr/lib/postgresql/9.3/bin/postgres --single -D
/etc/postgresql/9.3/main/ your_database_namesudo service postgresql start
-Steve
On 12/24/2014 1:25 PM, yobiSource wrote:
Thanks for the response.
I have collected all the data in a Gist:
https://gist.github.com/yobiSource/ab024968aeb390452825in short:
Ubuntu 12.04 x64
Mem/Swap: 16GB/16GB
PostgreSQL 9.3.5 (PostGis 2.1.4)
shared_buffers = 3GB
edge_table (OSM data) count=1733763 size=246MB
smin=1 smax=1606592 tmin=1 tmax=1606590 gmin=1 gmax=1733763The query [1] uses the (optimized) function pgr_vidstodmatrix from dev.
After 5:20 hours, the most memory and swap is in use.
The memory is released only when the connection is closed, even when the
execution is terminated!I'm creating a ticket after Christmas.
I noticed in k_targets_boost_wrapper line 36-49 possibly a solution but
i can't compile on the production-server.[1] https://gist.github.com/yobiSource/ab024968aeb390452825#file-sql
[2]
https://github.com/pgRouting/pgrouting/blob/develop/src/kdijkstra/src/k_targets_boost_wrapper.cppAm 23.12.2014 um 02:31 schrieb Stephen Woodbridge:
On 12/22/2014 8:06 PM, yobiSource wrote:
Hi,
is there any progress in memory management? [1]
I create 5000 times a distance matrix (pgr_kdijkstra) with ~300 points
and run oom. (>20 GB)I avoid that by just split the data and open multiple connections (one
after another) but it is a bit complicated.[1]
https://github.com/pgRouting/pgrouting/issues?q=is%3Aissue+is%3Aopen+memoryHi yobi,
I can appreciate how frustrating issues are like this. They are for the
developers also. Maybe you should open a ticket for this issue and
decribe what you are doing in some detail. Ideally getting a small
database dump and a query that reproduces problem would be very helpful.While there are other commands that have memory issues, they are most
likely not related to your OOM issue. Information you should post in the
ticket includes:OS: Windows, Linux
Hardware: 32/64 bit, Memory, amount of swapfile space
How many shared_buffers in your postgresql.conf file?
How many edges in your graph?
select count(*) from edge_table;edge and node ids:
select min(source) as smin, max(source) as smaxm, min(target) as
tmin, max(target) as tmax, min(gid) as gmin, max(gid) as gmax
from edge_table;What is your query that is running out on memory?
Describe the steps to reproduce the problem.
Then you can refer to the ticket number when asking about the issue and
all the data is in one place.As far as progress, we have been tied up on other projects because we
get very little funding for pgrouting, but with some more detailed
information we might be able to help you work around the issue.Also you might want to look at the tools in github branch develop in
src/common/sql/pgrouting_conversion_tools.sql which has a lot of tools
for building distance matrices.Also src/kdijkstra/src/k_targets_sp.c which has some new stuff that is
not in the tagged 2.0.0 release.-Steve
_______________________________________________
pgrouting-dev mailing list
pgrouting-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/pgrouting-dev_______________________________________________
pgrouting-dev mailing list
pgrouting-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/pgrouting-dev
--
Cheers
yobiSource