[pgrouting-users] std::bad_alloc with USA OSM dataset

Ben_Brehmer · December 12, 2010, 5:54am

Hello,

Currently attempting to get PgRouting to work on a USA dataset. I have created the routing table (source, target, id, the_geom etc) with osm2po as all other routing converters (ie. osm2pgrouting) failed due to out of memory. Currently I am attempting to run a shortest_path_astar query. The result is a crashed Database and the following Postgres log:

terminate called after throwing an instance of ‘std::bad_alloc’
what(): St9bad_alloc
LOG: server process (PID 1170) was terminated by signal 6: Aborted
LOG: terminating any other active server processes
FATAL: the database system is in recovery mode
LOG: all server processes terminated; reinitializing
LOG: database system was interrupted; last known up at 2010-12-12 05:17:42 UTC
LOG: database system was not properly shut down; automatic recovery in progress
LOG: record with zero length at 1F/FCF370A8
LOG: redo is not required

I suspect my data may be the problem as it is all of USA (from Cloudmade OSM file). My max source/target ids are quite large:
Max source/target id: 25,638,274

I have confirmed length and valid geometries on all rows in my sample query. I have confirmed valid source/target/ids on all rows in test query.

Without being a knowledgable C programmer, how would you suggest I go about debugging this? I have tried changing ‘int’ types in a_star.c to ‘long’ types, without luck. Without setting up an entire C development environment, is there a way to add debug statements to the C code (and then re-compile)? Tried increasing Postgres logging to the Maximum (debug5) and didn’t receive any more useful information.

Any help is greatly appreciated! For more detailed system info, please read below.

Thanks!

Ben

Currently running on a 64 bit amazon (Large instance):
Linux version 2.6.34.7-56.40.amzn1.x86_64 (mockbuild@build-31005.build) (gcc version 4.4.4 20100525 (Red Hat 4.4.4-5) (GCC) ) #1 SMP Fri Oct 22 18:48:49 UTC 2010

Also tried the Amazon 32 bit Linux instance (Small instance type) and received a “std::length_error, vector::_M_fill_insert” error:
Linux version 2.6.34.7-56.40.amzn1.i686 (mockbuild@build-31003.build) (gcc version 4.4.4 20100525 (Red Hat 4.4.4-5) (GCC) ) #1 SMP Fri Oct 22 18:48:33 UTC 2010

Tried both Pgrouting 1.05 and 1.04.

I am running Postgres 8.4, PostGIS 1.5 as follows:
#select version();

version
------------------------------------------------------------------------------------------------------------------
PostgreSQL 8.4.5 on x86_64-redhat-linux-gnu, compiled by GCC gcc (GCC) 4.1.2 20080704 (Red Hat 4.1.2-48), 64-bit

#select postgis_version();

postgis_version
---------------------------------------
1.5 USE_GEOS=1 USE_PROJ=1 USE_STATS=1

Sample Query:

SELECT * FROM shortest_path_astar(’
SELECT
roads.osm_id as id, roads.source::integer AS source, roads.target::integer AS target, (CASE WHEN roads.cost =0 THEN 1 ELSE roads.cost END)::float8 AS cost, roads.x1, roads.y1, roads.x2, roads.y2
FROM
ways roads
WHERE
(the_geom && ST_Expand(ST_SetSRID(ST_MakeBox2d(ST_Point(-73.945798873896,40.644763731167), ST_Point(-73.955540657038,40.649745616898)),4326), 0.01) )
', 42479854, 42497359, false, false) route ;

Ben_Brehmer · December 12, 2010, 7:06am

As a follow up to my previous email,

I created a new routing table containing the edges that would have been considered in my previous query (on the entire USA). Then ran assign_vertix_id() to rebuild the source/target ids, and issued the same query with success.

This seems to indicate that the source, target and/or id fields are too large in my USA level routing table. This goes inline with this ticket, which has the same error I encountered on my previous 32 bit instance: http://download.osgeo.org/pgrouting/forum/pgrouting.postlbs.org/ticket/149.html

Cheers,
Ben

On 2010-12-11, at 9:54 PM, Ben Brehmer wrote:

Hello,

Currently attempting to get PgRouting to work on a USA dataset. I have created the routing table (source, target, id, the_geom etc) with osm2po as all other routing converters (ie. osm2pgrouting) failed due to out of memory. Currently I am attempting to run a shortest_path_astar query. The result is a crashed Database and the following Postgres log:

terminate called after throwing an instance of ‘std::bad_alloc’
what(): St9bad_alloc
LOG: server process (PID 1170) was terminated by signal 6: Aborted
LOG: terminating any other active server processes
FATAL: the database system is in recovery mode
LOG: all server processes terminated; reinitializing
LOG: database system was interrupted; last known up at 2010-12-12 05:17:42 UTC
LOG: database system was not properly shut down; automatic recovery in progress
LOG: record with zero length at 1F/FCF370A8
LOG: redo is not required

I suspect my data may be the problem as it is all of USA (from Cloudmade OSM file). My max source/target ids are quite large:
Max source/target id: 25,638,274

I have confirmed length and valid geometries on all rows in my sample query. I have confirmed valid source/target/ids on all rows in test query.

Without being a knowledgable C programmer, how would you suggest I go about debugging this? I have tried changing ‘int’ types in a_star.c to ‘long’ types, without luck. Without setting up an entire C development environment, is there a way to add debug statements to the C code (and then re-compile)? Tried increasing Postgres logging to the Maximum (debug5) and didn’t receive any more useful information.

Any help is greatly appreciated! For more detailed system info, please read below.

Thanks!

Ben

Currently running on a 64 bit amazon (Large instance):
Linux version 2.6.34.7-56.40.amzn1.x86_64 (mockbuild@build-31005.build) (gcc version 4.4.4 20100525 (Red Hat 4.4.4-5) (GCC) ) #1 SMP Fri Oct 22 18:48:49 UTC 2010

Also tried the Amazon 32 bit Linux instance (Small instance type) and received a “std::length_error, vector::_M_fill_insert” error:
Linux version 2.6.34.7-56.40.amzn1.i686 (mockbuild@build-31003.build) (gcc version 4.4.4 20100525 (Red Hat 4.4.4-5) (GCC) ) #1 SMP Fri Oct 22 18:48:33 UTC 2010

Tried both Pgrouting 1.05 and 1.04.

I am running Postgres 8.4, PostGIS 1.5 as follows:
#select version();

version
------------------------------------------------------------------------------------------------------------------
PostgreSQL 8.4.5 on x86_64-redhat-linux-gnu, compiled by GCC gcc (GCC) 4.1.2 20080704 (Red Hat 4.1.2-48), 64-bit

#select postgis_version();

postgis_version
---------------------------------------
1.5 USE_GEOS=1 USE_PROJ=1 USE_STATS=1

Sample Query:

SELECT * FROM shortest_path_astar(’
SELECT
roads.osm_id as id, roads.source::integer AS source, roads.target::integer AS target, (CASE WHEN roads.cost =0 THEN 1 ELSE roads.cost END)::float8 AS cost, roads.x1, roads.y1, roads.x2, roads.y2
FROM
ways roads
WHERE
(the_geom && ST_Expand(ST_SetSRID(ST_MakeBox2d(ST_Point(-73.945798873896,40.644763731167), ST_Point(-73.955540657038,40.649745616898)),4326), 0.01) )
', 42479854, 42497359, false, false) route ;

dkastl · December 12, 2010, 3:28pm

Hi Ben,

It’s probably the ID being to large, yes.
osm2po generates an ID that also contains some information about the grid it belongs to, as far as I remember.

So if your total number of road links is OK and it would fir with integer, then it’s better to renumber ID’s, because it will also affect calculation speed. You can keep the ID that osm2po generated and just add a new column and renumber also source and target column accordingly.

Daniel

2010/12/12 Ben Brehmer <ben@geooasis.com>

As a follow up to my previous email,

I created a new routing table containing the edges that would have been considered in my previous query (on the entire USA). Then ran assign_vertix_id() to rebuild the source/target ids, and issued the same query with success.

This seems to indicate that the source, target and/or id fields are too large in my USA level routing table. This goes inline with this ticket, which has the same error I encountered on my previous 32 bit instance: http://download.osgeo.org/pgrouting/forum/pgrouting.postlbs.org/ticket/149.html

Cheers,
Ben

On 2010-12-11, at 9:54 PM, Ben Brehmer wrote:

Hello,

Currently attempting to get PgRouting to work on a USA dataset. I have created the routing table (source, target, id, the_geom etc) with osm2po as all other routing converters (ie. osm2pgrouting) failed due to out of memory. Currently I am attempting to run a shortest_path_astar query. The result is a crashed Database and the following Postgres log:

terminate called after throwing an instance of ‘std::bad_alloc’
what(): St9bad_alloc
LOG: server process (PID 1170) was terminated by signal 6: Aborted
LOG: terminating any other active server processes
FATAL: the database system is in recovery mode
LOG: all server processes terminated; reinitializing
LOG: database system was interrupted; last known up at 2010-12-12 05:17:42 UTC
LOG: database system was not properly shut down; automatic recovery in progress
LOG: record with zero length at 1F/FCF370A8
LOG: redo is not required

I suspect my data may be the problem as it is all of USA (from Cloudmade OSM file). My max source/target ids are quite large:
Max source/target id: 25,638,274

I have confirmed length and valid geometries on all rows in my sample query. I have confirmed valid source/target/ids on all rows in test query.

Without being a knowledgable C programmer, how would you suggest I go about debugging this? I have tried changing ‘int’ types in a_star.c to ‘long’ types, without luck. Without setting up an entire C development environment, is there a way to add debug statements to the C code (and then re-compile)? Tried increasing Postgres logging to the Maximum (debug5) and didn’t receive any more useful information.

Any help is greatly appreciated! For more detailed system info, please read below.

Thanks!

Ben

Currently running on a 64 bit amazon (Large instance):
Linux version 2.6.34.7-56.40.amzn1.x86_64 (mockbuild@build-31005.build) (gcc version 4.4.4 20100525 (Red Hat 4.4.4-5) (GCC) ) #1 SMP Fri Oct 22 18:48:49 UTC 2010

Also tried the Amazon 32 bit Linux instance (Small instance type) and received a “std::length_error, vector::_M_fill_insert” error:
Linux version 2.6.34.7-56.40.amzn1.i686 (mockbuild@build-31003.build) (gcc version 4.4.4 20100525 (Red Hat 4.4.4-5) (GCC) ) #1 SMP Fri Oct 22 18:48:33 UTC 2010

Tried both Pgrouting 1.05 and 1.04.

I am running Postgres 8.4, PostGIS 1.5 as follows:
#select version();

version
------------------------------------------------------------------------------------------------------------------
PostgreSQL 8.4.5 on x86_64-redhat-linux-gnu, compiled by GCC gcc (GCC) 4.1.2 20080704 (Red Hat 4.1.2-48), 64-bit

#select postgis_version();

postgis_version
---------------------------------------
1.5 USE_GEOS=1 USE_PROJ=1 USE_STATS=1

Sample Query:

SELECT * FROM shortest_path_astar(’
SELECT
roads.osm_id as id, roads.source::integer AS source, roads.target::integer AS target, (CASE WHEN roads.cost =0 THEN 1 ELSE roads.cost END)::float8 AS cost, roads.x1, roads.y1, roads.x2, roads.y2
FROM
ways roads
WHERE
(the_geom && ST_Expand(ST_SetSRID(ST_MakeBox2d(ST_Point(-73.945798873896,40.644763731167), ST_Point(-73.955540657038,40.649745616898)),4326), 0.01) )
', 42479854, 42497359, false, false) route ;

Pgrouting-users mailing list
Pgrouting-users@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/pgrouting-users

–
Georepublic UG & Georepublic Japan
eMail: daniel.kastl@georepublic.de
Web: http://georepublic.de