[pgrouting-dev] [pgrouting] need help with std::bad_alloc issue

Hi devs,

I have run into a strange problem with some pgrouting functions that you guys might have already seen in postgis.

We have stored procedures that are C and C++ and in general they work fine, but if I create a database, connect to it, create extensions and run some commands I get std::bad_alloc error. If I simply reconnect to the database, the same command does not generate an error, and, in fact, if I reconnect after installing the extension, I never get this error.

I have traced this down to the particular statement that is throwing the error, but there is nothing unique or particular about it. And we have seen this behavior in multiple commands.

So I have to conclude that:

1. we use the same pattern for most of our commands so that might be flawed in some basic way regarding memory

2. that there is something strange about create extension in that the libraries are not getting initialized correctly (or completely?) until a connection is made. We are trying to verify this is or is not unique to pg 9.2.

3. pgrouting installs multiple shared libraries in our extension and maybe postgresql assumes there is only going to be one shared library

4. or something else totally different that we are missing

So has anyone seen anything like this with postgis code?
Any thoughts on what this might be? or how to run it down?

I did post post a inquiry to the postgresql list and Tom responded with not enough information and to compile the server with --enable-cassert which I did (assuming my Debian recompile worked correctly), but since this is a C++ error and not a postgresql palloc issue we have not seen and cassert errors.

Thoughts?

-Steve

Hello all,

In case it helps:

I was having problems with C++ memory allocations seeming to clobber memory allocated in the C code, e.g. the array of edges from the SPI query. When the C++ vector of nodes was resized in my version of the cpp_core function in bd_astar, the array of edges from C turned to garbage. I got around this by allocating another array of edges in the C++ code and copying the data prior to the vector resize, but I assume something improper still lurks beneath the surface.

If it's relevant, I was also unable to use openMP because it crashes at the first new of a C++ copy constructor when making a copy of the routing class for each thread. With just one thread, the copy constructor is still used and works fine. I was able to route ~500k trips in less than an hour using this code single threaded but with multiple simultaneous queries, so it does "generally" work. These behaviors seem the same on Mac 10.8 and Ubuntu 13, both Postgres 9.2.4.

I did see a post that it's possible to run the entire postgres process under valgrind to get more info on memory, but I haven't had the time to attempt it.

Alec

On Jun 11, 2013, at 10:33 AM, Stephen Woodbridge <woodbri@swoodbridge.com> wrote:

Hi devs,

I have run into a strange problem with some pgrouting functions that you guys might have already seen in postgis.

We have stored procedures that are C and C++ and in general they work fine, but if I create a database, connect to it, create extensions and run some commands I get std::bad_alloc error. If I simply reconnect to the database, the same command does not generate an error, and, in fact, if I reconnect after installing the extension, I never get this error.

I have traced this down to the particular statement that is throwing the error, but there is nothing unique or particular about it. And we have seen this behavior in multiple commands.

So I have to conclude that:

1. we use the same pattern for most of our commands so that might be flawed in some basic way regarding memory

2. that there is something strange about create extension in that the libraries are not getting initialized correctly (or completely?) until a connection is made. We are trying to verify this is or is not unique to pg 9.2.

3. pgrouting installs multiple shared libraries in our extension and maybe postgresql assumes there is only going to be one shared library

4. or something else totally different that we are missing

So has anyone seen anything like this with postgis code?
Any thoughts on what this might be? or how to run it down?

I did post post a inquiry to the postgresql list and Tom responded with not enough information and to compile the server with --enable-cassert which I did (assuming my Debian recompile worked correctly), but since this is a C++ error and not a postgresql palloc issue we have not seen and cassert errors.

Thoughts?

-Steve
_______________________________________________
pgrouting-dev mailing list
pgrouting-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/pgrouting-dev

Dave - moving this over to the pgrouting list.

Hi Dave,

I'm not sure the Daniel used the example data network for this example. The data looks like one of the matrix I used in my test cases.

src/tsp/test/*.test

I just made up numbers for a symmetric matrix. It was not taken from an actual network.

So the sly part is that its NOT from a graph :wink:

Daniel - please correct me it you did something else.

-Steve

On 6/13/2013 6:11 PM, Dave Potts wrote:

In the travel sales person problem an example is provided, I having
problems understanding it

The example route network is posted as
http://docs.pgrouting.org/dev/doc/src/developer/sampledata.html#sampledata

I have assume the distance between red node 1 and red node 3 is 2, ie
the unit distance 1 +1 which I make to be 2

given that

I think the route matrix should be

     1 2 3 4

1 | 0 1 2 3
2 | 1 0 1 2
3 | 2 1 0 1
4 | 3 2 1 0

distance 1 to 4 is 1+1 +1 ==3
distance 2 to 4 is 1+1 ==2
distance 3 to 4 is 1
etc

But the give value is (please see
http://docs.pgrouting.org/dev/src/tsp/doc/index.html)
     1 2 3 4

1 {0,1,2,3},
2 {1,0,3,2},
3 {2,3,0,4},
4 {3,2,4,0}

I just don't understand how the distance between 3 and 4 can be the
value 4, ditto he values between 2 and 3

Have I missed something sly ?

Dave.

_______________________________________________
postgis-devel mailing list
postgis-devel@lists.osgeo.org
http://lists.osgeo.org/cgi-bin/mailman/listinfo/postgis-devel

_______________________________________________
postgis-devel mailing list
postgis-devel@lists.osgeo.org
http://lists.osgeo.org/cgi-bin/mailman/listinfo/postgis-devel

Hi Dave,

Thanks for reporting. Maybe I just mixed up something with copy & paste.
I remember that I needed to add a vertex table to the sample data for the new TSP with distance matrix and got the ID’s a bit wrong first.
Will take a look.

Daniel

···

On Fri, Jun 14, 2013 at 7:40 AM, Stephen Woodbridge <woodbri@swoodbridge.com> wrote:

Dave - moving this over to the pgrouting list.

Hi Dave,

I’m not sure the Daniel used the example data network for this example. The data looks like one of the matrix I used in my test cases.

src/tsp/test/*.test

I just made up numbers for a symmetric matrix. It was not taken from an actual network.

So the sly part is that its NOT from a graph :wink:

Daniel - please correct me it you did something else.

-Steve

On 6/13/2013 6:11 PM, Dave Potts wrote:

In the travel sales person problem an example is provided, I having
problems understanding it

The example route network is posted as
http://docs.pgrouting.org/dev/doc/src/developer/sampledata.html#sampledata

I have assume the distance between red node 1 and red node 3 is 2, ie
the unit distance 1 +1 which I make to be 2

given that

I think the route matrix should be

1 2 3 4

1 | 0 1 2 3
2 | 1 0 1 2
3 | 2 1 0 1
4 | 3 2 1 0

distance 1 to 4 is 1+1 +1 ==3
distance 2 to 4 is 1+1 ==2
distance 3 to 4 is 1
etc

But the give value is (please see
http://docs.pgrouting.org/dev/src/tsp/doc/index.html)
1 2 3 4

1 {0,1,2,3},
2 {1,0,3,2},
3 {2,3,0,4},
4 {3,2,4,0}

I just don’t understand how the distance between 3 and 4 can be the
value 4, ditto he values between 2 and 3

Have I missed something sly ?

Dave.


postgis-devel mailing list
postgis-devel@lists.osgeo.org
http://lists.osgeo.org/cgi-bin/mailman/listinfo/postgis-devel


postgis-devel mailing list
postgis-devel@lists.osgeo.org
http://lists.osgeo.org/cgi-bin/mailman/listinfo/postgis-devel


pgrouting-dev mailing list
pgrouting-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/pgrouting-dev


Georepublic UG & Georepublic Japan
eMail: daniel.kastl@georepublic.de
Web: http://georepublic.de

On 14/06/13 02:59, Daniel Kastl wrote:
Hi Daniel, List

I must admit I found the entire process of creating a postgress array in a suitable form for the pgr_tsp function to be a exercise in slow torture!

Would anybody have any major objections if I attempted to rewrite the interface so it took a normal sql querry?

Something like

pgr_costResultpgr_tsp('SELECT source,target distance FROM vertex_table',2);

That way the problem with creating a square array in postgres would disappear.

Dave

<https://www.google.nl/search?client=ubuntu&hs=uQi&channel=fs&gl=uk&q=exercise+in+slow+torture&spell=1&sa=X&ei=BKa6UYnAF5G20QXc_4HQCg&ved=0CCcQvwUoAA&gt;

Hi Dave,

Thanks for reporting. Maybe I just mixed up something with copy & paste.
I remember that I needed to add a vertex table to the sample data for the new TSP with distance matrix and got the ID's a bit wrong first.
Will take a look.

Daniel

On Fri, Jun 14, 2013 at 7:40 AM, Stephen Woodbridge <woodbri@swoodbridge.com <mailto:woodbri@swoodbridge.com>> wrote:

    Dave - moving this over to the pgrouting list.

    Hi Dave,

    I'm not sure the Daniel used the example data network for this
    example. The data looks like one of the matrix I used in my test
    cases.

    src/tsp/test/*.test

    I just made up numbers for a symmetric matrix. It was not taken
    from an actual network.

    So the sly part is that its NOT from a graph :wink:

    Daniel - please correct me it you did something else.

    -Steve

    On 6/13/2013 6:11 PM, Dave Potts wrote:

        In the travel sales person problem an example is provided, I
        having
        problems understanding it

        The example route network is posted as
        http://docs.pgrouting.org/dev/doc/src/developer/sampledata.html#sampledata

        I have assume the distance between red node 1 and red node 3
        is 2, ie
        the unit distance 1 +1 which I make to be 2

        given that

        I think the route matrix should be

             1 2 3 4
        ======
        1 | 0 1 2 3
        2 | 1 0 1 2
        3 | 2 1 0 1
        4 | 3 2 1 0

        distance 1 to 4 is 1+1 +1 ==3
        distance 2 to 4 is 1+1 ==2
        distance 3 to 4 is 1
        etc

        But the give value is (please see
        http://docs.pgrouting.org/dev/src/tsp/doc/index.html)
             1 2 3 4

        1 {0,1,2,3},
        2 {1,0,3,2},
        3 {2,3,0,4},
        4 {3,2,4,0}

        I just don't understand how the distance between 3 and 4 can
        be the
        value 4, ditto he values between 2 and 3

        Have I missed something sly ?

        Dave.

        _______________________________________________
        postgis-devel mailing list
        postgis-devel@lists.osgeo.org
        <mailto:postgis-devel@lists.osgeo.org>
        http://lists.osgeo.org/cgi-bin/mailman/listinfo/postgis-devel

    _______________________________________________
    postgis-devel mailing list
    postgis-devel@lists.osgeo.org <mailto:postgis-devel@lists.osgeo.org>
    http://lists.osgeo.org/cgi-bin/mailman/listinfo/postgis-devel
    _______________________________________________
    pgrouting-dev mailing list
    pgrouting-dev@lists.osgeo.org <mailto:pgrouting-dev@lists.osgeo.org>
    http://lists.osgeo.org/mailman/listinfo/pgrouting-dev

--
Georepublic UG & Georepublic Japan
eMail: daniel.kastl@georepublic.de <mailto:daniel.kastl@georepublic.de>
Web: http://georepublic.de/&gt;

_______________________________________________
pgrouting-dev mailing list
pgrouting-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/pgrouting-dev

Hi Dave,

Somehow we already discussed this in May:
http://pgrouting.974090.n3.nabble.com/pgrouting-dev-New-TSP-API-td4025094.html

I think the current function pgr_tsp function is good as it is, but some alternative with SQL query is a good idea.
Then there would be two ways how to call TSP, for example

pgr_tsp(matrix float, start integer)

pgr_tsp(sql text, start integer)

···

But instead of rewriting it’s better to write a wrapper function that accepts SQL and transforms it into the matrix (and ensures that the matrix is valid).

Daniel

On Fri, Jun 14, 2013 at 2:17 PM, Dave Potts <dave.potts@pinan.co.uk> wrote:

On 14/06/13 02:59, Daniel Kastl wrote:
Hi Daniel, List

I must admit I found the entire process of creating a postgress array in a suitable form for the pgr_tsp function to be a exercise in slow torture!

Would anybody have any major objections if I attempted to rewrite the interface so it took a normal sql querry?

Something like

pgr_costResult[] pgr_tsp('SELECT source,target distance FROM vertex_table',2);

That way the problem with creating a square array in postgres would disappear. 

Dave

Hi Dave,

Thanks for reporting. Maybe I just mixed up something with copy & paste.
I remember that I needed to add a vertex table to the sample data for the new TSP with distance matrix and got the ID’s a bit wrong first.
Will take a look.

Daniel

_______________________________________________
pgrouting-dev mailing list
[pgrouting-dev@lists.osgeo.org](mailto:pgrouting-dev@lists.osgeo.org)
[http://lists.osgeo.org/mailman/listinfo/pgrouting-dev](http://lists.osgeo.org/mailman/listinfo/pgrouting-dev)


pgrouting-dev mailing list
pgrouting-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/pgrouting-dev


Georepublic UG & Georepublic Japan
eMail: daniel.kastl@georepublic.de
Web: http://georepublic.de

On Fri, Jun 14, 2013 at 7:40 AM, Stephen Woodbridge <woodbri@swoodbridge.com> wrote:

Dave - moving this over to the pgrouting list.

Hi Dave,

I’m not sure the Daniel used the example data network for this example. The data looks like one of the matrix I used in my test cases.

src/tsp/test/*.test

I just made up numbers for a symmetric matrix. It was not taken from an actual network.

So the sly part is that its NOT from a graph :wink:

Daniel - please correct me it you did something else.

-Steve

On 6/13/2013 6:11 PM, Dave Potts wrote:

In the travel sales person problem an example is provided, I having
problems understanding it

The example route network is posted as
http://docs.pgrouting.org/dev/doc/src/developer/sampledata.html#sampledata

I have assume the distance between red node 1 and red node 3 is 2, ie
the unit distance 1 +1 which I make to be 2

given that

I think the route matrix should be

1 2 3 4

1 | 0 1 2 3
2 | 1 0 1 2
3 | 2 1 0 1
4 | 3 2 1 0

distance 1 to 4 is 1+1 +1 ==3
distance 2 to 4 is 1+1 ==2
distance 3 to 4 is 1
etc

But the give value is (please see
http://docs.pgrouting.org/dev/src/tsp/doc/index.html)
1 2 3 4

1 {0,1,2,3},
2 {1,0,3,2},
3 {2,3,0,4},
4 {3,2,4,0}

I just don’t understand how the distance between 3 and 4 can be the
value 4, ditto he values between 2 and 3

Have I missed something sly ?

Dave.


postgis-devel mailing list
postgis-devel@lists.osgeo.org
http://lists.osgeo.org/cgi-bin/mailman/listinfo/postgis-devel


postgis-devel mailing list
postgis-devel@lists.osgeo.org
http://lists.osgeo.org/cgi-bin/mailman/listinfo/postgis-devel


pgrouting-dev mailing list
pgrouting-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/pgrouting-dev


Georepublic UG & Georepublic Japan
eMail: daniel.kastl@georepublic.de
Web: http://georepublic.de

Hi Dave,

I'm not totally opposed to this, but I would like to discuss how you plan to structure the data and what your query would look like.

There is another possibility that I like better and that would be to write stored procedure that takes you sql query and returns a matrix.

matrix pgr_matrix(sql text)

Then you can do something like:

select * from pgr_tsp(pgr_matrix('select * from matrix_table'), 27);

This is like to be reusable elsewhere as a utility.

-Steve

On 6/14/2013 1:38 AM, Daniel Kastl wrote:

Hi Dave,

Somehow we already discussed this in May:
http://pgrouting.974090.n3.nabble.com/pgrouting-dev-New-TSP-API-td4025094.html

I think the current function pgr_tsp function is good as it is, but some
alternative with SQL query is a good idea.
Then there would be two ways how to call TSP, for example

pgr_tsp(matrix float, start integer)
pgr_tsp(sql text, start integer)

But instead of rewriting it's better to write a wrapper function that
accepts SQL and transforms it into the matrix (and ensures that the
matrix is valid).

Daniel

On Fri, Jun 14, 2013 at 2:17 PM, Dave Potts <dave.potts@pinan.co.uk
<mailto:dave.potts@pinan.co.uk>> wrote:

    On 14/06/13 02:59, Daniel Kastl wrote:
    Hi Daniel, List

    I must admit I found the entire process of creating a postgress
    array in a suitable form for the pgr_tsp function to be a exercise
    in slow torture!

    Would anybody have any major objections if I attempted to rewrite
    the interface so it took a normal sql querry?

    Something like

    pgr_costResultpgr_tsp('SELECT source,target distance FROM vertex_table',2);

    That way the problem with creating a square array in postgres would disappear.

    Dave

    <https://www.google.nl/search?client=ubuntu&hs=uQi&channel=fs&gl=uk&q=exercise+in+slow+torture&spell=1&sa=X&ei=BKa6UYnAF5G20QXc_4HQCg&ved=0CCcQvwUoAA&gt;

    Hi Dave,

    Thanks for reporting. Maybe I just mixed up something with copy &
    paste.
    I remember that I needed to add a vertex table to the sample data
    for the new TSP with distance matrix and got the ID's a bit wrong
    first.
    Will take a look.

    Daniel

    On Fri, Jun 14, 2013 at 7:40 AM, Stephen Woodbridge
    <woodbri@swoodbridge.com <mailto:woodbri@swoodbridge.com>> wrote:

        Dave - moving this over to the pgrouting list.

        Hi Dave,

        I'm not sure the Daniel used the example data network for this
        example. The data looks like one of the matrix I used in my
        test cases.

        src/tsp/test/*.test

        I just made up numbers for a symmetric matrix. It was not
        taken from an actual network.

        So the sly part is that its NOT from a graph :wink:

        Daniel - please correct me it you did something else.

        -Steve

        On 6/13/2013 6:11 PM, Dave Potts wrote:

            In the travel sales person problem an example is provided,
             I having
            problems understanding it

            The example route network is posted as
            http://docs.pgrouting.org/dev/doc/src/developer/sampledata.html#sampledata

            I have assume the distance between red node 1 and red node
            3 is 2, ie
            the unit distance 1 +1 which I make to be 2

            given that

            I think the route matrix should be

                 1 2 3 4
            ======
            1 | 0 1 2 3
            2 | 1 0 1 2
            3 | 2 1 0 1
            4 | 3 2 1 0

            distance 1 to 4 is 1+1 +1 ==3
            distance 2 to 4 is 1+1 ==2
            distance 3 to 4 is 1
            etc

            But the give value is (please see
            http://docs.pgrouting.org/dev/src/tsp/doc/index.html)
                 1 2 3 4

            1 {0,1,2,3},
            2 {1,0,3,2},
            3 {2,3,0,4},
            4 {3,2,4,0}

            I just don't understand how the distance between 3 and 4
            can be the
            value 4, ditto he values between 2 and 3

            Have I missed something sly ?

            Dave.

            _______________________________________________
            postgis-devel mailing list
            postgis-devel@lists.osgeo.org
            <mailto:postgis-devel@lists.osgeo.org>
            http://lists.osgeo.org/cgi-bin/mailman/listinfo/postgis-devel

        _______________________________________________
        postgis-devel mailing list
        postgis-devel@lists.osgeo.org
        <mailto:postgis-devel@lists.osgeo.org>
        http://lists.osgeo.org/cgi-bin/mailman/listinfo/postgis-devel
        _______________________________________________
        pgrouting-dev mailing list
        pgrouting-dev@lists.osgeo.org
        <mailto:pgrouting-dev@lists.osgeo.org>
        http://lists.osgeo.org/mailman/listinfo/pgrouting-dev

    --
    Georepublic UG & Georepublic Japan
    eMail: daniel.kastl@georepublic.de
    <mailto:daniel.kastl@georepublic.de>
    Web: http://georepublic.de/&gt;

    _______________________________________________
    pgrouting-dev mailing list
    pgrouting-dev@lists.osgeo.org <mailto:pgrouting-dev@lists.osgeo.org>
    http://lists.osgeo.org/mailman/listinfo/pgrouting-dev

    _______________________________________________
    pgrouting-dev mailing list
    pgrouting-dev@lists.osgeo.org <mailto:pgrouting-dev@lists.osgeo.org>
    http://lists.osgeo.org/mailman/listinfo/pgrouting-dev

--
Georepublic UG & Georepublic Japan
eMail: daniel.kastl@georepublic.de <mailto:daniel.kastl@georepublic.de>
Web: http://georepublic.de/&gt;

_______________________________________________
pgrouting-dev mailing list
pgrouting-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/pgrouting-dev

On Fri, Jun 14, 2013 at 10:07 PM, Stephen Woodbridge <
woodbri@swoodbridge.com> wrote:

Hi Dave,

I'm not totally opposed to this, but I would like to discuss how you plan
to structure the data and what your query would look like.

There is another possibility that I like better and that would be to write
stored procedure that takes you sql query and returns a matrix.

matrix pgr_matrix(sql text)

Then you can do something like:

select * from pgr_tsp(pgr_matrix('select * from matrix_table'), 27);

This idea I really like, because we might also be able to use it then for
Razequl's VRP solver.

Daniel

--
Georepublic UG & Georepublic Japan
eMail: daniel.kastl@georepublic.de
Web: http://georepublic.de

On 14/06/13 14:19, Daniel Kastl wrote:

I can try and have a play, my problem being that I am more a C/java programmer than an sql programmer. I prefer to deal with standard sql constructs rather that data base specfic stuff.

I can see why you got for the matrix solution, if avoid all those node/hash lookup problems.

If coding an select * from xyz solution , you have to deal with the case that a node may be any number in the range 1-32^2 and that has to be mapped on to an array of given size by using some type of hashing function and a reverse lookup has to be done on the return.

The store procedure has its attractions because its version neutral, it will work as well on linix box as well as a Windoz box.

I see what I can come up with.

DAve.
//

On Fri, Jun 14, 2013 at 10:07 PM, Stephen Woodbridge <woodbri@swoodbridge.com <mailto:woodbri@swoodbridge.com>> wrote:

    Hi Dave,

    I'm not totally opposed to this, but I would like to discuss how
    you plan to structure the data and what your query would look like.

    There is another possibility that I like better and that would be
    to write stored procedure that takes you sql query and returns a
    matrix.

    matrix pgr_matrix(sql text)

    Then you can do something like:

    select * from pgr_tsp(pgr_matrix('select * from matrix_table'), 27);

This idea I really like, because we might also be able to use it then for Razequl's VRP solver.

Daniel

--
Georepublic UG & Georepublic Japan
eMail: daniel.kastl@georepublic.de <mailto:daniel.kastl@georepublic.de>
Web: http://georepublic.de/&gt;

_______________________________________________
pgrouting-dev mailing list
pgrouting-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/pgrouting-dev

Dave,

I thought a lot about this problem before coming up with the matrix. The problem really boils down to how do you define your data that you want to fetch with "select * from xyz", ie: your inputs and what you need to solve TSP, ie: your outputs. In this case a symmetrix NxN matrix that is fully populated.

One idea I had was to structure xyz something like:

node_a, node_b, cost

but then you have to count the distinct nodes, and create a map to indices and a reverse map as you as you mention. But this is all pretty trivial to do in SQL.

create temporary table xyz_map (
   id serial not null primary key,
   nid integer
);

insert into xyz_map values (nid) select nid from (
   select distinct node_a as nid from xyz
   union
   select distinct node_b as nid from xyz
) as foo;

select array_agg(row) as matrix from (
   select a.id, array_agg(cost) as row
     from xyz a, xyz_map b, xyz_map c
     where a.node_a=b.nid and a.node_b=c.nid
     group by a.id
     order by c.id
   ) foo order by foo.id;

I have not tested this, but I think it is close to what you would work. I would then do some checking of matrix to make sure it is symmetric and filled out.

-Steve

On 6/15/2013 1:37 AM, Dave Potts wrote:

On 14/06/13 14:19, Daniel Kastl wrote:

I can try and have a play, my problem being that I am more a C/java
programmer than an sql programmer. I prefer to deal with standard sql
constructs rather that data base specfic stuff.

I can see why you got for the matrix solution, if avoid all those
node/hash lookup problems.

If coding an select * from xyz solution , you have to deal with the case
that a node may be any number in the range 1-32^2 and that has to be
mapped on to an array of given size by using some type of hashing
function and a reverse lookup has to be done on the return.

The store procedure has its attractions because its version neutral, it
will work as well on linix box as well as a Windoz box.

I see what I can come up with.

DAve.
//

On Fri, Jun 14, 2013 at 10:07 PM, Stephen Woodbridge
<woodbri@swoodbridge.com <mailto:woodbri@swoodbridge.com>> wrote:

    Hi Dave,

    I'm not totally opposed to this, but I would like to discuss how
    you plan to structure the data and what your query would look like.

    There is another possibility that I like better and that would be
    to write stored procedure that takes you sql query and returns a
    matrix.

    matrix pgr_matrix(sql text)

    Then you can do something like:

    select * from pgr_tsp(pgr_matrix('select * from matrix_table'), 27);

This idea I really like, because we might also be able to use it then
for Razequl's VRP solver.

Daniel

--
Georepublic UG & Georepublic Japan
eMail: daniel.kastl@georepublic.de <mailto:daniel.kastl@georepublic.de>
Web: http://georepublic.de/&gt;

_______________________________________________
pgrouting-dev mailing list
pgrouting-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/pgrouting-dev

_______________________________________________
pgrouting-dev mailing list
pgrouting-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/pgrouting-dev

On 15/06/13 14:28, Stephen Woodbridge wrote:

Nice Idea, but one of the problems I have is that I am VERY lazy!

When I create a source/target/cost table I only tend to populate it with existing routes, so if I have path at cost of 42 betweens nodes 2 and 3

I will create an entry like

     2 3 42

I have no desire to create an entry for journey between nodes 5 6 that does not exist or having to provide a value for journey between myself and myself eg a journey from node 2 to node 2, I think the suggested solution would require me to create entries like

    2 2 0
    5 6 0 etc

So I had a crack at writing a function based on Steves original screenplay. I think it needs a lot more work, when I create the temporary tables, I get a diagnostic from psql saying that its created a table. There must be a better way of doing it

Its invoked as
select mktsparray(' select source,target ,cost from edge_table2');

-- Make a distance matrix for the traveling salesman problem
--
-- Input is an sql string that provides source, target and the cost
-- of getting between them
--
--
CREATE or REPLACE FUNCTION mktsparray(tag_name text) RETURNS float
     LANGUAGE plpgsql
     AS $$
DECLARE
         ret_array float ;
         array_size integer;
         tsp_details record;
BEGIN

         ret_array := null;
         -- create a copy of the source
         EXECUTE 'DROP TABLE IF EXISTS tsp_src';
         EXECUTE 'DROP TABLE IF EXISTS tsp_map';

         create temporary table tsp_src ( id serial not null,source integer not null,
                                 target integer not null, cost float not null);
         -- create a tsp node to matrix lookup table
         create temporary table tsp_map ( id serial not null primary key, nid integer);

         EXECUTE 'insert into tsp_src (source, target, cost)'|| tag_name;

         insert into tsp_map (nid) select nid from (select distinct source as nid from edge_table2 union select distinct target as nid from edge_table2) as foo;
         -- should generate a matrix count(*) X count(*) of tsp_map

         select count(*) INTO array_size from tsp_map;
         -- by default populate the return results with zero, so any unknown
         -- routes will not be listed
         ret_array :=array_fill(0,ARRAY [ array_size,array_size]);

         -- Loop trough the input looking for any listed routes
         FOR tsp_details IN EXECUTE
                         'select c.nid as src ,b.nid as tar,a.cost as cos
                                 from tsp_src a, tsp_map b, tsp_map c
                                 where a.target= b.nid and a.source= c.nid and a.cost > 0
                                 group by b.id,c.id,a.cost' LOOP
ret_array[tsp_details.src][tsp_details.tar]:=tsp_details.cos;

         END LOOP;
         return ret_array;
END
$$;

Dave,

I thought a lot about this problem before coming up with the matrix. The problem really boils down to how do you define your data that you want to fetch with "select * from xyz", ie: your inputs and what you need to solve TSP, ie: your outputs. In this case a symmetrix NxN matrix that is fully populated.

One idea I had was to structure xyz something like:

node_a, node_b, cost

but then you have to count the distinct nodes, and create a map to indices and a reverse map as you as you mention. But this is all pretty trivial to do in SQL.

create temporary table xyz_map (
  id serial not null primary key,
  nid integer
);

insert into xyz_map values (nid) select nid from (
  select distinct node_a as nid from xyz
  union
  select distinct node_b as nid from xyz
) as foo;

select array_agg(row) as matrix from (
  select a.id, array_agg(cost) as row
    from xyz a, xyz_map b, xyz_map c
    where a.node_a=b.nid and a.node_b=c.nid
    group by a.id
    order by c.id
  ) foo order by foo.id;

I have not tested this, but I think it is close to what you would work. I would then do some checking of matrix to make sure it is symmetric and filled out.

-Steve

On 6/15/2013 1:37 AM, Dave Potts wrote:

On 14/06/13 14:19, Daniel Kastl wrote:

I can try and have a play, my problem being that I am more a C/java
programmer than an sql programmer. I prefer to deal with standard sql
constructs rather that data base specfic stuff.

I can see why you got for the matrix solution, if avoid all those
node/hash lookup problems.

If coding an select * from xyz solution , you have to deal with the case
that a node may be any number in the range 1-32^2 and that has to be
mapped on to an array of given size by using some type of hashing
function and a reverse lookup has to be done on the return.

The store procedure has its attractions because its version neutral, it
will work as well on linix box as well as a Windoz box.

I see what I can come up with.

DAve.
//

On Fri, Jun 14, 2013 at 10:07 PM, Stephen Woodbridge
<woodbri@swoodbridge.com <mailto:woodbri@swoodbridge.com>> wrote:

    Hi Dave,

    I'm not totally opposed to this, but I would like to discuss how
    you plan to structure the data and what your query would look like.

    There is another possibility that I like better and that would be
    to write stored procedure that takes you sql query and returns a
    matrix.

    matrix pgr_matrix(sql text)

    Then you can do something like:

    select * from pgr_tsp(pgr_matrix('select * from matrix_table'), 27);

This idea I really like, because we might also be able to use it then
for Razequl's VRP solver.

Daniel

--
Georepublic UG & Georepublic Japan
eMail: daniel.kastl@georepublic.de <mailto:daniel.kastl@georepublic.de>
Web: http://georepublic.de/&gt;

_______________________________________________
pgrouting-dev mailing list
pgrouting-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/pgrouting-dev

_______________________________________________
pgrouting-dev mailing list
pgrouting-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/pgrouting-dev

_______________________________________________
pgrouting-dev mailing list
pgrouting-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/pgrouting-dev

On 6/15/2013 2:34 PM, Dave Potts wrote:

On 15/06/13 14:28, Stephen Woodbridge wrote:

Nice Idea, but one of the problems I have is that I am VERY lazy!

When I create a source/target/cost table I only tend to populate it with
existing routes, so if I have path at cost of 42 betweens nodes 2 and 3

I will create an entry like

     2 3 42

I have no desire to create an entry for journey between nodes 5 6 that
does not exist or having to provide a value for journey between myself
and myself eg a journey from node 2 to node 2, I think the suggested
solution would require me to create entries like

    2 2 0
    5 6 0 etc

So I had a crack at writing a function based on Steves original
screenplay. I think it needs a lot more work, when I create the
temporary tables, I get a diagnostic from psql saying that its created
a table. There must be a better way of doing it

Its invoked as
select mktsparray(' select source,target ,cost from edge_table2');

-- Make a distance matrix for the traveling salesman problem
--
-- Input is an sql string that provides source, target and the cost
-- of getting between them
--

Dave,

You have to do the work somewhere :wink: Either in the table entries or when you initial the array. And the TSP code will not work if it is not a symmetric matrix with all values filled in. My note about checking the matrix was a hint check for these issues. Symmetric means the 2-3 has to be equal to 3-2 and the diagonal is all zeros.

-Steve

On 15/06/13 14:28, Stephen Woodbridge wrote:
Hi Steve,

I have had another at this problem, I managed to write some code to generate a matrix, small problem, since your mapping node values so fits in to the size values of the matrix, you also have to perform the same mapping on the start node and return value as well.

Dave.

Dave,

I thought a lot about this problem before coming up with the matrix. The problem really boils down to how do you define your data that you want to fetch with "select * from xyz", ie: your inputs and what you need to solve TSP, ie: your outputs. In this case a symmetrix NxN matrix that is fully populated.

One idea I had was to structure xyz something like:

node_a, node_b, cost

but then you have to count the distinct nodes, and create a map to indices and a reverse map as you as you mention. But this is all pretty trivial to do in SQL.

create temporary table xyz_map (
  id serial not null primary key,
  nid integer
);

insert into xyz_map values (nid) select nid from (
  select distinct node_a as nid from xyz
  union
  select distinct node_b as nid from xyz
) as foo;

select array_agg(row) as matrix from (
  select a.id, array_agg(cost) as row
    from xyz a, xyz_map b, xyz_map c
    where a.node_a=b.nid and a.node_b=c.nid
    group by a.id
    order by c.id
  ) foo order by foo.id;

I have not tested this, but I think it is close to what you would work. I would then do some checking of matrix to make sure it is symmetric and filled out.

-Steve

On 6/15/2013 1:37 AM, Dave Potts wrote:

On 14/06/13 14:19, Daniel Kastl wrote:

I can try and have a play, my problem being that I am more a C/java
programmer than an sql programmer. I prefer to deal with standard sql
constructs rather that data base specfic stuff.

I can see why you got for the matrix solution, if avoid all those
node/hash lookup problems.

If coding an select * from xyz solution , you have to deal with the case
that a node may be any number in the range 1-32^2 and that has to be
mapped on to an array of given size by using some type of hashing
function and a reverse lookup has to be done on the return.

The store procedure has its attractions because its version neutral, it
will work as well on linix box as well as a Windoz box.

I see what I can come up with.

DAve.
//

On Fri, Jun 14, 2013 at 10:07 PM, Stephen Woodbridge
<woodbri@swoodbridge.com <mailto:woodbri@swoodbridge.com>> wrote:

    Hi Dave,

    I'm not totally opposed to this, but I would like to discuss how
    you plan to structure the data and what your query would look like.

    There is another possibility that I like better and that would be
    to write stored procedure that takes you sql query and returns a
    matrix.

    matrix pgr_matrix(sql text)

    Then you can do something like:

    select * from pgr_tsp(pgr_matrix('select * from matrix_table'), 27);

This idea I really like, because we might also be able to use it then
for Razequl's VRP solver.

Daniel

--
Georepublic UG & Georepublic Japan
eMail: daniel.kastl@georepublic.de <mailto:daniel.kastl@georepublic.de>
Web: http://georepublic.de/&gt;

_______________________________________________
pgrouting-dev mailing list
pgrouting-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/pgrouting-dev

_______________________________________________
pgrouting-dev mailing list
pgrouting-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/pgrouting-dev

_______________________________________________
pgrouting-dev mailing list
pgrouting-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/pgrouting-dev