[GRASS5] G_socks_* unix socket wrapper functions...

GRASS GRASS developer
1

I've been playing around with a few small library type functions for
creating and using UNIX domain sockets in GRASS. The basic idea is
that all such sockets could have the socket file created in the user's
current mapset tempdir, then detached processes or modules could use
these functions to set-up communications channels (I'm thinking alot
about this FIFO vs. IPC thing for the Raster/Display libes).

Anyway, here's a short run-down on the functions. I've only done a
simple test of them, but so far they work right. All functions that
take a "name" paramater want a full path, except G_sock_get_fname()
which will build the path. This way, code can just say:

   mon = G_sock_get_fname ("x0");

and then use "mon" for other calls. I'll upload this to libes/gis if
there's interest. Will require adding <sys/un.h> as an include in
gis.h for the G_sock_accept() function.

/* ----------------------------------------------------------------------
* G_sock_get_fname(), builds the full path for a UNIX socket using the
* G__temp_element() routine (tempfile.c). Caller should free() the
* return value when it is no longer needed.
* ---------------------------------------------------------------------*/
char *
G_sock_get_fname (char *name)

/* -------------------------------------------------------------------
* G_sock_exists(char *): Returns 1 if path is to a UNIX socket that
* already exists, 0 otherwise.
* -------------------------------------------------------------------*/
int
G_sock_exists (char *name)

/* -----------------------------------------------------------------
* G_sock_bind (char *): Takes the full pathname for a UNIX socket
* and returns the file descriptor to the socket after a successful
* call to bind(). On error, it returns -1
* ----------------------------------------------------------------*/
int
G_sock_bind (char *name)

/* ---------------------------------------------------------------------
* G_sock_listen(int, unsigned int): Wrapper around the listen()
* function.
* --------------------------------------------------------------------*/
int
G_sock_listen (int sockfd, unsigned int queue_len)

/* -----------------------------------------------------------------------
* G_sock_accept (int sockfd, struct sockaddr_un *caddr, socklen_t *clen):
* Wrapper around the accept() function. Both *caddr, and *clen can be
* NULL if the server doesn't care about this info. Function returns
* the file descriptor or an error code generated by accept(). Note,
* this call will usually block until a connection arrives. You can use
* select() for a time out on the call.
* ---------------------------------------------------------------------*/
int
G_sock_accept (int sockfd, struct sockaddr_un *caddr, socklen_t *clen)

/* ----------------------------------------------------------------------
* G_sock_connect (char *name): Tries to connect to the unix socket
* specified by "name". Returns the file descriptor if successful, or
* -1 if unsuccessful. Global errno is set by connect() if return is -1.
* --------------------------------------------------------------------*/
int
G_sock_connect (char *name)

--
Eric G. Miller <egm2@jps.net>

----------------------------------------
If you want to unsubscribe from GRASS Development Team mailing list write to:
minordomo@geog.uni-hannover.de with
subject 'unsubscribe grass5'

2

Okay, I know there haven't been any comments yet but here's a follow-up.

I implemented these G_sock_* functions here locally, then started the
hard work of hacking on the XDRIVER, the raster library and d.mon so I
could test it out.

The good news, it seems to work well. Speed seems comparable to IPC
(which is noticeable faster than FIFOs).

The bad news (longer...):

1) It breaks stuff.
  *) I know and expected the HTMLMAP, and CELL drivers would be broken.
     Those should be easy enough (yea right!) to update...

  *) All modules or anything else that have both $(GISLIB) and
     $(RASTERLIB) must have $(GISLIB) listed *after* $(RASTERLIB) in the
     Gmakefile. This is easily fixed and necessary so the G_socks
     functions are resolved before the linker looks at the raster
     library.

  *) At least one namespace conflict arises from the accept() function
     (p.map.new). Fixable with a rename. (hmm, ps.map has an accept()
     as well -- make that two).

  *) Some other variable name/function name conflicts showed up (notably
     the use of "static int index" in Font2.c (there's an "index"
     function declared in <string.h> here... -- apparently compatibility
     stuff for BSD [not part of ISO standard...]).

2) Still have to work out monitor management stuff, since FIFO and
   monitor locks are no longer used. Really a d.mon thing here. Note,
   with the changes I've made, the XDRIVER exits by click just fine (but
   I haven't looked at getting d.mon to handle this properly).

   Note: Monitor locks are no longer needed in the set-up since all
   socket files exist in the $GISDASE/$LOCATION/$MAPSET/.tmp dir for
   the current users mapset, and, at least the XDRIVER process will
   only accept one connection at a time (and the listen queue is 1 so
   other processes will get rejected or timeout or something...).

3) There may be an issue with the socket file name scheme I've used,
   since pathnames may be restricted to 104 characters. This could be
   fixed by using something like $HOME/.grass5/ so the path *should*
   always be short enough.
   
I don't intend to upload these things (boy, now my CVS tree is really
out of wack though...). I just wanted to see if I could get such a
thing to work. I see Huidae has made a bunch of uploads for the IPC
stuff, so if it's working well enough we should probably stick with it
for 5.0 "stable".

P.S. I can make my changes available if someone wants to look at them.
I just didn't want to gum up the GRASS sources with more stuff that
isn't being used...

--
Eric G. Miller <egm2@jps.net>

----------------------------------------
If you want to unsubscribe from GRASS Development Team mailing list write to:
minordomo@geog.uni-hannover.de with
subject 'unsubscribe grass5'

3

Hi Eric,

On Mon, Jan 29, 2001 at 12:07:14AM -0800, Eric G . Miller wrote:

Okay, I know there haven't been any comments yet but here's a follow-up.

I am not expert for sockets etc. but as far as I know sockets are
fully portable. So we might be able to drop fifos as soon as possible
when the IPC is stable (hi all, please test this! It's so easy to
test now!).

I implemented these G_sock_* functions here locally, then started the
hard work of hacking on the XDRIVER, the raster library and d.mon so I
could test it out.

The good news, it seems to work well. Speed seems comparable to IPC
(which is noticeable faster than FIFOs).

Say, we get IPC stable quickly and make it default. Then sockets
could be implemented optionally using the same mechanism/scripts
which IPC is using now. Let's remove fifos asap after getting o.k.
from the relevant platform tests.

The bad news (longer...):

1) It breaks stuff.
  *) I know and expected the HTMLMAP, and CELL drivers would be broken.
     Those should be easy enough (yea right!) to update...

I've heard that someone has written a PNG driver to replace CELL
driver? Probably that could be done in one step.

  *) All modules or anything else that have both $(GISLIB) and
     $(RASTERLIB) must have $(GISLIB) listed *after* $(RASTERLIB) in the
     Gmakefile. This is easily fixed and necessary so the G_socks
     functions are resolved before the linker looks at the raster
     library.

Yes, possible (I have a nice find/sed script for that).

  *) At least one namespace conflict arises from the accept() function
     (p.map.new). Fixable with a rename. (hmm, ps.map has an accept()
     as well -- make that two).

Seems to be simple as well.

  *) Some other variable name/function name conflicts showed up (notably
     the use of "static int index" in Font2.c (there's an "index"
     function declared in <string.h> here... -- apparently compatibility
     stuff for BSD [not part of ISO standard...]).

Mhh no idea.

2) Still have to work out monitor management stuff, since FIFO and
   monitor locks are no longer used. Really a d.mon thing here. Note,
   with the changes I've made, the XDRIVER exits by click just fine (but
   I haven't looked at getting d.mon to handle this properly).

:slight_smile: This is what the world is waiting for!

   Note: Monitor locks are no longer needed in the set-up since all
   socket files exist in the $GISDASE/$LOCATION/$MAPSET/.tmp dir for
   the current users mapset, and, at least the XDRIVER process will
   only accept one connection at a time (and the listen queue is 1 so
   other processes will get rejected or timeout or something...).

Perfect. Even a long term wish to improve security (no longer rwx in
/usr/...)

3) There may be an issue with the socket file name scheme I've used,
   since pathnames may be restricted to 104 characters. This could be
   fixed by using something like $HOME/.grass5/ so the path *should*
   always be short enough.

If it is too long. maybe an interim .tmp could be created?

I don't intend to upload these things (boy, now my CVS tree is really
out of wack though...). I just wanted to see if I could get such a
thing to work. I see Huidae has made a bunch of uploads for the IPC
stuff, so if it's working well enough we should probably stick with it
for 5.0 "stable".

I agree.

P.S. I can make my changes available if someone wants to look at them.
I just didn't want to gum up the GRASS sources with more stuff that
isn't being used...

You are on the right way as sockets obviously allow the features we want.

Thanks for working on that,

Markus

----------------------------------------
If you want to unsubscribe from GRASS Development Team mailing list write to:
minordomo@geog.uni-hannover.de with
subject 'unsubscribe grass5'

4

Hi Eric

I basically agree with what Markus said - sockets are a good idea!

--
Sincerely,

Jazzman (a.k.a. Justin Hickey) e-mail: jhickey@hpcc.nectec.or.th
High Performance Computing Center
National Electronics and Computer Technology Center (NECTEC)
Bangkok, Thailand

People who think they know everything are very irritating to those
of us who do. ---Anonymous

Jazz and Trek Rule!!!

----------------------------------------
If you want to unsubscribe from GRASS Development Team mailing list write to:
minordomo@geog.uni-hannover.de with
subject 'unsubscribe grass5'

5

I agree that it is a good plan to move to sockets. As Markus says, they are very
portable. I have checked the Cygwin API docs and it looks like the standard
sockets library functions are supported. I haven't used them (directly) but am
certainly willing to test this out and help with coding/porting/etc. Since this
is integrated into the core libraries, rather than being an add on like IPC, it
is probably more reliable.

As to dropping fifos, we should make the IPC changes to the CELL and HTMLMAP
sources, if that is to be the standard. So far this is only implemented in the
XDRIVER.

Also, are IPC message queues implemented on all of the platforms? It is on
UNIX's and now works on Cygwin for WinNT, but what about other platforms, e.g..
Mac OS X? Maybe people using platforms that don't support IPC can respond to
this (if no one responds can we assume that all support this?)

Malcolm

Markus Neteler wrote:

Hi Eric,

On Mon, Jan 29, 2001 at 12:07:14AM -0800, Eric G . Miller wrote:
> Okay, I know there haven't been any comments yet but here's a follow-up.

I am not expert for sockets etc. but as far as I know sockets are
fully portable. So we might be able to drop fifos as soon as possible
when the IPC is stable (hi all, please test this! It's so easy to
test now!).

> I implemented these G_sock_* functions here locally, then started the
> hard work of hacking on the XDRIVER, the raster library and d.mon so I
> could test it out.
>
> The good news, it seems to work well. Speed seems comparable to IPC
> (which is noticeable faster than FIFOs).

Say, we get IPC stable quickly and make it default. Then sockets
could be implemented optionally using the same mechanism/scripts
which IPC is using now. Let's remove fifos asap after getting o.k.
from the relevant platform tests.

> The bad news (longer...):
>
> 1) It breaks stuff.
> *) I know and expected the HTMLMAP, and CELL drivers would be broken.
> Those should be easy enough (yea right!) to update...
I've heard that someone has written a PNG driver to replace CELL
driver? Probably that could be done in one step.

> *) All modules or anything else that have both $(GISLIB) and
> $(RASTERLIB) must have $(GISLIB) listed *after* $(RASTERLIB) in the
> Gmakefile. This is easily fixed and necessary so the G_socks
> functions are resolved before the linker looks at the raster
> library.
Yes, possible (I have a nice find/sed script for that).

> *) At least one namespace conflict arises from the accept() function
> (p.map.new). Fixable with a rename. (hmm, ps.map has an accept()
> as well -- make that two).
Seems to be simple as well.

> *) Some other variable name/function name conflicts showed up (notably
> the use of "static int index" in Font2.c (there's an "index"
> function declared in <string.h> here... -- apparently compatibility
> stuff for BSD [not part of ISO standard...]).
Mhh no idea.

> 2) Still have to work out monitor management stuff, since FIFO and
> monitor locks are no longer used. Really a d.mon thing here. Note,
> with the changes I've made, the XDRIVER exits by click just fine (but
> I haven't looked at getting d.mon to handle this properly).
:slight_smile: This is what the world is waiting for!

> Note: Monitor locks are no longer needed in the set-up since all
> socket files exist in the $GISDASE/$LOCATION/$MAPSET/.tmp dir for
> the current users mapset, and, at least the XDRIVER process will
> only accept one connection at a time (and the listen queue is 1 so
> other processes will get rejected or timeout or something...).
Perfect. Even a long term wish to improve security (no longer rwx in
/usr/...)

> 3) There may be an issue with the socket file name scheme I've used,
> since pathnames may be restricted to 104 characters. This could be
> fixed by using something like $HOME/.grass5/ so the path *should*
> always be short enough.
If it is too long. maybe an interim .tmp could be created?

> I don't intend to upload these things (boy, now my CVS tree is really
> out of wack though...). I just wanted to see if I could get such a
> thing to work. I see Huidae has made a bunch of uploads for the IPC
> stuff, so if it's working well enough we should probably stick with it
> for 5.0 "stable".
I agree.

> P.S. I can make my changes available if someone wants to look at them.
> I just didn't want to gum up the GRASS sources with more stuff that
> isn't being used...
You are on the right way as sockets obviously allow the features we want.

Thanks for working on that,

Markus

----------------------------------------
If you want to unsubscribe from GRASS Development Team mailing list write to:
minordomo@geog.uni-hannover.de with
subject 'unsubscribe grass5'

----------------------------------------
If you want to unsubscribe from GRASS Development Team mailing list write to:
minordomo@geog.uni-hannover.de with
subject 'unsubscribe grass5'

6

On Mon, Jan 29, 2001 at 09:03:43PM -0400, Malcolm Blue wrote:

I agree that it is a good plan to move to sockets. As Markus says,
they are very portable. I have checked the Cygwin API docs and it
looks like the standard sockets library functions are supported. I
haven't used them (directly) but am certainly willing to test this out
and help with coding/porting/etc. Since this is integrated into the
core libraries, rather than being an add on like IPC, it is probably
more reliable.

As to dropping fifos, we should make the IPC changes to the CELL and
HTMLMAP sources, if that is to be the standard. So far this is only
implemented in the XDRIVER.

Also, are IPC message queues implemented on all of the platforms? It
is on UNIX's and now works on Cygwin for WinNT, but what about other
platforms, e.g.. Mac OS X? Maybe people using platforms that don't
support IPC can respond to this (if no one responds can we assume that
all support this?)

Perhaps some older BSD's don't have IPC message queues ???

If you want to look at some of the changes I made, I put a little
tarball at http://www.jps.net/egm2/sockets.tar.gz. This has the major
files modified (SWITCHER.c, io.c, unix_socks.c [new]). It's by no means
a plug-in, since there were a lot of little other tweaks (like to d.mon,
for instance, to change calling convention for XDRIVER). Generally, I
think it's a bit cleaner (just a bit, though), but I'm biased :wink:

--
Eric G. Miller <egm2@jps.net>

----------------------------------------
If you want to unsubscribe from GRASS Development Team mailing list write to:
minordomo@geog.uni-hannover.de with
subject 'unsubscribe grass5'

7

On Mon, Jan 29, 2001 at 09:03:43PM -0400, Malcolm Blue wrote:
[...]

Also, are IPC message queues implemented on all of the platforms? It is
on UNIX's and now works on Cygwin for WinNT, but what about other
platforms, e.g.. Mac OS X? Maybe people using platforms that don't
support IPC can respond to this (if no one responds can we assume that all
support this?)

Malcolm

Hi all,

just checked some machines with "ipcs -q" to see if IPC is present:

Linux: yes
Solaris: yes
CRAY: yes

Looks good!

Markus

----------------------------------------
If you want to unsubscribe from GRASS Development Team mailing list write to:
minordomo@geog.uni-hannover.de with
subject 'unsubscribe grass5'

8

Markus Neteler wrote:

On Mon, Jan 29, 2001 at 09:03:43PM -0400, Malcolm Blue wrote:
[...]
> Also, are IPC message queues implemented on all of the platforms? It is
> on UNIX's and now works on Cygwin for WinNT, but what about other
> platforms, e.g.. Mac OS X? Maybe people using platforms that don't
> support IPC can respond to this (if no one responds can we assume that all
> support this?)
>
> Malcolm

Hi all,

just checked some machines with "ipcs -q" to see if IPC is present:

Linux: yes
Solaris: yes
CRAY: yes

HP-UX 10.2+ : yes

Looks good!

Markus

----------------------------------------
If you want to unsubscribe from GRASS Development Team mailing list write to:
minordomo@geog.uni-hannover.de with
subject 'unsubscribe grass5'

--
Robert Lagacé, professeur
Pavillon Comtois
Université Laval
Ste-Foy, Québec, G1K 7P4
tel : (418)-656-2131#2276
Fax : (418)-656-3723
E-mail : lagace@grr.ulaval.ca

----------------------------------------
If you want to unsubscribe from GRASS Development Team mailing list write to:
minordomo@geog.uni-hannover.de with
subject 'unsubscribe grass5'

9

IRIX64 6.5

IPC status from /dev/kmem as of Tue Jan 30 18:06:04 2001
T ID KEY MODE OWNER GROUP
Message Queues:

Look ok?

> just checked some machines with "ipcs -q" to see if IPC is present:
>
> Linux: yes
> Solaris: yes
> CRAY: yes
>

HP-UX 10.2+ : yes

> Looks good!
>
> Markus

--
+=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=+
| Eric B. Mitchell mailto:emitchell@altaira.com |
| tel: (301) 809 - 3534 Altair Aerospace Corporation |
| tel: (800) 7 - ALTAIR 4201 Northview Dr. Suite 410 |
| fax: (301) 805 - 8122 Bowie, MD 20716 |
+=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=+
              ,___
          /"\ / o=\ /"""---===/
         / \_/ \__/ ---===/
         | //\ || /""TT""/ //\ || ||""\
         | // \ || || // \ || ||__/
         | //--==\ |L--/ || //--==\ || || "=,
          \ ---===/
           \____---===/

----------------------------------------
If you want to unsubscribe from GRASS Development Team mailing list write to:
minordomo@geog.uni-hannover.de with
subject 'unsubscribe grass5'

10

IRIX looks good. It wouldn't have had the ipcs command otherwise and wouldn't
have known what to report on the queues. Some of the systems (like Linux seem to
set up queue on startup - I don't know why). Just for fun, compile grass using
IPC, start a couple of monitors and there should be 2 queues for each monitor.
Then you can tell us for sure that IRIX supports IPC, plus you can test the IPC
XDRIVER.

Cheers,
Malcolm

Eric Mitchell wrote:

IRIX64 6.5

IPC status from /dev/kmem as of Tue Jan 30 18:06:04 2001
T ID KEY MODE OWNER GROUP
Message Queues:

Look ok?

> > just checked some machines with "ipcs -q" to see if IPC is present:
> >
> > Linux: yes
> > Solaris: yes
> > CRAY: yes
> >
>
> HP-UX 10.2+ : yes
>
> > Looks good!
> >
> > Markus

--
+=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=+
| Eric B. Mitchell mailto:emitchell@altaira.com |
| tel: (301) 809 - 3534 Altair Aerospace Corporation |
| tel: (800) 7 - ALTAIR 4201 Northview Dr. Suite 410 |
| fax: (301) 805 - 8122 Bowie, MD 20716 |
+=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=+
              ,___
          /"\ / o=\ /"""---===/
         / \_/ \__/ ---===/
         | //\ || /""TT""/ //\ || ||""\
         | // \ || || // \ || ||__/
         | //--==\ |L--/ || //--==\ || || "=,
          \ ---===/
           \____---===/

----------------------------------------
If you want to unsubscribe from GRASS Development Team mailing list write to:
minordomo@geog.uni-hannover.de with
subject 'unsubscribe grass5'

----------------------------------------
If you want to unsubscribe from GRASS Development Team mailing list write to:
minordomo@geog.uni-hannover.de with
subject 'unsubscribe grass5'

11

Hi all

Malcolm Blue wrote:

IRIX looks good. It wouldn't have had the ipcs command otherwise and
wouldn't have known what to report on the queues. Some of the systems
(like Linux seem to set up queue on startup - I don't know why). Just
for fun, compile grass using IPC, start a couple of monitors and there
should be 2 queues for each monitor. Then you can tell us for sure
that IRIX supports IPC, plus you can test the IPC XDRIVER.

Here is a very brief report of IPC usage on IRIX. First, it works
although I didn't notice any speed differences. Second, the monitor does
not redraw if another window covers it and then reveals it. This problem
does not exist with the fifo version.

Not very extensive testing (no time) but I thought I'd report it anyway.

--
Sincerely,

Jazzman (a.k.a. Justin Hickey) e-mail: jhickey@hpcc.nectec.or.th
High Performance Computing Center
National Electronics and Computer Technology Center (NECTEC)
Bangkok, Thailand

People who think they know everything are very irritating to those
of us who do. ---Anonymous

Jazz and Trek Rule!!!

----------------------------------------
If you want to unsubscribe from GRASS Development Team mailing list write to:
minordomo@geog.uni-hannover.de with
subject 'unsubscribe grass5'

12

I added the unix_socks.c file to the libes/gis directory but haven't
hooked it up. Also added the prototypes to include/gisdefs.h, but they
are commented out at the moment.

I thought that my little tests using these functions for the XDRIVER
worked pretty well, and such functions could be generally useful. So,
even if it's not used right now, they're available for future use...

I made a small modification to the G_sock_accept() call, so it only
takes the listening file descriptor as an argument. The other two
arguments to accept() are actually return values for the client socket
and socket length (neither of which are particularly useful for local
sockets -- we already know this info). This simplifies a few things for
useage, making it possible to have these functions available without
adding socket headers to gis.h or to any module that uses them.

All the functions are real basic, mostly just wrappers around the
standard calls. They do free the user from having to worry about
setting up socket addresses and such -- only need a *unique* key name,
like "x0".

--
Eric G. Miller <egm2@jps.net>

----------------------------------------
If you want to unsubscribe from GRASS Development Team mailing list write to:
minordomo@geog.uni-hannover.de with
subject 'unsubscribe grass5'