[Geoserver-devel] fun with sphinx

[Justin_Deoliveira]

Hi folks,

So I spent some time writing developer docs for trunk today, and wrote up developer notes about the web ui with a short tutorial. I would provide the link but docs.geoserver.org seems to be down

Regardless, in doing so I found my self wanting a feature again with sphinx that was a feature I always wanted out of confluence as well: The ability to reference code directly out of svn instead of copying it manually into a code block.

I have looked for the ability to do this in sphinx, and it seems this sort of feature is only supported with python. Boo. The closest thing I have found is the "literalinclude" directive:

http://sphinx.pocoo.org/markup/code.html?highlight=literalinclude#dir-literalinclude

Which is almost exactly what I want, however it requires files to be local. It might work mucking with file paths referencing into the source part of the tree via an external link... but that seemed hacky and external links are painful.

So I decided to look into the sphinx extension system to see how hard it would be to write a plugin that did this. And I met with some success. I created an custom directive called "gsinclude" which basically looks like this:

.. gsinclude:: CatalogInfo
    :module: main
    :package: org.geoserver.catalog

And the result looks like the following:

http://skitch.com/jdeolive/bs723/skunkworks-v1.0-documentation

And it works with all the line filtering goodness as well:

.. gsinclude:: CatalogInfo
    :module: main
    :package: org.geoserver.catalog
    :lines: 15-18

Leads to:

http://skitch.com/jdeolive/bs727/skunkworks-v1.0-documentation

Fun stuff. What I am wondering is what other people think about this sort of custom extension? Useful at all?

It is also very possible that the sphinx gurus out there have a better way to solve this, so I am all ears if they do. But a good exercise none the less to become familiar with some the internal workings of sphinx.

-Justin

--
Justin Deoliveira
OpenGeo - http://opengeo.org
Enterprise support for open source geospatial.

[Andrea_Aime3]

Justin Deoliveira ha scritto:

Hi folks,

So I spent some time writing developer docs for trunk today, and wrote up developer notes about the web ui with a short tutorial. I would provide the link but docs.geoserver.org seems to be down

Regardless, in doing so I found my self wanting a feature again with sphinx that was a feature I always wanted out of confluence as well: The ability to reference code directly out of svn instead of copying it manually into a code block.

I have looked for the ability to do this in sphinx, and it seems this sort of feature is only supported with python. Boo. The closest thing I have found is the "literalinclude" directive:

http://sphinx.pocoo.org/markup/code.html?highlight=literalinclude#dir-literalinclude

Which is almost exactly what I want, however it requires files to be local. It might work mucking with file paths referencing into the source part of the tree via an external link... but that seemed hacky and external links are painful.

So I decided to look into the sphinx extension system to see how hard it would be to write a plugin that did this. And I met with some success. I created an custom directive called "gsinclude" which basically looks like this:

.. gsinclude:: CatalogInfo
    :module: main
    :package: org.geoserver.catalog

And the result looks like the following:

http://skitch.com/jdeolive/bs723/skunkworks-v1.0-documentation

And it works with all the line filtering goodness as well:

.. gsinclude:: CatalogInfo
    :module: main
    :package: org.geoserver.catalog
    :lines: 15-18

Leads to:

http://skitch.com/jdeolive/bs727/skunkworks-v1.0-documentation

Fun stuff. What I am wondering is what other people think about this sort of custom extension? Useful at all?

Well, first off, kudos to your python-fu

It seems to be useful all right.

You cited before that literalinclude requires
the file to be on the local file system so is this building
the file lookup (../../../src/main/src/main/java/... eek) in
a much more manageable manner?

To be extra sure: it does not require one to be online to build
the docs right? (e.g., it does not access svn via http and the like, right?).

It is also very possible that the sphinx gurus out there have a better way to solve this, so I am all ears if they do. But a good exercise none the less to become familiar with some the internal workings of sphinx.

The question that remains is, how is this extension integrating with
our docs? Is it a file that's sitting along with the docs and just gets
picked up during the build, or does it need to be installed in some
special location?

Cheers
Andrea

--
Andrea Aime
OpenGeo - http://opengeo.org
Expert service straight from the developers.

[Gabriel_Roldan2]

That looks really great, congrats. It is indeed a long awaited feature wrt docs.

It would be _perfect_ imho, if instead of having to reference code snipets through line numbers it could be done through some sort of tagging (so the file can evolve without breaking the docs). But I've no idea whether that's possible or how

excellent work

Gabriel

Justin Deoliveira wrote:

Hi folks,

So I spent some time writing developer docs for trunk today, and wrote up developer notes about the web ui with a short tutorial. I would provide the link but docs.geoserver.org seems to be down

Regardless, in doing so I found my self wanting a feature again with sphinx that was a feature I always wanted out of confluence as well: The ability to reference code directly out of svn instead of copying it manually into a code block.

I have looked for the ability to do this in sphinx, and it seems this sort of feature is only supported with python. Boo. The closest thing I have found is the "literalinclude" directive:

http://sphinx.pocoo.org/markup/code.html?highlight=literalinclude#dir-literalinclude

Which is almost exactly what I want, however it requires files to be local. It might work mucking with file paths referencing into the source part of the tree via an external link... but that seemed hacky and external links are painful.

So I decided to look into the sphinx extension system to see how hard it would be to write a plugin that did this. And I met with some success. I created an custom directive called "gsinclude" which basically looks like this:

.. gsinclude:: CatalogInfo
    :module: main
    :package: org.geoserver.catalog

And the result looks like the following:

http://skitch.com/jdeolive/bs723/skunkworks-v1.0-documentation

And it works with all the line filtering goodness as well:

.. gsinclude:: CatalogInfo
    :module: main
    :package: org.geoserver.catalog
    :lines: 15-18

Leads to:

http://skitch.com/jdeolive/bs727/skunkworks-v1.0-documentation

Fun stuff. What I am wondering is what other people think about this sort of custom extension? Useful at all?

It is also very possible that the sphinx gurus out there have a better way to solve this, so I am all ears if they do. But a good exercise none the less to become familiar with some the internal workings of sphinx.

-Justin

--
Gabriel Roldan
OpenGeo - http://opengeo.org
Expert service straight from the developers.

[Michael_Melvin]

Does this mean that everyone who wants to build the GeoServer docs have to also check out the full GeoServer source code as well?

Thanks,
Mike Pumphrey
OpenGeo - http://opengeo.org

Justin Deoliveira wrote:

Hi folks,

So I spent some time writing developer docs for trunk today, and wrote up developer notes about the web ui with a short tutorial. I would provide the link but docs.geoserver.org seems to be down

Regardless, in doing so I found my self wanting a feature again with sphinx that was a feature I always wanted out of confluence as well: The ability to reference code directly out of svn instead of copying it manually into a code block.

I have looked for the ability to do this in sphinx, and it seems this sort of feature is only supported with python. Boo. The closest thing I have found is the "literalinclude" directive:

http://sphinx.pocoo.org/markup/code.html?highlight=literalinclude#dir-literalinclude

Which is almost exactly what I want, however it requires files to be local. It might work mucking with file paths referencing into the source part of the tree via an external link... but that seemed hacky and external links are painful.

So I decided to look into the sphinx extension system to see how hard it would be to write a plugin that did this. And I met with some success. I created an custom directive called "gsinclude" which basically looks like this:

.. gsinclude:: CatalogInfo
    :module: main
    :package: org.geoserver.catalog

And the result looks like the following:

http://skitch.com/jdeolive/bs723/skunkworks-v1.0-documentation

And it works with all the line filtering goodness as well:

.. gsinclude:: CatalogInfo
    :module: main
    :package: org.geoserver.catalog
    :lines: 15-18

Leads to:

http://skitch.com/jdeolive/bs727/skunkworks-v1.0-documentation

Fun stuff. What I am wondering is what other people think about this sort of custom extension? Useful at all?

It is also very possible that the sphinx gurus out there have a better way to solve this, so I am all ears if they do. But a good exercise none the less to become familiar with some the internal workings of sphinx.

-Justin

[Justin_Deoliveira]

Andrea Aime wrote:

Justin Deoliveira ha scritto:

Hi folks,

So I spent some time writing developer docs for trunk today, and wrote up developer notes about the web ui with a short tutorial. I would provide the link but docs.geoserver.org seems to be down

Regardless, in doing so I found my self wanting a feature again with sphinx that was a feature I always wanted out of confluence as well: The ability to reference code directly out of svn instead of copying it manually into a code block.

I have looked for the ability to do this in sphinx, and it seems this sort of feature is only supported with python. Boo. The closest thing I have found is the "literalinclude" directive:

http://sphinx.pocoo.org/markup/code.html?highlight=literalinclude#dir-literalinclude

Which is almost exactly what I want, however it requires files to be local. It might work mucking with file paths referencing into the source part of the tree via an external link... but that seemed hacky and external links are painful.

So I decided to look into the sphinx extension system to see how hard it would be to write a plugin that did this. And I met with some success. I created an custom directive called "gsinclude" which basically looks like this:

.. gsinclude:: CatalogInfo
    :module: main
    :package: org.geoserver.catalog

And the result looks like the following:

http://skitch.com/jdeolive/bs723/skunkworks-v1.0-documentation

And it works with all the line filtering goodness as well:

.. gsinclude:: CatalogInfo
    :module: main
    :package: org.geoserver.catalog
    :lines: 15-18

Leads to:

http://skitch.com/jdeolive/bs727/skunkworks-v1.0-documentation

Fun stuff. What I am wondering is what other people think about this sort of custom extension? Useful at all?

Well, first off, kudos to your python-fu

It seems to be useful all right.

You cited before that literalinclude requires
the file to be on the local file system so is this building
the file lookup (../../../src/main/src/main/java/... eek) in
a much more manageable manner?

Actually it is pulling the file in over the web from the svn repo. It works by setting a default baseurl in the conf.py file, which looks like:

gsinclude_baseurl = "http://svn.codehaus.org/geoserver/trunk"

And the gsinclude directive uses that + all the info such as class name, package name and module to locate the file. And then pulls it in via http. If it can't find the file or is offline it simply issues a warning and continues on, and the code block is empty.

We could change this so that is builds up the path into the source tree. The downside of which would be that you could not build the docs separately from the sources, the upside would be it would work offline..

To be extra sure: it does not require one to be online to build
the docs right? (e.g., it does not access svn via http and the like, right?).

It is also very possible that the sphinx gurus out there have a better way to solve this, so I am all ears if they do. But a good exercise none the less to become familiar with some the internal workings of sphinx.

The question that remains is, how is this extension integrating with
our docs? Is it a file that's sitting along with the docs and just gets
picked up during the build, or does it need to be installed in some
special location?

No additional installation required. I created a "sphinxext" directory which sits parallel do the "developer","user", and "docguide" directories. For each conf.py you need to point it at the extension directory, and enable the specific extension. So once someone who develops the extension commits it and updates any conf.py files, those updating won't have to do anything to enable it.

Cheers
Andrea

--
Justin Deoliveira
OpenGeo - http://opengeo.org
Enterprise support for open source geospatial.

[Justin_Deoliveira]

Gabriel Roldan wrote:

That looks really great, congrats. It is indeed a long awaited feature wrt docs.

It would be _perfect_ imho, if instead of having to reference code snipets through line numbers it could be done through some sort of tagging (so the file can evolve without breaking the docs). But I've no idea whether that's possible or how

Agreed, and this is possible if the target language is python... but unfortunately not java

excellent work

Gabriel

Justin Deoliveira wrote:

Hi folks,

So I spent some time writing developer docs for trunk today, and wrote up developer notes about the web ui with a short tutorial. I would provide the link but docs.geoserver.org seems to be down

Regardless, in doing so I found my self wanting a feature again with sphinx that was a feature I always wanted out of confluence as well: The ability to reference code directly out of svn instead of copying it manually into a code block.

I have looked for the ability to do this in sphinx, and it seems this sort of feature is only supported with python. Boo. The closest thing I have found is the "literalinclude" directive:

http://sphinx.pocoo.org/markup/code.html?highlight=literalinclude#dir-literalinclude

Which is almost exactly what I want, however it requires files to be local. It might work mucking with file paths referencing into the source part of the tree via an external link... but that seemed hacky and external links are painful.

So I decided to look into the sphinx extension system to see how hard it would be to write a plugin that did this. And I met with some success. I created an custom directive called "gsinclude" which basically looks like this:

.. gsinclude:: CatalogInfo
    :module: main
    :package: org.geoserver.catalog

And the result looks like the following:

http://skitch.com/jdeolive/bs723/skunkworks-v1.0-documentation

And it works with all the line filtering goodness as well:

.. gsinclude:: CatalogInfo
    :module: main
    :package: org.geoserver.catalog
    :lines: 15-18

Leads to:

http://skitch.com/jdeolive/bs727/skunkworks-v1.0-documentation

Fun stuff. What I am wondering is what other people think about this sort of custom extension? Useful at all?

It is also very possible that the sphinx gurus out there have a better way to solve this, so I am all ears if they do. But a good exercise none the less to become familiar with some the internal workings of sphinx.

-Justin

--
Justin Deoliveira
OpenGeo - http://opengeo.org
Enterprise support for open source geospatial.

[Justin_Deoliveira]

Nope, it references the sources online directly from the svn repo. So you need to be online or else you will get an empty code block.

Mike Pumphrey wrote:

Does this mean that everyone who wants to build the GeoServer docs have to also check out the full GeoServer source code as well?

Thanks,
Mike Pumphrey
OpenGeo - http://opengeo.org

Justin Deoliveira wrote:

Hi folks,

So I spent some time writing developer docs for trunk today, and wrote up developer notes about the web ui with a short tutorial. I would provide the link but docs.geoserver.org seems to be down

Regardless, in doing so I found my self wanting a feature again with sphinx that was a feature I always wanted out of confluence as well: The ability to reference code directly out of svn instead of copying it manually into a code block.

I have looked for the ability to do this in sphinx, and it seems this sort of feature is only supported with python. Boo. The closest thing I have found is the "literalinclude" directive:

http://sphinx.pocoo.org/markup/code.html?highlight=literalinclude#dir-literalinclude

Which is almost exactly what I want, however it requires files to be local. It might work mucking with file paths referencing into the source part of the tree via an external link... but that seemed hacky and external links are painful.

So I decided to look into the sphinx extension system to see how hard it would be to write a plugin that did this. And I met with some success. I created an custom directive called "gsinclude" which basically looks like this:

.. gsinclude:: CatalogInfo
    :module: main
    :package: org.geoserver.catalog

And the result looks like the following:

http://skitch.com/jdeolive/bs723/skunkworks-v1.0-documentation

And it works with all the line filtering goodness as well:

.. gsinclude:: CatalogInfo
    :module: main
    :package: org.geoserver.catalog
    :lines: 15-18

Leads to:

http://skitch.com/jdeolive/bs727/skunkworks-v1.0-documentation

Fun stuff. What I am wondering is what other people think about this sort of custom extension? Useful at all?

It is also very possible that the sphinx gurus out there have a better way to solve this, so I am all ears if they do. But a good exercise none the less to become familiar with some the internal workings of sphinx.

-Justin

--
Justin Deoliveira
OpenGeo - http://opengeo.org
Enterprise support for open source geospatial.

[Andrea_Aime3]

Justin Deoliveira ha scritto:

Nope, it references the sources online directly from the svn repo. So you need to be online or else you will get an empty code block.

Given the choice, and from the perspective of someone that has
a slow ping line (160ms to ping the svn repo), I'd very much prefer
it to hit the local checkout as opposed to a network connection.

To give you an example:

time curl https://svn.codehaus.org/geoserver/trunk/src/main/src/main/java/org/geoserver/data/CatalogReader.java > /dev/null
   % Total % Received % Xferd Average Speed Time Time Time Current
                                  Dload Upload Total Spent Left Speed
100 5818 100 5818 0 0 6623 0 --:--:-- --:--:-- --:--:-- 36136

real 0m0.885s
user 0m0.020s
sys 0m0.004s

So I would basically a bit less than an extra second in the build per
snippet included in the docs. If the docs start making heavy use
of it we may add up to a few extra minutes of build time (every time
the build is done).

I don't know how typical this is, but persuaded it's common enough
for people in southern Europe, and may be worse in other countries.
Also, this keeps on downloading small amounts at each build, which
is not so good in countries where you pay per MB downloaded (and not
so good either if you're paying per hour since you can't build
without being online).

Finally, we're talking about the developer guide here, not the user one,
so it does not seem so odd to require a full checkout of GeoServer
with sources included.

Cheers
Andrea

--
Andrea Aime
OpenGeo - http://opengeo.org
Expert service straight from the developers.

[Justin_Deoliveira]

Andrea Aime wrote:

Justin Deoliveira ha scritto:

Nope, it references the sources online directly from the svn repo. So you need to be online or else you will get an empty code block.

Given the choice, and from the perspective of someone that has
a slow ping line (160ms to ping the svn repo), I'd very much prefer
it to hit the local checkout as opposed to a network connection.

To give you an example:

time curl https://svn.codehaus.org/geoserver/trunk/src/main/src/main/java/org/geoserver/data/CatalogReader.java > /dev/null
  % Total % Received % Xferd Average Speed Time Time Time Current
                                 Dload Upload Total Spent Left Speed
100 5818 100 5818 0 0 6623 0 --:--:-- --:--:-- --:--:-- 36136

real 0m0.885s
user 0m0.020s
sys 0m0.004s

So I would basically a bit less than an extra second in the build per
snippet included in the docs. If the docs start making heavy use
of it we may add up to a few extra minutes of build time (every time
the build is done).

You would only pay the full cost if you do a clean build, which from my experiences is not very often. The downloaded files are stored locally and not redownloaded over and over again.

That said it should be easy enough to do the check for the local sources, and use them if they are available. Should provide the best of both worlds.

I don't know how typical this is, but persuaded it's common enough
for people in southern Europe, and may be worse in other countries.
Also, this keeps on downloading small amounts at each build, which
is not so good in countries where you pay per MB downloaded (and not
so good either if you're paying per hour since you can't build
without being online).

Finally, we're talking about the developer guide here, not the user one,
so it does not seem so odd to require a full checkout of GeoServer
with sources included.

Cheers
Andrea

--
Justin Deoliveira
OpenGeo - http://opengeo.org
Enterprise support for open source geospatial.

[Andrea_Aime3]

Justin Deoliveira ha scritto:

So I would basically a bit less than an extra second in the build per
snippet included in the docs. If the docs start making heavy use
of it we may add up to a few extra minutes of build time (every time
the build is done).

You would only pay the full cost if you do a clean build, which from my experiences is not very often. The downloaded files are stored locally and not redownloaded over and over again.

That said it should be easy enough to do the check for the local sources, and use them if they are available. Should provide the best of both worlds.

Yup, sounds like a winner.

Cheers
Andrea

--
Andrea Aime
OpenGeo - http://opengeo.org
Expert service straight from the developers.

[Michael_Melvin]

Great. Can you add this handy new method to the docguide? Do you want me to assign a JIRA to you to do that?

Thanks,
Mike Pumphrey
OpenGeo - http://opengeo.org

Andrea Aime wrote:

Justin Deoliveira ha scritto:

So I would basically a bit less than an extra second in the build per
snippet included in the docs. If the docs start making heavy use
of it we may add up to a few extra minutes of build time (every time
the build is done).

You would only pay the full cost if you do a clean build, which from my experiences is not very often. The downloaded files are stored locally and not redownloaded over and over again.

That said it should be easy enough to do the check for the local sources, and use them if they are available. Should provide the best of both worlds.

Yup, sounds like a winner.

Cheers
Andrea