[Geoserver-devel] GeoTools PR reviews and downstream builds

GeoServer GeoServer Developer
1

Hi,
a common annoyance when reviewing a GeoTools PR is that, while it can be green on Travis,
it can still cause a downstream failure in GeoWebCache and GeoServer.

I don’t always checkout the PR, build it, and then run downstream builds locally when reviewing PRs… because well,
it takes quite a bit of time, and maintainer time is what we are short of.

Was wondering if this could be delegated to the build server, rough idea:

  • Add a entry in the build matrix that runs a custom shell script
  • The script build geotools without tests (the other builds cover that) in order to have fresh local GT jars in the maven repository
  • The script then check out GeoServer sources, and builds it with tests
  • Repeat the above steps for GWC as well
    However, I have no idea if a build matrix entry is even allowed to do the above.
    Has anyone experiences trying to do anything similar

Cheers
Andrea

···

GeoServer Professional Services from the experts! Visit http://goo.gl/it488V for more information. == Ing. Andrea Aime @geowolf Technical Lead GeoSolutions S.A.S. Via di Montramito 3/A 55054 Massarosa (LU) phone: +39 0584 962313 fax: +39 0584 1660272 mob: +39 339 8844549 http://www.geo-solutions.it http://twitter.com/geosolutions_it ------------------------------------------------------- Con riferimento alla normativa sul trattamento dei dati personali (Reg. UE 2016/679 - Regolamento generale sulla protezione dei dati “GDPR”), si precisa che ogni circostanza inerente alla presente email (il suo contenuto, gli eventuali allegati, etc.) è un dato la cui conoscenza è riservata al/i solo/i destinatario/i indicati dallo scrivente. Se il messaggio Le è giunto per errore, è tenuta/o a cancellarlo, ogni altra operazione è illecita. Le sarei comunque grato se potesse darmene notizia. This email is intended only for the person or entity to which it is addressed and may contain information that is privileged, confidential or otherwise protected from disclosure. We remind that - as provided by European Regulation 2016/679 “GDPR” - copying, dissemination or use of this e-mail or the information herein by anyone other than the intended recipient is prohibited. If you have received this email by mistake, please notify us immediately by telephone or e-mail.

2

Another possibility would be to run the custom script on another build platform. Has anyone experience for example with circle-ci?
I see they provide 4 build containers to open source projects:

https://circleci.com/open-source/

Open to other free alternatives too, bring them in.

Cheers
Andrea

···

GeoServer Professional Services from the experts! Visit http://goo.gl/it488V for more information. == Ing. Andrea Aime @geowolf Technical Lead GeoSolutions S.A.S. Via di Montramito 3/A 55054 Massarosa (LU) phone: +39 0584 962313 fax: +39 0584 1660272 mob: +39 339 8844549 http://www.geo-solutions.it http://twitter.com/geosolutions_it ------------------------------------------------------- Con riferimento alla normativa sul trattamento dei dati personali (Reg. UE 2016/679 - Regolamento generale sulla protezione dei dati “GDPR”), si precisa che ogni circostanza inerente alla presente email (il suo contenuto, gli eventuali allegati, etc.) è un dato la cui conoscenza è riservata al/i solo/i destinatario/i indicati dallo scrivente. Se il messaggio Le è giunto per errore, è tenuta/o a cancellarlo, ogni altra operazione è illecita. Le sarei comunque grato se potesse darmene notizia. This email is intended only for the person or entity to which it is addressed and may contain information that is privileged, confidential or otherwise protected from disclosure. We remind that - as provided by European Regulation 2016/679 “GDPR” - copying, dissemination or use of this e-mail or the information herein by anyone other than the intended recipient is prohibited. If you have received this email by mistake, please notify us immediately by telephone or e-mail.

3

Is any part of the build chain Jenkins? I know Jenkins can call a build script from another branch of the same repository, but it may also work to call a build job from a different repository. I plan on investigating this process for our in-house builds for a project I moved to. Might be worth someone with access to the build servers to investigate cross-repository builds?

Chris Snider

Senior Software Engineer

pa-logo-email

(attachments)

image002.jpg

···

On Mon, May 13, 2019 at 5:29 PM Andrea Aime <andrea.aime@…1268…> wrote:

Was wondering if this could be delegated to the build server, rough idea:

  • Add a entry in the build matrix that runs a custom shell script
  • The script build geotools without tests (the other builds cover that) in order to have fresh local GT jars in the maven repository
  • The script then check out GeoServer sources, and builds it with tests
  • Repeat the above steps for GWC as well

However, I have no idea if a build matrix entry is even allowed to do the above.

Has anyone experiences trying to do anything similar

Another possibility would be to run the custom script on another build platform. Has anyone experience for example with circle-ci?

I see they provide 4 build containers to open source projects:

https://circleci.com/open-source/

Open to other free alternatives too, bring them in.

Cheers

Andrea

==

GeoServer Professional Services from the experts! Visit http://goo.gl/it488V for more information. == Ing. Andrea Aime @geowolf Technical Lead GeoSolutions S.A.S. Via di Montramito 3/A 55054 Massarosa (LU) phone: +39 0584 962313 fax: +39 0584 1660272 mob: +39 339 8844549 http://www.geo-solutions.it http://twitter.com/geosolutions_it ------------------------------------------------------- Con riferimento alla normativa sul trattamento dei dati personali (Reg. UE 2016/679 - Regolamento generale sulla protezione dei dati “GDPR”), si precisa che ogni circostanza inerente alla presente email (il suo contenuto, gli eventuali allegati, etc.) è un dato la cui conoscenza è riservata al/i solo/i destinatario/i indicati dallo scrivente. Se il messaggio Le è giunto per errore, è tenuta/o a cancellarlo, ogni altra operazione è illecita. Le sarei comunque grato se potesse darmene notizia. This email is intended only for the person or entity to which it is addressed and may contain information that is privileged, confidential or otherwise protected from disclosure. We remind that - as provided by European Regulation 2016/679 “GDPR” - copying, dissemination or use of this e-mail or the information herein by anyone other than the intended recipient is prohibited. If you have received this email by mistake, please notify us immediately by telephone or e-mail.

4

Nope, we are talking about the PR validations performed by Travis and Appveyor, there is no Jenkins involved.
Not sure if it could make it involved, but also, even if technically feasible if we’d have the resources to make that happen.
One nice thing about Travis and Appveyor is that they provide the hardware for free (and circle-ci too, that’s why I was wondering about it).

Cheers
Andrea

(attachments)

image002.jpg

···

== GeoServer Professional Services from the experts! Visit http://goo.gl/it488V for more information. == Ing. Andrea Aime @geowolf Technical Lead GeoSolutions S.A.S. Via di Montramito 3/A 55054 Massarosa (LU) phone: +39 0584 962313 fax: +39 0584 1660272 mob: +39 339 8844549 http://www.geo-solutions.it http://twitter.com/geosolutions_it ------------------------------------------------------- Con riferimento alla normativa sul trattamento dei dati personali (Reg. UE 2016/679 - Regolamento generale sulla protezione dei dati “GDPR”), si precisa che ogni circostanza inerente alla presente email (il suo contenuto, gli eventuali allegati, etc.) è un dato la cui conoscenza è riservata al/i solo/i destinatario/i indicati dallo scrivente. Se il messaggio Le è giunto per errore, è tenuta/o a cancellarlo, ogni altra operazione è illecita. Le sarei comunque grato se potesse darmene notizia. This email is intended only for the person or entity to which it is addressed and may contain information that is privileged, confidential or otherwise protected from disclosure. We remind that - as provided by European Regulation 2016/679 “GDPR” - copying, dissemination or use of this e-mail or the information herein by anyone other than the intended recipient is prohibited. If you have received this email by mistake, please notify us immediately by telephone or e-mail.

5

When build.geoserver.org is back we should be able to configure it to check pull request branches and do a long build chain as you describe Andrea.

(attachments)

image002.jpg

···

–
Jody Garnett

6

Hi Jody,
this worries me… the build server is showing long build queues with the current workload,
if we add more, we should also provision more build power imho?

Cheers
Andrea

(attachments)

image002.jpg

···

Regards, Andrea Aime == GeoServer Professional Services from the experts! Visit http://goo.gl/it488V for more information. == Ing. Andrea Aime @geowolf Technical Lead GeoSolutions S.A.S. Via di Montramito 3/A 55054 Massarosa (LU) phone: +39 0584 962313 fax: +39 0584 1660272 mob: +39 339 8844549 http://www.geo-solutions.it http://twitter.com/geosolutions_it ------------------------------------------------------- Con riferimento alla normativa sul trattamento dei dati personali (Reg. UE 2016/679 - Regolamento generale sulla protezione dei dati “GDPR”), si precisa che ogni circostanza inerente alla presente email (il suo contenuto, gli eventuali allegati, etc.) è un dato la cui conoscenza è riservata al/i solo/i destinatario/i indicati dallo scrivente. Se il messaggio Le è giunto per errore, è tenuta/o a cancellarlo, ogni altra operazione è illecita. Le sarei comunque grato se potesse darmene notizia. This email is intended only for the person or entity to which it is addressed and may contain information that is privileged, confidential or otherwise protected from disclosure. We remind that - as provided by European Regulation 2016/679 “GDPR” - copying, dissemination or use of this e-mail or the information herein by anyone other than the intended recipient is prohibited. If you have received this email by mistake, please notify us immediately by telephone or e-mail.

7

Hi all,

While at the OSGeo code sprint, I spent some time working with CircleCI. The goal was to see if one build could chain multiple repos together. That is, could a CircleCI job build GeoTools, GeoWebCache, and GeoServer?

The good news is that CircleCI is flexible enough to chain builds. I put together a dummy example of that with JTS and Hatbox[1]. I was able to see a failure on a branch of JTS where I changed the name of class that Hatbox uses.

When I tried to setup a GeoTools build[2], I ran into problems where I was unable to get the project to build. I may have screwed up something with the job config or we need a bigger box from CircleCI (which would cost money).

Since I wasn’t finding good examples and was generally having trouble, I moved on to working with Jody on ImageN.

Anyone who is interested is more than welcome to my branches. Also, I’m happy to answer any questions. Wish I had better news…

Cheers,

Jim

  1. https://github.com/jnh5y/jts/blob/circleci/.circleci/config.yml
  2. https://github.com/jnh5y/geotools/blob/circleci/.circleci/config.yml
(attachments)

image002.jpg

···

On 5/14/2019 2:49 AM, Andrea Aime wrote:

Hi Jody,
this worries me… the build server is showing long build queues with the current workload,
if we add more, we should also provision more build power imho?

Cheers
Andrea

On Mon, May 13, 2019 at 10:54 PM Jody Garnett <jody.garnett@anonymised.com> wrote:

When build.geoserver.org is back we should be able to configure it to check pull request branches and do a long build chain as you describe Andrea.

–
Jody Garnett

On Mon, 13 May 2019 at 14:22, Chris Snider <chris.snider@anonymised.com> wrote:

Is any part of the build chain Jenkins? I know Jenkins can call a build script from another branch of the same repository, but it may also work to call a build job from a different repository. I plan on investigating this process for our in-house builds for a project I moved to. Might be worth someone with access to the build servers to investigate cross-repository builds?

Chris Snider

Senior Software Engineer

pa-logo-email

From: Andrea Aime <andrea.aime@anonymised.com>
Sent: Monday, May 13, 2019 10:37 AM
To: Geotools-Devel list <geotools-devel@lists.sourceforge.net>; Geoserver-devel <geoserver-devel@lists.sourceforge.net>
Subject: Re: [Geoserver-devel] GeoTools PR reviews and downstream builds

On Mon, May 13, 2019 at 5:29 PM Andrea Aime <andrea.aime@anonymised.com> wrote:

Was wondering if this could be delegated to the build server, rough idea:

  • Add a entry in the build matrix that runs a custom shell script
  • The script build geotools without tests (the other builds cover that) in order to have fresh local GT jars in the maven repository
  • The script then check out GeoServer sources, and builds it with tests
  • Repeat the above steps for GWC as well

However, I have no idea if a build matrix entry is even allowed to do the above.

Has anyone experiences trying to do anything similar

Another possibility would be to run the custom script on another build platform. Has anyone experience for example with circle-ci?

I see they provide 4 build containers to open source projects:

https://circleci.com/open-source/

Open to other free alternatives too, bring them in.

Cheers

Andrea

==

GeoServer Professional Services from the experts! Visit http://goo.gl/it488V for more information. == Ing. Andrea Aime @geowolf Technical Lead GeoSolutions S.A.S. Via di Montramito 3/A 55054 Massarosa (LU) phone: +39 0584 962313 fax: +39 0584 1660272 mob: +39 339 8844549 http://www.geo-solutions.it http://twitter.com/geosolutions_it ------------------------------------------------------- Con riferimento alla normativa sul trattamento dei dati personali (Reg. UE 2016/679 - Regolamento generale sulla protezione dei dati “GDPR”), si precisa che ogni circostanza inerente alla presente email (il suo contenuto, gli eventuali allegati, etc.) è un dato la cui conoscenza è riservata al/i solo/i destinatario/i indicati dallo scrivente. Se il messaggio Le è giunto per errore, è tenuta/o a cancellarlo, ogni altra operazione è illecita. Le sarei comunque grato se potesse darmene notizia. This email is intended only for the person or entity to which it is addressed and may contain information that is privileged, confidential or otherwise protected from disclosure. We remind that - as provided by European Regulation 2016/679 “GDPR” - copying, dissemination or use of this e-mail or the information herein by anyone other than the intended recipient is prohibited. If you have received this email by mistake, please notify us immediately by telephone or e-mail.

GeoTools-Devel mailing list
GeoTools-Devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geotools-devel

–

Regards, Andrea Aime == GeoServer Professional Services from the experts! Visit http://goo.gl/it488V for more information. == Ing. Andrea Aime @geowolf Technical Lead GeoSolutions S.A.S. Via di Montramito 3/A 55054 Massarosa (LU) phone: +39 0584 962313 fax: +39 0584 1660272 mob: +39 339 8844549 http://www.geo-solutions.it http://twitter.com/geosolutions_it ------------------------------------------------------- Con riferimento alla normativa sul trattamento dei dati personali (Reg. UE 2016/679 - Regolamento generale sulla protezione dei dati “GDPR”), si precisa che ogni circostanza inerente alla presente email (il suo contenuto, gli eventuali allegati, etc.) è un dato la cui conoscenza è riservata al/i solo/i destinatario/i indicati dallo scrivente. Se il messaggio Le è giunto per errore, è tenuta/o a cancellarlo, ogni altra operazione è illecita. Le sarei comunque grato se potesse darmene notizia. This email is intended only for the person or entity to which it is addressed and may contain information that is privileged, confidential or otherwise protected from disclosure. We remind that - as provided by European Regulation 2016/679 “GDPR” - copying, dissemination or use of this e-mail or the information herein by anyone other than the intended recipient is prohibited. If you have received this email by mistake, please notify us immediately by telephone or e-mail.

_______________________________________________
Geoserver-devel mailing list
[Geoserver-devel@lists.sourceforge.net](mailto:Geoserver-devel@lists.sourceforge.net)
[https://lists.sourceforge.net/lists/listinfo/geoserver-devel](https://lists.sourceforge.net/lists/listinfo/geoserver-devel)
8

Not sure how practical it would be, but it would be nice to take it a
step further and detect related PRs on the downstream projects. For
instance if a GWC PR references a GS PR or vis versa, the downstream GS
build would use the downstream PR rather than master.

On 2019-05-16 12:15 p.m., Jim Hughes wrote:

Hi all,

While at the OSGeo code sprint, I spent some time working with
CircleCI. The goal was to see if one build could chain multiple repos
together. That is, could a CircleCI job build GeoTools, GeoWebCache,
and GeoServer?

The good news is that CircleCI is flexible enough to chain builds. I
put together a dummy example of that with JTS and Hatbox[1]. I was
able to see a failure on a branch of JTS where I changed the name of
class that Hatbox uses.

When I tried to setup a GeoTools build[2], I ran into problems where I
was unable to get the project to build. I may have screwed up
something with the job config or we need a bigger box from CircleCI
(which would cost money).

Since I wasn't finding good examples and was generally having trouble,
I moved on to working with Jody on ImageN.

Anyone who is interested is more than welcome to my branches. Also,
I'm happy to answer any questions. Wish I had better news...

Cheers,

Jim

1. https://github.com/jnh5y/jts/blob/circleci/.circleci/config.yml
2. https://github.com/jnh5y/geotools/blob/circleci/.circleci/config.yml

On 5/14/2019 2:49 AM, Andrea Aime wrote:

Hi Jody,
this worries me... the build server is showing long build queues with
the current workload,
if we add more, we should also provision more build power imho?

Cheers
Andrea

On Mon, May 13, 2019 at 10:54 PM Jody Garnett <jody.garnett@anonymised.com
<mailto:jody.garnett@anonymised.com>> wrote:

    When build.geoserver.org <http://build.geoserver.org> is back we
    should be able to configure it to check pull request branches and
    do a long build chain as you describe Andrea.
    --
    Jody Garnett

    On Mon, 13 May 2019 at 14:22, Chris Snider
    <chris.snider@anonymised.com
    <mailto:chris.snider@anonymised.com>> wrote:

        Is any part of the build chain Jenkins? I know Jenkins can
        call a build script from another branch of the same
        repository, but it may also work to call a build job from a
        different repository. I plan on investigating this process
        for our in-house builds for a project I moved to. Might be
        worth someone with access to the build servers to investigate
        cross-repository builds?

        

        Chris Snider

        Senior Software Engineer

        pa-logo-email

        

        *From:* Andrea Aime <andrea.aime@anonymised.com
        <mailto:andrea.aime@anonymised.com>>
        *Sent:* Monday, May 13, 2019 10:37 AM
        *To:* Geotools-Devel list
        <geotools-devel@lists.sourceforge.net
        <mailto:geotools-devel@lists.sourceforge.net>>;
        Geoserver-devel <geoserver-devel@lists.sourceforge.net
        <mailto:geoserver-devel@lists.sourceforge.net>>
        *Subject:* Re: [Geoserver-devel] GeoTools PR reviews and
        downstream builds

        

        On Mon, May 13, 2019 at 5:29 PM Andrea Aime
        <andrea.aime@anonymised.com
        <mailto:andrea.aime@anonymised.com>> wrote:

            Was wondering if this could be delegated to the build
            server, rough idea:

              * Add a entry in the build matrix that runs a custom
                shell script
              * The script build geotools without tests (the other
                builds cover that) in order to have fresh local GT
                jars in the maven repository
              * The script then check out GeoServer sources, and
                builds it with tests
              * Repeat the above steps for GWC as well

            However, I have no idea if a build matrix entry is even
            allowed to do the above.

            Has anyone experiences trying to do anything similar

        

        Another possibility would be to run the custom script on
        another build platform. Has anyone experience for example
        with circle-ci?

        I see they provide 4 build containers to open source projects:

        

        https://circleci.com/open-source/

        

        Open to other free alternatives too, bring them in.

        

        Cheers

        Andrea

        

        ==

        GeoServer Professional Services from the experts! Visit
        http://goo.gl/it488V for more information. == Ing. Andrea
        Aime @geowolf Technical Lead GeoSolutions S.A.S. Via di
        Montramito 3/A 55054 Massarosa (LU) phone: +39 0584 962313
        fax: +39 0584 1660272 mob: +39 339 8844549
        http://www.geo-solutions.it
        http://twitter.com/geosolutions_it
        ------------------------------------------------------- /Con
        riferimento alla normativa sul trattamento dei dati personali
        (Reg. UE 2016/679 - Regolamento generale sulla protezione dei
        dati “GDPR”), si precisa che ogni circostanza inerente alla
        presente email (il suo contenuto, gli eventuali allegati,
        etc.) è un dato la cui conoscenza è riservata al/i solo/i
        destinatario/i indicati dallo scrivente. Se il messaggio Le è
        giunto per errore, è tenuta/o a cancellarlo, ogni altra
        operazione è illecita. Le sarei comunque grato se potesse
        darmene notizia. This email is intended only for the person
        or entity to which it is addressed and may contain
        information that is privileged, confidential or otherwise
        protected from disclosure. We remind that - as provided by
        European Regulation 2016/679 “GDPR” - copying, dissemination
        or use of this e-mail or the information herein by anyone
        other than the intended recipient is prohibited. If you have
        received this email by mistake, please notify us immediately
        by telephone or e-mail./

        _______________________________________________
        GeoTools-Devel mailing list
        GeoTools-Devel@lists.sourceforge.net
        <mailto:GeoTools-Devel@lists.sourceforge.net>
        https://lists.sourceforge.net/lists/listinfo/geotools-devel

--

Regards, Andrea Aime == GeoServer Professional Services from the
experts! Visit http://goo.gl/it488V for more information. == Ing.
Andrea Aime @geowolf Technical Lead GeoSolutions S.A.S. Via di
Montramito 3/A 55054 Massarosa (LU) phone: +39 0584 962313 fax: +39
0584 1660272 mob: +39 339 8844549 http://www.geo-solutions.it
http://twitter.com/geosolutions_it
------------------------------------------------------- /Con
riferimento alla normativa sul trattamento dei dati personali (Reg.
UE 2016/679 - Regolamento generale sulla protezione dei dati “GDPR”),
si precisa che ogni circostanza inerente alla presente email (il suo
contenuto, gli eventuali allegati, etc.) è un dato la cui conoscenza
è riservata al/i solo/i destinatario/i indicati dallo scrivente. Se
il messaggio Le è giunto per errore, è tenuta/o a cancellarlo, ogni
altra operazione è illecita. Le sarei comunque grato se potesse
darmene notizia. This email is intended only for the person or entity
to which it is addressed and may contain information that is
privileged, confidential or otherwise protected from disclosure. We
remind that - as provided by European Regulation 2016/679 “GDPR” -
copying, dissemination or use of this e-mail or the information
herein by anyone other than the intended recipient is prohibited. If
you have received this email by mistake, please notify us immediately
by telephone or e-mail./

_______________________________________________
Geoserver-devel mailing list
Geoserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel

_______________________________________________
Geoserver-devel mailing list
Geoserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel

--
Kevin Michael Smith
<smithkm@anonymised.com>

(attachments)

image002.jpg