[GRASS-dev] [GRASS GIS] #3179: g.extension not installing addons behind proxy

#3179: g.extension not installing addons behind proxy
----------------------------------------+-------------------------
Reporter: madi | Owner: grass-dev@…
     Type: defect | Status: new
Priority: normal | Milestone: 7.4.0
Component: Addons | Version: unspecified
Keywords: g.extension, proxy, addons | CPU: Unspecified
Platform: Unspecified |
----------------------------------------+-------------------------
Related to https://lists.osgeo.org/pipermail/grass-
user/2016-October/075337.html

when i try to use g.extension behind a proxy, following the example from
the manual, it gives me the following error:

{{{
g.extension ext=i.segment.hierarchical proxy="http://xx.xxx.xxx.xx:xxxx"
Traceback (most recent call last):
   File "/usr/local/grass-7.3.svn/scripts/g.extension", line 1730, in
<module>
     sys.exit(main())
   File "/usr/local/grass-7.3.svn/scripts/g.extension", line 1675, in main
     for ptype, purl in (p.split('=') for p in
options['proxy'].split(',')):
ValueError: need more than 1 value to unpack
}}}

Using the form suggested by Pietro:

{{{
proxy="http=<value>,ftp=<value>"
}}}

{{{
g.extension ext=i.segment.hierarchical
proxy="http=http://xx.xxx.xxx.xx:xxxx"
}}}

gives:

{{{
Fetching <i.segment.hierarchical> from GRASS GIS Addons repository (be
patient)...
ERROR: Extension <i.segment.hierarchical> not found
}}}

which is exactly the same as if I don't indicate the proxy at all.

FWIW, I normally use urllib2 to handle the proxy. However I have some
difficulties to apply this simple method to the 1700+ rows code (handling
several different addons sources for several os's) of g.extension, but I'm
behind a proxy and I'm available to test if you have a patch.

Here's what I do in python:

{{{
import urllib2
import os
proxy = urllib2.ProxyHandler({'http':os.environ['HTTP_PROXY']})
opener = urllib2.build_opener(proxy)
urllib2.install_opener(opener)
urllib2.urlopen('http://www.google.com')
}}}

Thanks

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/3179&gt;
GRASS GIS <https://grass.osgeo.org>

#3179: g.extension not installing addons behind proxy
--------------------------+----------------------------------------
  Reporter: madi | Owner: grass-dev@…
      Type: defect | Status: new
  Priority: normal | Milestone: 7.4.0
Component: Addons | Version: unspecified
Resolution: | Keywords: g.extension, proxy, addons
       CPU: Unspecified | Platform: Unspecified
--------------------------+----------------------------------------

Comment (by neteler):

Proxy ticket also in #1434

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/3179#comment:1&gt;
GRASS GIS <https://grass.osgeo.org>

#3179: g.extension not installing addons behind proxy
--------------------------+----------------------------------------
  Reporter: madi | Owner: grass-dev@…
      Type: defect | Status: new
  Priority: normal | Milestone: 7.4.0
Component: Addons | Version: unspecified
Resolution: | Keywords: g.extension, proxy, addons
       CPU: Unspecified | Platform: Unspecified
--------------------------+----------------------------------------

Comment (by neteler):

Please indicate the operating system, the output of `g.version -reg` and
the Python version.

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/3179#comment:2&gt;
GRASS GIS <https://grass.osgeo.org>

#3179: g.extension not installing addons behind proxy
--------------------------+----------------------------------------
  Reporter: madi | Owner: grass-dev@…
      Type: defect | Status: new
  Priority: normal | Milestone: 7.4.0
Component: Addons | Version: unspecified
Resolution: | Keywords: g.extension, proxy, addons
       CPU: Unspecified | Platform: Unspecified
--------------------------+----------------------------------------

Comment (by madi):

Can't check this now, as I'm currently not behind a proxy. Operating
system at the time of reporting was CentOS.

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/3179#comment:3&gt;
GRASS GIS <https://grass.osgeo.org>

#3179: g.extension not installing addons behind proxy
--------------------------+----------------------------------------
  Reporter: madi | Owner: grass-dev@…
      Type: defect | Status: new
  Priority: normal | Milestone: 7.4.0
Component: Addons | Version: unspecified
Resolution: | Keywords: g.extension, proxy, addons
       CPU: Unspecified | Platform: Unspecified
--------------------------+----------------------------------------

Comment (by mmetz):

Replying to [ticket:3179 madi]:
> Related to https://lists.osgeo.org/pipermail/grass-
user/2016-October/075337.html
>
> when i try to use g.extension behind a proxy, following the example from
the manual, it gives me the following error:
>
> {{{
> g.extension ext=i.segment.hierarchical proxy="http://xx.xxx.xxx.xx:xxxx"
> Traceback (most recent call last):
> File "/usr/local/grass-7.3.svn/scripts/g.extension", line 1730, in
<module>
> sys.exit(main())
> File "/usr/local/grass-7.3.svn/scripts/g.extension", line 1675, in
main
> for ptype, purl in (p.split('=') for p in
options['proxy'].split(',')):
> ValueError: need more than 1 value to unpack
> }}}
>
> Using the form suggested by Pietro:
>
> {{{
> proxy="http=<value>,ftp=<value>"
> }}}
>
> {{{
> g.extension ext=i.segment.hierarchical
proxy="http=http://xx.xxx.xxx.xx:xxxx"
> }}}
>
> gives:
>
> {{{
> Fetching <i.segment.hierarchical> from GRASS GIS Addons repository (be
> patient)...
> ERROR: Extension <i.segment.hierarchical> not found
> }}}
>
> which is exactly the same as if I don't indicate the proxy at all.

According to Python docs, "Proxies which require authentication for use
are not currently supported; this is considered an implementation
limitation" in urllib, and `g.extension` uses urllib, not urllib2 to open
urls.

>
> FWIW, I normally use urllib2 to handle the proxy. However I have some
difficulties to apply this simple method to the 1700+ rows code (handling
several different addons sources for several os's) of g.extension, but I'm
behind a proxy and I'm available to test if you have a patch.
>
> Here's what I do in python:
>
> {{{
> import urllib2
> import os
> proxy = urllib2.ProxyHandler({'http':os.environ['HTTP_PROXY']})
> opener = urllib2.build_opener(proxy)
> urllib2.install_opener(opener)
> urllib2.urlopen('http://www.google.com')
> }}}

That should also work for `g.extension`, including authentification.

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/3179#comment:4&gt;
GRASS GIS <https://grass.osgeo.org>

#3179: g.extension not installing addons behind proxy
--------------------------+----------------------------------------
  Reporter: madi | Owner: grass-dev@…
      Type: defect | Status: new
  Priority: normal | Milestone: 7.4.5
Component: Addons | Version: unspecified
Resolution: | Keywords: g.extension, proxy, addons
       CPU: Unspecified | Platform: Unspecified
--------------------------+----------------------------------------

Comment (by neteler):

Fix attempt via https://github.com/GRASS-GIS/grass-ci/pull/6 submitted as
r74116.
Note the different proxy syntax in the manual page.

If ok, to be backported.

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/3179#comment:10&gt;
GRASS GIS <https://grass.osgeo.org>

#3179: g.extension not installing addons behind proxy
--------------------------+----------------------------------------
  Reporter: madi | Owner: grass-dev@…
      Type: defect | Status: new
  Priority: normal | Milestone: 7.4.5
Component: Addons | Version: unspecified
Resolution: | Keywords: g.extension, proxy, addons
       CPU: Unspecified | Platform: Unspecified
--------------------------+----------------------------------------

Comment (by mmetz):

Replying to [comment:10 neteler]:
> Fix attempt via https://github.com/GRASS-GIS/grass-ci/pull/6 submitted
as r74116.
> Note the different proxy syntax in the manual page.
>
> If ok, to be backported.

Note, this is introducing a new python dependency, on RHEL 7 and
derivates, this is python-requests.

Do we have a list of required python 2 components and required python 3
components that can be updated? Ideally such a list would also indicate
which python components are required only for the GUI (systems such as HPC
may use GRASS without GUI).

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/3179#comment:11&gt;
GRASS GIS <https://grass.osgeo.org>

#3179: g.extension not installing addons behind proxy
--------------------------+----------------------------------------
  Reporter: madi | Owner: grass-dev@…
      Type: defect | Status: new
  Priority: normal | Milestone: 7.4.5
Component: Addons | Version: unspecified
Resolution: | Keywords: g.extension, proxy, addons
       CPU: Unspecified | Platform: Unspecified
--------------------------+----------------------------------------

Comment (by neteler):

Replying to [comment:11 mmetz]:
> Replying to [comment:10 neteler]:
> > Fix attempt via https://github.com/GRASS-GIS/grass-ci/pull/6 submitted
as r74116.
> > Note the different proxy syntax in the manual page.
> >
> > If ok, to be backported.
>
> Note, this is introducing a new python dependency, on RHEL 7 and
derivates, this is python-requests.

Yes, I realized that today.

> Do we have a list of required python 2 components and required python 3
components that can be updated?

Also yes. See r74121, it is REQUIREMENTS.html.

> Ideally such a list would also indicate which python components are
required only for the GUI (systems such as HPC may use GRASS without GUI).

That's more or less indicated in that file.

So, I am undecided now about the backport.

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/3179#comment:12&gt;
GRASS GIS <https://grass.osgeo.org>

#3179: g.extension not installing addons behind proxy
--------------------------+----------------------------------------
  Reporter: madi | Owner: grass-dev@…
      Type: defect | Status: new
  Priority: normal | Milestone: 7.4.5
Component: Addons | Version: unspecified
Resolution: | Keywords: g.extension, proxy, addons
       CPU: Unspecified | Platform: Unspecified
--------------------------+----------------------------------------

Comment (by mlennert):

Replying to [comment:12 neteler]:
> Replying to [comment:11 mmetz]:
> > Replying to [comment:10 neteler]:
> > > Fix attempt via https://github.com/GRASS-GIS/grass-ci/pull/6
submitted as r74116.
> > > Note the different proxy syntax in the manual page.
> > >
> > > If ok, to be backported.
> >
> > Note, this is introducing a new python dependency, on RHEL 7 and
derivates, this is python-requests.
>
> Yes, I realized that today.
>
> > Do we have a list of required python 2 components and required python
3 components that can be updated?
>
> Also yes. See r74121, it is REQUIREMENTS.html.
>
> > Ideally such a list would also indicate which python components are
required only for the GUI (systems such as HPC may use GRASS without GUI).
>
> That's more or less indicated in that file.
>
> So, I am undecided now about the backport.

I would plead for reworking this to use urllib2 (Python 2) which is
already imported in the module and which seems to work according to madi.
IIUC, authenticated proxy handling is part of urllib in Python 3, or ?

Let's try to avoid the endless dependency hell just out of easy
availability of small helper packages. See the [https://hackernoon.com
/whats-really-wrong-with-node-modules-and-why-this-is-your-fault-
8ac9fa893823 experience] with npm in the Javascript world as a case in
point. I would plead to keep most modules down to standard python library
dependency, unless there really is no other way.

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/3179#comment:13&gt;
GRASS GIS <https://grass.osgeo.org>

#3179: g.extension not installing addons behind proxy
--------------------------+----------------------------------------
  Reporter: madi | Owner: grass-dev@…
      Type: defect | Status: new
  Priority: normal | Milestone: 7.4.5
Component: Addons | Version: unspecified
Resolution: | Keywords: g.extension, proxy, addons
       CPU: Unspecified | Platform: Unspecified
--------------------------+----------------------------------------

Comment (by mmetz):

Replying to [comment:13 mlennert]:
> Replying to [comment:12 neteler]:
> > Replying to [comment:11 mmetz]:
> > > Replying to [comment:10 neteler]:
> > > > Fix attempt via https://github.com/GRASS-GIS/grass-ci/pull/6
submitted as r74116.
> > > > Note the different proxy syntax in the manual page.
> > > >
> > > > If ok, to be backported.
> > >
> > > Note, this is introducing a new python dependency, on RHEL 7 and
derivates, this is python-requests.
> >
> > Yes, I realized that today.
> >
> > > Do we have a list of required python 2 components and required
python 3 components that can be updated?
> >
> > Also yes. See r74121, it is REQUIREMENTS.html.
> >
> > > Ideally such a list would also indicate which python components are
required only for the GUI (systems such as HPC may use GRASS without GUI).
> >
> > That's more or less indicated in that file.
> >
> > So, I am undecided now about the backport.
>
> I would plead for reworking this to use urllib2 (Python 2) which is
already imported in the module and which seems to work according to madi.

+1
>
> Let's try to avoid the endless dependency hell just out of easy
availability of small helper packages. See the [https://hackernoon.com
/whats-really-wrong-with-node-modules-and-why-this-is-your-fault-
8ac9fa893823 experience] with npm in the Javascript world as a case in
point. I would plead to keep most modules down to standard python library
dependency, unless there really is no other way.

+1

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/3179#comment:14&gt;
GRASS GIS <https://grass.osgeo.org>

#3179: g.extension not installing addons behind proxy
--------------------------+----------------------------------------
  Reporter: madi | Owner: grass-dev@…
      Type: defect | Status: new
  Priority: normal | Milestone: 7.4.5
Component: Addons | Version: unspecified
Resolution: | Keywords: g.extension, proxy, addons
       CPU: Unspecified | Platform: Unspecified
--------------------------+----------------------------------------

Comment (by annakrat):

In [changeset:"74374" 74374]:
{{{
#!CommitTicketReference repository="" revision="74374"
g.extension: avoid python requests import, see #3179
}}}

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/3179#comment:15&gt;
GRASS GIS <https://grass.osgeo.org>

#3179: g.extension not installing addons behind proxy
--------------------------+----------------------------------------
  Reporter: madi | Owner: grass-dev@…
      Type: defect | Status: new
  Priority: normal | Milestone: 7.4.5
Component: Addons | Version: unspecified
Resolution: | Keywords: g.extension, proxy, addons
       CPU: Unspecified | Platform: Unspecified
--------------------------+----------------------------------------

Comment (by annakrat):

If somebody could test it with proxy, that would be great, I couldn't do
that now. This should work with Python 2/3, it uses six package (which we
need to require unlike requests).

If this works, we need to revert r74121.

As a side note, since we are moving to GitHub and we expect to get more
pull requests, we should be more strict about what we accept. People
typically care about getting a specific issue fixed, but this will often
introduce more mess in the code or even break other functionality.

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/3179#comment:16&gt;
GRASS GIS <https://grass.osgeo.org>

(just a comment on testing)

On Sun, Apr 14, 2019 at 5:49 AM GRASS GIS <trac@osgeo.org> wrote:
...

As a side note, since we are moving to GitHub and we expect to get more
pull requests, we should be more strict about what we accept. People
typically care about getting a specific issue fixed, but this will often
introduce more mess in the code or even break other functionality.

Yes, we need to have tests for much more code and instant testing.

Markus

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/3179#comment:16&gt;

#3179: g.extension not installing addons behind proxy
--------------------------+----------------------------------------
  Reporter: madi | Owner: grass-dev@…
      Type: defect | Status: new
  Priority: normal | Milestone: 7.4.5
Component: Addons | Version: unspecified
Resolution: | Keywords: g.extension, proxy, addons
       CPU: Unspecified | Platform: Unspecified
--------------------------+----------------------------------------

Comment (by pmav99):

{{{requests}}} need to be removed from {{{REQUIREMENTS.html}}} and the
{{{Dockerfile}}}, too.

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/3179#comment:17&gt;
GRASS GIS <https://grass.osgeo.org>

#3179: g.extension not installing addons behind proxy
--------------------------+----------------------------------------
  Reporter: madi | Owner: grass-dev@…
      Type: defect | Status: new
  Priority: normal | Milestone: 7.4.5
Component: Default | Version: unspecified
Resolution: | Keywords: g.extension, proxy, addons
       CPU: Unspecified | Platform: Unspecified
--------------------------+----------------------------------------
Changes (by martinl):

* component: Addons => Default

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/3179#comment:18&gt;
GRASS GIS <https://grass.osgeo.org>

#3179: g.extension not installing addons behind proxy
--------------------------+----------------------------------------
  Reporter: madi | Owner: grass-dev@…
      Type: defect | Status: new
  Priority: normal | Milestone:
Component: Default | Version: unspecified
Resolution: | Keywords: g.extension, proxy, addons
       CPU: Unspecified | Platform: Unspecified
--------------------------+----------------------------------------

Comment (by mmetz):

Replying to [comment:17 pmav99]:
> {{{requests}}} need to be removed from {{{REQUIREMENTS.html}}} and the
{{{Dockerfile}}}, too.

Done in trunk r74454

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/3179#comment:20&gt;
GRASS GIS <https://grass.osgeo.org>

#3179: g.extension not installing addons behind proxy
--------------------------+----------------------------------------
  Reporter: madi | Owner: grass-dev@…
      Type: defect | Status: new
  Priority: normal | Milestone: 7.8.3
Component: Default | Version: unspecified
Resolution: | Keywords: g.extension, proxy, addons
       CPU: Unspecified | Platform: Unspecified
--------------------------+----------------------------------------

Comment (by sbl):

Is there anything left here? Can the ticket be closed? Installation
through proxies should work now...

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/3179#comment:22&gt;
GRASS GIS <https://grass.osgeo.org>