[GeoNetwork-devel] GeoNetwork API calls to /records

Hi all,

I’m working on a Python API call to upload records from a fancy metadata submission form. I’m using the api/0.1/records post method, but I can’t seem to get it to work. It always gives me a 403 error, which if I understand the Swagger UI doc correctly, indicates I don’t have sufficient access. But this is a fresh GeoNetwork server, I’ve opened a session with the default admin username and password, and I can do an empty GET and receive a HTTP 200 response (and changing login details causes that to change to a 401 error).

I’ve tried adding auth details into the post command, but that didn’t change anything.

Any thoughts or ideas?

Code below

···

MERIDIAN on blue circle containing many numbers, with an orange wave pulse to the right.

Kim Mortimer

Data Manager

MERIDIAN - Marine Environmental Research Infrastructure for Data Integration and Application Network

Institute for Big Data Analytics, Faculty of Computer Sciences, Dalhousie University

p: + 1 902 494 1812 m: +1 902 880 1863

a: 6050 University Ave, Halifax, NS, B3H 4R2, Canada

w: https://meridian.cs.dal.ca e: k.mortimer@anonymised.com

Hi,

Did you check the CSRF token too?

https://geonetwork-opensource.org/manuals/3.6.x/en/customizing-application/misc.html

El mar., 25 jun. 2019 17:54, Kim Mortimer <K.Mortimer@anonymised.com.1219…> escribió:

Hi all,

I’m working on a Python API call to upload records from a fancy metadata submission form. I’m using the api/0.1/records post method, but I can’t seem to get it to work. It always gives me a 403 error, which if I understand the Swagger UI doc correctly, indicates I don’t have sufficient access. But this is a fresh GeoNetwork server, I’ve opened a session with the default admin username and password, and I can do an empty GET and receive a HTTP 200 response (and changing login details causes that to change to a 401 error).

I’ve tried adding auth details into the post command, but that didn’t change anything.

Any thoughts or ideas?

Code below

-- coding: utf-8 --

“”"

Accesses the GeoNetwork REST API to upload an xml metadata record.

Uses the Requests package for convenience

Made in Spyder Editor

Kim Mortimer, June 2019

“”"

import requests

from xml.dom import minidom

def gn_api_xml_push(XMLFilePath):

GN_BaseURL = ‘http://[! YOUR GEONETWORK HERE]/geonetwork/’ #removed local IP address

GN_API_URL = GN_BaseURL + ‘api/0.1/records’

session = requests.Session()

session.auth=(‘admin’, ‘admin’) #Your username/password here

loginResponse = session.get(GN_API_URL)

OpenedFile = open(XMLFilePath, ‘r’)

XMLDoc = minidom.parse(OpenedFile) #turns a file object into an XML document

OpenedFile.close()

parameters = {

‘file’: [XMLDoc], #Unsure if this is the Array format needed, but the error is the same even if I pass an empty array in

‘rejectIfInvalid’: True

}

requestResponse = session.post(GN_API_URL, data=parameters)

if requestResponse.status_code == 201:

responseDict = requestResponse.json()

if responseDict[‘numberOfRecordsNotFound’] > 0:

return {‘response’: “not found”}

elif responseDict[‘numberOfNullRecords’] > 0:

return {‘response’: “empty”}

elif responseDict[‘numberofRecordsWithErrors’] > 0:

errorString = ‘’

for error in responseDict[‘errors’]:

errorString = errorString + error[‘message’] + ‘\n’

metadataErrorString = ‘’

for value in responseDict[‘metadataErrors’].values():

metadataErrorString = metadataErrorString + value[0][‘message’] + ‘\n’

return {‘response’:“Errors in metadata record”,

‘errors’: errorString,

‘metadataErrors’: metadataErrorString

}

elif responseDict[‘numberOfRecordsProcessed’] > 0:

return {‘response’: ‘processed’}

else:

return {‘response’: ‘Unknown response - HTTP 201 success’}

elif requestResponse.status_code == 403:

return {‘response’: ‘HTTP 403 - insufficient access’,

‘loginResponse’: loginResponse}

else:

return {‘response’: ‘Unknown’,

‘statuscode’: requestResponse.status_code,

‘loginResponse’: loginResponse}


PS: The Swagger UI “try it out” feature for this request also doesn’t seem to work… I just used a known working XML document but the response is 400 -

{
  "message": "MultipartException",
  "code": "unsatisfied_request_parameter",
  "description": "The current request is not a multipart request"
}

Feels weird to me, but might just mean that the Swagger UI is out of date?

Thanks in advance,
Kim

MERIDIAN on blue circle containing many numbers, with an orange wave pulse to the right.

Kim Mortimer

Data Manager

MERIDIAN - Marine Environmental Research Infrastructure for Data Integration and Application Network

Institute for Big Data Analytics, Faculty of Computer Sciences, Dalhousie University

p: + 1 902 494 1812 m: +1 902 880 1863

a: 6050 University Ave, Halifax, NS, B3H 4R2, Canada

w: https://meridian.cs.dal.ca e: k.mortimer@anonymised.com


GeoNetwork-devel mailing list
GeoNetwork-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geonetwork-devel
GeoNetwork OpenSource is maintained at http://sourceforge.net/projects/geonetwork

Hi Maria,

Thanks for the tip! That was what I was missing, and that has gotten me into the POST processing now.

My new issue is that I can’t figure out what format is wanted for the ‘file’ parameter. It feels like everything I try gives me an HTTP 400 error “Could not find acceptable representation”, which implies to me I’m sending the wrong ‘thing’.

From the SwaggerUI doc - I can see that it wants “array [file] (formData)”… but I’ve seen many examples of FormData and it’s unclear whether that’s meaningful (other attributes say “query”, for example, but query is irrelevant to how they’re constructed/filled)

Since I’m only working with a single file right now, in Python I can just wrap whatever ‘file’ is in a list. But it’s not clear what ‘file’ is and how ‘formData’ plays a part. Some of the things I’ve tried…
A Python dictionary

A single string
The file path
The opened file object (using the built in open function)
The opened file object then read using default python (probably a bitwise string?)
A Python document created via DOM
A Python ElementTree
A Python Element (both of these are part of Python’s built in XML parsing)

JSON

But none of these seemed to work. From the manual https://geonetwork-opensource.org/manuals/trunk/eng/users/api/the-geonetwork-api.html I thought a string inside the array would work, but no such luck. I’m probably misinterpreting the example on that page, though.

Kim

···

From: María Arias de Reyna delawen@anonymised.com
Sent: 25 June 2019 13:04
To: Kim Mortimer
Cc: Devel geonetwork-devel@lists.sourceforge.net
Subject: Re: [GeoNetwork-devel] GeoNetwork API calls to /records

Hi,

Did you check the CSRF token too?

https://geonetwork-opensource.org/manuals/3.6.x/en/customizing-application/misc.html

El mar., 25 jun. 2019 17:54, Kim Mortimer <K.Mortimer@anonymised.com> escribió:

Hi all,

I’m working on a Python API call to upload records from a fancy metadata submission form. I’m using the api/0.1/records post method, but I can’t seem to get it to work. It always gives me a 403 error, which if I understand the Swagger UI doc correctly, indicates I don’t have sufficient access. But this is a fresh GeoNetwork server, I’ve opened a session with the default admin username and password, and I can do an empty GET and receive a HTTP 200 response (and changing login details causes that to change to a 401 error).

I’ve tried adding auth details into the post command, but that didn’t change anything.

Any thoughts or ideas?

Code below

-- coding: utf-8 --

“”"

Accesses the GeoNetwork REST API to upload an xml metadata record.

Uses the Requests package for convenience

Made in Spyder Editor

Kim Mortimer, June 2019

“”"

import requests

from xml.dom import minidom

def gn_api_xml_push(XMLFilePath):

GN_BaseURL = ‘http://[! YOUR GEONETWORK HERE]/geonetwork/’ #removed local IP address

GN_API_URL = GN_BaseURL + ‘api/0.1/records’

session = requests.Session()

session.auth=(‘admin’, ‘admin’) #Your username/password here

loginResponse = session.get(GN_API_URL)

OpenedFile = open(XMLFilePath, ‘r’)

XMLDoc = minidom.parse(OpenedFile) #turns a file object into an XML document

OpenedFile.close()

parameters = {

‘file’: [XMLDoc], #Unsure if this is the Array format needed, but the error is the same even if I pass an empty array in

‘rejectIfInvalid’: True

}

requestResponse = session.post(GN_API_URL, data=parameters)

if requestResponse.status_code == 201:

responseDict = requestResponse.json()

if responseDict[‘numberOfRecordsNotFound’] > 0:

return {‘response’: “not found”}

elif responseDict[‘numberOfNullRecords’] > 0:

return {‘response’: “empty”}

elif responseDict[‘numberofRecordsWithErrors’] > 0:

errorString = ‘’

for error in responseDict[‘errors’]:

errorString = errorString + error[‘message’] + ‘\n’

metadataErrorString = ‘’

for value in responseDict[‘metadataErrors’].values():

metadataErrorString = metadataErrorString + value[0][‘message’] + ‘\n’

return {‘response’:“Errors in metadata record”,

‘errors’: errorString,

‘metadataErrors’: metadataErrorString

}

elif responseDict[‘numberOfRecordsProcessed’] > 0:

return {‘response’: ‘processed’}

else:

return {‘response’: ‘Unknown response - HTTP 201 success’}

elif requestResponse.status_code == 403:

return {‘response’: ‘HTTP 403 - insufficient access’,

‘loginResponse’: loginResponse}

else:

return {‘response’: ‘Unknown’,

‘statuscode’: requestResponse.status_code,

‘loginResponse’: loginResponse}


PS: The Swagger UI “try it out” feature for this request also doesn’t seem to work… I just used a known working XML document but the response is 400 -

{
  "message": "MultipartException",
  "code": "unsatisfied_request_parameter",
  "description": "The current request is not a multipart request"
}

Feels weird to me, but might just mean that the Swagger UI is out of date?

Thanks in advance,
Kim

MERIDIAN on blue circle containing many numbers, with an orange wave pulse to the right.

Kim Mortimer

Data Manager

MERIDIAN - Marine Environmental Research Infrastructure for Data Integration and Application Network

Institute for Big Data Analytics, Faculty of Computer Sciences, Dalhousie University

p: + 1 902 494 1812 m: +1 902 880 1863

a: 6050 University Ave, Halifax, NS, B3H 4R2, Canada

w: https://meridian.cs.dal.ca e: k.mortimer@anonymised.com


GeoNetwork-devel mailing list
GeoNetwork-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geonetwork-devel
GeoNetwork OpenSource is maintained at http://sourceforge.net/projects/geonetwork

Maybe this mailing list thread could help you: http://osgeo-org.1560.x6.nabble.com/API-HttpMediaTypeNotAcceptableException-td5386516.html

···


Vriendelijke groeten / Kind regards,

Juan Luis Rodríguez.


Veenderweg 13
6721 WD Bennekom
The Netherlands
T: +31 (0)318 416664

Please consider the environment before printing this email.

I guess the main problem is that you need to send a multipart/form-data request. You can check some examples here:
https://stackoverflow.com/questions/12385179/how-to-send-a-multipart-form-data-with-requests-in-python
https://requests.readthedocs.io/en/latest/user/quickstart/#post-a-multipart-encoded-file

···


Vriendelijke groeten / Kind regards,

Juan Luis Rodríguez.


Veenderweg 13
6721 WD Bennekom
The Netherlands
T: +31 (0)318 416664

Please consider the environment before printing this email.

Hi Juan Luis Rodriguez,

Thanks for the suggestions. That old mailing list thread ended up having most of the answers in it.
My working code looks like this…

···

From: Juan Luis Rodríguez Ponce juanluisrp@anonymised.com
Sent: 25 June 2019 18:40
To: Kim Mortimer
Cc: María Arias de Reyna; Devel geonetwork-devel@anonymised.com.net
Subject: Re: [GeoNetwork-devel] GeoNetwork API calls to /records

On Tue, Jun 25, 2019 at 11:33 PM Juan Luis Rodríguez Ponce <juanluisrp@anonymised.com> wrote:

On Tue, Jun 25, 2019 at 8:34 PM Kim Mortimer <K.Mortimer@anonymised.com> wrote:

Hi Maria,

Thanks for the tip! That was what I was missing, and that has gotten me into the POST processing now.

My new issue is that I can’t figure out what format is wanted for the ‘file’ parameter. It feels like everything I try gives me an HTTP 400 error “Could not find acceptable representation”, which implies to me I’m sending the wrong ‘thing’.

From the SwaggerUI doc - I can see that it wants “array [file] (formData)”… but I’ve seen many examples of FormData and it’s unclear whether that’s meaningful (other attributes say “query”, for example, but query is irrelevant to how they’re constructed/filled)

Since I’m only working with a single file right now, in Python I can just wrap whatever ‘file’ is in a list. But it’s not clear what ‘file’ is and how ‘formData’ plays a part. Some of the things I’ve tried…
A Python dictionary

A single string
The file path
The opened file object (using the built in open function)
The opened file object then read using default python (probably a bitwise string?)
A Python document created via DOM
A Python ElementTree
A Python Element (both of these are part of Python’s built in XML parsing)

JSON

But none of these seemed to work. From the manual https://geonetwork-opensource.org/manuals/trunk/eng/users/api/the-geonetwork-api.html I thought a string inside the array would work, but no such luck. I’m probably misinterpreting the example on that page, though.

Maybe this mailing list thread could help you: http://osgeo-org.1560.x6.nabble.com/API-HttpMediaTypeNotAcceptableException-td5386516.html

I guess the main problem is that you need to send a multipart/form-data request. You can check some examples here:
https://stackoverflow.com/questions/12385179/how-to-send-a-multipart-form-data-with-requests-in-python
https://requests.readthedocs.io/en/latest/user/quickstart/#post-a-multipart-encoded-file


Vriendelijke groeten / Kind regards,

Juan Luis Rodríguez.


Veenderweg 13
6721 WD Bennekom
The Netherlands
T: +31 (0)318 416664

Please consider the environment before printing this email.