[GeoNetwork-users] Command-line Batch Import

Yoshiyuki_Kudo · August 29, 2012, 8:45am

Hi,

Is there a way to batch-import from command line (not GN Admin GUI menu) ?

I have tens of thousands of directories each holding hundreds of
ISO19139 formatted files.
I could move all the files in those directories into one directory,
but that would make the number in the directory too large.
So I was thinking one way would be to make a shell script that does
the batch(one directory at a time) import for each directory.

Thanks so much for your help.
Yoshi

Pinner_Luke · August 30, 2012, 12:03am

Have a look at the following python script, it (optionally recursively) batch uploads MEFs from a directory. If you are uploading xml instead of mef, change line 81 from '*.mef' to '*.xml'

http://code.google.com/p/metageta/source/browse/tools/batchuploadmef.py

Luke

-----Original Message-----
From: Yoshi NA [mailto:tombitake1979@anonymised.com]
Sent: Wednesday, 29 August 2012 6:45 PM
To: geonetwork-users@lists.sourceforge.net
Subject: [GeoNetwork-users] Command-line Batch Import

Hi,

Is there a way to batch-import from command line (not GN Admin GUI menu) ?

I have tens of thousands of directories each holding hundreds of
ISO19139 formatted files.
I could move all the files in those directories into one directory, but that would make the number in the directory too large.
So I was thinking one way would be to make a shell script that does the batch(one directory at a time) import for each directory.

Thanks so much for your help.
Yoshi

------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
GeoNetwork-users mailing list
GeoNetwork-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geonetwork-users
GeoNetwork OpenSource is maintained at http://sourceforge.net/projects/geonetwork

If you have received this transmission in error please notify us immediately by return e-mail and delete all copies. If this e-mail or any attachments have been sent to you in error, that error does not constitute waiver of any confidentiality, privilege or copyright in respect of information in the e-mail or attachments.

Please consider the environment before printing this email.

Yoshiyuki_Kudo · August 30, 2012, 2:41am

Hi Luke,

Thanks. But when I run it with the change on line 81 from '*.mef' to
'*.xml' made, I got error messages.

---
# python batchuploadxml.py -d /mnt/00 -s http://xxx.xxx.xxx -u admin -p xxx -r
Login successfull!
Uploading /mnt/00/01/iso_A2GL10304051419OD1_PS1A0000000.xml
Traceback (most recent call last):
  File "batchuploadxml.py", line 186, in ?
    main(**kwargs)
  File "batchuploadxml.py", line 84, in main
    uploadmef(mef,handler,proxy,cj)
  File "batchuploadxml.py", line 89, in uploadmef
    formvalues['mefFile']=fo
NameError: global name 'formvalues' is not defined
---

Are there more changes needed to accommodate this script for ISO19139 files ?

And please let me know what I give 'group' to grant "Public" access
from "Guest".

        formvalues={'insert_mode':'1',
                  'file_type':'mef',
                  'data':'',
                  'template':'n',
                  'title':'',
                  'uuidAction':'overwrite',
                  'styleSheet':'_none_',
                  'group':'2',
                  'category':'_none_'
                  }

Thanks a lot for your help,
Yoshi

On Thu, Aug 30, 2012 at 9:03 AM, Pinner, Luke
<Luke.Pinner@anonymised.com> wrote:

Have a look at the following python script, it (optionally recursively)
batch uploads MEFs from a directory. If you are uploading xml instead of
mef, change line 81 from '*.mef' to '*.xml'

http://code.google.com/p/metageta/source/browse/tools/batchuploadmef.py

Luke

-----Original Message-----
From: Yoshi NA [mailto:tombitake1979@anonymised.com]
Sent: Wednesday, 29 August 2012 6:45 PM
To: geonetwork-users@lists.sourceforge.net
Subject: [GeoNetwork-users] Command-line Batch Import

Hi,

Is there a way to batch-import from command line (not GN Admin GUI menu) ?

I have tens of thousands of directories each holding hundreds of
ISO19139 formatted files.
I could move all the files in those directories into one directory, but that
would make the number in the directory too large.
So I was thinking one way would be to make a shell script that does the
batch(one directory at a time) import for each directory.

Thanks so much for your help.
Yoshi

------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and threat
landscape has changed and how IT managers can respond. Discussions will
include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
GeoNetwork-users mailing list
GeoNetwork-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geonetwork-users
GeoNetwork OpenSource is maintained at
http://sourceforge.net/projects/geonetwork

If you have received this transmission in error please notify us immediately
by return e-mail and delete all copies. If this e-mail or any attachments
have been sent to you in error, that error does not constitute waiver of any
confidentiality, privilege or copyright in respect of information in the
e-mail or attachments. Please consider the environment before printing this
email.

Pinner_Luke · August 30, 2012, 2:45am

It's designed for mefs, not xml as that is all we use. I didn't actually test with xml files.
If you get it to work, let me know and I'll look at incorporating your changes into the script.

Luke

-----Original Message-----
From: Yoshi NA [mailto:tombitake1979@anonymised.com]
Sent: Thursday, 30 August 2012 12:41 PM
To: Pinner, Luke
Cc: geonetwork-users@lists.sourceforge.net
Subject: Re: [GeoNetwork-users] Command-line Batch Import

Hi Luke,

Thanks. But when I run it with the change on line 81 from '*.mef' to '*.xml' made, I got error messages.

---
# python batchuploadxml.py -d /mnt/00 -s http://xxx.xxx.xxx -u admin -p xxx -r Login successfull!
Uploading /mnt/00/01/iso_A2GL10304051419OD1_PS1A0000000.xml
Traceback (most recent call last):
  File "batchuploadxml.py", line 186, in ?
    main(**kwargs)
  File "batchuploadxml.py", line 84, in main
    uploadmef(mef,handler,proxy,cj)
  File "batchuploadxml.py", line 89, in uploadmef
    formvalues['mefFile']=fo
NameError: global name 'formvalues' is not defined
---

Are there more changes needed to accommodate this script for ISO19139 files ?

And please let me know what I give 'group' to grant "Public" access from "Guest".

        formvalues={'insert_mode':'1',
                  'file_type':'mef',
                  'data':'',
                  'template':'n',
                  'title':'',
                  'uuidAction':'overwrite',
                  'styleSheet':'_none_',
                  'group':'2',
                  'category':'_none_'
                  }

Thanks a lot for your help,
Yoshi

On Thu, Aug 30, 2012 at 9:03 AM, Pinner, Luke <Luke.Pinner@anonymised.com> wrote:

Have a look at the following python script, it (optionally
recursively) batch uploads MEFs from a directory. If you are uploading
xml instead of mef, change line 81 from '*.mef' to '*.xml'

http://code.google.com/p/metageta/source/browse/tools/batchuploadmef.p
y

Luke

-----Original Message-----
From: Yoshi NA [mailto:tombitake1979@anonymised.com]
Sent: Wednesday, 29 August 2012 6:45 PM
To: geonetwork-users@lists.sourceforge.net
Subject: [GeoNetwork-users] Command-line Batch Import

Hi,

Is there a way to batch-import from command line (not GN Admin GUI menu) ?

I have tens of thousands of directories each holding hundreds of
ISO19139 formatted files.
I could move all the files in those directories into one directory,
but that would make the number in the directory too large.
So I was thinking one way would be to make a shell script that does
the batch(one directory at a time) import for each directory.

Thanks so much for your help.
Yoshi

----------------------------------------------------------------------
--------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond.
Discussions will include endpoint security, mobile security and the
latest in malware threats.
http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
GeoNetwork-users mailing list
GeoNetwork-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geonetwork-users
GeoNetwork OpenSource is maintained at
http://sourceforge.net/projects/geonetwork

If you have received this transmission in error please notify us
immediately by return e-mail and delete all copies. If this e-mail or
any attachments have been sent to you in error, that error does not
constitute waiver of any confidentiality, privilege or copyright in
respect of information in the e-mail or attachments. Please consider
the environment before printing this email.

If you have received this transmission in error please notify us immediately by return e-mail and delete all copies. If this e-mail or any attachments have been sent to you in error, that error does not constitute waiver of any confidentiality, privilege or copyright in respect of information in the e-mail or attachments.

Please consider the environment before printing this email.

Pinner_Luke · August 30, 2012, 2:57am

I've fixed the formvalues error, small bug that crept in when I made the script recursive. It still won't handle xml though.

Luke

-----Original Message-----
From: Yoshi NA [mailto:tombitake1979@anonymised.com]
Sent: Thursday, 30 August 2012 12:41 PM
To: Pinner, Luke
Cc: geonetwork-users@lists.sourceforge.net
Subject: Re: [GeoNetwork-users] Command-line Batch Import

Hi Luke,

Thanks. But when I run it with the change on line 81 from '*.mef' to
'*.xml' made, I got error messages.

---
# python batchuploadxml.py -d /mnt/00 -s http://xxx.xxx.xxx -u admin -p xxx -r
Login successfull!
Uploading /mnt/00/01/iso_A2GL10304051419OD1_PS1A0000000.xml
Traceback (most recent call last):
  File "batchuploadxml.py", line 186, in ?
    main(**kwargs)
  File "batchuploadxml.py", line 84, in main
    uploadmef(mef,handler,proxy,cj)
  File "batchuploadxml.py", line 89, in uploadmef
    formvalues['mefFile']=fo
NameError: global name 'formvalues' is not defined
---

Are there more changes needed to accommodate this script for ISO19139 files ?

And please let me know what I give 'group' to grant "Public" access
from "Guest".

        formvalues={'insert_mode':'1',
                  'file_type':'mef',
                  'data':'',
                  'template':'n',
                  'title':'',
                  'uuidAction':'overwrite',
                  'styleSheet':'_none_',
                  'group':'2',
                  'category':'_none_'
                  }

Thanks a lot for your help,
Yoshi

On Thu, Aug 30, 2012 at 9:03 AM, Pinner, Luke
<Luke.Pinner@anonymised.com> wrote:

Have a look at the following python script, it (optionally recursively)
batch uploads MEFs from a directory. If you are uploading xml instead of
mef, change line 81 from '*.mef' to '*.xml'

http://code.google.com/p/metageta/source/browse/tools/batchuploadmef.py

Luke

-----Original Message-----
From: Yoshi NA [mailto:tombitake1979@anonymised.com]
Sent: Wednesday, 29 August 2012 6:45 PM
To: geonetwork-users@lists.sourceforge.net
Subject: [GeoNetwork-users] Command-line Batch Import

Hi,

Is there a way to batch-import from command line (not GN Admin GUI menu) ?

I have tens of thousands of directories each holding hundreds of
ISO19139 formatted files.
I could move all the files in those directories into one directory, but that
would make the number in the directory too large.
So I was thinking one way would be to make a shell script that does the
batch(one directory at a time) import for each directory.

Thanks so much for your help.
Yoshi

------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and threat
landscape has changed and how IT managers can respond. Discussions will
include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
GeoNetwork-users mailing list
GeoNetwork-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geonetwork-users
GeoNetwork OpenSource is maintained at
http://sourceforge.net/projects/geonetwork

If you have received this transmission in error please notify us immediately
by return e-mail and delete all copies. If this e-mail or any attachments
have been sent to you in error, that error does not constitute waiver of any
confidentiality, privilege or copyright in respect of information in the
e-mail or attachments. Please consider the environment before printing this
email.

If you have received this transmission in error please notify us immediately by return e-mail and delete all copies. If this e-mail or any attachments have been sent to you in error, that error does not constitute waiver of any confidentiality, privilege or copyright in respect of information in the e-mail or attachments.

Please consider the environment before printing this email.