Hi Andrea,
Andrea Aime escribió:
Hi Andrea, first at all, sorry for causing this trouble.
We should ask Justin before proceeding, on how to manage compiled version of
dlls/libraries.
As you know we are working on an integration with spatialite. We have worked
on compiling sqlite and spatialite to work on all platforms, and main job on
that task is not related to output format itself (that is simply java and
couple of files). Our main concern up to now was to detect how to get
libraries working correctly.
Example: windows 64 bits was not working and we found way to compile files
to make them working, but..here is the question: how to do when you should
work with binaries that depends on third-party libraries ? (what do you
prefer ? the binary files or the sources of that libraries that allow to get
binaries ?, in that case, we should upload code, but...where ? do you have
some folder exclusively assigned for that third-party libraries ?)
I think you can follow up the imageio-ext example. The project provides
binary bundles that people can download per platform, plus installation
instructions.
In order to install the extension users have to install both the jar files
in GeoServer and the native libs somewhere the JVM can pick them
up (where that is, it's platform dependent)
great ! we will follow that “model” (to keep inline with your current way to manage it)
We are agree that it's not the best upload that binaries to a source
repository. Our mistake was not to ask before proceeding, and believe me
that it's my fault not to check with your team.
Now, how do you prefer to manage that ?
First off, remove them from svn. Anyone working from a slow connection
is going to be in a world of pain if they try to checkout GeoServer.
Unfortunately anyone using "git svn" (like most core developers do) will have to
keep those 50MB around, as git keeps full history in the checkout.
Done. We have removed native libraries.
Your idea of getting jars for each OS is ok, but it will be still 50 MB of
"data", so we should move that jars to a repository folder . How can we get
access to those repositories ? must we ask for permissions ?.
I guess the easiest approach is to follow the imageio-ext lead:
you just have to find some site that would host your files.
Sourceforge could be a hosting platform for the natives for example.
About sourceforge, i’m agree to use it as a place where to place native/compiled files. Do you have currently any other place you usually use ?
We will use the imageio-ext example in order to get a similar schema on how you manage this cases on projects.
Also, reviewing developer commit notes, i should ask you. How do you believe
it should be convinient to manage files on testing ? (we are working on
unit-testing on spatialite output-format, so where you should place files
for that ?. As you know, we should upload "generated output artifacts" in
order to do comparations when running testings....test folder sounds "good",
but files are not going to be 1 or 2, probably will me a couple and i dont
know how much space it would involve).
We have many output formats, and we keep pre-generated outputs only
for the image cases in which it's almost impossible to test any other way
(and even in that case we're talking files that are no larger than 10-50KB).
If you are going to generate a database the way to test it is to connect
to it and make queries checking its contents.
The databases you create for testing should also promptly be removed once
the test is done, and normally they are saved in the /target directory because
if anything goes wrong the first "mvn clean install" will wipe them out anyways.
You should also use the existing test data to avoid adding extra weight.
The normal svn/build rocess is not the place for large high
volume/high computation
time tests, those have to be performed only manually on your own machine.
About testing data, i’m agree that it’s not nice to upload big files into SVN. In that case we can manage an initialization process when executing maven tests, that would generate testing databases for comparation using data located at data folder.
My question was related to let us know how do you manage that, and if you can clarify what is the way you currently manage that
If you have some documented guide or reference that defines it under geoserver environment, it will be welcomed.
E.g., many of us have datasets ranging in the hundred of GB for volume tests,
but no one would even considering committing a MB of binary data to the repo
I understand this, and i’m totally agree.
Dont worry…we are not going to upload 1GB of testing data :-).
Data folder is where we would check for example information ? (so, we can use those example files in order to generate our own databases, without need to add more testing data).
(people have been yelled at for a anything above 100KB in the last years, and
with good reason).
Main reason when you use a SVN is to have a complete, replicable and versionable way to allow developers/team to access easily artifacts related to development and project (that includes documentation, tests, databases, necessary binary files, like images,excels files if used, docs, etc etc etc).
This issue it’s simply a difference on criteria about how to proceed on project. Our mistake: We should ask before uploading.
We are open to work with different rules to our usual way, cause it’s not a problem, simply it’s a different way.
For us, a very good reason for uploading those type of files is related to allow developers -with different developed skills- to access easily to all necessary files to start to develop on a new platform/project/environment (it’s a tradeoff between make it easier for people to understand development environment vs. space at repository).
Ie: if someone need those binary files, he/she will have to download them from a place anyway - sourceforge, and ftp, a link, etc.
In order to let someone know where to get files you can:
- to make them available easily (since we dont have restrictions when working on our svn, it’s not a major problem), or
- to document clearly on a well-defined way where to get files, and to indicate people where they should read instructions for that
Each way have pros and contras, and both are valid.
Anyway…as i mentioned, it’s simply a difference on criteria, and we should asked before proceeding. Our fault.
Team is now a warned and developers known that we cannot do the usual way we do.
Regards
Jose