Release 3.5.0-beta - Release Notes - Converter for kitodo_opax.xml

[stefanCCS]

In the Release Notes there is given a hint, that a "converter" should exists, which converts existing kitodo_opac.xml to new ImportConfiguration, with reference to Projektauswahl-und-Katalogsuche.
Unfortunately, I cannot find a hint to a converter tool behind this link.
Please clarify.

[solth]

The converter can be reached via "ImportConfiguration" tab on the "Projects" page:

Please keep in mind that the new permissions to see the "ImportConfiguration" and "MappingFile" tabs on the "Projects" page are not assigned to any role by default, so you should add them to any role you find appropriate (e.g. "Administrator" or similar) or create a new role for these permissions.

[stefanCCS]

Ok, many thanks - will try out ... (and give an update here ...)

[solth]

Unfortunately, I cannot find a hint to a converter tool behind this link.

I added a section decribing the catalog configuration converter to https://github.com/kitodo/kitodo-production/wiki/Projektauswahl-und-Katalogsuche

Since this issue only describes the missing documentation of that tool, I will close it. If you encounter a functional problem with the converter, please open a new issue for that or reopen this issue if you think the documentation is not yet sufficient.

[stefanCCS]

@solth:
I have tried to use the 'convertion' feature using the button "Import catalogue configurations".
Unfortunately, I get an empty dialog like this:

My /usr/local/kitodo/config folder contains

With kitodo_opac.xml being the default one, provided with this new Release 3.5.0-beta,
and kitodo_opac_old.xml being my old kitodo_opac.xml.

Somehow, I have expected to see a choice of XML-Files (where I could have chosen for kitodo_opac_old.xml

--> please explain, what I should do?

P.S I do not have the rights to re-open this issue, maybe you would like to do this.

[solth]

The kitodo_opac.xml file in this release is empty because it is not used for configurations anymore. The converter is meant as a convenient way to transform existing legacy files. That's what I tried to document under https://github.com/kitodo/kitodo-production/wiki/Projektauswahl-und-Katalogsuche with the following:

"Zur Umwandlung bestehender Katalogkonfigurationen aus der kitodo_opac.xml bietet Kitodo einen Konverter an. "

So you should use your kitodo_opac_old.xml file with your custom configurations and rename it back to kitodo_opac.xml. Than the converter will find it.

[stefanCCS]

@solth:
Thanks again.
Now my three catalogues are found :-)

But next riddle to be solved starts with next dialog:

What should I choose here?
In my current kitodo_opac.xml I have

   <catalogue title="Kalliope" description="Kalliope Verbundkatalog">
        <interfaceType>sru</interfaceType>
        <returnFormat>xml</returnFormat>
        <metadataFormat>MODS</metadataFormat>
        <mappingFiles>
            <file>kalliope_slub2kitodo.xsl</file>
        </mappingFiles>
        <fileUpload>false</fileUpload>
        <config>
        ...

Based on this it is not clear for me, what should I choose for input and output, and especially, what is the meaning of Prestructured Import.

--> please clarify.

[stefanCCS]

Looking at this documentation, I would guess to use in my case (see above):

• For "import": MODS
• For "output": KITODO
  and keep "Prestructed Import" as un-checked.

[solth]

Correct

[stefanCCS]

Nice, looks better :-)

[stefanCCS]

Well, next issue concerning this new feature:
I will not be asked for a catalogue import during process generation.

Then, I have found a new configuration possibility for a project:

Which could be set e.g. to "K10Plus" in my case.
If I do this, "K10Plus" is chosen as default catalogue and I have the possibility to chose also the other catalogues.

So far, so good, but maybe another hint in the documentation would be fine, like:
"You need to set in Project setting the Default Import Configuration" (Maybe incl. Screenshot)

[stefanCCS]

Doing this I get this error here:

Which is a bit strange, as I have chosen "K10Plus" and behind "K10Plus" my mapping file is "pica2kitodo_k10plus_CCS.xsl".
Whereas "pica2kitodo.xsl" is the default one, provided with the Release.

[stefanCCS]

Checking the setting in the UI for "K10Plus" I have found this:

--> it looks like that the "converter" does not set the correct mapping file.
--> in my opinion this is a bug
--> please clarify

[solth]

So far, so good, but maybe another hint in the documentation would be fine

There is already a note in the documentation describing the option to set a default import configuration for a project:

[stefanCCS]

I have set to do the mapping correction manually, then I get this error, when pressing "save"

[solth]

--> it looks like that the "converter" does not set the correct mapping file.

Yes, this is something I noticed as well. If there is another mapping file in the system providing the same mapping as the one configured in the catalog configuration in kitodo_opac.xml, the wrong mapping file might get assigned. This warrants a separate bug ticket.

[stefanCCS]

Found this on the other other Tab ("Details"):

--> looks like a non-mandatory field (without "*") might be mandatory ...

[stefanCCS]

--> it looks like that the "converter" does not set the correct mapping file.

Yes, this is something I noticed as well. If there is another mapping file in the system providing the same mapping as the one configured in the catalog configuration in kitodo_opac.xml, the wrong mapping file might get assigned. This warrants a separate bug ticket.

Should I create a new Bug Issue, or will you do it by yourself?

[solth]

Should I create a new Bug Issue, or will you do it by yourself?

I will do that.

[solth]

--> looks like a non-mandatory field (without "*") might be mandatory ...

This issue is more or less already documented at #5238 (although its the other way round in that case): the error message is lost when the error during saving occurs on another tab of the import configuration edit form, so we cannot update the active tab automatically.

[stefanCCS]

--> looks like a non-mandatory field (without "*") might be mandatory ...

This issue is more or less already documented at #5238 (although its the other way round in that case): the error message is lost when the error during saving occurs on another tab of the import configuration edit form, so we cannot update the active tab automatically.

In my opinion this is just one part of the error ("not seeing the error cause").
The other part is in my case here: Why do I get an error at all? (I only have changed the mapping file, and a non-mandatory field is marked "red" when pressing "save).

[solth]

@stefanCCS Transformed this issue into a discussion because it covers too many aspects, IMO. Individual issues can than be opened for individual problems that are still present.

[stefanCCS]

Would you like me to go on with discussion here (for all aspects in this area), or should I try to create issues for each topic, from which I think is a separate one?

[stefanCCS]

@solth:
IMO we have the following (more or less independent) issues (in "Import Configurations" area):

1. The one already identified, where you plan to create a issue for ("If there is another mapping file in the system providing the same mapping as the one configured in the catalog configuration in kitodo_opac.xml, the wrong mapping file might get assigned").
2. The other one already defined as issue: Switch to "Mapping Files" tab when saving failes #5238: When an error occurs during saving in "Edit Import Configurations" the error is not visible (is on the "Details tab", UI stays on "Mapping files tab").
3. "Metadata for catalog ID" looks to be mandatory, but is not marked with a "*" (looks like just a small UI issue)
4. Now it gets complicated: Because of 1. a wrong "mapping file" was set during automatic import configuration generation from kitodo_opac.xml. As I tried to correct this manually, I could not save it anymore, as mandatory field "Metadata for catalog ID" was not filled.
   This leads to the questions:

• If 1. is corrected, would then the "Metadata for catalog ID" also be filled (I cannot believe this, because this information is not available automatically IMO) ?
• If this field cannot be automatically filled, would it still work without having this field set? (I have set it now manually, as I could not save it otherwise, so I could not try out)
• If it will not work without this field set, IMO the automatic import configuration generation always would need additional manual work (here: setting the "Metadata for catalog ID") - which would be ok, just needs to be clearly marked in documentation (similar situation as with setting the "Default catalogue" in "Project settings").

[stefanCCS]

Just to give you also some good news:
After I have set the "Metadata for catalog ID", the catalogue imports on a first glance (not tested intensively) has worked fine for my 3 configured catalogues.

[solth]

@stefanCCS thanks a lot for the list above. I will create issues for the points that still need to be resolved and that are not covered by a different issue, yet.

[solth]

1. The one already identified, where you plan to create a issue for ("If there is another mapping file in the system providing the same mapping as the one configured in the catalog configuration in kitodo_opac.xml, the wrong mapping file might get assigned").

-> #5458

[solth]

• If 1. is corrected, would then the "Metadata for catalog ID" also be filled (I cannot believe this, because this information is not available automatically IMO) ?

• If this field cannot be automatically filled, would it still work without having this field set? (I have set it now manually, as I could not save it otherwise, so I could not try out)

• If it will not work without this field set, IMO the automatic import configuration generation always would need additional manual work (here: setting the "Metadata for catalog ID") - which would be ok, just needs to be clearly marked in documentation (similar situation as with setting the "Default catalogue" in "Project settings").

-> Metadata for catalog ID and mapping files are (or at least should be) totally unrelated. See my response below: #5448 (reply in thread)

[stefanCCS]

@solth:
Just to make things even a bit more complicated:
In 4. of above post:
I only needed to fill the field "Metadata for catalog ID" manually from 2 out of my 3 catalogue configurations.
For one catalogue this was already filled somehow ...

[solth]

I only needed to fill the field "Metadata for catalog ID" manually from 2 out of my 3 catalogue configurations.
For one catalogue this was already filled somehow ...

The "Metadata for catalog ID" denotes the metadata used to store the original ID from the catalog. It is created from the XML subelement identifierMetadata of a <catalogue> node in the kitodo_opac.xml file. So one of your catalog configurations in that file probably contained this element while the other two did not.

Historically and by default (so if no such element was found in the catalog configuration in kitodo_opac.xml) the "magic" metadata CatalogIDDigital has been used for this purpose. This is still supposed to be the default value of this property for newly created ImportConfigurations: see

kitodo-production/Kitodo-DataManagement/src/main/resources/db/migration/V2_115__Make_identifier_metadata_nullable.sql

Line 13 in 0c6e6f6

ALTER TABLE importconfiguration MODIFY identifier_metadata varchar(255) NULL DEFAULT 'CatalogIDDigital';

but apparently this does not work.

This property sticks out from the current ImportConfiguration schema, because ideally it should be configured in the ruleset (with a new use= setting) instead of the ImportConfiguration, since it always has to be set to a metadata key that is actually present in the ruleset used.

A short term adjustment should be to fix the default behaviour so that CatalogIDDigital is indeed set as the default value for new ImportConfigurations. A better - potentially incremental - solution could then be to move this configuration from the ImportConfiguration class to the ruleset.

[stefanCCS]

@solth : Many thanks for these explanations.
This lead to my next question:
I have seen in my K10Plus Import Configuration settings for Metadata record ID XPath and Metadata record title XPath.
How this was set during automatic Import Configuration creation?
(it looks like it was taken from the configured mapping file, but even then, why this (TitleDocMain and again CatalogIDDigital ).

[solth]

The Metadata record ID XPath and the Metadata record title XPath where formerly hard-coded in the individual XML response handlers like MODSResponseHandler and MARCResponseHandler and are used to display relevant information for hitlists (hits on that list need at least a title - to display the hit - and an id - for creating a query for a specific record when the user clicks one hit).

Hard-coding these information was wrong, though, and was done under the wrong assumption that MODS and MARC always need corresponding information like identifier in the same XML fields. Since this is not the case, corresponding XPaths need to be configurable.

Since these paths were not configurable in the past, these values are not taken from the kitodo_opac.xml configuration file during transformation into ImportConfiguration instances, but instead the previous static values from the now defunct XML handlers is inserted. (see #5317 and #5356 for further information)

[solth]

Added new issue for missing default value of identifier_metadata/"Metadata for catalog ID": #5470