I believe there is something wrong in UNIMARC default framework (https://github.com/Koha-Community/Koha/blob/main/installer/data/mysql/en/marcflavour/unimarc/mandatory/unimarc_framework_DEFAULT.yml ) We have 2 fields for "local" data (specific to Koha) at biblio level : 099 and 942 And the same pieces of information are present in subfields of both fields : - 099$x and 942$0 (Number of issues => biblioitems.totalissues) - 099$s and 942$s (Serial record => biblio.serial) - 099$t and 942$c (Koha item type => biblioitems.itemtype) I suppose it is an error, because it makes no sense... And note that the 099 field in UNIMARC framework is called "local dates", so it should not contain information about item type or number of issue. Do you think we could remove safely from the default templates 099$x, 099$s, 099$t ? I don't know if those values may be harcoded somewhere in the code.
*** Bug 37788 has been marked as a duplicate of this bug. ***
There's a little hint about what's going on in bug 34266 - apparently Biblibre is doing something that makes them want to have a biblio-level itemtype-like thing which is not the same as item-level itemtypes, and so probably the existence of both 099$t and 942$c in the default framework is the expression of a longtime "discussion" between joubu and cait.
Oh, and we should probably consider the Fast Add framework too, since it only has 099$t and not 942$c - if the reason for TYPEDOC is something like FRBR, umm, Fast Added FRBR? Neato!
hell of a headache, isn't it ? But I am not sure it is the same scope as the current bz : we could already clean 099$x, 099$s and 099$t, and keep for later the question raised by "942$c (Koha item type => biblioitems.itemtype)" For history record : Regarding the fact that in UNIMARC biblioitems.itemtype and items.itype are not always mapped to itemtypes.itemtype, it is certainly a very old move. I was administrating a Koha 3.x in 2011 (with Biblibre support), and this was already our configuration. I believe it was a choice made when Koha was introduced in France in the early 2000s. So a lot of french libraries with Biblibre support have been using for many years items.itype to define the type of loan (short / long / no loan...). The same values could not meaningflully be used at record level, and fill itemtypes.itemtype with other values (book / dvd / ...) Changing this behavior would be a huge move backwards for all of us.
I had time to do some archeology : Before bug 30373 (conversion to yaml, and suppression of the french framework) the french UNIMARC framework was quite different from the english one. The new yaml framework seems to have inherited from both, in a way that is not completely correct. * History of the french UNIMARC framework before its conversion to yaml : In 2008 : At biblio level : - no 942 field - 099 field "information locales" (local information) only used for creation and modification dates - 200$b field "type de document" (document type) <=> biblioitems.itemtype, linked to itemtypes AV - 686$a field "type (indice)" linked with itemtypes AV At item level : 995$r field "type de prêt" (type of loan) mapped with items.itype, linked to itemtypes AV Note that the label was "type of loan" and not "Type of item and material" like in the english framework Bug 6557 in 2012 : - creation of a 942 field "total des prêts" (number of checkouts) with only one subfield $0 Bug 11586 in 2014 : - 686$a and 200$b fields no longer mapped with biblioitems.itemtype neither linked to any AV - new subfield 099$t "Type de document" (document type) <=> biblioitems.itemtype, not linked to any AV - new subfield 099$x "Nombre de prêts" (number of checkouts) <=> biblioitems.totalissues => duplicate of 942$0 ! At item level : 995$r field "type de prêt" has now a default value "PRETLIV" (loan of book) => weird because it did not fit the values defined in sample_itemtypes.sql At that point (just before the framework conversion to yaml) 099$t and 995$r were both mapped to biblioitems.itemtype, BUT only 995$r were linked to itemtypes AV. French libraries were free to linked their 099$t to the AV of their choice (like "TYPEDOC") ======== Changes to english UNIMARC framework : In 2008 : At biblio level : 990 field "ADDED ENTRY ELEMENTS (KOHA)", with: 990$0 : Koha issues (borrowed) <=> biblioitems.totalissues 990$c : Koha item type <=> biblioitems.itemtype ; linked to itemtypes AV At item level : 995$r field "Type of item and material", mapped with nothing, not linked to any AV Bug 5128 in 2010 (at the initiative of the Italian Zeno Tajoli https://lists.koha-community.org/pipermail/koha-devel/2010-July/034235.html ) - Removal of 990 field - Creation of 099 field "local dates" : same use as 099 of fr-FR framework, but with a different name, more restrictive (in french it was "local information" - Creation of 942 field "ADDED ENTRY ELEMENTS (KOHA)", with : 942$0 : Koha issues (borrowed) <=> biblioitems.totalissues 942$c : Koha item type <=> biblioitems.itemtype ; linked with itemtypes AV At item level : the 995$r field (Type of item and material) is mapped with 'items.itype' and linked with 'itemtypes' At that point (just before the framework conversion to yaml) 942$c and 995$r were both mapped to biblioitems.itemtype, AND linked to itemtypes AV (contrary to the french framework). The use of 099 was more restricted than in french framework, the use of 942 much broader. And the wording of 099, 942 and 995$r were different from french framework.
By the way, this bug is linked : https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=38042
(In reply to Mathieu Saby from comment #6) > By the way, this bug is linked : > https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=38042 I have just added a comment there explaining a bit why I think UNIMARC should change that and introduce a proper field for document type instead. The current mapping requires patching again and again and you STILL have broken features available to MARC21 users that rely on it being an itemtype in the record and not something else.
After all those discussions it seems to me that the logic of defaut UNIMARC framework must be aligned with that of MARC21, and TYPEDOC values should be removed from default installations. There are no technical or standards-related reasons for the current difference. If vendors or libraries want to change the default behavior, the question of the extent to which these changes are compatible with the smooth running of Koha is an other subject. But I would appreciate some feedback from other people from the UNIMARC community.
As France uses UNIMARC (as well as some other countries), I would suggest getting some feedback from the KohaLa user group (https://koha-fr.org/liste-koha-infos/) and perhaps from the chat channel (koha-Francophone).
Oh I tried the list, being a member of Kohala. There seems to be no interest for the subject, or more probably a lack of cataloging skills in our librairies, due to changes in recruitment policy... And I asked on Mattermost too. If there are no objection I will provide a patch this weekend
After checking, I hadn't asked a question on this specific subject on thé list, so I'm going to do so...
I've added Fridolin from BibLibre to the cc list. Maybe, he can ask someone from BibLibre to have a look or provide some comment (I think they may look after one or two UNIMARC libraries 8-)).
I wrote this message to the french list: Unless there are serious objections, I'd like to propose a patch making the following changes: - 099 will be used only for local dates (which have no equivalent in MARC21) - 942 for other local information (aligned as far as possible with 942 MARC21) - 942$c will be linked to the list of document types in “itemtypes”. - the authorized value TYPEDOC will be removed - Zebra and Elasticsearch indexes will be revised to adapt to these changes The fast cataloging grid will then have to be revised. Of course, this will only impact new installations. If service providers or libraries wish to adapt the grids and link them to the authorized TYPEDOC value, they are free to do so, but this will no longer be part of the standard settings.
+1
I am writing the patch
Created attachment 178126 [details] [review] Bug 38336: Fix confusion between 099 and 942 UNIMARC fields This patch fixes 099 and 942 fields (and incidentally 200$b) in the default and the FA UNIMARC frameworks and provides the changes that follow logically : - Zebra and Elasticsearch indexing - removal of the unused TYPEDOC authorized values category - UNIMARC facets in Koha.pm - test in TestBuilder.pm These changes are minimal and are intended to correct only the most serious problems. Subsequent work may bring improvements. Detail of changes : - The 099 field is renamed "Local dates (Koha)" and should be used in default configuration only for this storing creation and modification dates of record. - The 942 field is mandatory (because its $c subfield is) and should be used for all other Koha local information, like its MARC21 counterpart. - The 200$b field is no more indexed (it is now obsolete in cataloguing standards, and its MARC21 counterpart is not indexed). - Like in MARC21, ccode is now defined only at item level and itemtype/itype is now defined at item and biblio level. - The default values for itemtype at biblio and item level are the same, controlled by itype values (document type). - The TYPEDOC authorized values category, which was used for controlling the value if itemtype at record level, is removed. - In Koha.pm UNIMARC facets are fixed to be a mirror of MARC21 facets: a facet for itemtypes is created, and the facet for ccode is now based only on items. - A test on record-level itemtype in TestBuilder.pm is fixed in order to replace 099 with 942 field. Test plan: Note that sample data in ktd and sandbox will need to be fixed once this patch is pushed (there is no data in 942$c for example). It may cause warnings if you edit existing record but it is not a bug. A. Check framework changes 1/ In a UNIMARC k-t-d or sandbox, with Zebra search engine 2/ Apply the patch 3/ Check default and FA framework - 099 should be labelled "Local dates (Koha)" and contain 2 subfields:c and d - 942 should be mandatory, labelled "ADDED ENTRY ELEMENTS (Koha)" and contain 5 subfields : 0,2,6,c (mandatory),s 4/ Create a new record with default framework -- add a 200$a field, a 100$a field (with the cataloging plugin) -- in 942$c field, check that the values in dropdown list are the one defines for 'item types' in administration (Books, Computer Files, Continuing Resource, etc). -- Choose the value "Computer field" and Save the record -- Koha displays the item creation form. Check that the preselected value in 995$r (item type) is "Computer field" 5 / Do the same operation for a new record with FA framework B. Check Zebra indexing changes 6 / Search "livre" in the catalog. 419 results should be shown 7 / Check that an "Item type" facet is visible and functional 8 / Edit the first record of the result list, change its 942$c value and save 9 / Edit an item of that record and change the 995$r value (with a value different from the 942$c) and the 995$h value (3 possible values: Non-fiction, Fiction, Reference) 10 / Refresh the search for "livre" 11 / Check that the facet "Item types" has been updated and that a facet "Collections" is shown (from the 995$h) and functional C. Check Elasticsearch indexing changes 12 / I believe you cannot currently test Elasticsearch in a sandbox, so, open a new k-t-d with Elasticsearch search engine (ktd --es7 up) 13 / Repeat operations 6 to 11 in that new instance (277 results should be shown after the search in step 6, instead of 419 for Zebra, this is normal)
Hi Mathieu! I tried to test this today using K-T-D but the patch no longer applies cleanly on the current main branch (commit 8abf287e6d). More specifically: 1) there are trailing whitespace warnings, these appear to come from: --- a/installer/data/mysql/mandatory/auth_val_cat.sql +++ b/installer/data/mysql/mandatory/auth_val_cat.sql @@ -56,8 +56,7 @@ INSERT IGNORE INTO authorised_value_categories( category_name, is_system ) INSERT IGNORE INTO authorised_value_categories( category_name ) VALUES ('ETAT'), ('CAND'), - ('COUNTRY'), - ('TYPEDOC'), + ('COUNTRY'), <--------------- Here are the two extra spaces ('qualif'); 2) there are merge conflicts in two files: C4/Koha.pm t/lib/TestBuilder.pm Which are most likely related to the mass tidying of the codebase which was recently committed to main (Bug 38664). Also, I think the test plan would be a bit easier to read if it used a hanging indent, as per the Wiki instructions at: https://wiki.koha-community.org/wiki/Commit_messages#Test_plan So, instead of: A. Check framework changes 1/ In a UNIMARC k-t-d or sandbox, with Zebra search engine 2/ Apply the patch 3/ Check default and FA framework - 099 should be labelled "Local dates (Koha)" and contain 2 subfields:c and d - 942 should be mandatory, labelled "ADDED ENTRY ELEMENTS (Koha)" and contain 5 subfields : 0,2,6,c (mandatory),s 4/ Create a new record with default framework -- add a 200$a field, a 100$a field (with the cataloging plugin) -- in 942$c field, check that the values in dropdown list are the one defines for 'item types' in administration (Books, Computer Files, Continuing Resource, etc). -- Choose the value "Computer field" and Save the record -- Koha displays the item creation form. Check that the preselected value in 995$r (item type) is "Computer field" 5 / Do the same operation for a new record with FA framework Something like this, perhaps? A. Check framework changes 1/ In a UNIMARC k-t-d or sandbox, with Zebra search engine 2/ Apply the patch 3/ Check default and FA framework - 099 should be labelled "Local dates (Koha)" and contain 2 subfields:c and d - 942 should be mandatory, labelled "ADDED ENTRY ELEMENTS (Koha)" and contain 5 subfields : 0,2,6,c (mandatory),s 4/ Create a new record with default framework -- add a 200$a field, a 100$a field (with the cataloging plugin) -- in 942$c field, check that the values in dropdown list are the one defines for 'item types' in administration (Books, Computer Files, Continuing Resource, etc). -- Choose the value "Computer field" and Save the record -- Koha displays the item creation form. Check that the preselected value in 995$r (item type) is "Computer field" 5/ Do the same operation for a new record with FA framework Finally, do you know if there a way to test the changes made to the installer's YAML files? Thank you!
Created attachment 179007 [details] [review] Bug 38336: Fix confusion between 099 and 942 UNIMARC fields This patch fixes 099 and 942 fields (and incidentally 200$b) in the default and the FA UNIMARC frameworks and provides the changes that follow logically : - Zebra and Elasticsearch indexing - removal of the unused TYPEDOC authorized values category - UNIMARC facets in Koha.pm - test in TestBuilder.pm These changes are minimal and are intended to correct only the most serious problems. Subsequent work may bring improvements. Detail of changes : - The 099 field is renamed "Local dates (Koha)" and should be used in default configuration only for this storing creation and modification dates of record. - The 942 field is mandatory (because its $c subfield is) and should be used for all other Koha local information, like its MARC21 counterpart. - The 200$b field is no more indexed (it is now obsolete in cataloguing standards, and its MARC21 counterpart is not indexed). - Like in MARC21, ccode is now defined only at item level and itemtype/itype is now defined at item and biblio level. - The default values for itemtype at biblio and item level are the same, controlled by itype values (document type). - The TYPEDOC authorized values category, which was used for controlling the value if itemtype at record level, is removed. - In Koha.pm UNIMARC facets are fixed to be a mirror of MARC21 facets: a facet for itemtypes is created, and the facet for ccode is now based only on items. - A test on record-level itemtype in TestBuilder.pm is fixed in order to replace 099 with 942 field. Test plan: Note that sample data in ktd and sandbox will need to be fixed once this patch is pushed (there is no data in 942$c for example). It may cause warnings if you edit existing record but it is not a bug. A. Check AV and framework changes 1/ In a UNIMARC k-t-d or sandbox, with Zebra search engine 2/ Apply the patch 3/ Check that there is no "TYPEDOC" authorized value category 4/ Check default and FA framework - 099 should be labelled "Local dates (Koha)" and contain 2 subfields:c and d - 942 should be mandatory, labelled "ADDED ENTRY ELEMENTS (Koha)" and contain 5 subfields : 0,2,6,c (mandatory),s 5/ Create a new record with default framework -- add a 200$a field, a 100$a field (with the cataloging plugin) -- in 942$c field, check that the values in dropdown list are the one defines for 'item types' in administration (Books, Computer Files, Continuing Resource, etc). -- Choose the value "Computer field" and Save the record -- Koha displays the item creation form. Check that the preselected value in 995$r (item type) is "Computer field" 6/ Do the same operation for a new record with FA framework B. Check Zebra indexing changes 7 / Search "livre" in the catalog. 419 results should be shown 8 / Check that an "Item type" facet is visible and functional 9 / Edit the first record of the result list, change its 942$c value and save 10 / Edit an item of that record and change the 995$r value (with a value different from the 942$c) and the 995$h value (3 possible values: Non-fiction, Fiction, Reference) 11 / Refresh the search for "livre" 12 / Check that the facet "Item types" has been updated and that a facet "Collections" is shown (from the 995$h) and functional C. Check Elasticsearch indexing changes 13 / You cannot currently test Elasticsearch in a sandbox, so, open a new k-t-d with Elasticsearch search engine (ktd --es7 up) 14 / Repeat operations 7 to 12 in that new instance (277 results should be shown after the search in step 7, instead of 419 for Zebra, this is normal)
It should be fixed, and I added a step in test plan to check that the TYPEDOC AV is removed. Could you test again please?