Bug 38336 - Confusion between 099 and 942 field in UNIMARC framework
Summary: Confusion between 099 and 942 field in UNIMARC framework
Status: Needs Signoff
Alias: None
Product: Koha
Classification: Unclassified
Component: Cataloging (show other bugs)
Version: Main
Hardware: All All
: P5 - low normal
Assignee: Bugs List
QA Contact: Testopia
URL:
Keywords:
: 37788 (view as bug list)
Depends on:
Blocks:
 
Reported: 2024-11-03 10:17 UTC by Mathieu Saby
Modified: 2025-03-07 14:43 UTC (History)
8 users (show)

See Also:
Change sponsored?: ---
Patch complexity: ---
Documentation contact:
Documentation submission:
Text to go in the release notes:
Version(s) released in:
Circulation function:


Attachments
Bug 38336: Fix confusion between 099 and 942 UNIMARC fields (19.13 KB, patch)
2025-02-15 19:35 UTC, Mathieu Saby
Details | Diff | Splinter Review
Bug 38336: Fix confusion between 099 and 942 UNIMARC fields (19.36 KB, patch)
2025-03-06 09:35 UTC, Mathieu Saby
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description Mathieu Saby 2024-11-03 10:17:52 UTC
I believe there is something wrong in UNIMARC default framework (https://github.com/Koha-Community/Koha/blob/main/installer/data/mysql/en/marcflavour/unimarc/mandatory/unimarc_framework_DEFAULT.yml )

We have 2 fields for "local" data (specific to Koha) at biblio level : 099 and 942
And the same pieces of information are present in subfields of both fields :
- 099$x and 942$0 (Number of issues => biblioitems.totalissues)
- 099$s and 942$s (Serial record => biblio.serial)
- 099$t and 942$c  (Koha item type => biblioitems.itemtype)

I suppose it is an error, because it makes no sense... And note that the 099 field in UNIMARC framework is called "local dates", so it should not contain information about item type or number of issue.

Do you think we could remove safely from the default templates 099$x, 099$s, 099$t ?  I don't know if those values may be harcoded somewhere in the code.
Comment 1 Phil Ringnalda 2024-11-08 21:14:26 UTC
*** Bug 37788 has been marked as a duplicate of this bug. ***
Comment 2 Phil Ringnalda 2024-11-08 21:38:15 UTC
There's a little hint about what's going on in bug 34266 - apparently Biblibre is doing something that makes them want to have a biblio-level itemtype-like thing which is not the same as item-level itemtypes, and so probably the existence of both 099$t and 942$c in the default framework is the expression of a longtime "discussion" between joubu and cait.
Comment 3 Phil Ringnalda 2024-11-08 21:40:46 UTC
Oh, and we should probably consider the Fast Add framework too, since it only has 099$t and not 942$c - if the reason for TYPEDOC is something like FRBR, umm, Fast Added FRBR? Neato!
Comment 4 Mathieu Saby 2024-11-09 17:01:50 UTC
hell of a headache,  isn't it ?

But I am not sure it is the same scope as the current bz : we could already clean 099$x, 099$s and 099$t, and keep for later the question raised by "942$c  (Koha item type => biblioitems.itemtype)"

For history record :

Regarding the fact that in UNIMARC biblioitems.itemtype and items.itype are not always mapped to itemtypes.itemtype, it is certainly a very old move. I was administrating a Koha 3.x in 2011 (with Biblibre support), and this was already our configuration. I believe it was a choice made when Koha was introduced in France in the early 2000s.

So a lot of french libraries with Biblibre support have been using for many years items.itype to define the type of loan (short / long / no loan...). The same values could not meaningflully be used at record level, and fill itemtypes.itemtype with other values (book / dvd / ...)
Changing this behavior would be a huge move backwards for all of us.
Comment 5 Mathieu Saby 2024-11-09 18:39:09 UTC
I had time to do some archeology :

Before bug 30373 (conversion to yaml, and suppression of the french framework) the french UNIMARC framework was quite different from the english one. The new yaml framework seems to have inherited from both, in a way that is not completely correct.

* History of the french UNIMARC framework before its conversion to yaml :
In 2008 : 
At biblio level : 
- no 942 field
- 099 field "information locales" (local information) only used for creation and modification dates
- 200$b field "type de document" (document type) <=> biblioitems.itemtype, linked to itemtypes AV
- 686$a field "type (indice)" linked with itemtypes AV
At item level : 995$r field "type de prêt" (type of loan) mapped with items.itype, linked to itemtypes AV

Note that the label was "type of loan" and not "Type of item and material" like in the english framework

Bug 6557 in 2012 :
- creation of a 942 field "total des prêts" (number of checkouts) with only one subfield $0 

Bug 11586 in 2014 :
- 686$a and 200$b fields no longer mapped with biblioitems.itemtype neither linked to any AV
- new subfield 099$t "Type de document" (document type) <=> biblioitems.itemtype, not linked to any AV
- new subfield 099$x "Nombre de prêts" (number of checkouts) <=> biblioitems.totalissues => duplicate of 942$0 !

At item level : 995$r field "type de prêt" has now a default value "PRETLIV" (loan of book) => weird because it did not fit the values defined in sample_itemtypes.sql

At that point (just before the framework conversion to yaml) 099$t and 995$r were both mapped to biblioitems.itemtype, BUT only 995$r were linked to itemtypes AV. French libraries were free to linked their 099$t to the AV of their choice (like "TYPEDOC")


========

Changes to english UNIMARC framework :
In 2008 :
At biblio level : 990 field "ADDED ENTRY ELEMENTS (KOHA)", with:
990$0 : Koha issues (borrowed) <=> biblioitems.totalissues
990$c : Koha item type <=> biblioitems.itemtype ; linked to itemtypes AV
At item level : 995$r field "Type of item and material", mapped with nothing, not linked to any AV

Bug 5128 in 2010 (at the initiative of the Italian Zeno Tajoli https://lists.koha-community.org/pipermail/koha-devel/2010-July/034235.html )
- Removal of 990 field
- Creation of 099 field "local dates" : same use as 099 of fr-FR framework, but with a different name, more restrictive (in french it was "local information"
- Creation of 942 field "ADDED ENTRY ELEMENTS (KOHA)", with : 
942$0 : Koha issues (borrowed) <=> biblioitems.totalissues
942$c : Koha item type <=> biblioitems.itemtype ; linked with itemtypes AV
At item level : the 995$r field (Type of item and material) is mapped with 'items.itype' and linked with 'itemtypes'

At that point (just before the framework conversion to yaml) 942$c and 995$r were both mapped to biblioitems.itemtype, AND  linked to itemtypes AV (contrary to the french framework). The use of 099 was more restricted than in french framework, the use of 942 much broader. And the wording of 099, 942 and 995$r were different from french framework.
Comment 6 Mathieu Saby 2024-11-09 19:34:04 UTC
By the way, this bug is linked : 
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=38042
Comment 7 Katrin Fischer 2024-11-11 10:25:11 UTC
(In reply to Mathieu Saby from comment #6)
> By the way, this bug is linked : 
> https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=38042

I have just added a comment there explaining a bit why I think UNIMARC should change that and introduce a proper field for document type instead. The current mapping requires patching again and again and you STILL have broken features available to MARC21 users that rely on it being an itemtype in the record and not something else.
Comment 8 Mathieu Saby 2024-12-14 11:32:28 UTC
After all those discussions it seems to me that the logic of defaut UNIMARC framework must be aligned with that of MARC21, and TYPEDOC values should be removed from default installations.
There are no technical or standards-related reasons for the current difference.

If vendors or libraries want to change the default behavior, the question of the extent to which these changes are compatible with the smooth running of Koha is an other subject.

But I would appreciate some feedback from other people from the UNIMARC community.
Comment 9 David Nind 2024-12-14 16:44:28 UTC
As France uses UNIMARC (as well as some other countries), I would suggest getting some feedback from the KohaLa user group (https://koha-fr.org/liste-koha-infos/) and perhaps from the chat channel (koha-Francophone).
Comment 10 Mathieu Saby 2024-12-17 17:06:23 UTC
Oh I tried the list, being a member of Kohala. There seems to be no interest for the subject, or more probably a lack of cataloging skills in our librairies, due to changes in recruitment policy...
And I asked on Mattermost too.
If there are no objection I will provide a patch this weekend
Comment 11 Mathieu Saby 2024-12-17 17:10:57 UTC
After checking, I hadn't asked a question on this specific subject on thé list, so I'm going to do so...
Comment 12 David Nind 2024-12-17 19:02:55 UTC
I've added Fridolin from BibLibre to the cc list. 

Maybe, he can ask someone from BibLibre to have a look or provide some comment (I think they may look after one or two UNIMARC libraries 8-)).
Comment 13 Mathieu Saby 2024-12-18 10:51:55 UTC
I wrote this message to the french list: 

Unless there are serious objections, I'd like to propose a patch making the following changes: 
- 099 will be used only for local dates (which have no equivalent in MARC21)
- 942 for other local information (aligned as far as possible with 942 MARC21)
- 942$c will be linked to the list of document types in “itemtypes”.
- the authorized value TYPEDOC will be removed
- Zebra and Elasticsearch indexes will be revised to adapt to these changes

The fast cataloging grid will then have to be revised.

Of course, this will only impact new installations. 

If service providers or libraries wish to adapt the grids and link them to the authorized TYPEDOC value, they are free to do so, but this will no longer be part of the standard settings.
Comment 14 David Nind 2024-12-18 20:56:51 UTC
+1
Comment 15 Mathieu Saby 2025-01-10 16:55:03 UTC
I am writing the patch
Comment 16 Mathieu Saby 2025-02-15 19:35:02 UTC
Created attachment 178126 [details] [review]
Bug 38336: Fix confusion between 099 and 942 UNIMARC fields

This patch fixes 099 and 942 fields (and incidentally 200$b)
in the default and the FA UNIMARC frameworks and provides
the changes that follow logically :
- Zebra and Elasticsearch indexing
- removal of the unused TYPEDOC authorized values category
- UNIMARC facets in Koha.pm
- test in TestBuilder.pm

These changes are minimal and are intended to correct only
the most serious problems. Subsequent work may bring improvements.

Detail of changes :
- The 099 field is renamed "Local dates (Koha)" and should be
used in default configuration only for this storing creation
and modification dates of record.
- The 942 field is mandatory (because its $c subfield is)
and should be used for all other Koha local information,
like its MARC21 counterpart.
- The 200$b field is no more indexed (it is now obsolete in
cataloguing standards, and its MARC21 counterpart is not indexed).
- Like in MARC21, ccode is now defined only at item level and
itemtype/itype is now defined at item and biblio level.
- The default values for itemtype at biblio and item level are
the same, controlled by itype values (document type).
- The TYPEDOC authorized values category, which was used for
controlling the value if itemtype at record level, is removed.
- In Koha.pm UNIMARC facets are fixed to be a mirror of MARC21
facets: a facet for itemtypes is created, and the facet for ccode
is now based only on items.
- A test on record-level itemtype in TestBuilder.pm is fixed
in order to replace 099 with 942 field.

Test plan:

Note that sample data in ktd and sandbox will need to be fixed once
this patch is pushed (there is no data in 942$c for example).
It may cause warnings if you edit existing record but it is not a bug.

A. Check framework changes
1/ In a UNIMARC k-t-d or sandbox, with Zebra search engine
2/ Apply the patch
3/ Check default and FA framework
- 099 should be labelled "Local dates (Koha)" and
contain 2 subfields:c and d
- 942 should be mandatory, labelled "ADDED ENTRY ELEMENTS (Koha)"
and contain 5 subfields : 0,2,6,c (mandatory),s
4/ Create a new record with default framework
-- add a 200$a field, a 100$a field (with the cataloging plugin)
-- in 942$c field, check that the values in dropdown list are
the one defines for 'item types' in administration (Books,
Computer Files, Continuing Resource, etc).
-- Choose the value "Computer field" and Save the record
-- Koha displays the item creation form. Check that the
preselected value in 995$r (item type) is "Computer field"
5 / Do the same operation for a new record with FA framework

B. Check Zebra indexing changes
6 / Search "livre" in the catalog. 419 results should be shown
7 / Check that an "Item type" facet is visible and functional
8 / Edit the first record of the result list, change its 942$c
value and save
9 / Edit an item of that record and change the 995$r value
(with a value different from the 942$c) and the 995$h value
(3 possible values: Non-fiction, Fiction, Reference)
10 / Refresh the search for "livre"
11 / Check that the facet "Item types" has been updated and
that a facet "Collections" is shown (from the 995$h) and functional

C. Check Elasticsearch indexing changes
12 / I believe you cannot currently test Elasticsearch in a sandbox,
so, open a new k-t-d with Elasticsearch search engine (ktd --es7 up)
13 / Repeat operations 6 to 11 in that new instance
(277 results should be shown after the search in step 6,
instead of 419 for Zebra, this is normal)
Comment 17 Andreas Roussos 2025-03-06 08:30:06 UTC
Hi Mathieu!

I tried to test this today using K-T-D but the patch no longer applies
cleanly on the current main branch (commit 8abf287e6d).

More specifically:

1) there are trailing whitespace warnings, these appear to come from:

--- a/installer/data/mysql/mandatory/auth_val_cat.sql
+++ b/installer/data/mysql/mandatory/auth_val_cat.sql
@@ -56,8 +56,7 @@ INSERT IGNORE INTO authorised_value_categories( category_name, is_system )
 INSERT IGNORE INTO authorised_value_categories( category_name ) VALUES
     ('ETAT'),
     ('CAND'),
-    ('COUNTRY'),
-    ('TYPEDOC'),
+    ('COUNTRY'),  <--------------- Here are the two extra spaces
     ('qualif');

2) there are merge conflicts in two files:

   C4/Koha.pm
   t/lib/TestBuilder.pm

   Which are most likely related to the mass tidying of the
   codebase which was recently committed to main (Bug 38664).

Also, I think the test plan would be a bit easier to read
if it used a hanging indent, as per the Wiki instructions at:

https://wiki.koha-community.org/wiki/Commit_messages#Test_plan

So, instead of:

A. Check framework changes
1/ In a UNIMARC k-t-d or sandbox, with Zebra search engine
2/ Apply the patch
3/ Check default and FA framework
- 099 should be labelled "Local dates (Koha)" and
contain 2 subfields:c and d
- 942 should be mandatory, labelled "ADDED ENTRY ELEMENTS (Koha)"
and contain 5 subfields : 0,2,6,c (mandatory),s
4/ Create a new record with default framework
-- add a 200$a field, a 100$a field (with the cataloging plugin)
-- in 942$c field, check that the values in dropdown list are
the one defines for 'item types' in administration (Books,
Computer Files, Continuing Resource, etc).
-- Choose the value "Computer field" and Save the record
-- Koha displays the item creation form. Check that the
preselected value in 995$r (item type) is "Computer field"
5 / Do the same operation for a new record with FA framework

Something like this, perhaps?

A. Check framework changes
1/ In a UNIMARC k-t-d or sandbox, with Zebra search engine
2/ Apply the patch
3/ Check default and FA framework
   - 099 should be labelled "Local dates (Koha)" and
     contain 2 subfields:c and d
   - 942 should be mandatory, labelled "ADDED ENTRY ELEMENTS (Koha)"
     and contain 5 subfields : 0,2,6,c (mandatory),s
4/ Create a new record with default framework
   -- add a 200$a field, a 100$a field (with the cataloging plugin)
   -- in 942$c field, check that the values in dropdown list are
      the one defines for 'item types' in administration (Books,
      Computer Files, Continuing Resource, etc).
   -- Choose the value "Computer field" and Save the record
   -- Koha displays the item creation form. Check that the
      preselected value in 995$r (item type) is "Computer field"
5/ Do the same operation for a new record with FA framework

Finally, do you know if there a way to test the changes made to
the installer's YAML files?

Thank you!
Comment 18 Mathieu Saby 2025-03-06 09:35:28 UTC
Created attachment 179007 [details] [review]
Bug 38336: Fix confusion between 099 and 942 UNIMARC fields

This patch fixes 099 and 942 fields (and incidentally 200$b)
in the default and the FA UNIMARC frameworks and provides
the changes that follow logically :
- Zebra and Elasticsearch indexing
- removal of the unused TYPEDOC authorized values category
- UNIMARC facets in Koha.pm
- test in TestBuilder.pm

These changes are minimal and are intended to correct only
the most serious problems. Subsequent work may bring improvements.

Detail of changes :
- The 099 field is renamed "Local dates (Koha)" and should be
used in default configuration only for this storing creation
and modification dates of record.
- The 942 field is mandatory (because its $c subfield is)
and should be used for all other Koha local information,
like its MARC21 counterpart.
- The 200$b field is no more indexed (it is now obsolete in
cataloguing standards, and its MARC21 counterpart is not indexed).
- Like in MARC21, ccode is now defined only at item level and
itemtype/itype is now defined at item and biblio level.
- The default values for itemtype at biblio and item level are
the same, controlled by itype values (document type).
- The TYPEDOC authorized values category, which was used for
controlling the value if itemtype at record level, is removed.
- In Koha.pm UNIMARC facets are fixed to be a mirror of MARC21
facets: a facet for itemtypes is created, and the facet for ccode
is now based only on items.
- A test on record-level itemtype in TestBuilder.pm is fixed
in order to replace 099 with 942 field.

Test plan:

Note that sample data in ktd and sandbox will need to be fixed once
this patch is pushed (there is no data in 942$c for example).
It may cause warnings if you edit existing record but it is not a bug.

A. Check AV and framework changes
1/ In a UNIMARC k-t-d or sandbox, with Zebra search engine
2/ Apply the patch
3/ Check that there is no "TYPEDOC" authorized value category
4/ Check default and FA framework
   - 099 should be labelled "Local dates (Koha)" and
     contain 2 subfields:c and d
   - 942 should be mandatory, labelled "ADDED ENTRY ELEMENTS (Koha)"
     and contain 5 subfields : 0,2,6,c (mandatory),s
5/ Create a new record with default framework
   -- add a 200$a field, a 100$a field (with the cataloging plugin)
   -- in 942$c field, check that the values in dropdown list are
      the one defines for 'item types' in administration (Books,
      Computer Files, Continuing Resource, etc).
   -- Choose the value "Computer field" and Save the record
   -- Koha displays the item creation form. Check that the
      preselected value in 995$r (item type) is "Computer field"
6/ Do the same operation for a new record with FA framework

B. Check Zebra indexing changes
7 / Search "livre" in the catalog. 419 results should be shown
8 / Check that an "Item type" facet is visible and functional
9 / Edit the first record of the result list, change its 942$c
value and save
10 / Edit an item of that record and change the 995$r value
(with a value different from the 942$c) and the 995$h value
(3 possible values: Non-fiction, Fiction, Reference)
11 / Refresh the search for "livre"
12 / Check that the facet "Item types" has been updated and
that a facet "Collections" is shown (from the 995$h) and functional

C. Check Elasticsearch indexing changes
13 / You cannot currently test Elasticsearch in a sandbox,
so, open a new k-t-d with Elasticsearch search engine (ktd --es7 up)
14 / Repeat operations 7 to 12 in that new instance
(277 results should be shown after the search in step 7,
instead of 419 for Zebra, this is normal)
Comment 19 Mathieu Saby 2025-03-06 09:36:42 UTC
It should be fixed, and I added a step in test plan to check that the TYPEDOC AV is removed. Could you test again please?