Bug 29533

Summary: Add additional fields to items
Product: Koha Reporter: Mathieu Saby <mathsabypro>
Component: CatalogingAssignee: Bugs List <koha-bugs>
Status: NEW --- QA Contact: Testopia <testopia>
Severity: enhancement    
Priority: P5 - low CC: fridolin.somers, kyle, m.de.rooy, michaela.sieber
Version: Main   
Hardware: All   
OS: All   
GIT URL: Change sponsored?: ---
Patch complexity: --- Documentation contact:
Documentation submission: Text to go in the release notes:
Version(s) released in:
Circulation function:

Description Mathieu Saby 2021-11-19 17:49:30 UTC
MARC tag dedicated to items (995 Unimarc / 952 Marc21) can contain a lot of subfields (0-9 ; a-z ; A-Z).
It frequently happens that a value stored in this tag cannot be mapped to items SQL table through "Koha => MARC" configuration page. The use of  "more_subfields_xml" to make something useful of these information is cumbersome.

Koha should provide a way to map to SQL more information related to items, for example by creation 10 new fields in items sql table (items.customfield1, items.customfield2, etc), or something more subtle.
Comment 1 Marcel de Rooy 2021-11-20 12:45:25 UTC
(In reply to mathieu saby from comment #0)
> MARC tag dedicated to items (995 Unimarc / 952 Marc21) can contain a lot of
> subfields (0-9 ; a-z ; A-Z).

Bad idea. Dont use these capital letters. Not MARC compliant.
Comment 2 Mathieu Saby 2021-11-20 13:58:45 UTC
Hi
Even with only lowercases in 952/995, do you believe that the idea of allowing more subfields to be mapped in SQL is a bad idea?

But I think uppercases are allowed in UNIMARC for subfields in local (9XX) fields, and I doubt it is forbidden in MARC21.

For Marc21 the alphabetic character must be lowercase:

Subfield codes - Two characters that distinguish the data elements within a field which require separate manipulation. A subfield code consists of a delimiter (ASCII 1F hex), represented in this document as a $, followed by a data element identifier. Data element identifiers may be a lowercase alphabetic or a numeric character. 
https://www.loc.gov/marc/bibliographic/bdintro.html

But there is no such precision for UNIMARC:
Subfield Identifier – A code consisting of two characters identifying individual subfields within a variable field. The first character, the delimiter, is always the same unique character specified in ISO 2709 and the second character, the subfield code, is either numeric or alphabetic
https://eprints.mdx.ac.uk/2655/1/UNIMARC-2008-AH_final.pdf


And in marcXML schema you can read this, so it seems that uppercase are allowed in MARC21 for local data
https://www.loc.gov/standards/marcxml/schema/MARC21slim.xsd

"May 21, 2009 - Version 1.2 - in subfieldcodeDataType the pattern "[\da-z!"#$%&'()*+,-./:;<=>?{}_^`~\[\]\\]{1}" changed to: "[\dA-Za-z!"#$%&'()*+,-./:;<=>?{}_^`~\[\]\\]{1}" i.e "A-Z" added after "[\d" before "a-z" to allow upper case. This change is for consistency with the documentation. ************************************************************ This schema supports XML markup of MARC21 records as specified in the MARC documentation (see www.loc.gov). It allows tags with alphabetics and subfield codes that are symbols, neither of which are as yet used in the MARC 21 communications formats, but are allowed by MARC 21 for local data."
Comment 3 Katrin Fischer 2021-11-21 22:12:25 UTC
Thx Mathieu, that comment in the schema is really interesting! Upper case subfields work in Koha, but as Marcel points out they are not standard in MARC21. Bug I know they are used in local data by us and other libraries as well.
Comment 4 Katrin Fischer 2021-11-21 22:12:54 UTC
I'd be interested in what your problem with more_subfields_xml is? We have quite a bit of data there and no issues.
Comment 5 Marcel de Rooy 2021-11-22 07:48:40 UTC
(In reply to mathieu saby from comment #2)
> Even with only lowercases in 952/995, do you believe that the idea of
> allowing more subfields to be mapped in SQL is a bad idea?
No, not necessarily.

> But I think uppercases are allowed in UNIMARC for subfields in local (9XX)
> fields, and I doubt it is forbidden in MARC21.
As you wrote for MARC21: Data element identifiers may be a lowercase alphabetic or a numeric character. Thats clear language?
Comment 6 Mathieu Saby 2021-11-25 16:40:06 UTC
(In reply to Katrin Fischer from comment #4)
> I'd be interested in what your problem with more_subfields_xml is? We have
> quite a bit of data there and no issues.

A collegue from an other library told me it was "clumsy". I cannot say something more precisely because we are only planning to migrate to Koha and I am doing some tests. Generally speaking I confess I am not fond of Koha's logic for managing items (some other ILS I know, and our current one, store item data in SQL and only map them in MARC for display or export...), but we will adapt.

What I fear mainly with more_subfield_xml is performance problems, if we need to use ExtractValue() for extracting information from this field in a report. Maybe am I worrying about nothing, but I have bad memories of ExtractValue() applied to MARCXML data.

Less important, it will also be more complicated if we want to develop plugins or patches for customizing or making more parametrable the display of items (add a new column in OPAC for a information stored in more_subfield_xml )
Comment 7 Fridolin Somers 2021-12-18 07:39:04 UTC
This would be a little bit like borrowers sort1 and sort2.
I'm not against it :/

We should consider using the "additional fields" feature like for subscriptions, Bug 10855
Comment 8 Katrin Fischer 2022-07-21 08:09:41 UTC
I think a separate key-value style table would be the best way to improve the situation.

If we offered a 'Koha only' storage without the need to map to map to a valid MARC subfield identifier, as we have run out of those. 
 
We could, for example, use capital letters only internally and do something like $9 + defined prefix for export/import.

It would also help with the situation where data gets indexed and searchable that should not be (see recurring internal notes discussion).
Comment 9 Kyle M Hall (khall) 2024-05-17 17:25:34 UTC
Something that uses the additional fields subsystem (https://schema.koha-community.org/23_11/tables/additional_fields.html) and is patterned after the Item Message plugin ( https://github.com/bywatersolutions/koha-plugin-item-messages ) seems like a good place to take this.