Bug 32722 - UNIMARC: Remove mandatory flag from some subfields and field in default bibliographic framework
Summary: UNIMARC: Remove mandatory flag from some subfields and field in default bibli...
Status: RESOLVED FIXED
Alias: None
Product: Koha
Classification: Unclassified
Component: MARC Bibliographic data support (show other bugs)
Version: Main
Hardware: All All
: P5 - low critical
Assignee: Mathieu Saby
QA Contact: Martin Renvoize (ashimema)
URL:
Keywords:
: 36576 (view as bug list)
Depends on: 30373
Blocks:
  Show dependency treegraph
 
Reported: 2023-01-25 12:15 UTC by Thibault Keromnès
Modified: 2025-04-18 18:57 UTC (History)
12 users (show)

See Also:
GIT URL:
Change sponsored?: ---
Patch complexity: ---
Documentation contact:
Documentation submission:
Text to go in the release notes:
This updates the default UNIMARC bibliographic record framework to remove the mandatory flag from some subfields and fields. For UNIMARC, several subfields are only mandatory if the field is actually used (MARC21 does not have this requirement). A change made to the default framework by bug 30373 in Koha 22.05 meant that if the mandatory subfield was empty, and the field itself was optional (not mandatory), you couldn't save the record. For example, if field 410 (Series) is used (this is an optional field), then subfield $t (Title) is required. However, the way the default framework was set up (subfield $t was marked as mandatory) you couldn't save the record - as subfield $t was mandatory, even though the 410 is optional. As Koha is not currently able to manage both the UNIMARC and MARC21 requirements without significant changes, a practical decision was made to configure the otherwise mandatory subfields as not mandatory. Important note: This only affects NEW UNIMARC installations. Existing installations should edit their default UNIMARC framework to make these changes (although, it is likely that they have already done so).
Version(s) released in:
25.05.00,24.11.02,24.05.08
Circulation function:


Attachments
List of mandatory subfields (2.28 KB, application/vnd.ms-excel)
2023-01-25 12:15 UTC, Thibault Keromnès
Details
Bug 32722: Don't block record save if subfield is mandatory but field is optional (11.37 KB, patch)
2023-03-29 12:19 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 32722: Don't block record save if subfield is mandatory but field is optional (11.38 KB, patch)
2024-07-11 12:48 UTC, Biblibre Sandboxes
Details | Diff | Splinter Review
Bug 32722: Make several subfields and a field non mandatory in default UNIMARC biblio framework (15.81 KB, patch)
2024-12-07 18:52 UTC, Mathieu Saby
Details | Diff | Splinter Review
Bug 32722: Make several subfields and a field non mandatory in default UNIMARC biblio framework (15.93 KB, patch)
2024-12-10 07:40 UTC, Victor Grousset/tuxayo
Details | Diff | Splinter Review
Bug 32722: Make several subfields and a field non mandatory in default UNIMARC biblio framework (16.00 KB, patch)
2024-12-17 07:06 UTC, Martin Renvoize (ashimema)
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description Thibault Keromnès 2023-01-25 12:15:15 UTC
Created attachment 145658 [details]
List of mandatory subfields

BZ30733 added many mandatory subfields in UNIMARC default framework, following the IFLA manual updates. 
Thing is, most of those mandatory subfields are within an optional field, so they're supposed to be mandatory only if the field is used. Yet Koha blocks the record's creation with ~60 mandatory subfields.

Most practical solution would probably be to remove the mandatory flag to said subfields (the list is attached to the bug). 

Exception are 100$a and 200$a (the fields being mandatory too it makes sense)
Also have a doubt concerning 099$t and 942$c, that are not actually Unimarc, but Koha specifics, storing biblioitems.itemtype.
Comment 1 Katrin Fischer 2023-01-25 12:17:21 UTC
You could also make them important maybe instead of mandatory. That gives a warning, but still allows to save.
Comment 2 Thibault Keromnès 2023-03-27 13:56:24 UTC
Hello Katrin,
I'm not sure it's the best way, most mandatory subfields are just not relevant, depending on the document type.
For instance: 120$a asks for the scale of a map, even if you're working on a movie.
Comment 3 Jonathan Druart 2023-03-29 12:19:04 UTC
Created attachment 148909 [details] [review]
Bug 32722: Don't block record save if subfield is mandatory but field is optional

It should only be considered mandatory if at least one of the other
subfield is not empty.
Comment 4 Jonathan Druart 2023-03-29 12:22:06 UTC
This code is a nightmare!
Thibault, can you confirm the patch works as you expect? I think it also impact "important" subfields, not sure it should...
Comment 5 Thibault Keromnès 2023-03-30 07:41:50 UTC
Hey Jonathan, 
It does work as expected, but I've talked about it with Katerin & Aude, and it seems it would be a problem for Marc21 users. 
They consider a subfield to be mandatory whether the field itself is mandatory or not, so with this patch we would risk having records created without key informations.
I should have started there, sorry.
Comment 6 Thibault Keromnès 2023-03-31 12:48:56 UTC
Need for additional features on this patch: 
- an update on the database : in existing frameworks, for every mandatory subfields, make the corresponding field mandatory too (so the behaviour doesn't change by default)
- when editing a subfield : add a note to warn the user that the subfield will be mandatory only if the field is mandatory or if another subfield is not empty
- reproduce this behaviour for the authorities framework
Comment 7 Phil Ringnalda 2024-04-05 17:01:04 UTC
Since nothing involving updating existing frameworks can ever be easy:

Unlike the basic editor, the advanced editor already does exactly what you want: if 120$a is mandatory but 120 is not mandatory, then if you have no 120 in the advanced editor you are free to save, but if you have 120$b you are required to have $a.

So a MARC21 library which only uses the advanced editor, never the basic editor, could have intentionally set 100$a as mandatory but not set 100 as mandatory, intending to have the advanced editor require that if you use a main entry personal name, you actually have the name, not just an accidental "$q(Philip Anthony)$d1963-" without the name itself in $a. Automatically changing that to require that every bib record have a main entry personal name would be quite a shock.
Comment 8 Victor Grousset/tuxayo 2024-04-11 14:30:15 UTC
*** Bug 36576 has been marked as a duplicate of this bug. ***
Comment 9 Victor Grousset/tuxayo 2024-04-15 02:34:16 UTC
Found via git bisect that it comes from commit: "Bug 30373: Fix visibility, authorised values and descriptions"

----

Raised severity because UNIMARC cataloguing doesn't work out of the box.
It's a roadblock for small libraries that don't have a support company that has their own framework. Or don't know how to customize frameworks especially with such an unexpected default state.
less severe effect: In the end, it makes impossible to setup a framework that follows the standard so one must manually enforce the mandatory subfields within an optional field.

More meta issue: it's makes a pain to test UNIMARC patches.

---

Current patch makes creation and edition of biblio records usable again :D

Is it new the situation of mandatory subfields within an optional field? Or was there other occurrences before these IFLA UNIMARC updates?

Another inconsistency in handling those cases is that even if thanks to the patch, the save isn't blocked, all the 4XX subfields have the red "Required" and are colored yellow. Undistinguishable from really mandatory ones.

---


(In reply to Thibault Keromnès from comment #5)
> It does work as expected, but I've talked about it with Katerin & Aude, and
> it seems it would be a problem for Marc21 users. 

Would it work to have the change only affect UNIMARC? If MARC21 needs any change related to mandatory subfields, it could be another ticket. It's not the same impact on both standards.


(In reply to Thibault Keromnès from comment #6)
> Need for additional features on this patch: 
> - an update on the database : in existing frameworks, for every mandatory
> subfields, make the corresponding field mandatory too (so the behaviour
> doesn't change by default)

Is the need of data migration only for existing libraries that didn't got installed with the recent (22.05/bug 30373) default framework?
And those recently installed but with a different framework, like from their provider.
Comment 10 Jonathan Druart 2024-04-16 07:51:02 UTC
Just tell me what's needed. But be fast, I won't rebase this patch if there is a conflict.
Comment 11 Phil Ringnalda 2024-04-18 20:44:49 UTC
I'm curious about the size of the affected population in need of a retroactive fix. It would be new installs of 22.05 or later, who don't have anyone to fix a framework and who have continued using Koha despite having to fill in ~60 required fields which don't actually apply to the record they are saving, every single time they save a record. Would that be 1, 10, 100, or 1000?
Comment 12 Phil Ringnalda 2024-04-18 20:46:09 UTC
s/installs/UNIMARC installs/
Comment 13 Phil Ringnalda 2024-05-17 01:01:09 UTC
FWIW, my suggestion would be three bugs:

1. Major: Revert the mandatory changes to the UNIMARC frameworks we ship, since they should be a "Mandatory if tag is used" which doesn't currently exist.

2. Enh: Add a third checkbox for subfields (in both UNIMARC and MARC21) for "Mandatory if tag is used".

3. Normal, dependent on 2: Change the UNIMARC frameworks we ship to check *that* box for the things reverted in 1.
Comment 14 Victor Grousset/tuxayo 2024-06-18 00:05:22 UTC
(In reply to Jonathan Druart from comment #10)
> Just tell me what's needed. But be fast, I won't rebase this patch if there
> is a conflict.

Don't do anything for now, just a question: is that a lot of changes to implement this?:
> 2. Enh: Add a third checkbox for subfields (in both UNIMARC and MARC21)
> for "Mandatory if tag is used".

Not sure but seems like we could get away with changing the condition here:

https://git.koha-community.org/Koha-community/Koha/src/commit/8ab3989654b900cef2954c469b12ddce4375c9b8/koha-tmpl/intranet-tmpl/prog/en/modules/cataloguing/addbiblio.tt#L581
Comment 15 Phil Ringnalda 2024-07-06 06:56:07 UTC
The UNIMARC authority types we ship are just as bad.
Comment 16 Mathieu Saby 2024-07-11 12:47:14 UTC
For UNIMARC, your patch is fixing our problems, except this little issue:
- add data in 410$t (mandatory subfield in non mandatory field)
- click on save
- during 1 or 2 seconds, all the 4XX$t are colored in yellow, with no reason
- but the record is saved

I believe it can be fixed in a follow-up, so I add myt sign-off.

BUT I cannot tell if your patch will cause problems to MARC21 installations, because I don't know well this format. Should the patch be tested and signed off also by a a member of MARC21 community ?

========================
Here is my test plan :


**** Before patch : 

- Impossible to create a record without filling mandatory subfields in non mandatory fields
Ex :
create a new record with only the value "test" in 200$a
Error message : 
"The following mandatory subfields aren't filled:
Tag 020 subfield a Country Code in tab 0 
Tag 021 subfield a Country Code in tab 0 
etc"

- Impossible to edit existing record without filling mandatory subfields in non mandatory fields
Ex :
- Identify a non mandatory subfield in a non mandatory field (for ex, 856$u)
- Find a record with no value at all in this field
- Edit the framework to make the subfield (but not the field) mandatory
- Try to edit the record and to save it => Impossible, with this message : "The following mandatory subfields aren't filled:Tag 856 subfield u "


**** After patch : 

- Identify a non mandatory subfield in a non mandatory field (for ex, 856$u)
- Find a record with no value at all in this field
- Edit the framework to make the subfield (but not the field) mandatory
- Try to edit the record and to save it => possible, without alert

Other tests  :

- Add data in non mandatory subfield of non mandatory field (205$a) => possible
- Add data in non mandatory subfield of mandatory field (200$i) => possible
- Delete data in non mandatory subfield of non mandatory field (210$c) => possible
- Delete data in non mandatory subfield of mandatory field (200$f) => possible
- Change data in mandatory subfield of mandatory field (200$a) => possible
- Change data in mandatory subfield of non mandatory field (410$t) => possible
- Delete a subfield of mandatory field but keeping the other subfields (810$b) => save
- Delete all subfields of mandatory field (810$b) => impossible to save : "The following mandatory subfields aren't filled: The following fields aren't filled: Field 801 is mandatory, at least one of its subfields must be filled. "
- Delete a mandatory subfield of a mandatory field (210$a) => impossible to save : "The following mandatory subfields aren't filled: Tag 200 subfield a Title Proper in tab 2 "
- Add data in non mandatory subfield of non mandatory field (942$2) , but keeping blank the mandatory subfield (942$c) => impossible to save : "The following mandatory subfields aren't filled:Tag 942 subfield c Koha item type in tab 9 "
Comment 17 Biblibre Sandboxes 2024-07-11 12:48:05 UTC
Created attachment 168801 [details] [review]
Bug 32722: Don't block record save if subfield is mandatory but field is optional

It should only be considered mandatory if at least one of the other
subfield is not empty.

Signed-off-by: msaby <mathsabypro@gmail.com>
Comment 18 Mathieu Saby 2024-07-11 12:51:33 UTC
And by the way, the list of mandatory fields and subfield defined in unimarc default framework seems correct to me (we could add a few more mandatory subfields, but it does not really matter)
Comment 19 Phil Ringnalda 2024-07-11 13:16:02 UTC
MARC21 has for some time shipped defaults with 942$c, the biblioitems.itemtype, mandatory, but 942 not mandatory. Having itemtypes be mandatory is kind of a big deal, since patches that change code which assumes they are always defined to instead check for them being defined to avoid crashing will be rejected. So for MARC21 just changing the meaning of mandatory subfield in non-mandatory field wouldn't be acceptable.
Comment 20 Mathieu Saby 2024-07-15 12:16:42 UTC
So the patch fixes Koha for Unimarc (I agree with Thibault), but causes problem for Marc21. What could be the solution?
Comment 21 Victor Grousset/tuxayo 2024-07-20 01:28:37 UTC
(In reply to Mathieu Saby from comment #20)
> So the patch fixes Koha for Unimarc (I agree with Thibault), but causes
> problem for Marc21. What could be the solution?

comment 14 might be one.
Which is a subpart of the proposal of comment 13. Which needs the addition that the advanced editor would also need to be changed to account for the "Mandatory if tag is used" switch.

-----

(In reply to Phil Ringnalda (back in August) from comment #19)
> So for MARC21 just changing the meaning of
> mandatory subfield in non-mandatory field wouldn't be acceptable.

It came in chat I think but I don't remember if there would be a problem:
What about changing the MARC21 frameworks in existing installs and the installer to set to mandatory the fields containing a mandatory subfield?
Then the current patch would have the basic editor behave identically to the advanced editor (and have a bit of code cleanup as bonus). And the difference of policies of the impact of mandatory subfields on fields will be managed by choosing or not to set the various fields as mandatory. Which will be already be done by the migration so current behavior wouldn't be changed.
Comment 22 Phil Ringnalda 2024-08-06 04:13:59 UTC
(In reply to Victor Grousset/tuxayo from comment #21)
> It came in chat I think but I don't remember if there would be a problem:
> What about changing the MARC21 frameworks in existing installs and the
> installer to set to mandatory the fields containing a mandatory subfield?

That's comment 7: you cannot tell the difference between an existing install which has a mandatory subfield in a non-mandatory field because it works fine to make the field mandatory in the basic editor which is all they use, and an existing install which has it that way because it works fine to make the subfield mandatory only if the field has any other subfield used in the advanced editor, which is all they use.
Comment 23 Phil Ringnalda 2024-11-07 23:15:30 UTC
I happened to be reading bug 30373 again for something else, and noticed that Bernardo asked a couple of times for any input on mandatories. Shame he didn't get any then, so we wouldn't have this "OMG, they are marked as mandatory in the framework, we must support that, we can't ever revert anything that has landed!" problem.

And in fact UNIMARC has a third flavor of mandatory, which he didn't include, "mandatory if another particular subfield is used." If we believe that Koha is required to enforce every use of "MA" (mandatory if applicable) in the Occurrence column of the UNIMARC spec, then along with making 036$a, 036$b, and 036$c mandatory if any 036 subfield is used, we also need to make 036$d, 036$m, and 036$2 mandatory if 036$p is used.

But wait, there's more! There are also subfields which are MA because they are mandatory *unless* another subfield is used: 016$a, International Standard Recording Code, is mandatory if 016 is used unless 016$z, Erroneous ISRC, is used.

And there's also mandatory content within mandatory fixed-length fields: if we are enforcing every use of the word "mandatory" in the UNIMARC spec, then we need to disallow blanks or fill characters in 100$a for positions 0-7, 22-24, and 26-29.

And there's MA related to both the type of material, and whether or not you are doing FRBR cataloging: 101 is mandatory if the material has language and you aren't doing FRBR.

So make my comment 13 step 2 into "Add a checkbox for 'Mandatory if any subfield is used,' a checkbox and text input for 'Mandatory if subfield [   ] is used,' a checkbox and text input for 'Mandatory unless subfield [   ] is used,' and for fixed-length fields a...

Retract my comment 13 plan. The only reasonable plan is to say that "M" means mandatory and Koha should enforce it as mandatory (that's only 001, 100 and 100$a, 200 and 200$a plus local-to-Koha fields, what Thibault said we should keep in comment 0 except that 001 which is unambiguously mandatory is not mandatory in Koha's framework), and all instances of "MA" mean that catalogers should know when it is applicable, and if they create their own framework for things where it is applicable, they are free to make it mandatory in that framework.

Of course, UNIMARC has so many flavors of mandatory that it has confused even them, so in the current spec (which is, amusingly, still not actually approved) they sometimes call a subfield which is mandatory if the optional field is used 'M' and most times call it 'MA', and in section 3.5.1 listing all the mandatory fields they include 801 despite the entire description of it saying that it is MA, only mandatory when it applies because you are sending your records to someone else and you need to at that point include the originating source field, because "In many cases this field will be generated automatically at the time of exchange." Of course, that's the UNIMARC equivalent of MARC21 040, where our default framework makes 040$c mandatory, though since my production install from 2014 has 040$d mandatory we apparently have had a hard time deciding just what we really should require (hint: nothing, since we don't actually use it for anything).
Comment 24 Phil Ringnalda 2024-11-07 23:42:22 UTC
And roughly the same for authories: 001, 100 and 100$a, 152, and the 2xx and its subfield a that's the appropriate one for that authority type, with the same "it's mandatory, but only at the time you export" for 801. Given that for authorities we replace whatever you put in 001 with Koha's authid, it's pointless making that mandatory.
Comment 25 Phil Ringnalda 2024-11-08 04:55:16 UTC
And something clearly went wildly wrong with the mandatories for authorities. We have 676$b as mandatory, which is the "DDC number, End of a Range" which is utterly meaningless without the not-mandatory $a start of a range. Neither 676 nor any of its subfields have ever been mandatory in UNIMARC.

We have 825$a as mandatory, which is a note saying that the current record was used as an example in a different record. That means you cannot create your first authority record, because it is required to have a note about how it was used as an example in another authority record, but this is your first. Stalemate.

Anyone who remembers The Coach in Letterkenny knows how I'm feeling.
Comment 26 Mathieu Saby 2024-11-08 07:07:12 UTC
Thanks for the analysis. All this seems very complicated...

A few days ago, the board of the French Koha users' association discussed the subject: if this is the simplest and most realistic solution, we would agree to modify the UNIMARC framework to remove any "mandatory" flag on subfields belonging to non mandatory fields.

In practice, mandatory subfields are not a very important feature for us, as most of our records are derived from external databases (national catalogue...).

In that case the manual should be amended to make it clear that users should not try to define mandatory subfields if the field is not mandatory.

But first we wanted to write a message on the international Koha list, as there are UNIMARC users outside France (maybe in Italy or Portugal?) and we wanted to get their opinion.
Comment 27 Phil Ringnalda 2024-11-08 20:16:35 UTC
Three more things for whoever patches this to be wary of:

Because bug 26820 exists, for fields below 010 the subfield has to be mandatory to have it enforced. For UNIMARC bibliographic frameworks that means that right now the mandatory "000" (the leader) and 001, the Record indentifier, are not being enforced. UNIMARC authority frameworks are not affected, because both the "000" field and its "@" subfield are marked mandatory, and 001 and its "@" subfield are correctly not marked mandatory because in authority records Koha will replace anything present in 001 with its own authid no matter what you do.

Authority records don't have the same "collapsed" feature that bibliographic records do, so the only subfields that are visible when you first open the editor are mandatory subfields. That means that even for a mandatory field which has only one subfield, like 100, you should make both the field and the subfield mandatory, so that it is open for editing from the start. Currently it feels like a trap when you are told you skipped the mandatory field 100, and even when the error message sends you to the field, there's still no open place to edit it.

The 099 versus 942 thing is confusing, but critical. In the default framework both 099$t and 942$c are mapped to biblioitems.itemtype; 942$c uses the authorized value itemtypes, which is correct, while 099$t uses the authorized value TYPEDOC which has no values. In the Fast Add framework, 099$t is mapped to biblioitems.itemtype but 942 and 942$c do not exist, and 099$t uses the authorized value TYPEDOC, which gives an error in "MARC bibiliographic framework test" because a framework has to have something mapped to bibilioitems.itemtype which uses the authorized value itemtypes. Given that the name of 099 is "local dates" and it is the only place that there is something mapped to biblio.datecreated and biblio.timestamp, it's a reasonable guess that it was created to have those two things, and then someone stuck in some but not all of the things from 942, and then things escalated with bug 30373 using the wrong authorized value for itemtypes, and now since 22.05 we've created an AV with no values in both UNIMARC and MARC21 installs. If you alter the AV for 099$t to itemtypes, and then select a different one than you select for 942$c, you get biblioitems.itemtype "BK | CF" which I bet blows up pretty spectacularly.
Comment 28 Phil Ringnalda 2024-11-08 20:36:00 UTC
Ah, 099 versus 942 is also bug 38336
Comment 29 Mathieu Saby 2024-11-30 16:55:37 UTC
Nice summary Phil!

But for the issue with controlfields, I suppose that MARC21 is affected in the same way, as 000 and 003 are mandatory in default MARC21 framework, isnt it?

Concerning the problem that is the subject of this ticket, it seems to me that the only reasonable action in the short to medium term is to forget the patch provided by Jonathan, and instead to edit the default UNIMARC framework in order to keep the "mandatory" flag on subfields only if the field itself is mandatory. But this course of action won't fit with the title of this bug. Should we close this one and create a new one with a better title ?
Comment 30 Phil Ringnalda 2024-12-06 01:18:52 UTC
No need for a new bug, summaries can be edited. And the current summary is actually still the one that Thibault used when he filed the bug going on two years ago, saying "Most practical solution would probably be to remove the mandatory flag to said subfields" so we're right back around to comment 0.
Comment 31 Mathieu Saby 2024-12-07 18:14:22 UTC
I am providing a patch removing mandatory subfields, except for 100$a and 200$a

I also remove mandatory flag for 801 field. It is only mandatory in data exchange context. I don't have access to the original version of the ifla manual, but here is the french version : "Mandatory when exchanging bibliographic data. In many cases, this field will be generated automatically during the exchange."

I am keeping 099 and 942 as they are. I believe they deserve their own bug, as they are not defined by UNIMARC standard, and their use need to be clarified in Koha.
Comment 32 Mathieu Saby 2024-12-07 18:52:09 UTC
Created attachment 175278 [details] [review]
Bug 32722: Make several subfields and a field non mandatory in default UNIMARC biblio framework

801 should not be mandatory except in data exchange context (IFLA Manual)
In UNIMARC several subfields are mandatory only if the field is used.
This possibility is not available in MAR21, and Koha is currently not
able to manage this information in a useful way : it blocks the
validation of a record if the mandatory subfield is void, regardless
of the mandatory status of the field.
In consequence those subfields must be declared non mandatory in default framework.

Test plan:

1/ Open a UNIMARC Koha without patch. Try to create a record with
minimal information : 200$a and 100$a. A lot a alerts are displayed,
preventing Koha to add the record
2/ Apply the patch
3/ Try to create a record with minimal information : 200$a and a 100$a.
There should only be 2 alerts, regarding 099$t and 942$c field.
Comment 33 Victor Grousset/tuxayo 2024-12-10 07:40:21 UTC
Created attachment 175340 [details] [review]
Bug 32722: Make several subfields and a field non mandatory in default UNIMARC biblio framework

801 should not be mandatory except in data exchange context (IFLA Manual)
In UNIMARC several subfields are mandatory only if the field is used.
This possibility is not available in MAR21, and Koha is currently not
able to manage this information in a useful way : it blocks the
validation of a record if the mandatory subfield is void, regardless
of the mandatory status of the field.
In consequence those subfields must be declared non mandatory in default framework.

Test plan:

1/ Open a UNIMARC Koha without patch. Try to create a record with
minimal information : 200$a and 100$a. A lot a alerts are displayed,
preventing Koha to add the record
2/ Apply the patch
3/ Run reset_all or restart KTD (to remake a fresh koha from scratch)
4/ Try to create a record with minimal information : 200$a and a 100$a.
There should only be 2 alerts, regarding 099$t and 942$c field.

Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Comment 34 Victor Grousset/tuxayo 2024-12-10 07:40:56 UTC
Thanks a lot Phil for the analysis, now it's very very clear to me that the full issue is way beyond the pieces here and there that I understand about cataloging 😵‍💫😵‍💫😵‍💫😵‍💫😵‍💫😵‍💫😵‍💫😵‍💫😵‍💫😵‍💫😵‍💫


(In reply to Phil Ringnalda from comment #22)
> (In reply to Victor Grousset/tuxayo from comment #21)
> > It came in chat I think but I don't remember if there would be a problem:
> > What about changing the MARC21 frameworks in existing installs and the
> > installer to set to mandatory the fields containing a mandatory subfield?
> 
> That's comment 7: you cannot tell the difference between an existing install
> which has a mandatory subfield in a non-mandatory field because it works
> fine to make the field mandatory in the basic editor which is all they use,
> and an existing install which has it that way because it works fine to make
> the subfield mandatory only if the field has any other subfield used in the
> advanced editor, which is all they use.

Oh no, gone the hope of keeping the patch to finally have both editors behaving consistently. T_T

I wonder how that didn't cause problems immediately to introduce the advanced editor with a different behavior than the basic one. (Perfect behavior actually IIUC. If "only" at the same time migrating the frameworks so MARC21 and UNIMARC have both their conformant handling of mandatory stuff) And having incompatibilities with the default frameworks and the preferred handling of mandatory subfields of the MARC flavors. Frameworks need to be in line with the editor to be fully in line with the MARC flavor. But it's not possible to be in line with both editors at the same time. 😵‍💫
So now the advanced editor has been used long enough and frameworks have been tuned to it so it's not possible to easily fix the issue. aaaaah >_<

Given this:

> Unlike the basic editor, the advanced editor already does exactly
> what you want: if 120$a is mandatory
> but 120 is not mandatory, then if you have no 120 in the
> advanced editor you are free to save,
> but if you have 120$b you are required to have $a.

And given this:
(In reply to Thibault Keromnès from comment #5)
> I've talked about it with Katerin & Aude, and
> it seems it would be a problem for Marc21 users. 
> They consider a subfield to be mandatory whether the field itself is
> mandatory or not, so with this patch we would risk having records created
> without key informations.

We really have the default framework buggy with the advanced editor 😱.
Because the same problematic thing that would happen with the 1st patch to the basic editor is happening since a long time with the advanced editor.


----------------------


(In reply to Phil Ringnalda from comment #23)
> so we wouldn't have this "OMG, they are marked as
> mandatory in the framework, we must support that, we can't ever revert
> anything that has landed!" problem.


Is that really what happened here? I get that there can be that impression from part of the situation[1]. But is the cause of the blocking here so overwhelmingly that? (without other significant factors) And the blocking reasons are so strong that it can be described with pretty much the strongest wording possible. Really?

What about trying to get the basic editor and the advanced editor in line in how they handle mandatory fields? Maybe I totally misunderstood that there is this issue and that Jonathan's patch would get us most of the way there.
Of course after some time, if there is no one that has the skills and availability to expand on that patch. (To make it switch behavior between UNIMARC and MARC21.) Then the fallback is to revert the changes in the default UNIMARC framework and resign about the 1st patch.

[1] I think I feel it for different reasons. About how we are stuck with frameworks that can't work well with both editors at the same time and work well with standard usage of MARC21 and UNIMARC. Because stuff has landed and instances have modified their framework and it's not possible to fix without giving them trouble when they upgrade.
Wait, thinking about it again, I don't get how until now for MARC21 and until bug 30373 for UNIMARC, the status quo wasn't already trouble. Which has to be weighed against making an framework migration that will cause some manual work for a not that big share of instances. For which a big part of these customized their framework so there is reasonable hope most will not have much trouble doing adjustments again.
Does that makes sense? (for a follow-up ticket) (keeping in mind the preface of this comment, there are big chances it doesn't make sense ^^")
Comment 35 Victor Grousset/tuxayo 2024-12-10 07:45:00 UTC
It works! :)

Testing notes, when trying to save the result, I get

Can't call method "metadata" on an undefined value at /kohadevbox/koha/cataloguing/additem.pl line 195

  194: our $tagslib = &GetMarcStructure( 1, $frameworkcode );
  195: my $record = $biblio->metadata->record;
  196: 
  197: output_and_exit_if_error(


Given the patch changes, that doesn't look related. That will be something else to look after this >_<

(In reply to Victor Grousset/tuxayo from comment #9)
> More meta issue: it[the current issue] makes a pain to test UNIMARC patches.

And to test if UNIMARC cataloguing still works at all it seems 🫠
Comment 36 Mathieu Saby 2024-12-14 11:23:14 UTC
For the correction of 099 and 942 fields, I think it should be done there : https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=38336
Comment 37 Martin Renvoize (ashimema) 2024-12-17 07:06:42 UTC
Created attachment 175589 [details] [review]
Bug 32722: Make several subfields and a field non mandatory in default UNIMARC biblio framework

801 should not be mandatory except in data exchange context (IFLA Manual)
In UNIMARC several subfields are mandatory only if the field is used.
This possibility is not available in MAR21, and Koha is currently not
able to manage this information in a useful way : it blocks the
validation of a record if the mandatory subfield is void, regardless
of the mandatory status of the field.
In consequence those subfields must be declared non mandatory in default framework.

Test plan:

1/ Open a UNIMARC Koha without patch. Try to create a record with
minimal information : 200$a and 100$a. A lot a alerts are displayed,
preventing Koha to add the record
2/ Apply the patch
3/ Run reset_all or restart KTD (to remake a fresh koha from scratch)
4/ Try to create a record with minimal information : 200$a and a 100$a.
There should only be 2 alerts, regarding 099$t and 942$c field.

Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Comment 38 Martin Renvoize (ashimema) 2024-12-17 07:07:11 UTC
A pragmatic solution, thanks.

Passing QA
Comment 39 Katrin Fischer 2024-12-20 16:53:42 UTC
I feel like the change turned out a bit different to what the bug title suggests. Can we get an update to communicate the change a little better?
Comment 40 Mathieu Saby 2024-12-20 17:08:02 UTC
What about using the patch description as bug title ?
Comment 41 Katrin Fischer 2024-12-20 17:37:51 UTC
What about...: UNIMARC: Remove mandatory flag from some subfields and field in default bibliographic framework
Comment 42 Victor Grousset/tuxayo 2024-12-22 06:22:38 UTC
Sounds good :)
Comment 43 Katrin Fischer 2024-12-23 08:37:52 UTC
Pushed for 25.05!

Well done everyone, thank you!
Comment 44 David Nind 2025-01-03 23:02:50 UTC
I've had a go at a release note. (I hope I got the example right.)

Feel free to edit it, if I haven't correctly understood the change!

(There are a lot of nuances here, but I tried not into too many details.)
Comment 45 Paul Derscheid 2025-02-03 19:08:51 UTC
Nice work everyone!

Pushed to 24.11.x for 24.11.01
Comment 46 Alex Buckley 2025-03-14 01:28:32 UTC
Nice work everyone!

Pushed to 24.05.x for 24.05.08
Comment 47 Fridolin Somers 2025-03-20 14:30:17 UTC
I prefer not to backport to "very-old-stable" 23.11.x to avoid behavior change.
Comment 48 Caroline Cyr La Rose 2025-04-18 18:57:13 UTC
Individual fields are not in the manual, so there is nothing to edit in the manual.