Bug 18458

Summary: Merging authority record incorrectly orders subfields
Product: Koha Reporter: Nick Clemens <nick>
Component: MARC Authority data supportAssignee: Janusz Kaczmarek <januszop>
Status: CLOSED FIXED QA Contact: Josef Moravec <josef.moravec>
Severity: normal    
Priority: P5 - low CC: fridolin.somers, januszop, josef.moravec, m.de.rooy, marjorie.barry-vila, patrick.robitaille
Version: Main   
Hardware: All   
OS: All   
Change sponsored?: --- Patch complexity: Small patch
Documentation contact: Documentation submission:
Text to go in the release notes:
Version(s) released in:
Attachments: Bug 18458 - Merging authority record incorrectly orders subfields
Bug 18458: (Follow-up) Add a subtest in Merge.t
Bug 18458 - Merging authority record incorrectly orders subfields
Bug 18458: Add a subtest in Merge.t
Bug 18458: Merging authority record incorrectly orders subfields

Description Nick Clemens 2017-04-19 14:27:13 UTC
When merging two authorities and updating the linked biblios MARC::Field->new() is called  with the tagnumber, indicators, and the new subfield $9

This causes this subfield to be the first in the field as a whole and is inconsistent with usualy ordering
Comment 1 Marcel de Rooy 2017-04-19 14:32:24 UTC
Will have a look.
Comment 2 Marcel de Rooy 2017-05-01 13:24:44 UTC
(In reply to Nick Clemens from comment #0)
> When merging two authorities and updating the linked biblios
> MARC::Field->new() is called  with the tagnumber, indicators, and the new
> subfield $9
> 
> This causes this subfield to be the first in the field as a whole and is
> inconsistent with usualy ordering

The old merge routine did actually do the same.
Also note that we normally have the $9 before the a..z subfields in the editor.

This does not mean that we should do it differently of course.
Comment 3 Janusz Kaczmarek 2017-11-24 21:17:04 UTC
I would add to this: if a subfield precedes the controlled subfields (like $i in 7XX, which comes before $a -- rare but will become more and more usual) after merging will be moved to the end.
Comment 4 Janusz Kaczmarek 2017-11-25 01:51:23 UTC
Created attachment 69345 [details] [review]
Bug 18458 - Merging authority record incorrectly orders  subfields

While doing a merge, if a subfield(s) precedes the controlled subfields
(like $i before $a in 7XX, which comes before $a -- rare but will
become more and more usual) after merging will be moved to the end.
This is not right.

The patch (with AuthorityMergeMode == loose) make merge consider the
subfields order: all subfields which originally were found before
the first controlled subfield (e.g. $i before $a in 7XX / MARC 21)
will remain in the front, the rest of not controlled subfields that
should remain in the field will come after the subfields copied
from authority rec.

As a bonus, $9 will be placed at the end.

To test:
0) Have AuthorityMergeMode == loose;
1) Have some field in bibliorecord, controlled by an authority, with
extra subfield(s) (i.e. not present in authority rec.) placed at the
beginning of the field;
2) Open (not necessarily edit) and save the connected authority;
3) See that the extra subfieds were moved to the end of the field
   (and $9 is in the front);
4) Apply the patch;
5) Reorder subfields in biblio field;
6) Open (not necessarily edit) and save the connected authority;
7) See that the order has been conserved, additionally $9 the last
   subfield in the field.
Comment 5 Marcel de Rooy 2017-11-27 10:21:56 UTC
Janusz,
Thx again for this patch.
I am not so sure though if your code really makes sub merge easier (it is hard enough already).
Couldnt we do the following in order to keep the order intact:

Copy all fields in original order including $9.
Overwrite the authority controlled fields. If $overwrite, delete a subfield that is not controlled (only skip $9). [A hypothetical question: what about repeated subfields?]
Update $9 finally.

Another thing:
-                    append_fields_ordered( $marcrecord, $field_to );
+                    $marcrecord->insert_fields_ordered($field_to);
This is a regression. Look at the comments in append_fields_ordered. You change the order of fields here.

Finally, we need some tests to support your changes. Have a look at t/db_dependent/Authority/Merge.t :)
Comment 6 Janusz Kaczmarek 2017-11-27 17:20:08 UTC
(In reply to Marcel de Rooy from comment #5)
> Janusz,
> Thx again for this patch.
> I am not so sure though if your code really makes sub merge easier (it is
> hard enough already).
> Couldnt we do the following in order to keep the order intact:
> 
> Copy all fields in original order including $9.
> Overwrite the authority controlled fields. If $overwrite, delete a subfield
> that is not controlled (only skip $9). [A hypothetical question: what about
> repeated subfields?]
> Update $9 finally.
> 
> Another thing:
> -                    append_fields_ordered( $marcrecord, $field_to );
> +                    $marcrecord->insert_fields_ordered($field_to);
> This is a regression. Look at the comments in append_fields_ordered. You
> change the order of fields here.
> 
> Finally, we need some tests to support your changes. Have a look at
> t/db_dependent/Authority/Merge.t :)

Hi Marcel,

Thank you for your comments.  From the bottom up:

You are right--but I have never done it (the t/ story) :(

append_fields_ordered -- you are right, I mixed it with append_fields -- and this is what I wouldn't like here, hence the correction that turns out to be regression. 

Hmmm, and what does it mean indeed "controlled" at all?  Because even with the current approach we do not "clear" obviously controlled subfields that had been introduced by mistake by a cataloger.  (E.g. if there is $a-$d in personal name authority record, but a cataloger by mistake put additional $c by hand, $c will not disappear after a "refreshing" merge; it disappears only if it was $a-$c first in authority and then becomes $a-$d).  But anyway, it is definitely better now than it was before your corrections.  But--wouldn't we need something like: C4/Heading/MARC21.pm (which is used for linking) to clearly state what is controlled?--I now, it is hardcoded, but MARC itself is hardcoded too, i.e. title is in 245 $a (resp. 200 $a) etc.

Again, I feel like my proposal is more error-proof.  I mean: controlled subfields form always a coherent block (do they?).  So, if a cataloger puts, by mistake, an uncontrolled subfield between controlled subfields (which is not very rare in standard Koha editor)--it will go to the end after my merge (which was most probably his/hers intention).  Only if he/she puts an uncontrolled introductory subfield at the beginning, it will remain there.

And, if $overwrite, it is no sense at all to copy the original field--it seems to me it even more complicated than my approach, that I like ;)  And even worse: with your proposal there is no simple way to cope with potentially repeated subdields (e.g. $x) that would be subject to changed. 

And as to $9--yes, we like to have it at the very end (since it it totally non standard and since it blemish the Labeled MARC view! :) 

So, to sum up: I would willingly return to append_fields_ordered, but if I could ask for suggestions on how to prepare t/, would be great. 

I would appreciate your comments.
Comment 7 Marcel de Rooy 2017-12-08 09:45:41 UTC
Working here again now
Comment 8 Marcel de Rooy 2017-12-08 13:05:16 UTC
Current approach was:
$9 Controlled fields Old extra fields
The controlled fields form a block; $9 should be moved to the end.
Your approach is:
Extra-prefix Controlled block Extra-postfix $9
With our current Koha records $9 will be the first one as long as it has not been saved with this patch. So we need to avoid $9 in this loop (amended).
Note that since $i from your 7XX example has been moved backwards, it will not automatically move to the front anymore. But new 7XXs with $i $a will be okay.
Adding a subtest in Merge.t
Comment 9 Marcel de Rooy 2017-12-08 13:05:44 UTC
Created attachment 69637 [details] [review]
Bug 18458: (Follow-up) Add a subtest in Merge.t

This test illustrates the problem we have if you run it without the
second patch. And it serves to demonstrate that we resolved the
situation if you run it after the second patch.

Test plan:
[1] Without the second patch: The last subtest should fail.

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Comment 10 Marcel de Rooy 2017-12-08 13:05:49 UTC
Created attachment 69638 [details] [review]
Bug 18458 - Merging authority record incorrectly orders subfields

While doing a merge, if a subfield(s) precedes the controlled subfields
(like $i before $a in 7XX, which comes before $a -- rare but will
become more and more usual) after merging will be moved to the end.
This is not right.

The patch (with AuthorityMergeMode == loose) make merge consider the
subfields order: all subfields which originally were found before
the first controlled subfield (e.g. $i before $a in 7XX / MARC 21)
will remain in the front, the rest of not controlled subfields that
should remain in the field will come after the subfields copied
from authority rec.

As a bonus, $9 will be placed at the end.

To test:
0) Have AuthorityMergeMode == loose;
1) Have some field in bibliorecord, controlled by an authority, with
extra subfield(s) (i.e. not present in authority rec.) placed at the
beginning of the field;
2) Open (not necessarily edit) and save the connected authority;
3) See that the extra subfieds were moved to the end of the field
   (and $9 is in the front);
4) Apply the patch;
5) Reorder subfields in biblio field;
6) Open (not necessarily edit) and save the connected authority;
7) See that the order has been conserved, additionally $9 the last
   subfield in the field.

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Amended:
Moved field creation to its original place. Changed $9 handling. Simplified the following add_subfields for loop. Edited comments.
Restored the append_fields_ordered call (see comment6).

With this patch, the Merge.t test now passes.
Comment 11 Josef Moravec 2017-12-22 09:06:26 UTC
Created attachment 70104 [details] [review]
Bug 18458: Add a subtest in Merge.t

This test illustrates the problem we have if you run it without the
second patch. And it serves to demonstrate that we resolved the
situation if you run it after the second patch.

Test plan:
[1] Without the second patch: The last subtest should fail.

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>

Signed-off-by: Josef Moravec <josef.moravec@gmail.com>
Comment 12 Josef Moravec 2017-12-22 09:06:29 UTC
Created attachment 70105 [details] [review]
Bug 18458: Merging authority record incorrectly orders subfields

While doing a merge, if a subfield(s) precedes the controlled subfields
(like $i before $a in 7XX, which comes before $a -- rare but will
become more and more usual) after merging will be moved to the end.
This is not right.

The patch (with AuthorityMergeMode == loose) make merge consider the
subfields order: all subfields which originally were found before
the first controlled subfield (e.g. $i before $a in 7XX / MARC 21)
will remain in the front, the rest of not controlled subfields that
should remain in the field will come after the subfields copied
from authority rec.

As a bonus, $9 will be placed at the end.

To test:
0) Have AuthorityMergeMode == loose;
1) Have some field in bibliorecord, controlled by an authority, with
extra subfield(s) (i.e. not present in authority rec.) placed at the
beginning of the field;
2) Open (not necessarily edit) and save the connected authority;
3) See that the extra subfieds were moved to the end of the field
   (and $9 is in the front);
4) Apply the patch;
5) Reorder subfields in biblio field;
6) Open (not necessarily edit) and save the connected authority;
7) See that the order has been conserved, additionally $9 the last
   subfield in the field.

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Amended:
Moved field creation to its original place. Changed $9 handling. Simplified the following add_subfields for loop. Edited comments.
Restored the append_fields_ordered call (see comment6).

With this patch, the Merge.t test now passes.

Signed-off-by: Josef Moravec <josef.moravec@gmail.com>
Comment 13 Josef Moravec 2017-12-22 09:08:21 UTC
Looks good
Comment 14 Jonathan Druart 2017-12-22 16:16:38 UTC
Pushed to master for 18.05, thanks to everybody involved!
Comment 15 Nick Clemens 2018-01-15 16:44:45 UTC
Awesome work all, pushed to stable for 17.11.02
Comment 16 Fridolin Somers 2018-01-19 12:37:16 UTC
Pushed to 17.05.x for v17.05.08