When merging two authorities and updating the linked biblios MARC::Field->new() is called with the tagnumber, indicators, and the new subfield $9 This causes this subfield to be the first in the field as a whole and is inconsistent with usualy ordering
Will have a look.
(In reply to Nick Clemens from comment #0) > When merging two authorities and updating the linked biblios > MARC::Field->new() is called with the tagnumber, indicators, and the new > subfield $9 > > This causes this subfield to be the first in the field as a whole and is > inconsistent with usualy ordering The old merge routine did actually do the same. Also note that we normally have the $9 before the a..z subfields in the editor. This does not mean that we should do it differently of course.
I would add to this: if a subfield precedes the controlled subfields (like $i in 7XX, which comes before $a -- rare but will become more and more usual) after merging will be moved to the end.
Created attachment 69345 [details] [review] Bug 18458 - Merging authority record incorrectly orders subfields While doing a merge, if a subfield(s) precedes the controlled subfields (like $i before $a in 7XX, which comes before $a -- rare but will become more and more usual) after merging will be moved to the end. This is not right. The patch (with AuthorityMergeMode == loose) make merge consider the subfields order: all subfields which originally were found before the first controlled subfield (e.g. $i before $a in 7XX / MARC 21) will remain in the front, the rest of not controlled subfields that should remain in the field will come after the subfields copied from authority rec. As a bonus, $9 will be placed at the end. To test: 0) Have AuthorityMergeMode == loose; 1) Have some field in bibliorecord, controlled by an authority, with extra subfield(s) (i.e. not present in authority rec.) placed at the beginning of the field; 2) Open (not necessarily edit) and save the connected authority; 3) See that the extra subfieds were moved to the end of the field (and $9 is in the front); 4) Apply the patch; 5) Reorder subfields in biblio field; 6) Open (not necessarily edit) and save the connected authority; 7) See that the order has been conserved, additionally $9 the last subfield in the field.
Janusz, Thx again for this patch. I am not so sure though if your code really makes sub merge easier (it is hard enough already). Couldnt we do the following in order to keep the order intact: Copy all fields in original order including $9. Overwrite the authority controlled fields. If $overwrite, delete a subfield that is not controlled (only skip $9). [A hypothetical question: what about repeated subfields?] Update $9 finally. Another thing: - append_fields_ordered( $marcrecord, $field_to ); + $marcrecord->insert_fields_ordered($field_to); This is a regression. Look at the comments in append_fields_ordered. You change the order of fields here. Finally, we need some tests to support your changes. Have a look at t/db_dependent/Authority/Merge.t :)
(In reply to Marcel de Rooy from comment #5) > Janusz, > Thx again for this patch. > I am not so sure though if your code really makes sub merge easier (it is > hard enough already). > Couldnt we do the following in order to keep the order intact: > > Copy all fields in original order including $9. > Overwrite the authority controlled fields. If $overwrite, delete a subfield > that is not controlled (only skip $9). [A hypothetical question: what about > repeated subfields?] > Update $9 finally. > > Another thing: > - append_fields_ordered( $marcrecord, $field_to ); > + $marcrecord->insert_fields_ordered($field_to); > This is a regression. Look at the comments in append_fields_ordered. You > change the order of fields here. > > Finally, we need some tests to support your changes. Have a look at > t/db_dependent/Authority/Merge.t :) Hi Marcel, Thank you for your comments. From the bottom up: You are right--but I have never done it (the t/ story) :( append_fields_ordered -- you are right, I mixed it with append_fields -- and this is what I wouldn't like here, hence the correction that turns out to be regression. Hmmm, and what does it mean indeed "controlled" at all? Because even with the current approach we do not "clear" obviously controlled subfields that had been introduced by mistake by a cataloger. (E.g. if there is $a-$d in personal name authority record, but a cataloger by mistake put additional $c by hand, $c will not disappear after a "refreshing" merge; it disappears only if it was $a-$c first in authority and then becomes $a-$d). But anyway, it is definitely better now than it was before your corrections. But--wouldn't we need something like: C4/Heading/MARC21.pm (which is used for linking) to clearly state what is controlled?--I now, it is hardcoded, but MARC itself is hardcoded too, i.e. title is in 245 $a (resp. 200 $a) etc. Again, I feel like my proposal is more error-proof. I mean: controlled subfields form always a coherent block (do they?). So, if a cataloger puts, by mistake, an uncontrolled subfield between controlled subfields (which is not very rare in standard Koha editor)--it will go to the end after my merge (which was most probably his/hers intention). Only if he/she puts an uncontrolled introductory subfield at the beginning, it will remain there. And, if $overwrite, it is no sense at all to copy the original field--it seems to me it even more complicated than my approach, that I like ;) And even worse: with your proposal there is no simple way to cope with potentially repeated subdields (e.g. $x) that would be subject to changed. And as to $9--yes, we like to have it at the very end (since it it totally non standard and since it blemish the Labeled MARC view! :) So, to sum up: I would willingly return to append_fields_ordered, but if I could ask for suggestions on how to prepare t/, would be great. I would appreciate your comments.
Working here again now
Current approach was: $9 Controlled fields Old extra fields The controlled fields form a block; $9 should be moved to the end. Your approach is: Extra-prefix Controlled block Extra-postfix $9 With our current Koha records $9 will be the first one as long as it has not been saved with this patch. So we need to avoid $9 in this loop (amended). Note that since $i from your 7XX example has been moved backwards, it will not automatically move to the front anymore. But new 7XXs with $i $a will be okay. Adding a subtest in Merge.t
Created attachment 69637 [details] [review] Bug 18458: (Follow-up) Add a subtest in Merge.t This test illustrates the problem we have if you run it without the second patch. And it serves to demonstrate that we resolved the situation if you run it after the second patch. Test plan: [1] Without the second patch: The last subtest should fail. Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Created attachment 69638 [details] [review] Bug 18458 - Merging authority record incorrectly orders subfields While doing a merge, if a subfield(s) precedes the controlled subfields (like $i before $a in 7XX, which comes before $a -- rare but will become more and more usual) after merging will be moved to the end. This is not right. The patch (with AuthorityMergeMode == loose) make merge consider the subfields order: all subfields which originally were found before the first controlled subfield (e.g. $i before $a in 7XX / MARC 21) will remain in the front, the rest of not controlled subfields that should remain in the field will come after the subfields copied from authority rec. As a bonus, $9 will be placed at the end. To test: 0) Have AuthorityMergeMode == loose; 1) Have some field in bibliorecord, controlled by an authority, with extra subfield(s) (i.e. not present in authority rec.) placed at the beginning of the field; 2) Open (not necessarily edit) and save the connected authority; 3) See that the extra subfieds were moved to the end of the field (and $9 is in the front); 4) Apply the patch; 5) Reorder subfields in biblio field; 6) Open (not necessarily edit) and save the connected authority; 7) See that the order has been conserved, additionally $9 the last subfield in the field. Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl> Amended: Moved field creation to its original place. Changed $9 handling. Simplified the following add_subfields for loop. Edited comments. Restored the append_fields_ordered call (see comment6). With this patch, the Merge.t test now passes.
Created attachment 70104 [details] [review] Bug 18458: Add a subtest in Merge.t This test illustrates the problem we have if you run it without the second patch. And it serves to demonstrate that we resolved the situation if you run it after the second patch. Test plan: [1] Without the second patch: The last subtest should fail. Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl> Signed-off-by: Josef Moravec <josef.moravec@gmail.com>
Created attachment 70105 [details] [review] Bug 18458: Merging authority record incorrectly orders subfields While doing a merge, if a subfield(s) precedes the controlled subfields (like $i before $a in 7XX, which comes before $a -- rare but will become more and more usual) after merging will be moved to the end. This is not right. The patch (with AuthorityMergeMode == loose) make merge consider the subfields order: all subfields which originally were found before the first controlled subfield (e.g. $i before $a in 7XX / MARC 21) will remain in the front, the rest of not controlled subfields that should remain in the field will come after the subfields copied from authority rec. As a bonus, $9 will be placed at the end. To test: 0) Have AuthorityMergeMode == loose; 1) Have some field in bibliorecord, controlled by an authority, with extra subfield(s) (i.e. not present in authority rec.) placed at the beginning of the field; 2) Open (not necessarily edit) and save the connected authority; 3) See that the extra subfieds were moved to the end of the field (and $9 is in the front); 4) Apply the patch; 5) Reorder subfields in biblio field; 6) Open (not necessarily edit) and save the connected authority; 7) See that the order has been conserved, additionally $9 the last subfield in the field. Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl> Amended: Moved field creation to its original place. Changed $9 handling. Simplified the following add_subfields for loop. Edited comments. Restored the append_fields_ordered call (see comment6). With this patch, the Merge.t test now passes. Signed-off-by: Josef Moravec <josef.moravec@gmail.com>
Looks good
Pushed to master for 18.05, thanks to everybody involved!
Awesome work all, pushed to stable for 17.11.02
Pushed to 17.05.x for v17.05.08