Bug 10811 - Add a 999$c record matching rule to the MARC21 defaults
Summary: Add a 999$c record matching rule to the MARC21 defaults
Status: CLOSED FIXED
Alias: None
Product: Koha
Classification: Unclassified
Component: MARC Bibliographic record staging/import (show other bugs)
Version: Main
Hardware: All All
: P5 - low enhancement (vote)
Assignee: David Cook
QA Contact: Testopia
URL:
Keywords:
: 8748 (view as bug list)
Depends on:
Blocks:
 
Reported: 2013-09-02 03:21 UTC by David Cook
Modified: 2019-04-28 12:01 UTC (History)
6 users (show)

See Also:
Change sponsored?: ---
Patch complexity: Small patch
Documentation contact:
Documentation submission:
Text to go in the release notes:
Version(s) released in:


Attachments
Bug 10811 - Add a 999$c record matching rule to the MARC21 defaults (3.52 KB, patch)
2013-09-02 03:47 UTC, David Cook
Details | Diff | Splinter Review
Bug 10811 - Add a 999$c record matching rule to the MARC21 defaults (3.54 KB, patch)
2013-09-18 23:51 UTC, David Cook
Details | Diff | Splinter Review
Bug 10811 - Add a 999$c record matching rule to the MARC21 defaults (3.78 KB, patch)
2013-09-29 07:00 UTC, Chris Cormack
Details | Diff | Splinter Review
Bug 10811 - Add a 999$c record matching rule to the MARC21 defaults (3.52 KB, patch)
2013-12-23 05:22 UTC, David Cook
Details | Diff | Splinter Review
Bug 10811 - Add a 999$c record matching rule to the MARC21 defaults (4.41 KB, patch)
2014-01-15 19:20 UTC, Mark Tompsett
Details | Diff | Splinter Review
[SIGNED OFF] Bug 10811 - Add a 999$c record matching rule to the MARC21 defaults (4.44 KB, patch)
2014-01-16 02:04 UTC, Mark Tompsett
Details | Diff | Splinter Review
Bug 10811 - Add a 999$c record matching rule to the MARC21 defaults (4.46 KB, patch)
2014-01-17 18:49 UTC, Kyle M Hall
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description David Cook 2013-09-02 03:21:03 UTC
Currently, the only default matching rules for MARC21 are ISBN and ISSN.

However, I'm sure that many people would profit from having a 999$c "Koha biblionumber" record matching rule as well. 

1) If you export your bib records to do some global/batch changes with another program (like MarcEdit), you can use this rule to replace the existing bib records with your modified records.

2) With my upcoming OAI-PMH harvesting feature, if you add a OAI-PMH repository with this rule in the configuration, you'll be able to automagically match updated records with the original records that you imported via OAI-PMH (the magic is that I check the incoming unique identifier against records in Koha, and if there is already a bib record in Koha matching that identifier, I write the biblionumber for that record back into the incoming record so that they match. It sounds convoluted but it works well).

3) People might have other ideas about this as well.
Comment 1 David Cook 2013-09-02 03:47:31 UTC Comment hidden (obsolete)
Comment 2 I'm just a bot 2013-09-18 09:57:18 UTC
Applying: Bug 10811 - Add a 999$c record matching rule to the MARC21 defaults
Using index info to reconstruct a base tree...
M	installer/data/mysql/updatedatabase.pl
Falling back to patching base and 3-way merge...
Auto-merging installer/data/mysql/updatedatabase.pl
CONFLICT (content): Merge conflict in installer/data/mysql/updatedatabase.pl
Patch failed at 0001 Bug 10811 - Add a 999$c record matching rule to the MARC21 defaults
The copy of the patch that failed is found in:
   /home/christopher/git/koha/.git/rebase-apply/patch
When you have resolved this problem, run "git am --continue".
If you prefer to skip this patch, run "git am --skip" instead.
To restore the original branch and stop patching, run "git am --abort".
Comment 3 David Cook 2013-09-18 23:51:37 UTC Comment hidden (obsolete)
Comment 4 I'm just a bot 2013-09-29 06:11:03 UTC
Applying: Bug 10811 - Add a 999$c record matching rule to the MARC21 defaults
Using index info to reconstruct a base tree...
M	installer/data/mysql/updatedatabase.pl
Falling back to patching base and 3-way merge...
Auto-merging installer/data/mysql/updatedatabase.pl
CONFLICT (content): Merge conflict in installer/data/mysql/updatedatabase.pl
Patch failed at 0001 Bug 10811 - Add a 999$c record matching rule to the MARC21 defaults
The copy of the patch that failed is found in:
   /home/christopher/git/koha/.git/rebase-apply/patch
When you have resolved this problem, run "git am --continue".
If you prefer to skip this patch, run "git am --skip" instead.
To restore the original branch and stop patching, run "git am --abort".
Comment 5 Chris Cormack 2013-09-29 07:00:26 UTC Comment hidden (obsolete)
Comment 6 I'm just a bot 2013-10-27 06:31:45 UTC
Patch applied cleanly, go forth and signoff
Comment 7 David Cook 2013-12-23 05:22:05 UTC Comment hidden (obsolete)
Comment 8 Mark Tompsett 2013-12-23 05:35:05 UTC
Comment on attachment 23772 [details] [review]
Bug 10811 - Add a 999$c record matching rule to the MARC21 defaults

Review of attachment 23772 [details] [review]:
-----------------------------------------------------------------

::: installer/data/mysql/updatedatabase.pl
@@ +7857,5 @@
> +if ( CheckVersion($DBversion) ) {
> +    if ( C4::Context->preference("marcflavour") eq 'MARC21' ) {
> +        $dbh->do("
> +    INSERT INTO marc_matchers (code, description, record_type, threshold)
> +        VALUES ('KohaBiblio', '999$c', 'biblio', 1000);

The $c is taken as a variable, causing this to fail the QA tool.
Comment 9 Mark Tompsett 2013-12-23 05:43:22 UTC
REVISED TEST PLAN
-----------------
1) Apply the patch
2) Run the QA test tool
3) Drop the koha database
4) Create the koha database
5) Go to the staff client page, and do a "fresh" install making sure to select lots of optional marc 21 matching rules so the installer/data/mysql/en/marcflavour/marc21/optional/marc21_default_matching_rules.sql gets triggered.
6) Check the mysql tables (marc_matchers, matchpoints, matcher_matchpoints, matchpoint_components, matchpoint_component_norms) to make sure the values were inserted.
7) restore an older database
8) Go to the staff client page to trigger an update.
9) Check the mysql tables to make sure the values were inserted.

I think there may be some way of checking one could do from within Koha to test the validity of the addition, but I don't know it off the top of my head.

Marked as Failed QA, because I ran up to the QA test tool step.
Comment 10 Mark Tompsett 2014-01-15 18:36:39 UTC
You can't do multiple statements in the $dbh->do();
http://search.cpan.org/~timb/DBI-1.630/DBI.pm#do
Comment 11 Mark Tompsett 2014-01-15 18:45:04 UTC
Comment on attachment 21587 [details] [review]
Bug 10811 - Add a 999$c record matching rule to the MARC21 defaults

This is closer to correct.
Comment 12 Mark Tompsett 2014-01-15 18:45:25 UTC
Comment on attachment 23772 [details] [review]
Bug 10811 - Add a 999$c record matching rule to the MARC21 defaults

This breaks the do() in addition to the $c issue.
Comment 13 Mark Tompsett 2014-01-15 19:20:37 UTC Comment hidden (obsolete)
Comment 14 David Cook 2014-01-15 22:34:47 UTC
(In reply to M. Tompsett from comment #13)
> Created attachment 24421 [details] [review] [review]
> Bug 10811 - Add a 999$c record matching rule to the MARC21 defaults
> 
> This patch adds a default record matching rule for biblionumber
> using the 999$c and the Local-Number index.
> 
> This is part of the optional marc21 SQL, as well as in the
> updatabase.pl script.
> 
> TEST PLAN
> ---------
>  1) Back up your DB, because you probably want it back.
>  2) Apply the patch
>  3) Run the QA test tool.
>  4) Run updatedatabase.pl
>  5) Export some MARC bibliographic records from Koha
>  6) Import those same MARC bibliographic records using the
>      "KohaBiblio" record matching rule.
>  7) The incoming records should match perfectly
>  8) Check the mysql tables (marc_matchers, matchpoints,
>       matcher_matchpoints, matchpoint_components,
>       matchpoint_component_norms) to make sure the values were
>      inserted as expected.
>  9) drop the koha database
> 10) create the koha database
> 11) Go to the staff client page, and do a "fresh" install making
>      sure to select lots of optional marc 21 matching rules so
>      the .../marc21/optional/marc21_default_matching_rules.sql
>      gets triggered.
> 12) Check the mysql tables (marc_matchers, matchpoints,
>       matcher_matchpoints, matchpoint_components,
>       matchpoint_component_norms) to make sure the values were
>      inserted as expected.
> 13) Restore your DB backup. :)
> 
> I wasn't able to do steps 5-7 properly, because of the lack
> of 999$c MARC records in my data as far as I know.

Thanks for the update. I've been putting this one off as a trivial enhancement for a little while.

You should be able to do steps 5-7. Your Koha biblionumber is stored as your 999$c. If you don't have one in your Koha records, something is seriously wrong.

That said, it won't show up for you in Koha, as it's a hidden field everywhere. You'd have to verify its existence by checking the marcxml in the biblioitems table.
Comment 15 Mark Tompsett 2014-01-16 01:47:22 UTC
There are multiple ways to export. The first way is probably the best.
    Home -> Tools -> Export data
AND
    Home -> Search the catalogue -> find a specific biblio -> Save -> MARC (Unicode / UTF-8)

(Only the UTF-8 and MARCXML ones will save 999$c data)

I also ran into a nasty save problem for a title:
  Identification of an ‘Unidentified’ Holle List (The): The Last Word (list)
I suspect the ticks or brackets caused the issue, but that is a different bug. :)
Comment 16 Mark Tompsett 2014-01-16 02:04:03 UTC Comment hidden (obsolete)
Comment 17 Kyle M Hall 2014-01-17 18:49:30 UTC
Created attachment 24528 [details] [review]
Bug 10811 - Add a 999$c record matching rule to the MARC21 defaults

This patch adds a default record matching rule for biblionumber
using the 999$c and the Local-Number index.

This is part of the optional marc21 SQL, as well as in the
updatabase.pl script.

TEST PLAN
---------
 1) Back up your DB, because you probably want it back.
 2) Apply the patch
 3) Run the QA test tool.
 4) Run updatedatabase.pl
 5) Export some MARC bibliographic records from Koha
 6) Import those same MARC bibliographic records using the
     "KohaBiblio" record matching rule.
 7) The incoming records should match perfectly
 8) Check the mysql tables (marc_matchers, matchpoints,
      matcher_matchpoints, matchpoint_components,
      matchpoint_component_norms) to make sure the values were
     inserted as expected.
 9) drop the koha database
10) create the koha database
11) Go to the staff client page, and do a "fresh" install making
     sure to select lots of optional marc 21 matching rules so
     the .../marc21/optional/marc21_default_matching_rules.sql
     gets triggered.
12) Check the mysql tables (marc_matchers, matchpoints,
      matcher_matchpoints, matchpoint_components,
      matchpoint_component_norms) to make sure the values were
     inserted as expected.
13) Restore your DB backup. :)

I corrected any problems I found with the patch, particularly the
$c issue.

Signed-off-by: Mark Tompsett <mtompset@hotmail.com>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Comment 18 Kyle M Hall 2014-01-17 18:50:08 UTC
I would question whether this match point should be added for upgrades, as the installer may have opted not to install the default match points. I believe that is a question for the release manager though.
Comment 19 Galen Charlton 2014-01-23 03:52:43 UTC
(In reply to Kyle M Hall from comment #18)
> I would question whether this match point should be added for upgrades, as
> the installer may have opted not to install the default match points. I
> believe that is a question for the release manager though.

Either the database update should be skipped, or the MARC matching rules SQL should be mandatory during installation.
Comment 20 Galen Charlton 2014-01-31 19:24:49 UTC
Pushed to master.  Note that I dropped the database update and added a follow-up to ensure that matching would work regardless of the QueryParser mode.

By the way, I wouldn't be opposed to making the default matching rules mandatory during installation.

Thanks David!
Comment 21 Katrin Fischer 2019-04-28 12:01:23 UTC
*** Bug 8748 has been marked as a duplicate of this bug. ***