Bug 7284 - Authority matching algorithm improvements
Summary: Authority matching algorithm improvements
Status: CLOSED FIXED
Alias: None
Product: Koha
Classification: Unclassified
Component: MARC Authority data support (show other bugs)
Version: 3.6
Hardware: All All
: P3 major (vote)
Assignee: Jared Camins-Esakov
QA Contact: Ian Walls
URL: http://wiki.koha-community.org/wiki/B...
Keywords:
: 5803 (view as bug list)
Depends on: 7421
Blocks: 5454 7417 7418 7419 7475
  Show dependency treegraph
 
Reported: 2011-12-01 17:34 UTC by Jared Camins-Esakov
Modified: 2013-12-05 19:53 UTC (History)
9 users (show)

See Also:
Change sponsored?: Sponsored
Patch complexity: ---
Documentation contact:
Documentation submission:
Text to go in the release notes:
Version(s) released in:


Attachments
Bug 7284: Clean up authority code and add tests (6.61 KB, patch)
2012-01-07 22:32 UTC, Jared Camins-Esakov
Details | Diff | Splinter Review
Bug 7284: Add heading match option to auth plugin (54.68 KB, patch)
2012-01-07 22:33 UTC, Jared Camins-Esakov
Details | Diff | Splinter Review
Bug 7284: Improvements to authority linker script (89.13 KB, patch)
2012-01-07 22:33 UTC, Jared Camins-Esakov
Details | Diff | Splinter Review
Bug 7284: Use the new linking for cataloguing (21.82 KB, patch)
2012-01-07 22:33 UTC, Jared Camins-Esakov
Details | Diff | Splinter Review
Bug 7284: Add UNIMARC handler for C4::Heading (9.57 KB, patch)
2012-01-07 22:33 UTC, Jared Camins-Esakov
Details | Diff | Splinter Review
Bug 7284: Add UNIMARC handler for C4::Heading (modified) (10.33 KB, patch)
2012-01-09 11:56 UTC, Frédéric Demians
Details | Diff | Splinter Review
Bug 7284: Authority matching improvements (160.80 KB, patch)
2012-01-21 15:00 UTC, Jared Camins-Esakov
Details | Diff | Splinter Review
Bug 7284: Authority matching improvements (160.53 KB, patch)
2012-01-24 17:09 UTC, Jared Camins-Esakov
Details | Diff | Splinter Review
Bug 7284: Authority matching improvements (160.63 KB, patch)
2012-02-04 02:09 UTC, Jared Camins-Esakov
Details | Diff | Splinter Review
Bug 7284 follow up: Cataloging module relinking (8.13 KB, patch)
2012-02-10 00:29 UTC, Jared Camins-Esakov
Details | Diff | Splinter Review
Bug 7284: Authority matching improvements (173.17 KB, patch)
2012-02-16 13:49 UTC, Jared Camins-Esakov
Details | Diff | Splinter Review
Bug 7284: Authority matching improvements (173.31 KB, patch)
2012-02-17 14:54 UTC, Jared Camins-Esakov
Details | Diff | Splinter Review
Bug 7284: Authority matching improvements (173.48 KB, patch)
2012-02-21 16:08 UTC, Jared Camins-Esakov
Details | Diff | Splinter Review
Bug 7284: Authority matching improvements (173.50 KB, patch)
2012-02-23 14:03 UTC, Jared Camins-Esakov
Details | Diff | Splinter Review
signed off Bug 7284: Authority matching improvements (173.52 KB, patch)
2012-02-29 12:24 UTC, David Schuster
Details | Diff | Splinter Review
Bug 7284: Improve UNIMARC Zebra configuration (14.40 KB, patch)
2012-03-07 17:20 UTC, Jared Camins-Esakov
Details | Diff | Splinter Review
Bug 7284: Fix unit test broken by erroneous POD (654 bytes, patch)
2012-03-07 23:56 UTC, Jared Camins-Esakov
Details | Diff | Splinter Review
[3.6.x] Bug 7284: Authority matching improvements (177.20 KB, patch)
2012-03-10 14:54 UTC, Jared Camins-Esakov
Details | Diff | Splinter Review
bug 7284: fix package build problem, move BEGIN into new (1.04 KB, patch)
2012-04-04 11:39 UTC, MJ Ray (software.coop)
Details | Diff | Splinter Review
bug 7284: fix package build problem, move BEGIN into new (1.41 KB, patch)
2012-04-04 11:48 UTC, Jared Camins-Esakov
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description Jared Camins-Esakov 2011-12-01 17:34:41 UTC
At present, the automatic authority matching for MARC21 is of limited use because it fails on headings with more than one subfield, doesn't take into account subfield codes, and considers punctuation significant. An improved matching algorithm should be able to match the following headings to the correct authorities (these particular examples are from a local authority file):
=650  #4$aHistory
=650  #4$aHistory
=650  #4$aHistory$xBibliography (the technique of bibliography as applied to the study of history)
=650  #4$aHistory$vBibliography (bibliographies about history)
=650  #4$aHistory$vBibliography.
=650  #4$aHistory$zGreek Empire$vBibliography
=650  #4$aHistory$zGreek Empire$vBibliography.
=650  #0$aHistory.
=650  #7$aHistory.$2abc

Those headings should match the following authorities:
=150  #4$aHistory.
=150  #4$aHistory$xBibliography.
=150  #4$aHistory$vBibliography.
=150  #4$aHistory$zGreek Empire$vBibliography.
=150  #0$aHistory.
=150  #7$aHistory.$2abc

Libraries with examples of problematic headings from other authority files are respectfully requested to provide them in comments for the purpose of testing.

There are a number of additional changes needed to make the link_bibs_to_authorities.pl script and the situation where BiblioAddsAuthorities=allow work properly:
* The option to link headings to the first matching authority, even if there is more than one (and provide some sort of warning about that fact)
* Verbose mode on link_bibs_to_authorities.pl should offer more information.
* link_bibs_to_authorities.pl should be able to process only a portion of the catalog.
* Allow machine-created authority records to be used for indexing.

Potential future changes that would make these features even more useful:
* A web interface to link_bibs_to_authorities.pl
* An authority record de-duplicator
* An option to correct punctuation when authorizing headings (either via link_bibs_to_authorities.pl or in the cataloging module)
Comment 1 Chris Cormack 2011-12-07 20:37:28 UTC
Hmm, this sounds like a really useful enhancement.
Comment 2 Ian Walls 2011-12-07 21:45:33 UTC
I'd recommend an even more robust linker algorithm in the case of multiple headings... perhaps something to check completeness of the record (more fields filled in is 'better'), or how many other records link to it (more popular is 'better').  Taking the first result would be easier, but not necessarily always the best for the cataloger.  Perhaps making this syspref-controlled?
Comment 3 lculber 2011-12-09 15:08:01 UTC
Corporate headings are a big issue for us and this enhancement would be useful.

For testing purposes, examples from our catalog for authorities:
100 1# $aWelty, Eudora$d1909-2001$xCharacters$xPoor 
110 1# $aUnited States.$bWork Projects Administration.$bService Division.$bWar Services Program
110 1# $aConfederate States of America.$bArmy.$bTexas Brigade
150 #7 $aSpectators$zMississippi$zJackson$y1960-1970$2lctgm
150 ## $aAmerican literature$vBibliography$vCatalogs
150 ## $aChoctaw Indians$zMississippi$xAntiquities$vCatalogs

to bibliographic records:
600 10 $$aWelty, Eudora$d1909-2001$xCharacters$xPoor.  
610 1# $aUnited States.$bWork Projects Administration.$bService Division.$bWar Services Program.
610 10 $aConfederate States of America.$b.Army.$bTexas Brigade.
650 #7 $aSpectators$zMississippi$zJackson$y1960-1970$2lctgm.
650 #0 $aAmerican literature$vBibliography$vCatalogs. 
650 #0 $aChoctaw Indians$zMississippi$xAntiquities$xCatalogs.
Comment 4 Jared Camins-Esakov 2012-01-07 22:32:58 UTC Comment hidden (obsolete)
Comment 5 Jared Camins-Esakov 2012-01-07 22:33:08 UTC Comment hidden (obsolete)
Comment 6 Jared Camins-Esakov 2012-01-07 22:33:23 UTC Comment hidden (obsolete)
Comment 7 Jared Camins-Esakov 2012-01-07 22:33:37 UTC Comment hidden (obsolete)
Comment 8 Jared Camins-Esakov 2012-01-07 22:33:45 UTC Comment hidden (obsolete)
Comment 9 Jared Camins-Esakov 2012-01-07 22:36:57 UTC
The five patches that make up the improved authority linker are attached to this bug, and the repository for ongoing work can be found at https://github.com/jcamins/koha/commits/bug_7284_v2

Please test.
Comment 10 Frédéric Demians 2012-01-09 11:56:18 UTC Comment hidden (obsolete)
Comment 11 Frédéric Demians 2012-01-09 12:08:01 UTC
(In reply to comment #2)
> I'd recommend an even more robust linker algorithm in the case of
> multiple headings... perhaps something to check completeness of the
> record (more fields filled in is 'better'), or how many other records
> link to it (more popular is 'better').  Taking the first result would
> be easier, but not necessarily always the best for the cataloger.
> Perhaps making this syspref-controlled?

There may be more than a unique heading match because the matching is
done on a global Zebra authority index. We could, for example, search
specifically on Zebra Personal-Name-Heading rather than on Heading-Main.
Comment 12 Jared Camins-Esakov 2012-01-11 19:18:59 UTC
(In reply to comment #11)
> There may be more than a unique heading match because the matching is
> done on a global Zebra authority index. We could, for example, search
> specifically on Zebra Personal-Name-Heading rather than on Heading-Main.

The benefit of the architecture I chose for the Linker is that it is easy to implement a new linker that does as granular matching as one could wish. From my point of view, using the more specific indexes greatly increases the complexity for minimal benefit. Patches adding a C4::Linker::PreciseIndex linker module are gratefully accepted.
Comment 13 Jared Camins-Esakov 2012-01-21 15:00:19 UTC Comment hidden (obsolete)
Comment 14 Jared Camins-Esakov 2012-01-24 17:09:04 UTC Comment hidden (obsolete)
Comment 15 Jared Camins-Esakov 2012-02-04 02:09:02 UTC Comment hidden (obsolete)
Comment 16 Jared Camins-Esakov 2012-02-04 03:25:27 UTC
TESTING PLAN

Note: all of these tests require that you have some authority records, preferably for headings that actually appear in your bibliographic data. At least one authority record must contain a "see from" reference (remember which one contains this, as you'll need it for some of the tests).

=== Setting up the patch ===

1. Run installer/data/mysql/atomicupdate/bug_7284_authority_linking_pt1
2. Make sure you install the following files: etc/zeradb/authorities/etc/bib1.att, etc/zebradb/marc_defs/marc21/authorities/authority-koha-indexdefs.xml, etc/zebradb/marc_defs/marc21/authorities/authority-zebra-indexdefs.xsl, etc/zebradb/marc_defs/marc21/authorities/koha-indexdefs-to-zebra.xsl, and etc/zebradb/marc_defs/unimarc/authorities/record.abs
3. Run misc/migration_tools/rebuild_zebra.pl -a -r

=== Testing the Heading match in the cataloging plugin ===

1. Create a new record, and open the cataloging plugin for an authority-controlled field.
2. Search for an authority by entering the "see from" term in the Heading Match box
3. Confirm that the appropriate heading shows up
4. Search for an authority by entering the preferred heading into the Main entry or Main entry ($a only) box (i.e., repeat the procedure you usually use for cataloging, whatever that may be)
5. Confirm that the appropriate heading shows up

=== Testing the cataloging interface ===

1. Turn off BiblioAddsAuthorities
2. Confirm that you cannot enter text directly in an authority-controlled field
3. Confirm that if you search for a heading using the authority control plugin the heading is inserted (note, however, that this patch does not AND IS NOT INTENDED TO fix the bugs in the authority plugin with duplicate subfields; those are wholly out of scope- this check is for regressions)
4. Turn on BiblioAddsAuthorities and AutoCreateAuthorities
5. Confirm that you can enter text directly into an authority-controlled field, and if you enter a heading that doesn't currently have an authority record, an authority record stub is automatically created, and the heading you entered linked
6. Confirm that if you enter a heading with only a subfield $a that *matches* an existing heading, the authid for that heading is inserted into subfield $9
7. Confirm that if you enter a heading with multiple subfields that *matches* an existing heading, the authid for that heading is inserted into subfield $9
8. Turn on BiblioAddsAuthorities and turn off AutoCreateAuthorities
9. Confirm that you can enter text directly into an authority-controlled field, and if you enter a heading that doesn't currently have an authority record, an authority record stub is *not* created
10. Confirm that if you enter a heading with only a subfield $a that *matches* an existing heading, the authid for that heading is inserted into subfield $9
11. Confirm that if you enter a heading with multiple subfields that *matches* an existing heading, the authid for that heading is inserted into subfield $9

=== Testing link_bibs_to_authorities.pl ===

1. Set LinkerModule to "Default," turn on LinkerRelink and BiblioAddsAuthorities, and turn AutoCreateAuthorities and LinkerKeepStale off
2. Edit one bib record so that an authority controlled field that has already been linked (i.e. has data in $9) has a heading that does not match any authority record in your database
3. Run misc/link_bibs_to_authorities.pl --link-report --verbose --test (you may want to pipe the output into less or a file, as the result is quite a lot of information)
4. Look over the report to see if the headings that you have authority records for report being matched, that the heading you modified in step 2 is reported as "unlinked," and confirm that no changes were actually made to the database (to check this, look at the bib record you edited earlier, and check that the authid in the field you edited hasn't changed)
5. Run misc/link_bibs_to_authorities.pl --link-report --verbose (you may want to pipe the output into less or a file, as the result is quite a lot of information)
6. Check that the heading you modified has been unlinked
7. Change the modified heading back to whatever it was, but don't use the authority control plugin to populate $9
8. Run misc/link_bibs_to_authorities.pl --link-report --verbose --bib-limit="biblionumber=${BIB}" (replacing ${BIB} with the biblionumber of the record you've been editing)
9. Confirm that the heading has been linked to the correct authority record
10. Turn LinkerKeepStale on
11. Change that heading to something else
12. Run misc/link_bibs_to_authorities.pl --link-report --verbose --bib-limit="biblionumber=${BIB}" (replacing ${BIB} with the biblionumber of the record you've been editing)
13. Confirm that the $9 has not changed
14. Turn LinkerKeepStale off
15. Create two authorities with the same heading
16. Run misc/migration_tools/rebuild_zebra.pl -a -z
17. Enter that heading into the bibliographic record you are working with
18. Run misc/link_bibs_to_authorities.pl --link-report --verbose --bib-limit="biblionumber=${BIB}" (replacing ${BIB} with the biblionumber of the record you've been editing)
19. Confirm that the heading has not been linked
20. Change LinkerModule to "FirstMatch"
21. Run misc/link_bibs_to_authorities.pl --link-report --verbose --bib-limit="biblionumber=${BIB}" (replacing ${BIB} with the biblionumber of the record you've been editing)
22. Confirm that the heading has been linked to the first authority record it matches
23. Change LinkerModule to "LastMatch"
24. Run misc/link_bibs_to_authorities.pl --link-report --verbose --bib-limit="biblionumber=${BIB}" (replacing ${BIB} with the biblionumber of the record you've been editing)
25. Confirm that the heading has been linked to the second authority record it matches
26. Run misc/link_bibs_to_authorities.pl --link-report --verbose --auth-limit="authid=${AUTH}" (replacing ${AUTH} with an authid)
27. Confirm that only that heading is displayed in the report, and only those bibs with that heading have been changed

=== Conclusion ===

If all those things worked, good news! You're ready to sign off on the patch for bug 7284.
Comment 17 Jared Camins-Esakov 2012-02-07 21:38:54 UTC
There is one feature that is not documented in the commit message:

Enter "broader_headings" in LinkerOptions. With this option, the linker will try to match the following heading as follows:
=600  10$aCamins-Esakov, Jared$xCoin collections$vCatalogs$vEarly works to 1800.

First: Camins-Esakov, Jared--Coin collections--Catalogs--Early works to 1800
Next: Camins-Esakov, Jared--Coin collections--Catalogs
Next: Camins-Esakov, Jared--Coin collections
Next: Camins-Esakov, Jared (matches! if a previous attempt had matched, it would not have tried this)

This is probably relevant only to MARC21 and LCSH, but could potentially be of great use to libraries that make heavy use of floating subdivisions.
Comment 18 Jared Camins-Esakov 2012-02-10 00:29:57 UTC Comment hidden (obsolete)
Comment 19 Jared Camins-Esakov 2012-02-16 13:49:40 UTC Comment hidden (obsolete)
Comment 20 Paul Poulain 2012-02-17 14:39:03 UTC
The script devel/misc/testbugzillapatches.pl has detected that this bug does not apply anymore, so i'm marking this bug as "does not apply".

The script is supposed to handle conflicts related to updatedatabase, but no other cases of merge conflict. So the conflict may be trivial to fix.

If it happens that the patch attached does apply cleanly, please send me an email, there's something wrong in the script. It's possible, as it's a recent script.
If the patch applies does not apply as I think, please rebase and resubmit, thanks
Comment 21 Jared Camins-Esakov 2012-02-17 14:54:05 UTC Comment hidden (obsolete)
Comment 22 Jared Camins-Esakov 2012-02-21 16:08:16 UTC Comment hidden (obsolete)
Comment 23 Jared Camins-Esakov 2012-02-23 14:03:28 UTC Comment hidden (obsolete)
Comment 24 David Schuster 2012-02-29 12:24:57 UTC Comment hidden (obsolete)
Comment 25 Jared Camins-Esakov 2012-02-29 12:49:52 UTC
Marking as signed off since David Schuster signed off on the patch.
Comment 26 Ian Walls 2012-03-01 18:50:24 UTC
Small QA comment:  Jared, be sure to add yourself to the copyright statements on:

C4/AuthoritiesMarc.pm
C4/Heading.pm
misc/link_bibs_to_authorities.pl

As you've contributed significantly to their functionality now.

Subroutine parameters:

$marc_handler->valid_bib_heading_tag() is changed to take frameworkcode as a second param.  In C4::Heading->new_from_bib_field, the framework is passed in and then on to valid_bib_heading_tag() only some of the time.
Comment 27 Ian Walls 2012-03-01 19:27:09 UTC
Argh, slipped and clicked save.

Subroutine parameters:

$marc_handler->valid_bib_heading_tag() is changed to take frameworkcode as a
second param, but it's not used.  In C4::Heading->new_from_bib_field, the framework is passed in and then on to valid_bib_heading_tag(), but not by C4/Linker/Default.pm.  So, if the code is eventually updated to make use of the frameworkcode in valid_bib_heading_tag, and the "Default Linker" is used, there could be an error with an undefined value.  Best to explicitly set $frameworkcode = shift || '' in valid_bib_heading_tag(), to future-proof this a bit.


Untranslatable Strings:

There are untranslatable strings in C4/Biblio.pm when creating the 'created by Koha' citation.  The fields chosen aren't adjustable, either, but the having  "Machine generated authority record." and "Work Cat." in the authority records automatically is less than optimal in any non-English environment.


No other coding issues found through Perl analysis.  All in all, I'm quite pleased with this patch and the new, well-documented functionality it includes.  I'd recommend taking a look at these noted issues in a followup patch, but as they're not deal-breakers, I'm going to mark this as Passed QA.
Comment 28 Paul Poulain 2012-03-07 17:04:49 UTC
Testing and trying to deal with this huge patch under UNIMARC.

What I did:
* update bib1.att to add +att 8900    Match
* copy record.abs on my install
* reindexed my authorities
* checked that I can find them (searching Science, getting 6 results, as expected. one "science", and others with subfields)
* updated the new sysprefs to set them to : 
  -CatalogModuleRelink= Do not
  -LinkerKeepStale = Do not
  -LinkerModule = First Match
  -LinkerOptions = empty
  -LinkerRelink  = Do
* added a biblio, with subject entered "Science"
* the Science is not linked + I get the following error in zebra:
17:54:28-07/03 zebrasrv(2) [request] Search authorities ERROR 114 1 1+0 RPN @attrset Bib-1 @and @attr 5=100 @attr 1=authtype SNC @attr 6=3 @attr 5=100 @attr 4=1 @attr 1=Match-heading Science

I can't find "Match-heading" index in my config files, where is it supposed to be ?
- it's not in record.abs
- I use zebra-authorities.cfg, not zebra-authorities-dom.cfg, if that matters
Comment 29 Jared Camins-Esakov 2012-03-07 17:20:09 UTC Comment hidden (obsolete)
Comment 30 Paul Poulain 2012-03-07 17:32:23 UTC
i'll test the unimarc follow-up soon, but 1 thing strikes me:
isn't the match& match-heading obsoleting Heading and Heading-main indexes ? If yes shouldn't they be removed ?

I grepped and feel they are obsoleted : I can find them only in marc def/config and nowhere in the code
Comment 31 Jared Camins-Esakov 2012-03-07 17:38:34 UTC
Paul,

(In reply to comment #30)
> i'll test the unimarc follow-up soon, but 1 thing strikes me:
> isn't the match& match-heading obsoleting Heading and Heading-main indexes ?
> If yes shouldn't they be removed ?
> 
> I grepped and feel they are obsoleted : I can find them only in marc
> def/config and nowhere in the code

Heading and Heading-main are still used by the interface, in SearchAuthorities. I added Match-heading so that my changes would not affect the existing authority searching functionality.
Comment 32 Paul Poulain 2012-03-07 22:10:29 UTC
Jared, I think i'm ready to push this patch. I've added a documentation page on the wiki, because this update *must* be explained a lot and announced during upgrade. Could you please check what I copy/pasted/rewrote from the comment, check if i've missed something of made something wrong ?
I'll add the link to the wiki page to the updatedatabase message, and announce the change widely when releasing 3.8 !
Comment 33 Jared Camins-Esakov 2012-03-07 22:16:28 UTC
Paul,

(In reply to comment #32)
> Jared, I think i'm ready to push this patch. I've added a documentation page
> on the wiki, because this update *must* be explained a lot and announced
> during upgrade. Could you please check what I copy/pasted/rewrote from the
> comment, check if i've missed something of made something wrong ?
> I'll add the link to the wiki page to the updatedatabase message, and
> announce the change widely when releasing 3.8 !

Looks good to me. I also added the LinkerOptions section, since that's a user-visible feature as well.
Comment 34 Jared Camins-Esakov 2012-03-07 23:56:43 UTC Comment hidden (obsolete)
Comment 35 Katrin Fischer 2012-03-08 06:28:39 UTC
Only s small note: The formatting in the updatedatabase message does not work, because it is esaced. So the output is:

Upgrade to 3.07.00.021 done (<b>Enhancement 7284, improved authority matching, see <a href='http://wiki.koha-community.org/wiki/Bug7284_authority_matching_improvement'> wiki page for configuration update needed</a></b>)
Comment 36 Jared Camins-Esakov 2012-03-10 14:54:53 UTC
Created attachment 8137 [details] [review]
[3.6.x] Bug 7284: Authority matching improvements

Squashed patch incorporating all previous patches (there is no functional
change compared to the previous version of this patch, this patch merely
squashes the original patch and follow-up, and rebases on latest master).

=== TL;DR VERSION ===
*** Installation ***
1. Run installer/data/mysql/atomicupdate/bug_7284_authority_linking_pt1
and installer/data/mysql/atomicupdate/bug_7284_authority_linking_pt2
2. Make sure you copy the following files from kohaclone to koha-dev:
etc/zeradb/authorities/etc/bib1.att,
etc/zebradb/marc_defs/marc21/authorities/authority-koha-indexdefs.xml,
etc/zebradb/marc_defs/marc21/authorities/authority-zebra-indexdefs.xsl,
etc/zebradb/marc_defs/marc21/authorities/koha-indexdefs-to-zebra.xsl, and
etc/zebradb/marc_defs/unimarc/authorities/record.abs
3. Run misc/migration_tools/rebuild_zebra.pl -a -r

*** New sysprefs ***
* AutoCreateAuthorities
* CatalogModuleRelink
* LinkerModule
* LinkerOptions
* LinkerRelink
* LinkerKeepStale

*** Important notes ***
You must have rebuild_zebra processing the zebraqueue for bibs when testing this
patch.

=== DESCRIPTION ===

*** Cataloging module ***
* Added an additional box to the authority finder plugin for "Heading match,"
  which consults not just the main entry but also See-from and See-also-from
  headings.

* With this patch, the automatic authority linking will actually work properly
  in the cataloging module. As Owen pointed out while testing the patch,
  though, longtime users of Koha will not be expecting that. In keeping with
  the principles of least surprise and maximum configurability, a new syspref,
  CatalogModuleRelink makes it possible to disable authority relinking in the
  cataloging module only (i.e. leaving it enabled for future runs of
  link_bibs_to_authorities.pl).  Note that though the default behavior matches
  the current behavior of Koha, it does not match the intended behavior.
  Libraries that want the intended behavior rather than the current behavior
  will need to adjust the CatalogModuleRelink syspref.

*** misc/link_bibs_to_authorities.pl ***
Added the following options to the misc/link_bibs_to_authorities.pl script:
--auth-limit        Only process those headings that match the authorities
                    matching the user-specified WHERE clause.
--bib-limit         Only process those bib records that match the
                    user-specified WHERE clause.
--commit            Commit the results to the database after every N records
                    are processed.
--link-report       Display a report of all the headings that were processed.

Converted misc/link_bibs_to_authorities.pl to use POD.

Added a detailed report of headings that linked, did not link, and linked
in a "fuzzy" fashion (the exact semantics of fuzzy are up to the individual
linker modules) during the run.

*** C4::Linker ***
Implemented new C4::Linker functionality to make it possible to easily add
custom authority linker algorithms. Currently available linker options are:
* Default: retains the current behavior of only creating links when there is
  an exact match to one and only one authority record; if the 'broader_headings'
  option is enabled, it will try to link to headings to authority records for
  broader headings by removing subfields from the end of the heading (NOTE:
  test the results before enabling broader_headings in a production system
  because its usefulness is very much dependent on individual sites' authority
  files)
* First Match: based on Default, creates a link to the *first* authority
  record that matches a given heading, even if there is more than one
  authority record that matches
* Last Match: based on Default, creates a link to the *last* authority
  record that matches a given heading, even if there is more than one record
  that matches

The API for linker modules is very simple. All modules should implement the
following two functions:
<get_link ($field)> - return the authid for the authority that should be
linked to the provided MARC::Field object, and a boolean to indicate whether
the match is "fuzzy" (the semantics of "fuzzy" are up to the individual plugin).
In order to handle authority limits, get_link should always end with:
    return $self->SUPER::_handle_auth_limit($authid), $fuzzy;

<flip_heading ($field)> - return a MARC::Field object with the heading flipped
to the preferred form. At present this routine is not used, and can be a stub.

Made the linking functionality use the SearchAuthorities in C4::AuthoritiesMarc
rather than SimpleSearch in C4::Search. Once C4::Search has been refactored,
SearchAuthorities should be rewritten to simply call into C4::Search. However,
at this time C4::Search cannot handle authority searching. Also fixed numerous
performance issues in SearchAuthorities and the Linker script:
* Correctly destroy ZOOM recordsets in SearchAuthorities when finished. If left
  undestroyed, efficiency appears to approach O(log n^n)
* Add an optional $skipmetadata flag to SearchAuthorities that can be used to
  avoid additional calls into Zebra when all that is wanted are authority
  records and not statistics about their use

*** New sysprefs ***
* AutoCreateAuthorities - When this and BiblioAddsAuthorities are both turned
  on, automatically create authority records for headings that don't have
  any authority link when cataloging. When BiblioAddsAuthorities is on and
  AutoCreateAuthorities is turned off, do not automatically generate authority
  records, but allow the user to enter headings that don't match an existing
  authority. When BiblioAddsAuthorities is off, this has no effect.
* CatalogModuleRelink - when turned on, the automatic linker will relink
  headings when a record is saved in the cataloging module when LinkerRelink
  is turned on, even if the headings were manually linked to a different
  authority by the cataloger. When turned off (the default), the automatic
  linker will not relink any headings that have already been linked when a
  record is saved.
* LinkerModule - Chooses which linker module to use for matching headings
  (current options are as described above in the section on linker options:
  "Default," "FirstMatch," and "LastMatch")
* LinkerOptions - A pipe-separated list of options to set for the authority
  linker (at the moment, the only option available is "broader_headings," which
  is described below)
* LinkerRelink - When turned on, the linker will confirm the links for headings
  that have previously been linked to an authority record when it runs. When
  turned off, any heading with an existing link will be ignored.
* LinkerKeepStale - When turned on, the linker will never *delete* a link to an
  authority record, though, depending on the value of LinkerRelink, it may
  change the link.

*** Other changes ***
* Cleaned up authorities code by removing unused functions and adding
  unimplemented functions and added some unit tests.

* This patch also modifies the authority indexing to remove trailing punctuation
  from Match indexes.

* Replace the old BiblioAddAuthorities subroutines with calls into the new
  C4::Linker routines.

* Add a simple implementation for C4::Heading::UNIMARC. (With thanks to F.
  Demians, 2011.01.09) Correct C4::Heading::UNIMARC class loading. Create
  biblio tag to authority types data structure at initialization rather than
  querying DB.

* Ran perltidy on all changed code.

*** Linker Options ***
Enter "broader_headings" in LinkerOptions. With this option, the linker will
try to match the following heading as follows:
=600  10$aCamins-Esakov, Jared$xCoin collections$vCatalogs$vEarly works to
1800.

First: Camins-Esakov, Jared--Coin collections--Catalogs--Early works to 1800
Next: Camins-Esakov, Jared--Coin collections--Catalogs
Next: Camins-Esakov, Jared--Coin collections
Next: Camins-Esakov, Jared (matches! if a previous attempt had matched, it
would not have tried this)

This is probably relevant only to MARC21 and LCSH, but could potentially be of
great use to libraries that make heavy use of floating subdivisions.

=== TESTING PLAN ===

Note: all of these tests require that you have some authority records,
preferably for headings that actually appear in your bibliographic data. At
least one authority record must contain a "see from" reference (remember which
one contains this, as you'll need it for some of the tests). The number shown
in the "Used in" column in the authority module is populated using Zebra
searches of the bibliographic database, so you *must* have
rebuild_zebra.pl -b -z [-x] running in cron, or manually run it after running
the linker.

*** Testing the Heading match in the cataloging plugin ***
1.  Create a new record, and open the cataloging plugin for an
    authority-controlled field.
2.  Search for an authority by entering the "see from" term in the Heading Match
    box
3.  Confirm that the appropriate heading shows up
4.  Search for an authority by entering the preferred heading into the Main
    entry or Main entry ($a only) box (i.e., repeat the procedure you usually
    use for cataloging, whatever that may be)
5.  Confirm that the appropriate heading shows up

*** Testing the cataloging interface ***
6.  Turn off BiblioAddsAuthorities
7.  Confirm that you cannot enter text directly in an authority-controlled field
8.  Confirm that if you search for a heading using the authority control plugin
    the heading is inserted (note, however, that this patch does not AND IS NOT
    INTENDED TO fix the bugs in the authority plugin with duplicate subfields;
    those are wholly out of scope- this check is for regressions)
9.  Turn on BiblioAddsAuthorities and AutoCreateAuthorities
10. Confirm that you can enter text directly into an authority-controlled field,
    and if you enter a heading that doesn't currently have an authority record,
    an authority record stub is automatically created, and the heading you
    entered linked
11. Confirm that if you enter a heading with only a subfield $a that fully
    *matches* an existing heading (i.e. the existing heading has only
    subfield $a populated), the authid for that heading is inserted into
    subfield $9
12. Confirm that if you enter a heading with multiple subfields that *matches*
    an existing heading, the authid for that heading is inserted into
    subfield $9
13. Turn on BiblioAddsAuthorities and turn off AutoCreateAuthorities
14. Confirm that you can enter text directly into an authority-controlled field,
    and if you enter a heading that doesn't currently have an authority record,
    an authority record stub is *not* created
15. Confirm that if you enter a heading with only a subfield $a that *matches*
    an existing heading, the authid for that heading is inserted into
    subfield $9
16. Confirm that if you enter a heading with multiple subfields that *matches*
    an existing heading, the authid for that heading is inserted into
    subfield $9
17. Create a record and link an authority record to an authorized field using
    the authority plugin.
18. Save the record. Ensure that the heading is linked to the appropriate
    authority.
19. Open the record. Change the heading manually to something else, leaving
    the link. Save the record.
20. Ensure that the heading remains linked to that same authority.
21. Change CatalogModuleRelink to "on."
22. Open the record. Use the authority plugin to link that heading to the
    same authority record you did earlier.
23. Save the record. Ensure that the heading is linked to the appropriate
    authority.
24. Open the record. Change the heading manually to something else, leaving
    the link. Save the record.
25. Ensure that the heading is no longer linked to the old authority record.

*** Testing link_bibs_to_authorities.pl ***
26. Set LinkerModule to "Default," turn on LinkerRelink and
    BiblioAddsAuthorities, and turn AutoCreateAuthorities and
    LinkerKeepStale off
27. Edit one bib record so that an authority controlled field that has already
    been linked (i.e. has data in $9) has a heading that does not match any
    authority record in your database
28. Run misc/link_bibs_to_authorities.pl --link-report --verbose --test (you may
    want to pipe the output into less or a file, as the result is quite a lot of
    information)
29. Look over the report to see if the headings that you have authority records
    for report being matched, that the heading you modified in step 2 is
    reported as "unlinked," and confirm that no changes were actually made to
    the database (to check this, look at the bib record you edited earlier, and
    check that the authid in the field you edited hasn't changed)
30. Run misc/link_bibs_to_authorities.pl --link-report --verbose (you may want
    to pipe the output into less or a file, as the result is quite a lot of
    information)
31. Check that the heading you modified has been unlinked
32. Change the modified heading back to whatever it was, but don't use the
    authority control plugin to populate $9
33. Run misc/link_bibs_to_authorities.pl --link-report --verbose
    --bib-limit="biblionumber=${BIB}" (replacing ${BIB} with the biblionumber
    of the record you've been editing)
34. Confirm that the heading has been linked to the correct authority record
35. Turn LinkerKeepStale on
36. Change that heading to something else
37. Run misc/link_bibs_to_authorities.pl --link-report --verbose
    --bib-limit="biblionumber=${BIB}" (replacing ${BIB} with the biblionumber
    of the record you've been editing)
38. Confirm that the $9 has not changed
39. Turn LinkerKeepStale off
40. Create two authorities with the same heading
41. Run misc/migration_tools/rebuild_zebra.pl -a -z
42. Enter that heading into the bibliographic record you are working with
43. Run misc/link_bibs_to_authorities.pl --link-report --verbose
    --bib-limit="biblionumber=${BIB}" (replacing ${BIB} with the biblionumber
    of the record you've been editing)
44. Confirm that the heading has not been linked
45. Change LinkerModule to "FirstMatch"
46. Run misc/link_bibs_to_authorities.pl --link-report --verbose
    --bib-limit="biblionumber=${BIB}" (replacing ${BIB} with the biblionumber
    of the record you've been editing)
47. Confirm that the heading has been linked to the first authority record it
    matches
48. Change LinkerModule to "LastMatch"
49. Run misc/link_bibs_to_authorities.pl --link-report --verbose
    --bib-limit="biblionumber=${BIB}" (replacing ${BIB} with the biblionumber
    of the record you've been editing)
50. Confirm that the heading has been linked to the second authority record it
    matches
51. Run misc/link_bibs_to_authorities.pl --link-report --verbose
    --auth-limit="authid=${AUTH}" (replacing ${AUTH} with an authid)
52. Confirm that only that heading is displayed in the report, and only those
    bibs with that heading have been changed

If all those things worked, good news! You're ready to sign off on the patch
for bug 7284.

Signed-off-by: Jared Camins-Esakov <jcamins@cpbibliography.com>
Rebased on latest master and squashed follow-up, 16 February 2012
Signed-off-by: Jared Camins-Esakov <jcamins@cpbibliography.com>
Rebased on latest master, 21 February 2012

Signed-off-by: schuster <dschust1@gmail.com>
Signed-off-by: Paul Poulain <paul.poulain@biblibre.com>
Tested with an UNIMARC setup that things work fine. They do

Signed-off-by: Jared Camins-Esakov <jcamins@cpbibliography.com>
Cherry-picked onto 3.6.x, 10 March 2012
Note for RMaint: this patch squashes all the bug 7284 patches from Master,
including the updatedatabase changes (but does not set the DB revision).
Comment 37 MJ Ray (software.coop) 2012-04-04 11:39:28 UTC Comment hidden (obsolete)
Comment 38 Jared Camins-Esakov 2012-04-04 11:48:36 UTC
Created attachment 8864 [details] [review]
bug 7284: fix package build problem, move BEGIN into new

Signed-off-by: Jared Camins-Esakov <jcamins@cpbibliography.com>
Comment 39 Paul Poulain 2012-04-04 16:17:31 UTC
QA comment: I feel this patch addresses a real problem, but I also feel the description is incomplete/unclear:

fix package build problem, move BEGIN into new

WHY is it moved ?

Can I rewrite it to:
"this patch retrieve $dbh when declaring a ->new() object, instead of doing this in BEGIN block, that make some problem for make" ?

I think I can, so passed QA, will push
Comment 40 Paul Poulain 2012-04-04 16:19:17 UTC
follow-up pushed, chris_n, you'll probably have to push the fix package build problem for 3.6 as well
Comment 41 Chris Cormack 2012-04-04 22:52:09 UTC
Currently this is not in 3.6.x but yes if it is pushed to that, the follow up will have to be pushed there too.
Comment 42 Jared Camins-Esakov 2012-05-09 16:52:16 UTC
This bug will be included in the Koha 3.6.5 release.
Comment 43 Jared Camins-Esakov 2012-05-28 19:15:53 UTC
*** Bug 5803 has been marked as a duplicate of this bug. ***