One of the problems that arises when maintaining an authority file long-term is inadvertent duplication of authority records. In order to combat this problem, Koha needs a script for bulk de-duplicating of authority records. The deduplicator should be able to automatically recognize machine-generated authority records, and choose official records. It should also recognize when one record supersedes another based on previous headings. The algorithm will also need to give greater weight to newer records.
Authority deduplication is similar to authority unlink/link. Consider two terms like "Bargioni, Stefano, 1959-" linked to N1 biblios and "Bargioni, Stefano" linked to N2 biblios. You need to unlink N2 biblios from the second term, i.e. link them to the first one. Deduplication can be automatic, while the step of choosing term-from and term-to for unlink/link biblios must be manual.
Hello Jared. We did a big work on a similar script. I put this ticket on top of rebase tasks. We try to provide a lot of options, few more tests and in few weeks you could test it.
Claire, (In reply to comment #2) > Hello Jared. We did a big work on a similar script. I put this ticket on top > of rebase tasks. We try to provide a lot of options, few more tests and in > few weeks you could test it. I actually borrowed from a deduplication script that hdl showed me, though I think it may be that you have a newer version. The script I have works for MARC21, and offers very basic options for UNIMARC, but I'd love to integrate my changes into your script (or vice versa) to provide more options.
Jared, I am talking about the initial Henri Damien script that evolved with more options and bug fix. You will find the last version (can change again a little but not a lot) here http://git.biblibre.com/?p=koha;a=shortlog;h=refs/heads/ft/dedup_authorities
Created attachment 11161 [details] [review] Bug 7419: Move indexation subs in C4::Search Move indexation-related subs from rebuild_zebra.pl to C4::Search and create two new subs IndexRecord and DeleteRecordIndex
Created attachment 11162 [details] [review] Bug 7419: New script dedup_authorities.pl This script allow to deduplicate authorities automatically. It works this way: 1) authorities are fetched from the database. You can limit fetched results by authtypecode, or directly by specifying WHERE clause 2) for each authority: 2.1) build a Zebra query, depending on --match parameter(s) 2.2) run the query, retrieve the results 2.3) for each result check if result is really a duplicate. check is customizable with option --check. 2.4) among real duplicates, choose the one we want to keep (use --choose-method option). 2.5) use C4::Authorities::merge to merge authorities 3) index updated biblios and authorities, and remove from index removed authorities Use --help for more informations on options Script is in misc/migration_tools/
Created attachment 11412 [details] [review] Bug 7419: Move indexation subs in C4::Search Rebased on master. Add support for --skip-index option of rebuild_zebra.pl
Created attachment 11413 [details] [review] Bug 7419: New script dedup_authorities.pl Rebased on master
Created attachment 11486 [details] [review] Bug 7419: Move indexation subs in C4::ZebraIndex Move indexation-related subs from rebuild_zebra.pl to new module C4::ZebraIndex and create two new subs IndexRecord and DeleteRecordIndex
Created attachment 11487 [details] [review] Bug 7419: Move indexation subs in C4::ZebraIndex Move indexation-related subs from rebuild_zebra.pl to new module C4::ZebraIndex and create two new subs IndexRecord and DeleteRecordIndex -- Same patch as above, but with the right commit message.
Created attachment 11488 [details] [review] Bug 7419: New script dedup_authorities.pl It works this way: 1) authorities are fetched from the database. You can limit fetched results by authtypecode, or directly by specifying WHERE clause 2) for each authority: 2.1) build a Zebra query, depending on --match parameter(s) 2.2) run the query, retrieve the results 2.3) for each result check if result is really a duplicate. check is customizable with option --check. 2.4) among real duplicates, choose the one we want to keep (use --choose-method option). 2.5) use C4::Authorities::merge to merge authorities 3) index updated biblios and authorities, and remove from index removed authorities Use --help for more informations on options Script is in misc/migration_tools/
Indexation subs are now in a new module C4::ZebraIndex, instead of C4::Search (better to not "overload" C4::Search).
I tried to test this patch with the sandbox system, but the system send back these errors : The sandbox you've requested is now ready. The patches attached to bugzilla 7419 have been applied, and the result is <h1>Something went wrong !</h1>Applying: Bug 7419: Move indexation subs in C4::ZebraIndex<br/> Using index info to reconstruct a base tree...<br/> Falling back to patching base and 3-way merge...<br/> Auto-merging misc/migration_tools/rebuild_zebra.pl<br/> CONFLICT (content): Merge conflict in misc/migration_tools/rebuild_zebra.pl<br/> Failed to merge in the changes.<br/> Patch failed at 0001 Bug 7419: Move indexation subs in C4::ZebraIndex<br/> When you have resolved this problem run git am --resolved.<br/> If you would prefer to skip this patch, instead run git am --skip.<br/> To restore the original branch and stop patching run git am --abort.<br/> Bug 7419 - Add authority deduplication script<br/> <br/> Bug 7419: Move indexation subs in C4::ZebraIndex<br/> Apply? [yn] <br/> Patch left in /tmp/Bug-7419-Move-indexation-subs-in-C4ZebraIndex-fNv1EB.patch<br/> Thanks for fixing it !
Created attachment 13073 [details] [review] Bug 7419: Move indexation subs in C4::ZebraIndex rebased on master
Created attachment 13074 [details] [review] Bug 7419: New script dedup_authorities.pl rebased on master
Pascale, you should be able to apply these patches now.
Hello Julian, if we don't use --verbose we have errors => Use of uninitialized value $verbose in numeric ge (>=) at ./dedup_authorities.pl line 126. the same for lines 143,145,151. For line 126 : if ($verbose >= 2) should be : if ($verbose AND $verbose >= 2) thx
Julian, it doesn't apply : Applying: Bug 7419: Move indexation subs in C4::ZebraIndex Using index info to reconstruct a base tree... Falling back to patching base and 3-way merge... Auto-merging misc/migration_tools/rebuild_zebra.pl CONFLICT (content): Merge conflict in misc/migration_tools/rebuild_zebra.pl Failed to merge in the changes. Patch failed at 0001 Bug 7419: Move indexation subs in C4::ZebraIndex
Created attachment 13629 [details] [review] Bug 7419: Move indexation subs in C4::ZebraIndex Move indexation-related subs from rebuild_zebra.pl to new module C4::ZebraIndex and create two new subs IndexRecord and DeleteRecordIndex
Created attachment 13630 [details] [review] Bug 7419: New script dedup_authorities.pl This script allow to deduplicate authorities automatically. It works this way: 1) authorities are fetched from the database. You can limit fetched results by authtypecode, or directly by specifying WHERE clause 2) for each authority: 2.1) build a Zebra query, depending on --match parameter(s) 2.2) run the query, retrieve the results 2.3) for each result check if result is really a duplicate. check is customizable with option --check. 2.4) among real duplicates, choose the one we want to keep (use --choose-method option). 2.5) use C4::Authorities::merge to merge authorities 3) index updated biblios and authorities, and remove from index removed authorities Use --help for more informations on options Script is in misc/migration_tools/
Patches rebased on master
Patch still applies. Minor POD errors on script. Missing license statement on script. The problem is, I can't figure how to test it. And I think this could be a very valuable tool. If you could you add some test cases, or an accompanying .txt that explains better the syntax it would be easier to test and use.
Just retested it with Leïla and found an error merging 25894 into 1198 failed :Can't call method "fields" on an undefined value at /home/koha/src/C4/AuthoritiesMarc.pm line 1392. Patch needs some work.
Created attachment 17790 [details] [review] Bug 7419: Move indexation subs in C4::ZebraIndex Rebased on master
Created attachment 17791 [details] [review] Bug 7419: New script dedup_authorities.pl Rebased on master
Created attachment 17792 [details] [review] Bug 7419: merge sub should retrieve marc records from DB if not given This should fix issue of comment 23
I tested this script and found that it did not work at all with MARC21 because of the way it was retrieving authid. I am working on a counter-patch which I hope to upload tomorrow which incorporates the matching options from your script but should work properly with both UNIMARC and MARC21.
Created attachment 18712 [details] Sample duplicate MARC21 authorities
Created attachment 18713 [details] [review] Bug 7419: correct LCCN/see from indexing This patch adds indexes for previous headings and LCCN for MARC21 authority records. See the following patch for a test plan to confirm that the indexing works correctly.
Created attachment 18714 [details] [review] Bug 7419: General-purpose record deduplicator This patch adds a script for deduplicating records. It is most useful for authority records but by design could be easily extended for use with bibliographic records, if someone had a good use case. To test using the sample records attached to the bug (MARC21 only): 1) Apply patches. 2) Import the sample records (sampleauths.mrc) file into Koha and make sure that they get indexed (by waiting until rebuild_zebra.pl runs automatically or by running rebuild_zebra.pl -a -z [-x] manually) 3) Deduplicate based on LCCN (replace {FIRSTAUTH} with the authid of the first imported record; on my system that number is 367123668; this is not actually necessary, but if you have a lot of authority records, the process could take quite a while otherwise): > misc/migration_tools/dedup_records.pl -t -v -a \ -l "authid >= {FIRSTAUTH}" -r -m "lc-card/010a" -s date 4) Check that you have 21 duplicate records replaced from amongst the new records (you will quite possibly have more than 21 duplicate records reported, depending on whether you have any of these authorities already, but you should have at least the 21). 5) Deduplicate based on genre heading, preferring Library of Congress authorities to local authorities: > misc/migration_tools/dedup_records.pl -t -v -a \ -l "authid >= {FIRSTAUTH}" -r -m "he/155a" -s "source=DLC" -s date 6) Check that you have 3 duplicate records replaced from amongst the new records. 7) Sign off. Complete POD documentation: SYNOPSIS dedup_records.pl --match=1 -a dedup_records.pl --match="LC-card-number/010a" --select="date" \ --limit="authid > 367123592" -a dedup_records.pl --match="Match/100abcdefghijklmnopqrstuvwxyz" \ --select="source=DLC" --select="date" \ --limit="authtypecode='PERSO_NAME'" -a DESCRIPTION This script will identify duplicate records, and either suggest that you merge them (in the case of bibliographic records) or automatically merge them for you (in the case of authority records). OPTIONS --help Prints this help -v|--verbose Print verbose log information (warning: very verbose!). -t|--test Do not actually make any changes to the database, just report what changes would be made. -r|--report Print a report of what happened during the run. -l|--limit=S Only process those records that match the user-specified WHERE clause (the WHERE is implied and should not be included on the command line). -a|--authorities Check for duplicate authorities rather that duplicate bibliographic records. -s|--select=s Repeatable. Specify how to identify which record to prefer. See the section on SELECTORS below. -m|--match=s Specifies the matching rule to use. This can be the numeric ID of a matching rule that you have already configured (preferred), or you can specify a matching rule on the command-line in the following format: <index1>/<tag1><subfield1>[##<index2>/<tag2><subfield2>[##...]] Examples: at/152b##he-main/2..a##he/2..bxyzt##ident/009@ authtype/152b##he-main,ext/2..a##he,ext/2..bxyz sn,ne,st-numeric/001##authtype/152b##he-main,ext/2..a##he,ext/2..bxyz -c|--check=s Only relevant when you are using a matching rule specified on the command line. Specifies sanity checks to use to ensure that the records are really duplicate. The format is <tag1><subfields1>[,<tag2><subfields2>[,...]] Examples: 200abxyz will check subfields a,b,x,y,z of 200 fields 009@,152b will check 009 data and 152$b subfields SELECTORS This script supports a number of selectors for choosing which record is "better." score Prefer the record which is the best match based on the specified matching rule. This will probably only be useful in cases where the matching rule will not match the source record, since the source record will automatically be given a score of 2 * the matching rule threshold if it wasn't picked up by the matcher. date Prefer the record which is newer based on the 005 field. source=ABC MARC21 only. Prefer records which come from ABC based on the 003 field. usage Authorities only. Prefer the record used in the most bibliographic records. ppn UNIMARC only. Prefer records which have a PPN in the 009 field.
The final two patches comprise my counter-patch, which will handle both MARC21 and UNIMARC data (though the sample data provided on this bug is MARC21-only).
Several people have asked about a test plan, so I am repeating my test plan from comment 30, since that comment is so long it can be hard to find: To test using the sample records attached to the bug (MARC21 only): 1) Apply patches. 2) Import the sample records (sampleauths.mrc) file into Koha and make sure that they get indexed (by waiting until rebuild_zebra.pl runs automatically or by running rebuild_zebra.pl -a -z [-x] manually) 3) Deduplicate based on LCCN (replace {FIRSTAUTH} with the authid of the first imported record; on my system that number is 367123668; this is not actually necessary, but if you have a lot of authority records, the process could take quite a while otherwise): > misc/migration_tools/dedup_records.pl -t -v -a \ -l "authid >= {FIRSTAUTH}" -r -m "lc-card/010a" -s date 4) Check that you have 21 duplicate records replaced from amongst the new records (you will quite possibly have more than 21 duplicate records reported, depending on whether you have any of these authorities already, but you should have at least the 21). 5) Deduplicate based on genre heading, preferring Library of Congress authorities to local authorities: > misc/migration_tools/dedup_records.pl -t -v -a \ -l "authid >= {FIRSTAUTH}" -r -m "he/155a" -s "source=DLC" -s date 6) Check that you have 3 duplicate records replaced from amongst the new records. 7) Sign off.
hi Tomas, could you update the ticket with info about your testing of this bug Jared mentioned you found some problems?
(In reply to Mason James from comment #33) > hi Tomas, > could you update the ticket with info about your testing of this bug > > Jared mentioned you found some problems? The sample data and test plan didn't match, there was a problem with 155$a vs. 150$a (the second proposed test) and some warnings that needed to be addressed.
Created attachment 20783 [details] [review] Bug 7419: General-purpose record deduplicator This patch adds a script for deduplicating records. It is most useful for authority records but by design could be easily extended for use with bibliographic records, if someone had a good use case. See the follow-up for an updated test plan. Complete POD documentation: SYNOPSIS dedup_records.pl --match=1 -a dedup_records.pl --match="LC-card-number/010a" --select="date" \ --limit="authid > 367123592" -a dedup_records.pl --match="Match/100abcdefghijklmnopqrstuvwxyz" \ --select="source=DLC" --select="date" \ --limit="authtypecode='PERSO_NAME'" -a DESCRIPTION This script will identify duplicate records, and either suggest that you merge them (in the case of bibliographic records) or automatically merge them for you (in the case of authority records). OPTIONS --help Prints this help -v|--verbose Print verbose log information (warning: very verbose!). -t|--test Do not actually make any changes to the database, just report what changes would be made. -r|--report Print a report of what happened during the run. -l|--limit=S Only process those records that match the user-specified WHERE clause (the WHERE is implied and should not be included on the command line). -a|--authorities Check for duplicate authorities rather that duplicate bibliographic records. -s|--select=s Repeatable. Specify how to identify which record to prefer. See the section on SELECTORS below. -m|--match=s Specifies the matching rule to use. This can be the numeric ID of a matching rule that you have already configured (preferred), or you can specify a matching rule on the command-line in the following format: <index1>/<tag1><subfield1>[##<index2>/<tag2><subfield2>[##...]] Examples: at/152b##he-main/2..a##he/2..bxyzt##ident/009@ authtype/152b##he-main,ext/2..a##he,ext/2..bxyz sn,ne,st-numeric/001##authtype/152b##he-main,ext/2..a##he,ext/2..bxyz -c|--check=s Only relevant when you are using a matching rule specified on the command line. Specifies sanity checks to use to ensure that the records are really duplicate. The format is <tag1><subfields1>[,<tag2><subfields2>[,...]] Examples: 200abxyz will check subfields a,b,x,y,z of 200 fields 009@,152b will check 009 data and 152$b subfields SELECTORS This script supports a number of selectors for choosing which record is "better." score Prefer the record which is the best match based on the specified matching rule. This will probably only be useful in cases where the matching rule will not match the source record, since the source record will automatically be given a score of 2 * the matching rule threshold if it wasn't picked up by the matcher. date Prefer the record which is newer based on the 005 field. source=ABC MARC21 only. Prefer records which come from ABC based on the 003 field. usage Authorities only. Prefer the record used in the most bibliographic records. ppn UNIMARC only. Prefer records which have a PPN in the 009 field.
Created attachment 20784 [details] [review] Bug 7419: Add more missing authority indexes Koha's CCL configuration did not allow for most of the indexes that were configured for authorities. This patch adds missing authority indexes to QP.
Created attachment 20785 [details] [review] Bug 7419: quiet warnings in deduplicator Quiet some warnings from the deduplicator script. Revised test plan: 1) Apply patches. 2) Import the sample records (sampleauths.mrc) file into Koha and make sure that they get indexed (by waiting until rebuild_zebra.pl runs automatically or by running rebuild_zebra.pl -a -z [-x] manually) 3) Deduplicate based on LCCN (replace {FIRSTAUTH} with the authid of the first imported record; on my system that number is 367123668): > misc/migration_tools/dedup_records.pl -t -v -a \ -l "authid >= {FIRSTAUTH}" -r -m "lc-card/010a" -s date 4) Check that you have 10 records that were chosen (you may have more, if you added more authorities after importing the sample data, but you should have at least the ten). 5) Deduplicate based on geographic heading, preferring Library of Congress authorities to local authorities: > misc/migration_tools/dedup_records.pl -t -v -a \ -l "authid >= {FIRSTAUTH}" -r -m "geographic/151a" -s "source=DLC" -s date 6) Check that you have 2 duplicate records replaced from amongst the new records. 7) Sign off.
Note that the new test plan does require you to be using QueryParser (and an updated etc/searchengine/queryparser.yaml).
Applying: Bug 7284: Authority matching improvements Using index info to reconstruct a base tree... M C4/AuthoritiesMarc.pm M C4/Biblio.pm M C4/Heading.pm M C4/Heading/MARC21.pm M C4/Search.pm M acqui/addorderiso2709.pl M acqui/neworderempty.pl M authorities/auth_finder.pl M cataloguing/addbiblio.pl M etc/zebradb/authorities/etc/bib1.att M etc/zebradb/marc_defs/marc21/authorities/authority-koha-indexdefs.xml M etc/zebradb/marc_defs/marc21/authorities/authority-zebra-indexdefs.xsl A etc/zebradb/marc_defs/marc21/authorities/koha-indexdefs-to-zebra.xsl M etc/zebradb/marc_defs/unimarc/authorities/record.abs M installer/data/mysql/sysprefs.sql M installer/data/mysql/updatedatabase.pl M koha-tmpl/intranet-tmpl/prog/en/includes/auth-finder-search.inc M koha-tmpl/intranet-tmpl/prog/en/modules/admin/preferences/authorities.pref M misc/link_bibs_to_authorities.pl A t/Heading_MARC21.t A t/db_dependent/lib/KohaTest/AuthoritiesMarc.pm Falling back to patching base and 3-way merge... CONFLICT (modify/delete): t/db_dependent/lib/KohaTest/AuthoritiesMarc.pm deleted in HEAD and modified in Bug 7284: Authority matching improvements. Version Bug 7284: Authority matching improvements of t/db_dependent/lib/KohaTest/AuthoritiesMarc.pm left in tree. Auto-merging t/db_dependent/Linker_FirstMatch.t CONFLICT (add/add): Merge conflict in t/db_dependent/Linker_FirstMatch.t Auto-merging t/Barcodes_annual.t CONFLICT (content): Merge conflict in t/Barcodes_annual.t Auto-merging misc/link_bibs_to_authorities.pl CONFLICT (content): Merge conflict in misc/link_bibs_to_authorities.pl Auto-merging koha-tmpl/intranet-tmpl/prog/en/modules/admin/preferences/authorities.pref CONFLICT (content): Merge conflict in koha-tmpl/intranet-tmpl/prog/en/modules/admin/preferences/authorities.pref Auto-merging koha-tmpl/intranet-tmpl/prog/en/includes/auth-finder-search.inc CONFLICT (content): Merge conflict in koha-tmpl/intranet-tmpl/prog/en/includes/auth-finder-search.inc Auto-merging installer/data/mysql/updatedatabase.pl CONFLICT (content): Merge conflict in installer/data/mysql/updatedatabase.pl Auto-merging installer/data/mysql/sysprefs.sql CONFLICT (content): Merge conflict in installer/data/mysql/sysprefs.sql Auto-merging C4/Search.pm CONFLICT (content): Merge conflict in C4/Search.pm Auto-merging C4/Linker/Default.pm CONFLICT (add/add): Merge conflict in C4/Linker/Default.pm Auto-merging C4/Heading/UNIMARC.pm CONFLICT (add/add): Merge conflict in C4/Heading/UNIMARC.pm Auto-merging C4/Heading.pm CONFLICT (content): Merge conflict in C4/Heading.pm Auto-merging C4/Biblio.pm CONFLICT (content): Merge conflict in C4/Biblio.pm Auto-merging C4/AuthoritiesMarc.pm CONFLICT (content): Merge conflict in C4/AuthoritiesMarc.pm Patch failed at 0001 Bug 7284: Authority matching improvements The copy of the patch that failed is found in: /home/christopher/git/koha/.git/rebase-apply/patch When you have resolved this problem, run "git am --continue". If you prefer to skip this patch, run "git am --skip" instead. To restore the original branch and stop patching, run "git am --abort".
Please note that the final four patches comprise a counter-patch to the first three, which are not MARC21-compatible.
Created attachment 22150 [details] [review] works with Julian Maurice patchs
Hi, I test this to dedup authorities. I begin with the 4 Jared patches ( attachment.cgi?id= 18713 20783 20784 20785 ) : this dev is more complete but didn't work to dedup my authorities. So I tested The 3 Julian's patches + 1 mine ( attachment.cgi?id= 17790 17791 17792 22150 ) : this dev is less complete (take only authorities) but works to dedup my authorities
(In reply to stephane.delaune from comment #42) > Hi, I test this to dedup authorities. I begin with the 4 Jared patches ( > attachment.cgi?id= 18713 20783 20784 20785 ) : this dev is more complete but > didn't work to dedup my authorities. What results did you see?
(In reply to Jared Camins-Esakov from comment #43) > (In reply to stephane.delaune from comment #42) > > Hi, I test this to dedup authorities. I begin with the 4 Jared patches ( > > attachment.cgi?id= 18713 20783 20784 20785 ) : this dev is more complete but > > didn't work to dedup my authorities. > > What results did you see? I had launched with the command line: ~/src/misc/migration_tools/dedup_records.pl -t -r -v -a -l "authtypecode='SNG'" -m 'at/152b##he/2..axyz' I just noticed a lot of error messages: "oAuth error: Unsupported Use attribute (114) Any Bib-1" However, I did not notice anything more and I'm not able to reproduce the same test conditions
Created attachment 22907 [details] [review] Zconn fix, works with Julian Maurice patchs Zconn fix, works with Julian Maurice patchs
Created attachment 25087 [details] [review] Bug 7419: Move indexation subs in C4::ZebraIndex Rebased on master
Created attachment 25088 [details] [review] Bug 7419: New script dedup_authorities.pl Rebased on master Removed all changes to C4::AuthoritiesMarc::merge (see bug 11700)
Created attachment 25089 [details] [review] Fix C4::Context::Zconn()
Comment on attachment 25089 [details] [review] Fix C4::Context::Zconn() Obsolete patch on Zconn (new bug 11701)
When I am testing dedup-records.pl with Match/100abc on PERSO_NAME, the script finds two duplicates Surname, C.-J. and Surname, C-J. It does not find: Surname, Fred and Surname, F. (Note: Could be a match, but maybe not.) Without diving deep into C4/Matcher.pm myself, could you give some hints how to get the script to report such ones? On the two patch sets in this report: I would suggest to move the two patches of Julian to another report. This is confusing. Also note that I would not recommend to refactor rebuild_zebra on a report about authorities; you'd better split them up.
(In reply to M. de Rooy from comment #50) > When I am testing dedup-records.pl with Match/100abc on PERSO_NAME, the > script finds two duplicates Surname, C.-J. and Surname, C-J. > It does not find: Surname, Fred and Surname, F. (Note: Could be a match, but > maybe not.) > > Without diving deep into C4/Matcher.pm myself, could you give some hints how > to get the script to report such ones? You would need to write your own matcher that identified Surname, Fred and Surname, F. as duplicates. If you have a 4xx field for Surname, F. in the new record you could use Previous-heading-see-from/100abcd. > On the two patch sets in this report: I would suggest to move the two > patches of Julian to another report. This is confusing. Also note that I > would not recommend to refactor rebuild_zebra on a report about authorities; > you'd better split them up. I agree. The two deduplicators are not mutually exclusive in my opinion, so perhaps it would make sense for me to create a separate bug for Julian's patches with a subject something like "Authority deduplicator for UNIMARC"?
The patch that move indexation subs out of rebuild_zebra.pl was moved to a new bug (Bug 13705)
Comment on attachment 25088 [details] [review] Bug 7419: New script dedup_authorities.pl Moved the dedup_authorities.pl patch to a new bug (bug 13706)
Created attachment 35886 [details] [review] Bug 7419: correct LCCN/see from indexing Rebased on master
Created attachment 35887 [details] [review] Bug 7419: General-purpose record deduplicator Rebased on master
Created attachment 35888 [details] [review] Bug 7419: Add more missing authority indexes Rebased on master
Created attachment 35889 [details] [review] Bug 7419: quiet warnings in deduplicator Rebased on master
I always have the same error while running the script: oAuth error: Unsupported Used attribute (114) Any Bib-1 Examples: perl misc/migration_tools/dedup_records.pl -t -v -a -r --match=he/200a Query: @or @attr 7=1 @attr 4=109 @attr 1=Local-Number 0 @attr 1=Any @attr 4=1 @attr 5=100 @attr 6=3 "X" (where X is the content of 200$a) perl misc/migration_tools/dedup_records.pl -t -v -a -r --match=at/152b Query: @or @attr 7=1 @attr 4=109 @attr 1=Local-Number 0 @attr 1=Any @attr 4=1 @attr 5=100 @attr 6=3 "X" (where X is 152$b) Also, the documentation says we could use --match=he/2..a but it doesn't seem to work (no zebra query is launched) Changing status to Failed QA
I cannot reproduce the error, so I presume I had a bad Zebra configuration. Switching back to Needs Signoff
Created attachment 42972 [details] [review] Bug 7419: Allow wildcards in tagspec Options --match and --check can now contains tagspec like "2.."
Patches apply, but there is are some problems noted by the qa script: FAIL misc/migration_tools/dedup_records.pl OK critic FAIL forbidden patterns forbidden pattern: merge marker (=======) (line 419) OK git manipulation OK pod OK spelling FAIL valid when is experimental
*** This bug has been marked as a duplicate of bug 13706 ***