Bug 7419 - Add authority deduplication script
Summary: Add authority deduplication script
Status: Failed QA
Alias: None
Product: Koha
Classification: Unclassified
Component: MARC Authority data support (show other bugs)
Version: master
Hardware: All All
: P5 - low enhancement (vote)
Target Milestone: ---
Assignee: Jared Camins-Esakov
QA Contact: Testopia
URL:
Whiteboard:
Keywords:
Depends on: 7284 11700
Blocks: 8212
  Show dependency treegraph
 
Reported: 2012-01-07 22:57 UTC by Jared Camins-Esakov
Modified: 2017-10-16 12:33 UTC (History)
20 users (show)

See Also:
Change sponsored?: Sponsored
Patch complexity: ---
Bot Control: Ignore this bug
When did the bot last check this:
Who signed the patch off:
Text to go in the release notes:
Version(s) released in:


Attachments
Bug 7419: Move indexation subs in C4::Search (33.01 KB, patch)
2012-07-27 12:22 UTC, Julian Maurice
Details | Diff | Splinter Review
Bug 7419: New script dedup_authorities.pl (26.92 KB, patch)
2012-07-27 12:23 UTC, Julian Maurice
Details | Diff | Splinter Review
Bug 7419: Move indexation subs in C4::Search (33.69 KB, patch)
2012-08-07 12:20 UTC, Julian Maurice
Details | Diff | Splinter Review
Bug 7419: New script dedup_authorities.pl (26.92 KB, patch)
2012-08-07 12:21 UTC, Julian Maurice
Details | Diff | Splinter Review
Bug 7419: Move indexation subs in C4::ZebraIndex (34.73 KB, patch)
2012-08-09 12:13 UTC, Julian Maurice
Details | Diff | Splinter Review
Bug 7419: Move indexation subs in C4::ZebraIndex (34.73 KB, patch)
2012-08-09 12:15 UTC, Julian Maurice
Details | Diff | Splinter Review
Bug 7419: New script dedup_authorities.pl (26.96 KB, patch)
2012-08-09 12:16 UTC, Julian Maurice
Details | Diff | Splinter Review
Bug 7419: Move indexation subs in C4::ZebraIndex (35.15 KB, patch)
2012-10-26 14:40 UTC, Julian Maurice
Details | Diff | Splinter Review
Bug 7419: New script dedup_authorities.pl (27.08 KB, patch)
2012-10-26 14:41 UTC, Julian Maurice
Details | Diff | Splinter Review
Bug 7419: Move indexation subs in C4::ZebraIndex (35.15 KB, patch)
2012-11-22 10:16 UTC, Julian Maurice
Details | Diff | Splinter Review
Bug 7419: New script dedup_authorities.pl (27.08 KB, patch)
2012-11-22 10:16 UTC, Julian Maurice
Details | Diff | Splinter Review
Bug 7419: Move indexation subs in C4::ZebraIndex (36.49 KB, patch)
2013-04-29 14:18 UTC, Julian Maurice
Details | Diff | Splinter Review
Bug 7419: New script dedup_authorities.pl (26.79 KB, patch)
2013-04-29 14:18 UTC, Julian Maurice
Details | Diff | Splinter Review
Bug 7419: merge sub should retrieve marc records from DB if not given (1.23 KB, patch)
2013-04-29 14:19 UTC, Julian Maurice
Details | Diff | Splinter Review
Sample duplicate MARC21 authorities (15.59 KB, application/octet-stream)
2013-06-06 16:02 UTC, Jared Camins-Esakov
Details
Bug 7419: correct LCCN/see from indexing (17.12 KB, patch)
2013-06-06 16:17 UTC, Jared Camins-Esakov
Details | Diff | Splinter Review
Bug 7419: General-purpose record deduplicator (18.86 KB, patch)
2013-06-06 16:17 UTC, Jared Camins-Esakov
Details | Diff | Splinter Review
Bug 7419: General-purpose record deduplicator (17.66 KB, patch)
2013-09-04 19:01 UTC, Jared Camins-Esakov
Details | Diff | Splinter Review
Bug 7419: Add more missing authority indexes (2.32 KB, patch)
2013-09-04 19:01 UTC, Jared Camins-Esakov
Details | Diff | Splinter Review
Bug 7419: quiet warnings in deduplicator (2.90 KB, patch)
2013-09-04 19:01 UTC, Jared Camins-Esakov
Details | Diff | Splinter Review
works with Julian Maurice patchs (1.71 KB, patch)
2013-10-21 07:40 UTC, stephane.delaune
Details | Diff | Splinter Review
Zconn fix, works with Julian Maurice patchs (1.92 KB, patch)
2013-11-13 14:13 UTC, stephane.delaune
Details | Diff | Splinter Review
Bug 7419: Move indexation subs in C4::ZebraIndex (36.49 KB, patch)
2014-02-06 15:32 UTC, Julian Maurice
Details | Diff | Splinter Review
Bug 7419: New script dedup_authorities.pl (20.31 KB, patch)
2014-02-06 15:33 UTC, Julian Maurice
Details | Diff | Splinter Review
Fix C4::Context::Zconn() (1.96 KB, patch)
2014-02-06 15:34 UTC, Julian Maurice
Details | Diff | Splinter Review
Bug 7419: correct LCCN/see from indexing (17.15 KB, patch)
2015-02-13 08:43 UTC, Julian Maurice
Details | Diff | Splinter Review
Bug 7419: General-purpose record deduplicator (17.65 KB, patch)
2015-02-13 08:44 UTC, Julian Maurice
Details | Diff | Splinter Review
Bug 7419: Add more missing authority indexes (2.31 KB, patch)
2015-02-13 08:44 UTC, Julian Maurice
Details | Diff | Splinter Review
Bug 7419: quiet warnings in deduplicator (2.89 KB, patch)
2015-02-13 08:44 UTC, Julian Maurice
Details | Diff | Splinter Review
Bug 7419: Allow wildcards in tagspec (1.36 KB, patch)
2015-09-30 13:08 UTC, Julian Maurice
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description Jared Camins-Esakov 2012-01-07 22:57:03 UTC
One of the problems that arises when maintaining an authority file long-term is inadvertent duplication of authority records. In order to combat this problem, Koha needs a script for bulk de-duplicating of authority records. The deduplicator should be able to automatically recognize machine-generated authority records, and choose official records. It should also recognize when one record supersedes another based on previous headings. The algorithm will also need to give greater weight to newer records.
Comment 1 Stefano Bargioni 2012-01-13 08:21:42 UTC
Authority deduplication is similar to authority unlink/link. Consider two terms like
"Bargioni, Stefano, 1959-" linked to N1 biblios
and
"Bargioni, Stefano" linked to N2 biblios.
You need to unlink N2 biblios from the second term, i.e. link them to the first one.
Deduplication can be automatic, while the step of choosing term-from and term-to for unlink/link biblios must be manual.
Comment 2 claire.hernandez@biblibre.com 2012-07-11 14:33:20 UTC
Hello Jared. We did a big work on a similar script. I put this ticket on top of rebase tasks. We try to provide a lot of options, few more tests and in few weeks you could test it.
Comment 3 Jared Camins-Esakov 2012-07-12 17:45:42 UTC
Claire,

(In reply to comment #2)
> Hello Jared. We did a big work on a similar script. I put this ticket on top
> of rebase tasks. We try to provide a lot of options, few more tests and in
> few weeks you could test it.

I actually borrowed from a deduplication script that hdl showed me, though I think it may be that you have a newer version. The script I have works for MARC21, and offers very basic options for UNIMARC, but I'd love to integrate my changes into your script (or vice versa) to provide more options.
Comment 4 claire.hernandez@biblibre.com 2012-07-13 07:28:48 UTC
Jared, I am talking about the initial Henri Damien script that evolved with more options and bug fix. You will find the last version (can change again a little but not a lot) here http://git.biblibre.com/?p=koha;a=shortlog;h=refs/heads/ft/dedup_authorities
Comment 5 Julian Maurice 2012-07-27 12:22:35 UTC Comment hidden (obsolete)
Comment 6 Julian Maurice 2012-07-27 12:23:23 UTC Comment hidden (obsolete)
Comment 7 Julian Maurice 2012-08-07 12:20:58 UTC Comment hidden (obsolete)
Comment 8 Julian Maurice 2012-08-07 12:21:27 UTC Comment hidden (obsolete)
Comment 9 Julian Maurice 2012-08-09 12:13:26 UTC Comment hidden (obsolete)
Comment 10 Julian Maurice 2012-08-09 12:15:12 UTC Comment hidden (obsolete)
Comment 11 Julian Maurice 2012-08-09 12:16:29 UTC Comment hidden (obsolete)
Comment 12 Julian Maurice 2012-08-09 12:18:09 UTC
Indexation subs are now in a new module C4::ZebraIndex, instead of C4::Search (better to not "overload" C4::Search).
Comment 13 Pascale Nalon 2012-10-23 13:27:02 UTC
I tried to test this patch with the sandbox system, but the system send back these errors :
The sandbox you've requested is now ready. The patches attached to bugzilla 7419 have been applied, and the result is
<h1>Something went wrong !</h1>Applying: Bug 7419: Move indexation subs in C4::ZebraIndex<br/>
Using index info to reconstruct a base tree...<br/>
Falling back to patching base and 3-way merge...<br/>
Auto-merging misc/migration_tools/rebuild_zebra.pl<br/>
CONFLICT (content): Merge conflict in misc/migration_tools/rebuild_zebra.pl<br/>
Failed to merge in the changes.<br/>
Patch failed at 0001 Bug 7419: Move indexation subs in C4::ZebraIndex<br/>
When you have resolved this problem run git am --resolved.<br/>
If you would prefer to skip this patch, instead run git am --skip.<br/>
To restore the original branch and stop patching run git am --abort.<br/>
Bug 7419 - Add authority deduplication script<br/>
<br/>
Bug 7419: Move indexation subs in C4::ZebraIndex<br/>
Apply? [yn] <br/>
Patch left in /tmp/Bug-7419-Move-indexation-subs-in-C4ZebraIndex-fNv1EB.patch<br/>

Thanks for fixing it !
Comment 14 Julian Maurice 2012-10-26 14:40:46 UTC Comment hidden (obsolete)
Comment 15 Julian Maurice 2012-10-26 14:41:08 UTC Comment hidden (obsolete)
Comment 16 Julian Maurice 2012-10-26 14:42:18 UTC
Pascale, you should be able to apply these patches now.
Comment 17 Christophe Croullebois 2012-11-12 15:36:43 UTC
Hello Julian,
if we don't use --verbose we have errors => Use of uninitialized value $verbose in numeric ge (>=) at ./dedup_authorities.pl line 126.
the same for lines 143,145,151.
For line 126 : if ($verbose >= 2) should be : if ($verbose AND $verbose >= 2)
thx
Comment 18 Christophe Croullebois 2012-11-15 15:18:45 UTC
Julian, it doesn't apply :

Applying: Bug 7419: Move indexation subs in C4::ZebraIndex
Using index info to reconstruct a base tree...
Falling back to patching base and 3-way merge...
Auto-merging misc/migration_tools/rebuild_zebra.pl
CONFLICT (content): Merge conflict in misc/migration_tools/rebuild_zebra.pl
Failed to merge in the changes.
Patch failed at 0001 Bug 7419: Move indexation subs in C4::ZebraIndex
Comment 19 Julian Maurice 2012-11-22 10:16:27 UTC Comment hidden (obsolete)
Comment 20 Julian Maurice 2012-11-22 10:16:36 UTC Comment hidden (obsolete)
Comment 21 Julian Maurice 2012-11-22 10:16:59 UTC
Patches rebased on master
Comment 22 Bernardo Gonzalez Kriegel 2013-02-23 01:44:27 UTC
Patch still applies.
Minor POD errors on script.
Missing license statement on script.

The problem is, I can't figure how to test it. And I think this could be a very valuable tool.

If you could you add some test cases, or an accompanying .txt that explains better the syntax it would be easier to test and use.
Comment 23 Julian Maurice 2013-03-21 13:51:38 UTC
Just retested it with Leïla and found an error

merging 25894 into 1198 failed :Can't call method "fields" on an undefined value at /home/koha/src/C4/AuthoritiesMarc.pm line 1392.

Patch needs some work.
Comment 24 Julian Maurice 2013-04-29 14:18:21 UTC Comment hidden (obsolete)
Comment 25 Julian Maurice 2013-04-29 14:18:41 UTC Comment hidden (obsolete)
Comment 26 Julian Maurice 2013-04-29 14:19:30 UTC Comment hidden (obsolete)
Comment 27 Jared Camins-Esakov 2013-06-05 13:28:17 UTC
I tested this script and found that it did not work at all with MARC21 because of the way it was retrieving authid. I am working on a counter-patch which I hope to upload tomorrow which incorporates the matching options from your script but should work properly with both UNIMARC and MARC21.
Comment 28 Jared Camins-Esakov 2013-06-06 16:02:31 UTC
Created attachment 18712 [details]
Sample duplicate MARC21 authorities
Comment 29 Jared Camins-Esakov 2013-06-06 16:17:21 UTC Comment hidden (obsolete)
Comment 30 Jared Camins-Esakov 2013-06-06 16:17:30 UTC Comment hidden (obsolete)
Comment 31 Jared Camins-Esakov 2013-06-06 16:18:42 UTC
The final two patches comprise my counter-patch, which will handle both MARC21 and UNIMARC data (though the sample data provided on this bug is MARC21-only).
Comment 32 Jared Camins-Esakov 2013-08-22 14:21:54 UTC
Several people have asked about a test plan, so I am repeating my test plan from comment 30, since that comment is so long it can be hard to find:
To test using the sample records attached to the bug (MARC21 only):
1) Apply patches.
2) Import the sample records (sampleauths.mrc) file into Koha and
   make sure that they get indexed (by waiting until rebuild_zebra.pl
   runs automatically or by running rebuild_zebra.pl -a -z [-x] manually)
3) Deduplicate based on LCCN (replace {FIRSTAUTH} with the authid of the
   first imported record; on my system that number is 367123668; this is
   not actually necessary, but if you have a lot of authority records,
   the process could take quite a while otherwise):

   > misc/migration_tools/dedup_records.pl -t -v -a \
     -l "authid >= {FIRSTAUTH}" -r -m "lc-card/010a" -s date

4) Check that you have 21 duplicate records replaced from amongst the
   new records (you will quite possibly have more than 21 duplicate
   records reported, depending on whether you have any of these
   authorities already, but you should have at least the 21).
5) Deduplicate based on genre heading, preferring Library of Congress
   authorities to local authorities:

   > misc/migration_tools/dedup_records.pl -t -v -a \
     -l "authid >= {FIRSTAUTH}" -r -m "he/155a" -s "source=DLC" -s date

6) Check that you have 3 duplicate records replaced from amongst the new
   records.
7) Sign off.
Comment 33 Mason James 2013-08-27 01:56:48 UTC
hi Tomas, 
could you update the ticket with info about your testing of this bug

Jared mentioned you found some problems?
Comment 34 Tomás Cohen Arazi 2013-08-27 16:24:24 UTC
(In reply to Mason James from comment #33)
> hi Tomas, 
> could you update the ticket with info about your testing of this bug
> 
> Jared mentioned you found some problems?

The sample data and test plan didn't match, there was a problem with 155$a vs. 150$a (the second proposed test) and some warnings that needed to be addressed.
Comment 35 Jared Camins-Esakov 2013-09-04 19:01:24 UTC Comment hidden (obsolete)
Comment 36 Jared Camins-Esakov 2013-09-04 19:01:37 UTC Comment hidden (obsolete)
Comment 37 Jared Camins-Esakov 2013-09-04 19:01:49 UTC Comment hidden (obsolete)
Comment 38 Jared Camins-Esakov 2013-09-04 19:03:41 UTC
Note that the new test plan does require you to be using QueryParser (and an updated etc/searchengine/queryparser.yaml).
Comment 39 I'm just a bot 2013-09-20 09:52:21 UTC
Applying: Bug 7284: Authority matching improvements
Using index info to reconstruct a base tree...
M	C4/AuthoritiesMarc.pm
M	C4/Biblio.pm
M	C4/Heading.pm
M	C4/Heading/MARC21.pm
M	C4/Search.pm
M	acqui/addorderiso2709.pl
M	acqui/neworderempty.pl
M	authorities/auth_finder.pl
M	cataloguing/addbiblio.pl
M	etc/zebradb/authorities/etc/bib1.att
M	etc/zebradb/marc_defs/marc21/authorities/authority-koha-indexdefs.xml
M	etc/zebradb/marc_defs/marc21/authorities/authority-zebra-indexdefs.xsl
A	etc/zebradb/marc_defs/marc21/authorities/koha-indexdefs-to-zebra.xsl
M	etc/zebradb/marc_defs/unimarc/authorities/record.abs
M	installer/data/mysql/sysprefs.sql
M	installer/data/mysql/updatedatabase.pl
M	koha-tmpl/intranet-tmpl/prog/en/includes/auth-finder-search.inc
M	koha-tmpl/intranet-tmpl/prog/en/modules/admin/preferences/authorities.pref
M	misc/link_bibs_to_authorities.pl
A	t/Heading_MARC21.t
A	t/db_dependent/lib/KohaTest/AuthoritiesMarc.pm
Falling back to patching base and 3-way merge...
CONFLICT (modify/delete): t/db_dependent/lib/KohaTest/AuthoritiesMarc.pm deleted in HEAD and modified in Bug 7284: Authority matching improvements. Version Bug 7284: Authority matching improvements of t/db_dependent/lib/KohaTest/AuthoritiesMarc.pm left in tree.
Auto-merging t/db_dependent/Linker_FirstMatch.t
CONFLICT (add/add): Merge conflict in t/db_dependent/Linker_FirstMatch.t
Auto-merging t/Barcodes_annual.t
CONFLICT (content): Merge conflict in t/Barcodes_annual.t
Auto-merging misc/link_bibs_to_authorities.pl
CONFLICT (content): Merge conflict in misc/link_bibs_to_authorities.pl
Auto-merging koha-tmpl/intranet-tmpl/prog/en/modules/admin/preferences/authorities.pref
CONFLICT (content): Merge conflict in koha-tmpl/intranet-tmpl/prog/en/modules/admin/preferences/authorities.pref
Auto-merging koha-tmpl/intranet-tmpl/prog/en/includes/auth-finder-search.inc
CONFLICT (content): Merge conflict in koha-tmpl/intranet-tmpl/prog/en/includes/auth-finder-search.inc
Auto-merging installer/data/mysql/updatedatabase.pl
CONFLICT (content): Merge conflict in installer/data/mysql/updatedatabase.pl
Auto-merging installer/data/mysql/sysprefs.sql
CONFLICT (content): Merge conflict in installer/data/mysql/sysprefs.sql
Auto-merging C4/Search.pm
CONFLICT (content): Merge conflict in C4/Search.pm
Auto-merging C4/Linker/Default.pm
CONFLICT (add/add): Merge conflict in C4/Linker/Default.pm
Auto-merging C4/Heading/UNIMARC.pm
CONFLICT (add/add): Merge conflict in C4/Heading/UNIMARC.pm
Auto-merging C4/Heading.pm
CONFLICT (content): Merge conflict in C4/Heading.pm
Auto-merging C4/Biblio.pm
CONFLICT (content): Merge conflict in C4/Biblio.pm
Auto-merging C4/AuthoritiesMarc.pm
CONFLICT (content): Merge conflict in C4/AuthoritiesMarc.pm
Patch failed at 0001 Bug 7284: Authority matching improvements
The copy of the patch that failed is found in:
   /home/christopher/git/koha/.git/rebase-apply/patch
When you have resolved this problem, run "git am --continue".
If you prefer to skip this patch, run "git am --skip" instead.
To restore the original branch and stop patching, run "git am --abort".
Comment 40 Jared Camins-Esakov 2013-09-20 12:38:29 UTC
Please note that the final four patches comprise a counter-patch to the first three, which are not MARC21-compatible.
Comment 41 stephane.delaune 2013-10-21 07:40:10 UTC Comment hidden (obsolete)
Comment 42 stephane.delaune 2013-10-21 07:48:22 UTC
Hi, I test this to dedup authorities. I begin with the 4 Jared patches ( attachment.cgi?id= 18713 20783 20784 20785 ) : this dev is more complete but didn't work to dedup my authorities.

So I tested The 3 Julian's patches + 1 mine ( attachment.cgi?id= 17790 17791 17792 22150 ) : this dev is less complete (take only authorities) but works to dedup my authorities
Comment 43 Jared Camins-Esakov 2013-10-21 12:33:35 UTC
(In reply to stephane.delaune from comment #42)
> Hi, I test this to dedup authorities. I begin with the 4 Jared patches (
> attachment.cgi?id= 18713 20783 20784 20785 ) : this dev is more complete but
> didn't work to dedup my authorities.

What results did you see?
Comment 44 stephane.delaune 2013-11-13 14:11:00 UTC
(In reply to Jared Camins-Esakov from comment #43)
> (In reply to stephane.delaune from comment #42)
> > Hi, I test this to dedup authorities. I begin with the 4 Jared patches (
> > attachment.cgi?id= 18713 20783 20784 20785 ) : this dev is more complete but
> > didn't work to dedup my authorities.
> 
> What results did you see?

I had launched with the command line:
~/src/misc/migration_tools/dedup_records.pl -t -r -v -a -l "authtypecode='SNG'" -m 'at/152b##he/2..axyz'
I just noticed a lot of error messages: "oAuth error: Unsupported Use attribute (114) Any Bib-1"
However, I did not notice anything more and I'm not able to reproduce the same test conditions
Comment 45 stephane.delaune 2013-11-13 14:13:14 UTC Comment hidden (obsolete)
Comment 46 Julian Maurice 2014-02-06 15:32:53 UTC Comment hidden (obsolete)
Comment 47 Julian Maurice 2014-02-06 15:33:59 UTC Comment hidden (obsolete)
Comment 48 Julian Maurice 2014-02-06 15:34:28 UTC Comment hidden (obsolete)
Comment 49 Julian Maurice 2014-02-06 16:17:01 UTC
Comment on attachment 25089 [details] [review]
Fix C4::Context::Zconn()

Obsolete patch on Zconn (new bug 11701)
Comment 50 Marcel de Rooy 2014-09-16 07:36:58 UTC
When I am testing dedup-records.pl with Match/100abc on PERSO_NAME, the script finds two duplicates Surname, C.-J. and Surname, C-J.
It does not find: Surname, Fred and Surname, F. (Note: Could be a match, but maybe not.)

Without diving deep into C4/Matcher.pm myself, could you give some hints how to get the script to report such ones?

On the two patch sets in this report: I would suggest to move the two patches of Julian to another report. This is confusing. Also note that I would not recommend to refactor rebuild_zebra on a report about authorities; you'd better split them up.
Comment 51 Jared Camins-Esakov 2014-10-09 13:24:36 UTC
(In reply to M. de Rooy from comment #50)
> When I am testing dedup-records.pl with Match/100abc on PERSO_NAME, the
> script finds two duplicates Surname, C.-J. and Surname, C-J.
> It does not find: Surname, Fred and Surname, F. (Note: Could be a match, but
> maybe not.)
> 
> Without diving deep into C4/Matcher.pm myself, could you give some hints how
> to get the script to report such ones?

You would need to write your own matcher that identified Surname, Fred and Surname, F. as duplicates. If you have a 4xx field for Surname, F. in the new record you could use Previous-heading-see-from/100abcd.

> On the two patch sets in this report: I would suggest to move the two
> patches of Julian to another report. This is confusing. Also note that I
> would not recommend to refactor rebuild_zebra on a report about authorities;
> you'd better split them up.

I agree. The two deduplicators are not mutually exclusive in my opinion, so perhaps it would make sense for me to create a separate bug for Julian's patches with a subject something like "Authority deduplicator for UNIMARC"?
Comment 52 Julian Maurice 2015-02-13 08:37:03 UTC
The patch that move indexation subs out of rebuild_zebra.pl was moved to a new bug (Bug 13705)
Comment 53 Julian Maurice 2015-02-13 08:41:09 UTC
Comment on attachment 25088 [details] [review]
Bug 7419: New script dedup_authorities.pl

Moved the dedup_authorities.pl patch to a new bug (bug 13706)
Comment 54 Julian Maurice 2015-02-13 08:43:51 UTC
Created attachment 35886 [details] [review]
Bug 7419: correct LCCN/see from indexing

Rebased on master
Comment 55 Julian Maurice 2015-02-13 08:44:12 UTC
Created attachment 35887 [details] [review]
Bug 7419: General-purpose record deduplicator

Rebased on master
Comment 56 Julian Maurice 2015-02-13 08:44:26 UTC
Created attachment 35888 [details] [review]
Bug 7419: Add more missing authority indexes

Rebased on master
Comment 57 Julian Maurice 2015-02-13 08:44:39 UTC
Created attachment 35889 [details] [review]
Bug 7419: quiet warnings in deduplicator

Rebased on master
Comment 58 Julian Maurice 2015-02-13 08:58:38 UTC
I always have the same error while running the script:

oAuth error: Unsupported Used attribute (114) Any Bib-1

Examples:

perl misc/migration_tools/dedup_records.pl -t -v -a -r --match=he/200a
Query: @or @attr 7=1 @attr 4=109 @attr 1=Local-Number 0  @attr 1=Any  @attr 4=1  @attr 5=100 @attr 6=3 "X" (where X is the content of 200$a)

perl misc/migration_tools/dedup_records.pl -t -v -a -r --match=at/152b
Query: @or @attr 7=1 @attr 4=109 @attr 1=Local-Number 0  @attr 1=Any  @attr 4=1  @attr 5=100 @attr 6=3 "X" (where X is 152$b)

Also, the documentation says we could use --match=he/2..a but it doesn't seem to work (no zebra query is launched)

Changing status to Failed QA
Comment 59 Julian Maurice 2015-09-30 13:03:03 UTC
I cannot reproduce the error, so I presume I had a bad Zebra configuration.
Switching back to Needs Signoff
Comment 60 Julian Maurice 2015-09-30 13:08:19 UTC
Created attachment 42972 [details] [review]
Bug 7419: Allow wildcards in tagspec

Options --match and --check can now contains tagspec like "2.."
Comment 61 Katrin Fischer 2017-05-01 19:55:55 UTC
Patches apply, but there is are some problems noted by the qa script:

 FAIL	misc/migration_tools/dedup_records.pl
   OK	  critic
   FAIL	  forbidden patterns
		forbidden pattern: merge marker (=======) (line 419)
   OK	  git manipulation
   OK	  pod
   OK	  spelling
   FAIL	  valid
		when is experimental