Bug 16493

Summary: acq matching on title and author
Product: Koha Reporter: Nicole C. Engard <nengard>
Component: AcquisitionsAssignee: Kyle M Hall <kyle>
Status: CLOSED FIXED QA Contact: Testopia <testopia>
Severity: critical    
Priority: P5 - low CC: amy, fridolin.somers, jonathan.druart, julian.maurice, katrin.fischer, koha, kyle, mtj, nicole
Version: unspecified   
Hardware: All   
OS: All   
See Also: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18482
Change sponsored?: --- Patch complexity: Small patch
Documentation contact: Documentation submission:
Text to go in the release notes:
Version(s) released in:
Bug Depends on: 12074    
Bug Blocks:    
Attachments: Bug 16493 - acq matching on title and author
Bug 16493 - acq matching on title and author
Bug 16493 - acq matching on title and author
Bug 16493 [QA Followup] - Restore title and author match as an option, make it the default
Bug 16493 - acq matching on title and author
Bug 16493 [QA Followup] - Restore title and author match as an option, make it the default
Bug 16493 - acq matching on title and author
Bug 16493 [QA Followup] - Restore title and author match as an option, make it the default
Bug 16493: acq matching on title and author
Bug 16493: [QA Followup] Restore title and author match as an option, make it the default

Description Nicole C. Engard 2016-05-10 17:14:39 UTC
When you order from a staged file you're getting duplicate warnings that are inaccurate.  For example, when you order a file of 50 DVDs for example and they don't have ISBNs they're matching on the books. And then you have to order them one by one.

----------

[11:42]  <cait> it seems to use FindDuplicate
[11:43]  <cait> http://git.koha-community.org/gitweb/?p=koha.git;a=blob;f=C4/Search.pm;hb=47dc39997198126be702c13c58053e793c609f90#l83
[11:44]  <cait> i might be misreading it, but looks like isbn when there is one - if there is none, title and author

-----------

This shouldn't happen, it's a big deal in libraries that do a lot of ordering and is just plain wrong based on cataloging rules.  Is there any way to override this without doing them one by one or a way to make it use the matching rules instead?
Comment 1 Jonathan Druart 2016-05-11 11:49:18 UTC
What do you suggest?
Do you think bug 12704 should be reverted because it's not valid (matching rules should be used instead)?
Comment 2 Kyle M Hall 2016-05-11 13:09:31 UTC
(In reply to Jonathan Druart from comment #1)
> What do you suggest?
> Do you think bug 12704 should be reverted because it's not valid (matching
> rules should be used instead)?

Jonathan, is that the right bug?
Comment 3 Jonathan Druart 2016-05-11 13:39:29 UTC
(In reply to Kyle M Hall from comment #2)
> (In reply to Jonathan Druart from comment #1)
> > What do you suggest?
> > Do you think bug 12704 should be reverted because it's not valid (matching
> > rules should be used instead)?
> 
> Jonathan, is that the right bug?

Not at all! :)
It is bug 12074.
Comment 4 Kyle M Hall 2016-05-11 14:26:58 UTC Comment hidden (obsolete)
Comment 5 Nicole C. Engard 2016-05-11 14:45:44 UTC Comment hidden (obsolete)
Comment 6 Koha Team University Lyon 3 2016-05-13 13:11:10 UTC
Hi,
I'm not sure to understand so I need more explanations :)
Now, since BZ 12074 was pushed to master, when someone adds a full batch from a staged file, there's an alert for duplicates. But this alert is based on what is found by FindDuplicate.
Could we change Findduplicate as it can find duplicate on another field than ISBN rather to avoid to use findDuplicate ?

I have no preference, all what I want is that we can find duplicates for books, DVDs or whatever materials when we're importing a batch records.

But, is it better to do it without FindDuplicates ?
Sonia
Comment 7 Nicole C. Engard 2016-05-13 13:14:29 UTC
(In reply to Koha Team Lyon 3 from comment #6)
> Hi,
> I'm not sure to understand so I need more explanations :)
> Now, since BZ 12074 was pushed to master, when someone adds a full batch
> from a staged file, there's an alert for duplicates. But this alert is based
> on what is found by FindDuplicate.
> Could we change Findduplicate as it can find duplicate on another field than
> ISBN rather to avoid to use findDuplicate ?
> 
> I have no preference, all what I want is that we can find duplicates for
> books, DVDs or whatever materials when we're importing a batch records.
> 
> But, is it better to do it without FindDuplicates ?
> Sonia

Sonia,

The problem was that FindDuplicates was hardcoded and the library had no way to change that.  So the following scenario is just one example of where FindDuplicates fails libraries:

1. Stage DVD Marc Records
2. Use a rule to match on 024 (the identifier on many DVDs)
3. Find duplicates
4. Go to Acq and add staged file to basket
5. Koha says there are duplicates but there aren't instead it's matching on Title & Author and matching DVDs to books because the DVDs don't have ISBNs

By using this patch we allow for libraries to choose how to find duplicates instead of forcing them to use a hard coded script that they have no control over.

Nicole
Comment 8 Koha Team University Lyon 3 2016-05-13 13:23:52 UTC
Ok, Thanks Nicole, I understand and I share the "setting principle" that allow librarians to choose the key to deduplicate.
I will test the patch with unimarc file to see if it's ok for us.
Comment 9 Jonathan Druart 2016-05-21 08:20:06 UTC
(In reply to Koha Team Lyon 3 from comment #8)
> Ok, Thanks Nicole, I understand and I share the "setting principle" that
> allow librarians to choose the key to deduplicate.
> I will test the patch with unimarc file to see if it's ok for us.

Do you think you will have time to test it before the release?
It's a critical one and we need it fixed soon.
Comment 10 Koha Team University Lyon 3 2016-05-21 09:04:16 UTC
Ok, I can test it on monday. Is it too late ?
Comment 11 Jonathan Druart 2016-05-21 09:27:02 UTC
(In reply to Koha Team Lyon 3 from comment #10)
> Ok, I can test it on monday. Is it too late ?

No, Koha 16.05 will be released on May 26th :)
Comment 12 Koha Team University Lyon 3 2016-05-23 14:46:28 UTC
Hi,
I have just finished my tests. The patch works as described.
I'm a little worried because I'm afraid that some of my colleagues forget to choose a matching rule when they add orders from a batch file. It will be a regression for them as they have nothing to choose by now to have a deduplicationw to have a deduplication.

So, I'm wondering if it would be possible to have a default matching rule selected when we add a orders from a batch file. Or, perhaps it would be better to keep a deduplication by FindDuplicate when no matching rules is chosen and to switch to a matching rule if the user choose one.

What do you think about it ?

PS : It's not related to this patch but I have some difficulties to set the matching rules. In the documentation, there's :
Threshold of 1000
Match Point on 020$a 1000
Match Point on 022$a 1000
In the example above, a match on either the 020$a or the 022$a will result in a successful match.
When I test this patch, I have to put a number in the match point greater than the threshold, if I put a number equal to the thershold, there's no deduplication alert...
And I try to put 2 matchpoints with a result greater to the threshold and if none of the two is greater than the threshold, there's no deduplication alert.
Comment 13 Katrin Fischer 2016-06-08 06:29:51 UTC
I think Sonia has some good points there - Kyle, could you take a look please?
Comment 14 Kyle M Hall 2016-06-08 10:54:48 UTC Comment hidden (obsolete)
Comment 15 Kyle M Hall 2016-06-08 10:54:55 UTC Comment hidden (obsolete)
Comment 16 Jonathan Druart 2016-06-11 13:35:39 UTC
Sonia, could you test it again with the follow-up please?
Comment 17 Koha Team University Lyon 3 2016-06-13 11:15:51 UTC
I have an error when I try to import the batch with the default option "Title and Author"

Software error:

Undefined subroutine &main::FindDuplicate called at /home/koha/src/acqui/addorderiso2709.pl line 163.
For help, please send mail to the webmaster (webmaster@test4-sandbox), giving this error message and the time and date of the error.

Sonia
Comment 18 Kyle M Hall 2016-06-27 16:07:22 UTC
Created attachment 52903 [details] [review]
Bug 16493 - acq matching on title and author

When you order from a staged file you're getting duplicate warnings that
are inaccurate.  For example, when you order a file of 50 DVDs for
example and they don't have ISBNs they're matching on the books. And
then you have to order them one by one.

This patch replaces the use of FindDuplicates with Koha's match point
system. This means you can select from the same match points defined
and used in the batch record importer, or you can opt to skip matching
altogether!

Test Plan:
1) Import a record with a title, isbn and author.
2) Delete the from the record and stage it again
3) Attempt to add it to a basket via the staged record
4) You should note the gives you the "No records imported" message
5) Apply this patch
6) Create a matcher for ISBN
7) Create a matcher for Author/Title
8) Attempt to add the record to your basket using the ISBN matcher
8) Koha should find no match and import the record to the basket
9) Stage the record again
10) Attempt to add the record to your basket using the Title/Author matcher
11) You should recieve the "No records imported" message.

Signed-off-by: Barbara Fondren <bfondren@roundrocktexas.gov>

Signed-off-by: Nicole C Engard <nengard@bywatersolutions.com>
Comment 19 Kyle M Hall 2016-06-27 16:07:29 UTC
Created attachment 52904 [details] [review]
Bug 16493 [QA Followup] - Restore title and author match as an option, make it the default
Comment 20 Kyle M Hall 2016-06-27 16:08:09 UTC
(In reply to Koha Team Lyon 3 from comment #17)
> I have an error when I try to import the batch with the default option
> "Title and Author"
> 
> Software error:
> 
> Undefined subroutine &main::FindDuplicate called at
> /home/koha/src/acqui/addorderiso2709.pl line 163.
> For help, please send mail to the webmaster (webmaster@test4-sandbox),
> giving this error message and the time and date of the error.
> 
> Sonia

My bad, I forgot to re-add the use line for FindDuplicate. Would you please retest? Thanks!
Comment 21 Mark Tompsett 2016-07-17 02:11:08 UTC
Comment on attachment 52904 [details] [review]
Bug 16493 [QA Followup] - Restore title and author match as an option, make it the default

Review of attachment 52904 [details] [review]:
-----------------------------------------------------------------

::: acqui/addorderiso2709.pl
@@ +160,4 @@
>          # 1st insert the biblio, or find it through matcher
>          unless ( $biblionumber ) {
>              if ($matcher_id) {
> +                if ( $matcher_id eq '_TITLE_AUTHOR_' ) {

Perhaps a $duplinbatch = undef; above this might be in order since this is in a loop?
Comment 22 Koha Team University Lyon 3 2016-10-10 15:13:35 UTC
Hi,
I've tested it on a bywaters sandbox (they work !)
But I don't manage to have correct deduplication.
The default choice for deduplication "author and title" don't actually deduplicate by author and title.
- I have imported a file in a basket
- I have changed the original file to remove ISBN
- I have imported this second file with the "title and author" matching checked : all the titles have been added to the basket, though title and author are exactly the same.
- I have tried to re-import the first fil with the ISBN with "title and author" matching : no record have been imported : the deduplication works.

I don't know what is exactly behind the matching key "title and author" ?  Perhaps should we call it "Title/author/ISBN" ?

Sonia
Comment 23 Kyle M Hall 2016-11-03 16:54:33 UTC
Created attachment 57139 [details] [review]
Bug 16493 - acq matching on title and author

When you order from a staged file you're getting duplicate warnings that
are inaccurate.  For example, when you order a file of 50 DVDs for
example and they don't have ISBNs they're matching on the books. And
then you have to order them one by one.

This patch replaces the use of FindDuplicates with Koha's match point
system. This means you can select from the same match points defined
and used in the batch record importer, or you can opt to skip matching
altogether!

Test Plan:
1) Import a record with a title, isbn and author.
2) Delete the from the record and stage it again
3) Attempt to add it to a basket via the staged record
4) You should note the gives you the "No records imported" message
5) Apply this patch
6) Create a matcher for ISBN
7) Create a matcher for Author/Title
8) Attempt to add the record to your basket using the ISBN matcher
8) Koha should find no match and import the record to the basket
9) Stage the record again
10) Attempt to add the record to your basket using the Title/Author matcher
11) You should recieve the "No records imported" message.

Signed-off-by: Barbara Fondren <bfondren@roundrocktexas.gov>

Signed-off-by: Nicole C Engard <nengard@bywatersolutions.com>
Comment 24 Kyle M Hall 2016-11-03 16:54:45 UTC
Created attachment 57140 [details] [review]
Bug 16493 [QA Followup] - Restore title and author match as an option, make it the default
Comment 25 Kyle M Hall 2016-11-03 16:56:20 UTC
Good Catch Sonia, there was a bug in my followup that was causing it to not actually use the Title/Author matching. Please give it another try if you like!

(In reply to Koha Team Lyon 3 from comment #22)
> Hi,
> I've tested it on a bywaters sandbox (they work !)
> But I don't manage to have correct deduplication.
> The default choice for deduplication "author and title" don't actually
> deduplicate by author and title.
> - I have imported a file in a basket
> - I have changed the original file to remove ISBN
> - I have imported this second file with the "title and author" matching
> checked : all the titles have been added to the basket, though title and
> author are exactly the same.
> - I have tried to re-import the first fil with the ISBN with "title and
> author" matching : no record have been imported : the deduplication works.
> 
> I don't know what is exactly behind the matching key "title and author" ? 
> Perhaps should we call it "Title/author/ISBN" ?
> 
> Sonia
Comment 26 Tomás Cohen Arazi 2016-11-04 13:54:36 UTC
Created attachment 57198 [details] [review]
Bug 16493: acq matching on title and author

When you order from a staged file you're getting duplicate warnings that
are inaccurate.  For example, when you order a file of 50 DVDs for
example and they don't have ISBNs they're matching on the books. And
then you have to order them one by one.

This patch replaces the use of FindDuplicates with Koha's match point
system. This means you can select from the same match points defined
and used in the batch record importer, or you can opt to skip matching
altogether!

Test Plan:
1) Import a record with a title, isbn and author.
2) Delete the from the record and stage it again
3) Attempt to add it to a basket via the staged record
4) You should note the gives you the "No records imported" message
5) Apply this patch
6) Create a matcher for ISBN
7) Create a matcher for Author/Title
8) Attempt to add the record to your basket using the ISBN matcher
8) Koha should find no match and import the record to the basket
9) Stage the record again
10) Attempt to add the record to your basket using the Title/Author matcher
11) You should recieve the "No records imported" message.

Signed-off-by: Barbara Fondren <bfondren@roundrocktexas.gov>
Signed-off-by: Nicole C Engard <nengard@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Comment 27 Tomás Cohen Arazi 2016-11-04 13:55:15 UTC
Created attachment 57199 [details] [review]
Bug 16493: [QA Followup] Restore title and author match as an option, make it the default

Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Comment 28 Kyle M Hall 2016-11-04 14:08:38 UTC
Pushed to master for 16.11!
Comment 29 Julian Maurice 2016-11-30 12:03:42 UTC
Pushed to 3.22.x, will be in 3.22.13
Comment 30 Mason James 2016-12-08 02:56:29 UTC
Pushed to 16.05.x, will be in 16.05.06
Comment 31 Nick Clemens 2017-07-19 17:47:45 UTC
*** Bug 16495 has been marked as a duplicate of this bug. ***