When you order from a staged file you're getting duplicate warnings that are inaccurate. For example, when you order a file of 50 DVDs for example and they don't have ISBNs they're matching on the books. And then you have to order them one by one. ---------- [11:42] <cait> it seems to use FindDuplicate [11:43] <cait> http://git.koha-community.org/gitweb/?p=koha.git;a=blob;f=C4/Search.pm;hb=47dc39997198126be702c13c58053e793c609f90#l83 [11:44] <cait> i might be misreading it, but looks like isbn when there is one - if there is none, title and author ----------- This shouldn't happen, it's a big deal in libraries that do a lot of ordering and is just plain wrong based on cataloging rules. Is there any way to override this without doing them one by one or a way to make it use the matching rules instead?
What do you suggest? Do you think bug 12704 should be reverted because it's not valid (matching rules should be used instead)?
(In reply to Jonathan Druart from comment #1) > What do you suggest? > Do you think bug 12704 should be reverted because it's not valid (matching > rules should be used instead)? Jonathan, is that the right bug?
(In reply to Kyle M Hall from comment #2) > (In reply to Jonathan Druart from comment #1) > > What do you suggest? > > Do you think bug 12704 should be reverted because it's not valid (matching > > rules should be used instead)? > > Jonathan, is that the right bug? Not at all! :) It is bug 12074.
Created attachment 51439 [details] [review] Bug 16493 - acq matching on title and author When you order from a staged file you're getting duplicate warnings that are inaccurate. For example, when you order a file of 50 DVDs for example and they don't have ISBNs they're matching on the books. And then you have to order them one by one. This patch replaces the use of FindDuplicates with Koha's match point system. This means you can select from the same match points defined and used in the batch record importer, or you can opt to skip matching altogether! Test Plan: 1) Import a record with a title, isbn and author. 2) Delete the from the record and stage it again 3) Attempt to add it to a basket via the staged record 4) You should note the gives you the "No records imported" message 5) Apply this patch 6) Create a matcher for ISBN 7) Create a matcher for Author/Title 8) Attempt to add the record to your basket using the ISBN matcher 8) Koha should find no match and import the record to the basket 9) Stage the record again 10) Attempt to add the record to your basket using the Title/Author matcher 11) You should recieve the "No records imported" message.
Created attachment 51441 [details] [review] Bug 16493 - acq matching on title and author When you order from a staged file you're getting duplicate warnings that are inaccurate. For example, when you order a file of 50 DVDs for example and they don't have ISBNs they're matching on the books. And then you have to order them one by one. This patch replaces the use of FindDuplicates with Koha's match point system. This means you can select from the same match points defined and used in the batch record importer, or you can opt to skip matching altogether! Test Plan: 1) Import a record with a title, isbn and author. 2) Delete the from the record and stage it again 3) Attempt to add it to a basket via the staged record 4) You should note the gives you the "No records imported" message 5) Apply this patch 6) Create a matcher for ISBN 7) Create a matcher for Author/Title 8) Attempt to add the record to your basket using the ISBN matcher 8) Koha should find no match and import the record to the basket 9) Stage the record again 10) Attempt to add the record to your basket using the Title/Author matcher 11) You should recieve the "No records imported" message. Signed-off-by: Barbara Fondren <bfondren@roundrocktexas.gov> Signed-off-by: Nicole C Engard <nengard@bywatersolutions.com>
Hi, I'm not sure to understand so I need more explanations :) Now, since BZ 12074 was pushed to master, when someone adds a full batch from a staged file, there's an alert for duplicates. But this alert is based on what is found by FindDuplicate. Could we change Findduplicate as it can find duplicate on another field than ISBN rather to avoid to use findDuplicate ? I have no preference, all what I want is that we can find duplicates for books, DVDs or whatever materials when we're importing a batch records. But, is it better to do it without FindDuplicates ? Sonia
(In reply to Koha Team Lyon 3 from comment #6) > Hi, > I'm not sure to understand so I need more explanations :) > Now, since BZ 12074 was pushed to master, when someone adds a full batch > from a staged file, there's an alert for duplicates. But this alert is based > on what is found by FindDuplicate. > Could we change Findduplicate as it can find duplicate on another field than > ISBN rather to avoid to use findDuplicate ? > > I have no preference, all what I want is that we can find duplicates for > books, DVDs or whatever materials when we're importing a batch records. > > But, is it better to do it without FindDuplicates ? > Sonia Sonia, The problem was that FindDuplicates was hardcoded and the library had no way to change that. So the following scenario is just one example of where FindDuplicates fails libraries: 1. Stage DVD Marc Records 2. Use a rule to match on 024 (the identifier on many DVDs) 3. Find duplicates 4. Go to Acq and add staged file to basket 5. Koha says there are duplicates but there aren't instead it's matching on Title & Author and matching DVDs to books because the DVDs don't have ISBNs By using this patch we allow for libraries to choose how to find duplicates instead of forcing them to use a hard coded script that they have no control over. Nicole
Ok, Thanks Nicole, I understand and I share the "setting principle" that allow librarians to choose the key to deduplicate. I will test the patch with unimarc file to see if it's ok for us.
(In reply to Koha Team Lyon 3 from comment #8) > Ok, Thanks Nicole, I understand and I share the "setting principle" that > allow librarians to choose the key to deduplicate. > I will test the patch with unimarc file to see if it's ok for us. Do you think you will have time to test it before the release? It's a critical one and we need it fixed soon.
Ok, I can test it on monday. Is it too late ?
(In reply to Koha Team Lyon 3 from comment #10) > Ok, I can test it on monday. Is it too late ? No, Koha 16.05 will be released on May 26th :)
Hi, I have just finished my tests. The patch works as described. I'm a little worried because I'm afraid that some of my colleagues forget to choose a matching rule when they add orders from a batch file. It will be a regression for them as they have nothing to choose by now to have a deduplicationw to have a deduplication. So, I'm wondering if it would be possible to have a default matching rule selected when we add a orders from a batch file. Or, perhaps it would be better to keep a deduplication by FindDuplicate when no matching rules is chosen and to switch to a matching rule if the user choose one. What do you think about it ? PS : It's not related to this patch but I have some difficulties to set the matching rules. In the documentation, there's : Threshold of 1000 Match Point on 020$a 1000 Match Point on 022$a 1000 In the example above, a match on either the 020$a or the 022$a will result in a successful match. When I test this patch, I have to put a number in the match point greater than the threshold, if I put a number equal to the thershold, there's no deduplication alert... And I try to put 2 matchpoints with a result greater to the threshold and if none of the two is greater than the threshold, there's no deduplication alert.
I think Sonia has some good points there - Kyle, could you take a look please?
Created attachment 52181 [details] [review] Bug 16493 - acq matching on title and author When you order from a staged file you're getting duplicate warnings that are inaccurate. For example, when you order a file of 50 DVDs for example and they don't have ISBNs they're matching on the books. And then you have to order them one by one. This patch replaces the use of FindDuplicates with Koha's match point system. This means you can select from the same match points defined and used in the batch record importer, or you can opt to skip matching altogether! Test Plan: 1) Import a record with a title, isbn and author. 2) Delete the from the record and stage it again 3) Attempt to add it to a basket via the staged record 4) You should note the gives you the "No records imported" message 5) Apply this patch 6) Create a matcher for ISBN 7) Create a matcher for Author/Title 8) Attempt to add the record to your basket using the ISBN matcher 8) Koha should find no match and import the record to the basket 9) Stage the record again 10) Attempt to add the record to your basket using the Title/Author matcher 11) You should recieve the "No records imported" message. Signed-off-by: Barbara Fondren <bfondren@roundrocktexas.gov> Signed-off-by: Nicole C Engard <nengard@bywatersolutions.com>
Created attachment 52182 [details] [review] Bug 16493 [QA Followup] - Restore title and author match as an option, make it the default
Sonia, could you test it again with the follow-up please?
I have an error when I try to import the batch with the default option "Title and Author" Software error: Undefined subroutine &main::FindDuplicate called at /home/koha/src/acqui/addorderiso2709.pl line 163. For help, please send mail to the webmaster (webmaster@test4-sandbox), giving this error message and the time and date of the error. Sonia
Created attachment 52903 [details] [review] Bug 16493 - acq matching on title and author When you order from a staged file you're getting duplicate warnings that are inaccurate. For example, when you order a file of 50 DVDs for example and they don't have ISBNs they're matching on the books. And then you have to order them one by one. This patch replaces the use of FindDuplicates with Koha's match point system. This means you can select from the same match points defined and used in the batch record importer, or you can opt to skip matching altogether! Test Plan: 1) Import a record with a title, isbn and author. 2) Delete the from the record and stage it again 3) Attempt to add it to a basket via the staged record 4) You should note the gives you the "No records imported" message 5) Apply this patch 6) Create a matcher for ISBN 7) Create a matcher for Author/Title 8) Attempt to add the record to your basket using the ISBN matcher 8) Koha should find no match and import the record to the basket 9) Stage the record again 10) Attempt to add the record to your basket using the Title/Author matcher 11) You should recieve the "No records imported" message. Signed-off-by: Barbara Fondren <bfondren@roundrocktexas.gov> Signed-off-by: Nicole C Engard <nengard@bywatersolutions.com>
Created attachment 52904 [details] [review] Bug 16493 [QA Followup] - Restore title and author match as an option, make it the default
(In reply to Koha Team Lyon 3 from comment #17) > I have an error when I try to import the batch with the default option > "Title and Author" > > Software error: > > Undefined subroutine &main::FindDuplicate called at > /home/koha/src/acqui/addorderiso2709.pl line 163. > For help, please send mail to the webmaster (webmaster@test4-sandbox), > giving this error message and the time and date of the error. > > Sonia My bad, I forgot to re-add the use line for FindDuplicate. Would you please retest? Thanks!
Comment on attachment 52904 [details] [review] Bug 16493 [QA Followup] - Restore title and author match as an option, make it the default Review of attachment 52904 [details] [review]: ----------------------------------------------------------------- ::: acqui/addorderiso2709.pl @@ +160,4 @@ > # 1st insert the biblio, or find it through matcher > unless ( $biblionumber ) { > if ($matcher_id) { > + if ( $matcher_id eq '_TITLE_AUTHOR_' ) { Perhaps a $duplinbatch = undef; above this might be in order since this is in a loop?
Hi, I've tested it on a bywaters sandbox (they work !) But I don't manage to have correct deduplication. The default choice for deduplication "author and title" don't actually deduplicate by author and title. - I have imported a file in a basket - I have changed the original file to remove ISBN - I have imported this second file with the "title and author" matching checked : all the titles have been added to the basket, though title and author are exactly the same. - I have tried to re-import the first fil with the ISBN with "title and author" matching : no record have been imported : the deduplication works. I don't know what is exactly behind the matching key "title and author" ? Perhaps should we call it "Title/author/ISBN" ? Sonia
Created attachment 57139 [details] [review] Bug 16493 - acq matching on title and author When you order from a staged file you're getting duplicate warnings that are inaccurate. For example, when you order a file of 50 DVDs for example and they don't have ISBNs they're matching on the books. And then you have to order them one by one. This patch replaces the use of FindDuplicates with Koha's match point system. This means you can select from the same match points defined and used in the batch record importer, or you can opt to skip matching altogether! Test Plan: 1) Import a record with a title, isbn and author. 2) Delete the from the record and stage it again 3) Attempt to add it to a basket via the staged record 4) You should note the gives you the "No records imported" message 5) Apply this patch 6) Create a matcher for ISBN 7) Create a matcher for Author/Title 8) Attempt to add the record to your basket using the ISBN matcher 8) Koha should find no match and import the record to the basket 9) Stage the record again 10) Attempt to add the record to your basket using the Title/Author matcher 11) You should recieve the "No records imported" message. Signed-off-by: Barbara Fondren <bfondren@roundrocktexas.gov> Signed-off-by: Nicole C Engard <nengard@bywatersolutions.com>
Created attachment 57140 [details] [review] Bug 16493 [QA Followup] - Restore title and author match as an option, make it the default
Good Catch Sonia, there was a bug in my followup that was causing it to not actually use the Title/Author matching. Please give it another try if you like! (In reply to Koha Team Lyon 3 from comment #22) > Hi, > I've tested it on a bywaters sandbox (they work !) > But I don't manage to have correct deduplication. > The default choice for deduplication "author and title" don't actually > deduplicate by author and title. > - I have imported a file in a basket > - I have changed the original file to remove ISBN > - I have imported this second file with the "title and author" matching > checked : all the titles have been added to the basket, though title and > author are exactly the same. > - I have tried to re-import the first fil with the ISBN with "title and > author" matching : no record have been imported : the deduplication works. > > I don't know what is exactly behind the matching key "title and author" ? > Perhaps should we call it "Title/author/ISBN" ? > > Sonia
Created attachment 57198 [details] [review] Bug 16493: acq matching on title and author When you order from a staged file you're getting duplicate warnings that are inaccurate. For example, when you order a file of 50 DVDs for example and they don't have ISBNs they're matching on the books. And then you have to order them one by one. This patch replaces the use of FindDuplicates with Koha's match point system. This means you can select from the same match points defined and used in the batch record importer, or you can opt to skip matching altogether! Test Plan: 1) Import a record with a title, isbn and author. 2) Delete the from the record and stage it again 3) Attempt to add it to a basket via the staged record 4) You should note the gives you the "No records imported" message 5) Apply this patch 6) Create a matcher for ISBN 7) Create a matcher for Author/Title 8) Attempt to add the record to your basket using the ISBN matcher 8) Koha should find no match and import the record to the basket 9) Stage the record again 10) Attempt to add the record to your basket using the Title/Author matcher 11) You should recieve the "No records imported" message. Signed-off-by: Barbara Fondren <bfondren@roundrocktexas.gov> Signed-off-by: Nicole C Engard <nengard@bywatersolutions.com> Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Created attachment 57199 [details] [review] Bug 16493: [QA Followup] Restore title and author match as an option, make it the default Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Pushed to master for 16.11!
Pushed to 3.22.x, will be in 3.22.13
Pushed to 16.05.x, will be in 16.05.06
*** Bug 16495 has been marked as a duplicate of this bug. ***