Bug 18482

Summary: False duplicates detected on adding a batch from a stage file
Product: Koha Reporter: Koha Team University Lyon 3 <koha>
Component: AcquisitionsAssignee: Koha Team University Lyon 3 <koha>
Status: CLOSED FIXED QA Contact: Testopia <testopia>
Severity: major    
Priority: P5 - low CC: jonathan.druart, katrin.fischer, koha, kyle, mtj
Version: Main   
Hardware: All   
OS: All   
See Also: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=12074
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=16493
Change sponsored?: --- Patch complexity: ---
Documentation contact: Documentation submission:
Text to go in the release notes:
Version(s) released in:
Bug Depends on: 12074    
Bug Blocks:    
Attachments: False duplicates detected on adding a batch from a stage file
TestFile1
TestFile2
Bug 18482 False duplicates detected on adding a batch from a stage file
Bug 18482 False duplicates detected on adding a batch from a stage file
Bug 18482 False duplicates detected on adding a batch from a stage file
Bug 18482 False duplicates detected on adding a batch [16.11.x]

Description Koha Team University Lyon 3 2017-04-22 16:43:20 UTC
Created attachment 62572 [details]
False duplicates detected on adding a batch from a stage file

When adding a batch from a stage file with defaut matching on
title/author, if a duplicate is detected, all following records
treated in the batch are discarded from import even if they are not duplicates.

Checking for duplicate now lies on duplinbatch value that is not reset at each loop (In the initial commit, discarding record was directly tight to FindDuplicate return). So from the point when duplinbatch is defined, all the following biblios are discarded as duplicate whatever FindDuplicate return is.
This patch adds a distinct flag.

Test plan :

You need the 2 test attached files TestFile1.mrc and TestFile2.elc (TestFile1 includes only the title "Amilec ou La graine d'hommes" that is also included in TestFile2). To save them, right click on bugzilla link and do 'Save target as'.

1) go to “Stage MARC records for import” page, upload TestFile1, select iso 5426 encoding and stage it for Import. 
2) Click again 'Stage MARC records' button and do the same for TestFile2.elc
3) Go to acquisitions module, enter a vendor name and create a new basket.
4) From your basket, click 'Add to basket', then in the “Add order to basket window/block” choose 'From a staged file'.
5) Then click ‘Add orders' button on File1.
6) Leave default matching on Title/Author, tick the title, add a location in second tag and fund in third tag and Save. 
7) Make sure that the new record is indexed (depending to your indexing
system and test platform).
8) Redo step 4 and step 5 this time on TestFile2.
9) Leave default matching on Title/Author, select all titles, add a location in second tag, a fund in third tag and Save. 
10) A warning is displayed saying that no records have been imported because they all match an existing record. The “Import all” block is not any more visible.
11) Apply the patch
12) Refresh 'Add orders from' page (File2) and redo step 9
13) You are redirected to the basket page : a warning is displayed to tell you that some duplicates have been found and skipped.
There's a link on the warning throughout you can go back to the list of remaining records and treat them individually if necesary.
14) Click the link : you fall upon the only title that is a very duplicate (record of TestFile1).
15) Be happy
16) Sign off

Olivier Crouzet
Comment 1 Koha Team University Lyon 3 2017-04-22 16:44:50 UTC
Created attachment 62573 [details]
TestFile1
Comment 2 Koha Team University Lyon 3 2017-04-22 16:45:42 UTC
Created attachment 62574 [details]
TestFile2
Comment 3 Koha Team University Lyon 3 2017-04-22 16:50:04 UTC
Created attachment 62575 [details] [review]
Bug 18482 False duplicates detected on adding a batch from a stage file
Comment 4 Koha Team University Lyon 3 2017-04-22 16:52:35 UTC
This patch must be added from version 3.22.13 and higher
Comment 5 Alex Buckley 2017-04-23 23:23:28 UTC
Created attachment 62581 [details] [review]
Bug 18482 False duplicates detected on adding a batch from a stage file

When adding a batch from a stage file with defaut matching on
title/author, if a duplicate is detected, all following records
treated in the batch are discarded from import even if they are not duplicates

Signed-off-by: Alex Buckley <alexbuckley@catalyst.net.nz>
Comment 6 Jonathan Druart 2017-04-24 17:30:09 UTC
Created attachment 62621 [details] [review]
Bug 18482 False duplicates detected on adding a batch from a stage file

When adding a batch from a stage file with defaut matching on
title/author, if a duplicate is detected, all following records
treated in the batch are discarded from import even if they are not duplicates

Signed-off-by: Alex Buckley <alexbuckley@catalyst.net.nz>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Comment 7 Kyle M Hall 2017-04-28 10:51:19 UTC
Pushed to master for 17.05, thanks Olivier!
Comment 8 Katrin Fischer 2017-05-13 12:06:25 UTC
There is a conflict in acqui/addorderiso2709.pl when applying this to 16.11.x, can you please check?
Comment 9 Koha Team University Lyon 3 2017-05-19 13:08:36 UTC
Created attachment 63579 [details] [review]
Bug 18482 False duplicates detected on adding a batch [16.11.x]

add specific patch for 16.11.x. Just remove the biblio_count line that seems to been added more lately
Comment 10 Katrin Fischer 2017-06-05 13:32:48 UTC
Thanks for the rebased patch!

This patch has been pushed to 16.11.x and will be in 16.11.09.
Comment 11 Mason James 2017-07-31 11:07:28 UTC
Pushed to 16.05.x, for 16.05.15 release