Bug 12074

Summary: Filter duplicates when adding a full batch from a staged file
Product: Koha Reporter: Koha Team University Lyon 3 <koha>
Component: AcquisitionsAssignee: Koha Team University Lyon 3 <koha>
Status: CLOSED FIXED QA Contact: Testopia <testopia>
Severity: enhancement    
Priority: P5 - low CC: aloi54, jonathan.druart, koha, m.de.rooy, olli-antti.kivilahti, sandboxes, tomascohen
Version: Main   
Hardware: All   
OS: All   
See Also: http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=6386
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=13182
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18482
Change sponsored?: --- Patch complexity: ---
Documentation contact: Documentation submission:
Text to go in the release notes:
Version(s) released in:
Circulation function:
Bug Depends on: 7180    
Bug Blocks: 16493, 16495, 18482    
Attachments: Bug 12074 Filter duplicates when adding a batch from a staged file
test file 1
test file 2
Bug 12074 - Filter duplicates when adding a full batch from a staged file
Bug 12074 Filter duplicates when adding a batch from a staged file
Bug 12074 Filter duplicates when adding a batch from a staged file

Description Koha Team University Lyon 3 2014-04-11 15:29:44 UTC
Until now, when you add order for a single record from a staged batch, duplicates are checked and if found you can choose 3 different treatments : adding a new order from existing record, adding a new record nevertheless or doing nothing.
But when importing the batch all in once (Import all), there's no duplicate checking.
This patch aims to solve this.
When adding a batch of records to a basket, duplicates are skipped and
an alert is displayed with a link to them so as they could be treated
individually.
In the case that all batch records match an existing record, you stay on the staged batch list, a warning is also visible to tell you that nothing had been imported. The "Import all" block is not any more displayed

Test plan :

You need the 2 test attached files TestFile1.mrc and TestFile2.elc (TestFile1 includes only the title "Amilec ou La graine d'hommes" that is also included in TestFile2)

1) go to “Stage MARC records for import” page, upload TestFile1 and stage it (select iso 5426 encoding).
2) Manage staged record and import the batch.
3) Make sure that the new record is indexed (depending to your indexing system and test platform).
4) Go back to go to “Stage MARC records for import” page upload TestFile2 and stage it (select iso 5426 encoding).
5) Go to acquisitions module and create a new basket.
6) From your basket, in the “Add order to basket block”  choose  'From a staged file'.
7) Then click File2 (‘addorder button').
8) Go down the "Import all" block and save.
9) You are redirected to the basket page : a warning is displayed to tell you that some duplicates have been found and skipped.
There's a link on the warning throughout you can go back to the list of remaining records and treat them individually if necesary.
10) Click the link : you fall upon the title of TestFile1 (of course as it's a duplicate).
11) Check that the imported records have been indexed.
11) Go down the "Import all" block and save.
12) A warning is displayed saying that no records have been imported because they all match an existing record. The “Import all” block is not any more visible.

Olivier Crouzet
Comment 1 Koha Team University Lyon 3 2014-04-11 15:39:18 UTC Comment hidden (obsolete)
Comment 2 Koha Team University Lyon 3 2014-04-11 15:45:46 UTC
Created attachment 27032 [details]
test file 1
Comment 3 Koha Team University Lyon 3 2014-04-11 15:46:15 UTC
Created attachment 27033 [details]
test file 2
Comment 4 Koha Team University Lyon 3 2014-05-12 11:03:09 UTC
*** Bug 6386 has been marked as a duplicate of this bug. ***
Comment 5 Jonathan Druart 2014-05-22 15:03:05 UTC
Bug 7180 has been (finally) pushed. This one does not apply anymore.
Comment 6 Olli-Antti Kivilahti 2014-10-31 08:57:05 UTC
Hi there!

Our librarians are having hard time because this new feature doesn't check for duplicates when making acquisitions from this staged file.

They share your pain, in addition to that we are providing a few more improvements to ease acquisition decisions from the staging file.

1. Provide visual cues of which lineitems are already imported and show a "map" of existing Items in the consortium (of each branch) non-disruptively.
2. Display the default price fetched from the MARC record in the staged file listing.
3. (Having found this bug) Add a checkbox for each matched record whether or not to overwrite the existing record with this file or not. Defaults to not overwriting.
As a possible improvement... we have an in-house patch which makes MARC transformations based on Matcher rules so you can automatically transform the imported MARC based on Matcher rules, for ex move the agerestriction from incoming Records 049c => 521a in Koha. Different vendors provide some information in different fields. If you want to test it I can rebase it for you. It might be better to do the transformations during the stage file process though.
Comment 7 Marcel de Rooy 2014-10-31 09:37:05 UTC
(In reply to Olli-Antti Kivilahti from comment #6)
> Hi there!
Goedemorgen

> As a possible improvement... we have an in-house patch which makes MARC
> transformations based on Matcher rules so you can automatically transform
> the imported MARC based on Matcher rules, for ex move the agerestriction
> from incoming Records 049c => 521a in Koha. 

Several people are also / have been working on improving the marc mod templates functionality..
Comment 8 Olli-Antti Kivilahti 2014-10-31 15:24:10 UTC
(In reply to M. de Rooy from comment #7)
> (In reply to Olli-Antti Kivilahti from comment #6)
> > Hi there!
> Goedemorgen
> 
> > As a possible improvement... we have an in-house patch which makes MARC
> > transformations based on Matcher rules so you can automatically transform
> > the imported MARC based on Matcher rules, for ex move the agerestriction
> > from incoming Records 049c => 521a in Koha. 
> 
> Several people are also / have been working on improving the marc mod
> templates functionality..

Hi!
I have heard of it as well, but the results are not available NOW() :)
Also I think their work is a bit too complex for our needs.
Shared our feature here: Bug 13181
Comment 9 Jonathan Druart 2014-11-05 15:13:41 UTC
Is this still valid?
Comment 10 Koha Team University Lyon 3 2014-11-10 15:37:37 UTC
We just need to find time to check ... I'll also take a look at Bug 13181
Comment 11 Olli-Antti Kivilahti 2014-12-15 15:23:41 UTC
Regarding Bug 13181 you should focus on the community effort of using MARC Modification Templates. My hack is obsoleted and will be replaced by the community tool.
Comment 12 Koha Team University Lyon 3 2014-12-15 15:32:19 UTC
In fact , the bug 13182 is most interesting for us.
We want to keep the possibility to see the duplicates and to choose to order it when adding a full batch in a order.
Comment 13 Koha Team University Lyon 3 2015-02-24 15:10:43 UTC Comment hidden (obsolete)
Comment 14 aloij 2015-03-06 09:31:33 UTC
the patch is applied correctly. When adding a batch of documents to a basket , duplicates are ignored and correctly
a visual alert is displayed with a link to them so they can be treated so
individual .
Signed off
Comment 15 Biblibre Sandboxes 2015-03-06 09:33:13 UTC
Patch tested with a sandbox, by JA <aloi54@live.fr>
Comment 16 Biblibre Sandboxes 2015-03-06 09:33:38 UTC Comment hidden (obsolete)
Comment 17 Jonathan Druart 2015-03-17 13:50:48 UTC
Created attachment 36948 [details] [review]
Bug 12074 Filter duplicates when adding a batch from a staged file

When adding a batch of records to a basket, duplicates are skipped and
an alert is displayed with a link to them so as they could be treated
individually.

Test plan :

You need the 2 test attached files TestFile1.mrc and TestFile2.elc
(TestFile1 includes only the title "Amilec ou La graine d'hommes" that
is also included in TestFile2)

1) go to “Stage MARC records for import” page, upload TestFile1 and
stage it (select iso 5426 encoding).
2) Manage staged record and import the batch.
3) Make sure that the new record is indexed (depending to your indexing
system and test platform).
4) Go back to go to “Stage MARC records for import” page upload
TestFile2 and stage it (select iso 5426 encoding).
5) Go to acquisitions module and create a new basket.
6) From your basket, in the “Add order to basket block”  choose  'From a
staged file'.
7) Then click File2 (‘addorder button').
8) Go down the "Import all" block and save.
9) You are redirected to the basket page : a warning is displayed to
tell you that some duplicates have been found and skipped.
There's a link on the warning throughout you can go back to the list of
remaining records and treat them individually if necesary.
10) Click the link : you fall upon the title of TestFile1 (of course as
it's a duplicate).
11) Check that the imported records have been indexed.
11) Go down the "Import all" block and save.
12) A warning is displayed saying that no records have been imported
because they all match an existing record. The “Import all” block is not
any more visible.

Signed-off-by: JA <aloi54@live.fr>

Signed-off-by: Jonathan Druart <jonathan.druart@biblibre.com>
Comment 18 Tomás Cohen Arazi (tcohen) 2015-06-24 14:35:10 UTC
Patch pushed to master.

Thanks Lyon3 Team!