Bug 12074 - Filter duplicates when adding a full batch from a staged file
Summary: Filter duplicates when adding a full batch from a staged file
Status: CLOSED FIXED
Alias: None
Product: Koha
Classification: Unclassified
Component: Acquisitions (show other bugs)
Version: master
Hardware: All All
: P5 - low enhancement (vote)
Assignee: Koha Team Lyon 3
QA Contact: Testopia
URL:
Whiteboard:
Keywords:
: 6386 (view as bug list)
Depends on: 7180
Blocks: 16493 18482 16495
  Show dependency treegraph
 
Reported: 2014-04-11 15:29 UTC by Koha Team Lyon 3
Modified: 2017-04-24 17:23 UTC (History)
8 users (show)

See Also:
Change sponsored?: ---
Patch complexity: ---
Bot Control: ---
When did the bot last check this:
Who signed the patch off:
Text to go in the release notes:


Attachments
Bug 12074 Filter duplicates when adding a batch from a staged file (5.98 KB, patch)
2014-04-11 15:39 UTC, Koha Team Lyon 3
Details | Diff | Splinter Review
test file 1 (1.79 KB, text/plain)
2014-04-11 15:45 UTC, Koha Team Lyon 3
Details
test file 2 (13.68 KB, text/plain)
2014-04-11 15:46 UTC, Koha Team Lyon 3
Details
Bug 12074 - Filter duplicates when adding a full batch from a staged file (5.50 KB, patch)
2015-02-24 15:10 UTC, Koha Team Lyon 3
Details | Diff | Splinter Review
Bug 12074 Filter duplicates when adding a batch from a staged file (5.54 KB, patch)
2015-03-06 09:33 UTC, sandboxes@biblibre.com
Details | Diff | Splinter Review
Bug 12074 Filter duplicates when adding a batch from a staged file (7.05 KB, patch)
2015-03-17 13:50 UTC, Jonathan Druart
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description Koha Team Lyon 3 2014-04-11 15:29:44 UTC
Until now, when you add order for a single record from a staged batch, duplicates are checked and if found you can choose 3 different treatments : adding a new order from existing record, adding a new record nevertheless or doing nothing.
But when importing the batch all in once (Import all), there's no duplicate checking.
This patch aims to solve this.
When adding a batch of records to a basket, duplicates are skipped and
an alert is displayed with a link to them so as they could be treated
individually.
In the case that all batch records match an existing record, you stay on the staged batch list, a warning is also visible to tell you that nothing had been imported. The "Import all" block is not any more displayed

Test plan :

You need the 2 test attached files TestFile1.mrc and TestFile2.elc (TestFile1 includes only the title "Amilec ou La graine d'hommes" that is also included in TestFile2)

1) go to “Stage MARC records for import” page, upload TestFile1 and stage it (select iso 5426 encoding).
2) Manage staged record and import the batch.
3) Make sure that the new record is indexed (depending to your indexing system and test platform).
4) Go back to go to “Stage MARC records for import” page upload TestFile2 and stage it (select iso 5426 encoding).
5) Go to acquisitions module and create a new basket.
6) From your basket, in the “Add order to basket block”  choose  'From a staged file'.
7) Then click File2 (‘addorder button').
8) Go down the "Import all" block and save.
9) You are redirected to the basket page : a warning is displayed to tell you that some duplicates have been found and skipped.
There's a link on the warning throughout you can go back to the list of remaining records and treat them individually if necesary.
10) Click the link : you fall upon the title of TestFile1 (of course as it's a duplicate).
11) Check that the imported records have been indexed.
11) Go down the "Import all" block and save.
12) A warning is displayed saying that no records have been imported because they all match an existing record. The “Import all” block is not any more visible.

Olivier Crouzet
Comment 1 Koha Team Lyon 3 2014-04-11 15:39:18 UTC Comment hidden (obsolete)
Comment 2 Koha Team Lyon 3 2014-04-11 15:45:46 UTC
Created attachment 27032 [details]
test file 1
Comment 3 Koha Team Lyon 3 2014-04-11 15:46:15 UTC
Created attachment 27033 [details]
test file 2
Comment 4 Koha Team Lyon 3 2014-05-12 11:03:09 UTC
*** Bug 6386 has been marked as a duplicate of this bug. ***
Comment 5 Jonathan Druart 2014-05-22 15:03:05 UTC
Bug 7180 has been (finally) pushed. This one does not apply anymore.
Comment 6 Olli-Antti Kivilahti 2014-10-31 08:57:05 UTC
Hi there!

Our librarians are having hard time because this new feature doesn't check for duplicates when making acquisitions from this staged file.

They share your pain, in addition to that we are providing a few more improvements to ease acquisition decisions from the staging file.

1. Provide visual cues of which lineitems are already imported and show a "map" of existing Items in the consortium (of each branch) non-disruptively.
2. Display the default price fetched from the MARC record in the staged file listing.
3. (Having found this bug) Add a checkbox for each matched record whether or not to overwrite the existing record with this file or not. Defaults to not overwriting.
As a possible improvement... we have an in-house patch which makes MARC transformations based on Matcher rules so you can automatically transform the imported MARC based on Matcher rules, for ex move the agerestriction from incoming Records 049c => 521a in Koha. Different vendors provide some information in different fields. If you want to test it I can rebase it for you. It might be better to do the transformations during the stage file process though.
Comment 7 Marcel de Rooy 2014-10-31 09:37:05 UTC
(In reply to Olli-Antti Kivilahti from comment #6)
> Hi there!
Goedemorgen

> As a possible improvement... we have an in-house patch which makes MARC
> transformations based on Matcher rules so you can automatically transform
> the imported MARC based on Matcher rules, for ex move the agerestriction
> from incoming Records 049c => 521a in Koha. 

Several people are also / have been working on improving the marc mod templates functionality..
Comment 8 Olli-Antti Kivilahti 2014-10-31 15:24:10 UTC
(In reply to M. de Rooy from comment #7)
> (In reply to Olli-Antti Kivilahti from comment #6)
> > Hi there!
> Goedemorgen
> 
> > As a possible improvement... we have an in-house patch which makes MARC
> > transformations based on Matcher rules so you can automatically transform
> > the imported MARC based on Matcher rules, for ex move the agerestriction
> > from incoming Records 049c => 521a in Koha. 
> 
> Several people are also / have been working on improving the marc mod
> templates functionality..

Hi!
I have heard of it as well, but the results are not available NOW() :)
Also I think their work is a bit too complex for our needs.
Shared our feature here: Bug 13181
Comment 9 Jonathan Druart 2014-11-05 15:13:41 UTC
Is this still valid?
Comment 10 Koha Team Lyon 3 2014-11-10 15:37:37 UTC
We just need to find time to check ... I'll also take a look at Bug 13181
Comment 11 Olli-Antti Kivilahti 2014-12-15 15:23:41 UTC
Regarding Bug 13181 you should focus on the community effort of using MARC Modification Templates. My hack is obsoleted and will be replaced by the community tool.
Comment 12 Koha Team Lyon 3 2014-12-15 15:32:19 UTC
In fact , the bug 13182 is most interesting for us.
We want to keep the possibility to see the duplicates and to choose to order it when adding a full batch in a order.
Comment 13 Koha Team Lyon 3 2015-02-24 15:10:43 UTC Comment hidden (obsolete)
Comment 14 aloij 2015-03-06 09:31:33 UTC
the patch is applied correctly. When adding a batch of documents to a basket , duplicates are ignored and correctly
a visual alert is displayed with a link to them so they can be treated so
individual .
Signed off
Comment 15 sandboxes@biblibre.com 2015-03-06 09:33:13 UTC
Patch tested with a sandbox, by JA <aloi54@live.fr>
Comment 16 sandboxes@biblibre.com 2015-03-06 09:33:38 UTC Comment hidden (obsolete)
Comment 17 Jonathan Druart 2015-03-17 13:50:48 UTC
Created attachment 36948 [details] [review]
Bug 12074 Filter duplicates when adding a batch from a staged file

When adding a batch of records to a basket, duplicates are skipped and
an alert is displayed with a link to them so as they could be treated
individually.

Test plan :

You need the 2 test attached files TestFile1.mrc and TestFile2.elc
(TestFile1 includes only the title "Amilec ou La graine d'hommes" that
is also included in TestFile2)

1) go to “Stage MARC records for import” page, upload TestFile1 and
stage it (select iso 5426 encoding).
2) Manage staged record and import the batch.
3) Make sure that the new record is indexed (depending to your indexing
system and test platform).
4) Go back to go to “Stage MARC records for import” page upload
TestFile2 and stage it (select iso 5426 encoding).
5) Go to acquisitions module and create a new basket.
6) From your basket, in the “Add order to basket block”  choose  'From a
staged file'.
7) Then click File2 (‘addorder button').
8) Go down the "Import all" block and save.
9) You are redirected to the basket page : a warning is displayed to
tell you that some duplicates have been found and skipped.
There's a link on the warning throughout you can go back to the list of
remaining records and treat them individually if necesary.
10) Click the link : you fall upon the title of TestFile1 (of course as
it's a duplicate).
11) Check that the imported records have been indexed.
11) Go down the "Import all" block and save.
12) A warning is displayed saying that no records have been imported
because they all match an existing record. The “Import all” block is not
any more visible.

Signed-off-by: JA <aloi54@live.fr>

Signed-off-by: Jonathan Druart <jonathan.druart@biblibre.com>
Comment 18 Tomás Cohen Arazi 2015-06-24 14:35:10 UTC
Patch pushed to master.

Thanks Lyon3 Team!