Bug 18273

Summary:	bulkmarcimport.pl inserts authority duplicates
Product:	Koha	Reporter:	Matthias Meusburger <matthias.meusburger>
Component:	Command-line Utilities	Assignee:	Matthias Meusburger <matthias.meusburger>
Status:	Pushed to oldoldstable ---	QA Contact:	Martin Renvoize (ashimema) <martin.renvoize>
Severity:	normal
Priority:	P5 - low	CC:	david, fridolin.somers, koha, koha, lucas, marjorie.barry-vila, martin.renvoize, robin
Version:	Main
Hardware:	All
OS:	All
Change sponsored?:	---	Patch complexity:	Trivial patch
Documentation contact:		Documentation submission:
Text to go in the release notes:	This fixes the misc/migration_tools/bulkmarcimport.pl script when importing authority records so that the "--match" option works as expected, and no duplicates are created. Previously, this option was not working for authority records and duplicate records were being created even when there was a match.	Version(s) released in:	24.11.00,24.05.06,23.11.11
Circulation function:
Attachments:	Bug 18273: Do not insert authority if there already are duplicates in koha Bug 18273: Do not insert authority if there already are duplicates in koha Bug 18273: Do not insert authority if there already are duplicates in koha Bug 18273: Do not insert authority if there already are duplicates in koha

Description Matthias Meusburger 2017-03-15 15:30:40 UTC

When importing authorities through bulkmarcimport with the -match option, there are 3 possibilities:

1) An existing authority has been found in koha according to the -match criteria. The authority is replaced if needed (005 are checked first). This is ok.

2) No authority is found in koha according to the -match criteria. The authority is inserted. This is also ok.

3) Multiple authorities have been found in koha according to the -match criteria.  The authority is inserted. 

=> Is this the expected behavior?

As the -match criteria is used for deduping, the existing authorities are very likely to be already existing duplicates. Do we really want to add another one?

My feeling is that in this case, the authority should NOT be inserted.

Can I have your opinion on this?

Comment 1 Matthias Meusburger 2017-03-15 16:12:04 UTC Comment hidden (obsolete)

Created attachment 61106 [details] [review]
Bug 18273: Do not insert authority if there already are duplicates in koha

Comment 2 Katrin Fischer 2017-03-19 21:40:03 UTC

The bibliographic import overlays the one with the lower (or higher?) biblionumber. Not sure if we should make it consistent.

Comment 3 Matthias Meusburger 2017-03-20 08:56:16 UTC

Are you sure about that Katrin?

From what I've seen, the behaviour is the same for biblios: if multiple matching biblios have been found, a new biblio is inserted.

Comment 4 Katrin Fischer 2017-03-20 09:00:05 UTC

Hm, I was pretty sure, but was thinking of the staged marc tool. Didn't realize then it will probably be different for bulkmarcimport. Makes sense then.

Comment 5 Fridolin Somers 2021-03-11 11:55:42 UTC

Need signoff ?

Comment 6 Fridolin Somers 2021-12-03 21:17:05 UTC

Needs an easy rebase

Comment 7 Koha Team University Lyon 3 2022-05-18 09:44:37 UTC

Hi,
it would be nice to have a follow-up on this one.
Thx
Sonia

Comment 8 David Nind 2022-11-23 18:30:08 UTC

Created attachment 144213 [details] [review]
Bug 18273: Do not insert authority if there already are duplicates in koha

Comment 9 David Nind 2022-11-23 18:36:26 UTC

I think I managed to successfully rebase this one, but I'm not sure how to test.

If we could get a test plan, I'll have a go at testing.

It would be great if the test plan could include:
- the bulkmarcimport command to use, and
- authority record samples to use to test the scenarios outlined in the bug description (or how to create).

Comment 10 Katrin Fischer 2023-09-10 16:06:20 UTC

Patch still applies, but is missing a test plan. 

I think some hints, especially on the CLI parameters to use with bulkmarcimport could help to get this further.

Comment 11 Matthias Meusburger 2024-09-12 12:17:00 UTC

Created attachment 171385 [details] [review]
Bug 18273: Do not insert authority if there already are duplicates in koha

Test plan:

1) Create an authority with a unique value in a searchable field, for instance a PERSO_NAME authority with 'BZ18273' in the 100$a field.

2) Create a second authority with the same value in the same field.

3) Export this second authority as iso2709 (Save -> MARC (Unicode/UTF-8))

4) Run bulkmarcimport.pl with the following parameters:
perl bulkmarcimport.pl -a --file=/kohadevbox/koha/auth-xxxx.utf8 --match='Heading,100$a' -l /kohadevbox/koha/bulk.log

5) Check in the log file and/or through search that a third authority has been added (despite using the --match option)

6) Apply the patch

7) Delete the 3 authorities and repeat step 1 to 4.

8) Check that a third authority has not been added (--match worked)

Comment 12 Matthias Meusburger 2024-09-12 12:17:51 UTC

Hi, I've added a test plan.

Comment 13 Martin Renvoize (ashimema) 2024-09-26 19:25:55 UTC

Created attachment 172049 [details] [review]
Bug 18273: Do not insert authority if there already are duplicates in koha

Test plan:

1) Create an authority with a unique value in a searchable field, for instance a PERSO_NAME authority with 'BZ18273' in the 100$a field.

2) Create a second authority with the same value in the same field.

3) Export this second authority as iso2709 (Save -> MARC (Unicode/UTF-8))

4) Run bulkmarcimport.pl with the following parameters:
perl bulkmarcimport.pl -a --file=/kohadevbox/koha/auth-xxxx.utf8 --match='Heading,100$a' -l /kohadevbox/koha/bulk.log

5) Check in the log file and/or through search that a third authority has been added (despite using the --match option)

6) Apply the patch

7) Delete the 3 authorities and repeat step 1 to 4.

8) Check that a third authority has not been added (--match worked)

Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>

Comment 14 Martin Renvoize (ashimema) 2024-09-26 19:26:47 UTC

Thanks Matts,

Super trivial patch working as I'd expect.  Passing QA

Comment 15 Katrin Fischer 2024-10-21 11:42:18 UTC

Updating this to "bug".

Comment 16 Katrin Fischer 2024-10-21 13:12:22 UTC

Pushed for 24.11!

Well done everyone, thank you!

Comment 17 Lucas Gass (lukeg) 2024-11-20 18:22:21 UTC

Backported to 24.05.x for upcoming 24.05.06

Comment 18 Fridolin Somers 2024-12-13 14:35:19 UTC

Pushed to 23.11.x for 23.11.11