Bug 15248

Summary: Make it possible to configure duplicate check on bibliographic records
Product: Koha Reporter: Paul Poulain <paul.poulain>
Component: ToolsAssignee: Julian Maurice <julian.maurice>
Status: Passed QA --- QA Contact: Andrew Fuerste-Henry <andrew>
Severity: enhancement    
Priority: P5 - low CC: andrew, dcook, jean-manuel.broust, julian.maurice, marie.hedbom, michaela.sieber, mspinney, sally.healey, severine.queune, veron
Version: Main   
Hardware: All   
OS: All   
See Also: http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=14973
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27278
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31472
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37871
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=35886
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=7132
GIT URL: Change sponsored?: ---
Patch complexity: --- Documentation contact:
Documentation submission: Text to go in the release notes:
Version(s) released in:
Circulation function:
Attachments: Bug 15248: Use record matching rules to find duplicate while cataloging
Bug 15248: Use record matching rules to find duplicate while cataloging
Bug 15248: Use record matching rules to find duplicate while cataloging
Bug 15248: Use record matching rules to find duplicate while cataloging
Bug 15248: Use record matching rules to find duplicate while cataloging

Description Paul Poulain 2015-11-24 15:46:48 UTC
Currently, FindDuplicate tries to find a duplicate by doing a search, that cannot be tuned by the library.
It would be better to have the opportunity to use the matching points system.
A "FindDuplicate" matching point entry could be added, set to title/author -current behaviour of FindDuplicate-, that the library can update/change to fit it's needs.
Comment 1 David Cook 2021-07-21 06:37:38 UTC
I like this idea!

I was just about to open a ticket because I wanted to add a few indexes (like identifier-standard for 024$a and 022$a and Control-number for 001)to FindDuplicate (which have already been implemented locally).

Using record matching rules might really be the best way to do this, since different libraries might have different needs/wants for how to find duplicates.
Comment 2 Katrin Fischer 2022-05-24 11:10:34 UTC
We'd love to add 001 to the checks as this is the most important field for our synchronization with the union catalog.
Comment 3 Katrin Fischer 2025-08-29 13:36:02 UTC
Just an idea: could we make it possible to use the matching rules for the duplicate check? I think it would probably work for our use case and would offer some flexibility.
Comment 4 Andrew Fuerste-Henry 2025-08-29 13:39:21 UTC
(In reply to Katrin Fischer from comment #3)
> Just an idea: could we make it possible to use the matching rules for the
> duplicate check? I think it would probably work for our use case and would
> offer some flexibility.

We've had a library ask us to implement exactly this; watch this space :)
Comment 5 Katrin Fischer 2025-08-29 13:54:39 UTC
(In reply to Andrew Fuerste-Henry from comment #4)
> (In reply to Katrin Fischer from comment #3)
> > Just an idea: could we make it possible to use the matching rules for the
> > duplicate check? I think it would probably work for our use case and would
> > offer some flexibility.
> 
> We've had a library ask us to implement exactly this; watch this space :)

Awesome news, will do!
Comment 6 Julian Maurice 2025-09-29 14:45:20 UTC
(In reply to Andrew Fuerste-Henry from comment #4)
> (In reply to Katrin Fischer from comment #3)
> > Just an idea: could we make it possible to use the matching rules for the
> > duplicate check? I think it would probably work for our use case and would
> > offer some flexibility.
> 
> We've had a library ask us to implement exactly this; watch this space :)

We had to do it too...
I'll submit my patch here. Hopefully this is similar to what you had in mind.
Comment 7 Julian Maurice 2025-09-29 14:45:49 UTC
Created attachment 187049 [details] [review]
Bug 15248: Use record matching rules to find duplicate while cataloging

This patch allows to configure how Koha searches for duplicates when
cataloging manually by using a record matching rule.

For each framework we can now set a record matching rule to use when
searching for duplicates.
This allow to use different matching rules while in acquisition for
instance (ACQ framework).

If a matching rule is selected, we also have the ability to forbid the
creation of duplicate: if a duplicate is found and this option is
enabled, the usual option to create a duplicate will be hidden.

If no matching rule is selected, then the default matching mechanism is
used.

Test plan:
1. Create a record matching rule matching on ISBN
    Threshold: 100
    Record type: biblio
    Match point:
        Search index: "isbn"
        Score: 100
        Tag: "020" (or "010" for UNIMARC)
        Subfields: "a"
        Offset: 0
        Length: 0
        Normalization rule: Remove spaces
    No match checks
2. Edit the default MARC framework and select the new matching rule and
   enable "Forbid duplicate creation"
3. Create a biblio with an ISBN
4. Make sure your new biblio is indexed in Zebra or ES
5. Try to create a biblio with the same ISBN. You should see a warning
   about the duplicate with no option to create the duplicate biblio
6. In the acquisitions module, in a basket add a new order from an
   external source (requires a configured Z39.50/SRU server)
7. Choose a record with an ISBN
8. Once the order is created, try to create a new order from an external
   source and choose the same record. You should see a warning about the
   duplicate.
9. Edit the record matching rule and disable "Forbid duplicate creation"
10. Redo step 5. Now there should be an option to create the duplicate
    record.
11. Redo step 8. Now there should be an option to create the duplicate
    record.
12. Run the following test scripts:
   - t/db_dependent/Search.t
   - t/db_dependent/Matcher.t

Sponsored-by: SCD Lyon2
Comment 8 Julian Maurice 2025-09-29 14:48:09 UTC
Created attachment 187050 [details] [review]
Bug 15248: Use record matching rules to find duplicate while cataloging

This patch allows to configure how Koha searches for duplicates when
cataloging manually by using a record matching rule.

For each framework we can now set a record matching rule to use when
searching for duplicates.
This allow to use different matching rules while in acquisition for
instance (ACQ framework).

If a matching rule is selected, we also have the ability to forbid the
creation of duplicate: if a duplicate is found and this option is
enabled, the usual option to create a duplicate will be hidden.

If no matching rule is selected, then the default matching mechanism is
used.

Test plan:
1. Create a record matching rule matching on ISBN
    Threshold: 100
    Record type: biblio
    Match point:
        Search index: "isbn"
        Score: 100
        Tag: "020" (or "010" for UNIMARC)
        Subfields: "a"
        Offset: 0
        Length: 0
        Normalization rule: Remove spaces
    No match checks
2. Edit the default MARC framework and select the new matching rule and
   enable "Forbid duplicate creation"
3. Create a biblio with an ISBN
4. Make sure your new biblio is indexed in Zebra or ES
5. Try to create a biblio with the same ISBN. You should see a warning
   about the duplicate with no option to create the duplicate biblio
6. In the acquisitions module, in a basket add a new order from an
   external source (requires a configured Z39.50/SRU server)
7. Choose a record with an ISBN
8. Once the order is created, try to create a new order from an external
   source and choose the same record. You should see a warning about the
   duplicate.
9. Edit the record matching rule and disable "Forbid duplicate creation"
10. Redo step 5. Now there should be an option to create the duplicate
    record.
11. Redo step 8. Now there should be an option to create the duplicate
    record.
12. Run the following test scripts:
   - t/db_dependent/Search.t
   - t/db_dependent/Matcher.t

Sponsored-by: SCD Lyon2
Comment 9 Andrew Fuerste-Henry 2025-09-29 15:36:56 UTC
(In reply to Julian Maurice from comment #6)
> (In reply to Andrew Fuerste-Henry from comment #4)
> > (In reply to Katrin Fischer from comment #3)
> > > Just an idea: could we make it possible to use the matching rules for the
> > > duplicate check? I think it would probably work for our use case and would
> > > offer some flexibility.
> > 
> > We've had a library ask us to implement exactly this; watch this space :)
> 
> We had to do it too...
> I'll submit my patch here. Hopefully this is similar to what you had in mind.

Testing now! Your test plan suggests this will work for our partner. I like the additional framework option to entirely block bib creation on a detected duplicate!
Comment 10 Andrew Fuerste-Henry 2025-09-29 15:59:25 UTC
Koha dies when I try to edit any framework. Editing tags and subfields is fine, but I cannot edit the framework itself. I get:

"DBIx::Class::Schema::source(): Can't find source for BiblioFrameworkMarcMatcher at /kohadevbox/koha/Koha/Objects.pm line 530
 at /usr/share/perl5/DBIx/Class/Exception.pm line 77"

I applied this patch, ran updatedatabase, ran yarn build, then restart_all. Am I missing some step?

I tried manually inserting values into the new biblio_framework_marc_matcher table from the db and then editing a framework where that table had an entry for that framework code, but still got the same error.
Comment 11 Julian Maurice 2025-09-30 05:54:39 UTC
(In reply to Andrew Fuerste-Henry from comment #10)
> I applied this patch, ran updatedatabase, ran yarn build, then restart_all.
> Am I missing some step?

Ah sorry I forgot to include it in the test plan: you need to run misc/devel/update_dbix_class_files.pl and restart koha
Comment 12 Andrew Fuerste-Henry 2025-09-30 13:47:44 UTC
Created attachment 187105 [details] [review]
Bug 15248: Use record matching rules to find duplicate while cataloging

This patch allows to configure how Koha searches for duplicates when
cataloging manually by using a record matching rule.

For each framework we can now set a record matching rule to use when
searching for duplicates.
This allow to use different matching rules while in acquisition for
instance (ACQ framework).

If a matching rule is selected, we also have the ability to forbid the
creation of duplicate: if a duplicate is found and this option is
enabled, the usual option to create a duplicate will be hidden.

If no matching rule is selected, then the default matching mechanism is
used.

Test plan:
1. Create a record matching rule matching on ISBN
    Threshold: 100
    Record type: biblio
    Match point:
        Search index: "isbn"
        Score: 100
        Tag: "020" (or "010" for UNIMARC)
        Subfields: "a"
        Offset: 0
        Length: 0
        Normalization rule: Remove spaces
    No match checks
2. Edit the default MARC framework and select the new matching rule and
   enable "Forbid duplicate creation"
3. Create a biblio with an ISBN
4. Make sure your new biblio is indexed in Zebra or ES
5. Try to create a biblio with the same ISBN. You should see a warning
   about the duplicate with no option to create the duplicate biblio
6. In the acquisitions module, in a basket add a new order from an
   external source (requires a configured Z39.50/SRU server)
7. Choose a record with an ISBN
8. Once the order is created, try to create a new order from an external
   source and choose the same record. You should see a warning about the
   duplicate.
9. Edit the record matching rule and disable "Forbid duplicate creation"
10. Redo step 5. Now there should be an option to create the duplicate
    record.
11. Redo step 8. Now there should be an option to create the duplicate
    record.
12. Run the following test scripts:
   - t/db_dependent/Search.t
   - t/db_dependent/Matcher.t

Sponsored-by: SCD Lyon2
Signed-off-by: Andrew Fuerste Henry <andrew@bywatersolutions.com>
Comment 13 Andrew Fuerste-Henry 2025-09-30 13:48:25 UTC
(In reply to Julian Maurice from comment #11)
> Ah sorry I forgot to include it in the test plan: you need to run
> misc/devel/update_dbix_class_files.pl and restart koha

Doh, I shoulda thought of that. Thanks!
I've signed this off, but if someone replaces my SO I'd be happy to QA.
Comment 14 Andrew Fuerste-Henry 2025-09-30 19:16:15 UTC
Created attachment 187119 [details] [review]
Bug 15248: Use record matching rules to find duplicate while cataloging

This patch allows to configure how Koha searches for duplicates when
cataloging manually by using a record matching rule.

For each framework we can now set a record matching rule to use when
searching for duplicates.
This allow to use different matching rules while in acquisition for
instance (ACQ framework).

If a matching rule is selected, we also have the ability to forbid the
creation of duplicate: if a duplicate is found and this option is
enabled, the usual option to create a duplicate will be hidden.

If no matching rule is selected, then the default matching mechanism is
used.

Test plan:
1. Create a record matching rule matching on ISBN
    Threshold: 100
    Record type: biblio
    Match point:
        Search index: "isbn"
        Score: 100
        Tag: "020" (or "010" for UNIMARC)
        Subfields: "a"
        Offset: 0
        Length: 0
        Normalization rule: Remove spaces
    No match checks
2. Edit the default MARC framework and select the new matching rule and
   enable "Forbid duplicate creation"
3. Create a biblio with an ISBN
4. Make sure your new biblio is indexed in Zebra or ES
5. Try to create a biblio with the same ISBN. You should see a warning
   about the duplicate with no option to create the duplicate biblio
6. In the acquisitions module, in a basket add a new order from an
   external source (requires a configured Z39.50/SRU server)
7. Choose a record with an ISBN
8. Once the order is created, try to create a new order from an external
   source and choose the same record. You should see a warning about the
   duplicate.
9. Edit the record matching rule and disable "Forbid duplicate creation"
10. Redo step 5. Now there should be an option to create the duplicate
    record.
11. Redo step 8. Now there should be an option to create the duplicate
    record.
12. Run the following test scripts:
   - t/db_dependent/Search.t
   - t/db_dependent/Matcher.t

Sponsored-by: SCD Lyon2
Signed-off-by: Andrew Fuerste Henry <andrew@bywatersolutions.com>

Signed-off-by: Phillip Berg <phillip.berg@mainlib.org>
Comment 15 Andrew Fuerste-Henry 2025-09-30 20:18:58 UTC
Created attachment 187134 [details] [review]
Bug 15248: Use record matching rules to find duplicate while cataloging

This patch allows to configure how Koha searches for duplicates when
cataloging manually by using a record matching rule.

For each framework we can now set a record matching rule to use when
searching for duplicates.
This allow to use different matching rules while in acquisition for
instance (ACQ framework).

If a matching rule is selected, we also have the ability to forbid the
creation of duplicate: if a duplicate is found and this option is
enabled, the usual option to create a duplicate will be hidden.

If no matching rule is selected, then the default matching mechanism is
used.

Test plan:
1. Create a record matching rule matching on ISBN
    Threshold: 100
    Record type: biblio
    Match point:
        Search index: "isbn"
        Score: 100
        Tag: "020" (or "010" for UNIMARC)
        Subfields: "a"
        Offset: 0
        Length: 0
        Normalization rule: Remove spaces
    No match checks
2. Edit the default MARC framework and select the new matching rule and
   enable "Forbid duplicate creation"
3. Create a biblio with an ISBN
4. Make sure your new biblio is indexed in Zebra or ES
5. Try to create a biblio with the same ISBN. You should see a warning
   about the duplicate with no option to create the duplicate biblio
6. In the acquisitions module, in a basket add a new order from an
   external source (requires a configured Z39.50/SRU server)
7. Choose a record with an ISBN
8. Once the order is created, try to create a new order from an external
   source and choose the same record. You should see a warning about the
   duplicate.
9. Edit the record matching rule and disable "Forbid duplicate creation"
10. Redo step 5. Now there should be an option to create the duplicate
    record.
11. Redo step 8. Now there should be an option to create the duplicate
    record.
12. Run the following test scripts:
   - t/db_dependent/Search.t
   - t/db_dependent/Matcher.t

Sponsored-by: SCD Lyon2
Signed-off-by: Andrew Fuerste Henry <andrew@bywatersolutions.com>

Signed-off-by: Jeanne Maurielle <jeanne.mauriello@mainlib.org>
Comment 16 Andrew Fuerste-Henry 2025-09-30 20:19:46 UTC
I gave sign-off credit to the wrong librarian!