Currently, FindDuplicate tries to find a duplicate by doing a search, that cannot be tuned by the library. It would be better to have the opportunity to use the matching points system. A "FindDuplicate" matching point entry could be added, set to title/author -current behaviour of FindDuplicate-, that the library can update/change to fit it's needs.
I like this idea! I was just about to open a ticket because I wanted to add a few indexes (like identifier-standard for 024$a and 022$a and Control-number for 001)to FindDuplicate (which have already been implemented locally). Using record matching rules might really be the best way to do this, since different libraries might have different needs/wants for how to find duplicates.
We'd love to add 001 to the checks as this is the most important field for our synchronization with the union catalog.
Just an idea: could we make it possible to use the matching rules for the duplicate check? I think it would probably work for our use case and would offer some flexibility.
(In reply to Katrin Fischer from comment #3) > Just an idea: could we make it possible to use the matching rules for the > duplicate check? I think it would probably work for our use case and would > offer some flexibility. We've had a library ask us to implement exactly this; watch this space :)
(In reply to Andrew Fuerste-Henry from comment #4) > (In reply to Katrin Fischer from comment #3) > > Just an idea: could we make it possible to use the matching rules for the > > duplicate check? I think it would probably work for our use case and would > > offer some flexibility. > > We've had a library ask us to implement exactly this; watch this space :) Awesome news, will do!
(In reply to Andrew Fuerste-Henry from comment #4) > (In reply to Katrin Fischer from comment #3) > > Just an idea: could we make it possible to use the matching rules for the > > duplicate check? I think it would probably work for our use case and would > > offer some flexibility. > > We've had a library ask us to implement exactly this; watch this space :) We had to do it too... I'll submit my patch here. Hopefully this is similar to what you had in mind.
Created attachment 187049 [details] [review] Bug 15248: Use record matching rules to find duplicate while cataloging This patch allows to configure how Koha searches for duplicates when cataloging manually by using a record matching rule. For each framework we can now set a record matching rule to use when searching for duplicates. This allow to use different matching rules while in acquisition for instance (ACQ framework). If a matching rule is selected, we also have the ability to forbid the creation of duplicate: if a duplicate is found and this option is enabled, the usual option to create a duplicate will be hidden. If no matching rule is selected, then the default matching mechanism is used. Test plan: 1. Create a record matching rule matching on ISBN Threshold: 100 Record type: biblio Match point: Search index: "isbn" Score: 100 Tag: "020" (or "010" for UNIMARC) Subfields: "a" Offset: 0 Length: 0 Normalization rule: Remove spaces No match checks 2. Edit the default MARC framework and select the new matching rule and enable "Forbid duplicate creation" 3. Create a biblio with an ISBN 4. Make sure your new biblio is indexed in Zebra or ES 5. Try to create a biblio with the same ISBN. You should see a warning about the duplicate with no option to create the duplicate biblio 6. In the acquisitions module, in a basket add a new order from an external source (requires a configured Z39.50/SRU server) 7. Choose a record with an ISBN 8. Once the order is created, try to create a new order from an external source and choose the same record. You should see a warning about the duplicate. 9. Edit the record matching rule and disable "Forbid duplicate creation" 10. Redo step 5. Now there should be an option to create the duplicate record. 11. Redo step 8. Now there should be an option to create the duplicate record. 12. Run the following test scripts: - t/db_dependent/Search.t - t/db_dependent/Matcher.t Sponsored-by: SCD Lyon2
Created attachment 187050 [details] [review] Bug 15248: Use record matching rules to find duplicate while cataloging This patch allows to configure how Koha searches for duplicates when cataloging manually by using a record matching rule. For each framework we can now set a record matching rule to use when searching for duplicates. This allow to use different matching rules while in acquisition for instance (ACQ framework). If a matching rule is selected, we also have the ability to forbid the creation of duplicate: if a duplicate is found and this option is enabled, the usual option to create a duplicate will be hidden. If no matching rule is selected, then the default matching mechanism is used. Test plan: 1. Create a record matching rule matching on ISBN Threshold: 100 Record type: biblio Match point: Search index: "isbn" Score: 100 Tag: "020" (or "010" for UNIMARC) Subfields: "a" Offset: 0 Length: 0 Normalization rule: Remove spaces No match checks 2. Edit the default MARC framework and select the new matching rule and enable "Forbid duplicate creation" 3. Create a biblio with an ISBN 4. Make sure your new biblio is indexed in Zebra or ES 5. Try to create a biblio with the same ISBN. You should see a warning about the duplicate with no option to create the duplicate biblio 6. In the acquisitions module, in a basket add a new order from an external source (requires a configured Z39.50/SRU server) 7. Choose a record with an ISBN 8. Once the order is created, try to create a new order from an external source and choose the same record. You should see a warning about the duplicate. 9. Edit the record matching rule and disable "Forbid duplicate creation" 10. Redo step 5. Now there should be an option to create the duplicate record. 11. Redo step 8. Now there should be an option to create the duplicate record. 12. Run the following test scripts: - t/db_dependent/Search.t - t/db_dependent/Matcher.t Sponsored-by: SCD Lyon2
(In reply to Julian Maurice from comment #6) > (In reply to Andrew Fuerste-Henry from comment #4) > > (In reply to Katrin Fischer from comment #3) > > > Just an idea: could we make it possible to use the matching rules for the > > > duplicate check? I think it would probably work for our use case and would > > > offer some flexibility. > > > > We've had a library ask us to implement exactly this; watch this space :) > > We had to do it too... > I'll submit my patch here. Hopefully this is similar to what you had in mind. Testing now! Your test plan suggests this will work for our partner. I like the additional framework option to entirely block bib creation on a detected duplicate!
Koha dies when I try to edit any framework. Editing tags and subfields is fine, but I cannot edit the framework itself. I get: "DBIx::Class::Schema::source(): Can't find source for BiblioFrameworkMarcMatcher at /kohadevbox/koha/Koha/Objects.pm line 530 at /usr/share/perl5/DBIx/Class/Exception.pm line 77" I applied this patch, ran updatedatabase, ran yarn build, then restart_all. Am I missing some step? I tried manually inserting values into the new biblio_framework_marc_matcher table from the db and then editing a framework where that table had an entry for that framework code, but still got the same error.
(In reply to Andrew Fuerste-Henry from comment #10) > I applied this patch, ran updatedatabase, ran yarn build, then restart_all. > Am I missing some step? Ah sorry I forgot to include it in the test plan: you need to run misc/devel/update_dbix_class_files.pl and restart koha
Created attachment 187105 [details] [review] Bug 15248: Use record matching rules to find duplicate while cataloging This patch allows to configure how Koha searches for duplicates when cataloging manually by using a record matching rule. For each framework we can now set a record matching rule to use when searching for duplicates. This allow to use different matching rules while in acquisition for instance (ACQ framework). If a matching rule is selected, we also have the ability to forbid the creation of duplicate: if a duplicate is found and this option is enabled, the usual option to create a duplicate will be hidden. If no matching rule is selected, then the default matching mechanism is used. Test plan: 1. Create a record matching rule matching on ISBN Threshold: 100 Record type: biblio Match point: Search index: "isbn" Score: 100 Tag: "020" (or "010" for UNIMARC) Subfields: "a" Offset: 0 Length: 0 Normalization rule: Remove spaces No match checks 2. Edit the default MARC framework and select the new matching rule and enable "Forbid duplicate creation" 3. Create a biblio with an ISBN 4. Make sure your new biblio is indexed in Zebra or ES 5. Try to create a biblio with the same ISBN. You should see a warning about the duplicate with no option to create the duplicate biblio 6. In the acquisitions module, in a basket add a new order from an external source (requires a configured Z39.50/SRU server) 7. Choose a record with an ISBN 8. Once the order is created, try to create a new order from an external source and choose the same record. You should see a warning about the duplicate. 9. Edit the record matching rule and disable "Forbid duplicate creation" 10. Redo step 5. Now there should be an option to create the duplicate record. 11. Redo step 8. Now there should be an option to create the duplicate record. 12. Run the following test scripts: - t/db_dependent/Search.t - t/db_dependent/Matcher.t Sponsored-by: SCD Lyon2 Signed-off-by: Andrew Fuerste Henry <andrew@bywatersolutions.com>
(In reply to Julian Maurice from comment #11) > Ah sorry I forgot to include it in the test plan: you need to run > misc/devel/update_dbix_class_files.pl and restart koha Doh, I shoulda thought of that. Thanks! I've signed this off, but if someone replaces my SO I'd be happy to QA.
Created attachment 187119 [details] [review] Bug 15248: Use record matching rules to find duplicate while cataloging This patch allows to configure how Koha searches for duplicates when cataloging manually by using a record matching rule. For each framework we can now set a record matching rule to use when searching for duplicates. This allow to use different matching rules while in acquisition for instance (ACQ framework). If a matching rule is selected, we also have the ability to forbid the creation of duplicate: if a duplicate is found and this option is enabled, the usual option to create a duplicate will be hidden. If no matching rule is selected, then the default matching mechanism is used. Test plan: 1. Create a record matching rule matching on ISBN Threshold: 100 Record type: biblio Match point: Search index: "isbn" Score: 100 Tag: "020" (or "010" for UNIMARC) Subfields: "a" Offset: 0 Length: 0 Normalization rule: Remove spaces No match checks 2. Edit the default MARC framework and select the new matching rule and enable "Forbid duplicate creation" 3. Create a biblio with an ISBN 4. Make sure your new biblio is indexed in Zebra or ES 5. Try to create a biblio with the same ISBN. You should see a warning about the duplicate with no option to create the duplicate biblio 6. In the acquisitions module, in a basket add a new order from an external source (requires a configured Z39.50/SRU server) 7. Choose a record with an ISBN 8. Once the order is created, try to create a new order from an external source and choose the same record. You should see a warning about the duplicate. 9. Edit the record matching rule and disable "Forbid duplicate creation" 10. Redo step 5. Now there should be an option to create the duplicate record. 11. Redo step 8. Now there should be an option to create the duplicate record. 12. Run the following test scripts: - t/db_dependent/Search.t - t/db_dependent/Matcher.t Sponsored-by: SCD Lyon2 Signed-off-by: Andrew Fuerste Henry <andrew@bywatersolutions.com> Signed-off-by: Phillip Berg <phillip.berg@mainlib.org>
Created attachment 187134 [details] [review] Bug 15248: Use record matching rules to find duplicate while cataloging This patch allows to configure how Koha searches for duplicates when cataloging manually by using a record matching rule. For each framework we can now set a record matching rule to use when searching for duplicates. This allow to use different matching rules while in acquisition for instance (ACQ framework). If a matching rule is selected, we also have the ability to forbid the creation of duplicate: if a duplicate is found and this option is enabled, the usual option to create a duplicate will be hidden. If no matching rule is selected, then the default matching mechanism is used. Test plan: 1. Create a record matching rule matching on ISBN Threshold: 100 Record type: biblio Match point: Search index: "isbn" Score: 100 Tag: "020" (or "010" for UNIMARC) Subfields: "a" Offset: 0 Length: 0 Normalization rule: Remove spaces No match checks 2. Edit the default MARC framework and select the new matching rule and enable "Forbid duplicate creation" 3. Create a biblio with an ISBN 4. Make sure your new biblio is indexed in Zebra or ES 5. Try to create a biblio with the same ISBN. You should see a warning about the duplicate with no option to create the duplicate biblio 6. In the acquisitions module, in a basket add a new order from an external source (requires a configured Z39.50/SRU server) 7. Choose a record with an ISBN 8. Once the order is created, try to create a new order from an external source and choose the same record. You should see a warning about the duplicate. 9. Edit the record matching rule and disable "Forbid duplicate creation" 10. Redo step 5. Now there should be an option to create the duplicate record. 11. Redo step 8. Now there should be an option to create the duplicate record. 12. Run the following test scripts: - t/db_dependent/Search.t - t/db_dependent/Matcher.t Sponsored-by: SCD Lyon2 Signed-off-by: Andrew Fuerste Henry <andrew@bywatersolutions.com> Signed-off-by: Jeanne Maurielle <jeanne.mauriello@mainlib.org>
I gave sign-off credit to the wrong librarian!