Bug 6713

Summary: marc import not matching
Product: Koha Reporter: Nicole C. Engard <nengard>
Component: CatalogingAssignee: laurenthdl <henridamien.laurent>
Status: CLOSED WORKSFORME QA Contact: Bugs List <koha-bugs>
Severity: critical    
Priority: P3 CC: 1joynelson, chris, henridamien.laurent, jcamins, katrin.fischer, koha.sekjal, wizzyrea
Version: master   
Hardware: All   
OS: All   
Change sponsored?: --- Patch complexity: ---
Documentation contact: Documentation submission:
Text to go in the release notes:
Version(s) released in:
Attachments: Bug 6713, editing matching rules broken
My marc records
marc files not matching
data to test : ? and !
Followup proposed.

Description Nicole C. Engard 2011-08-11 19:51:07 UTC
I did an export from my library collection and then went to import it using the ISBN match.  When the import completed it had found NO matches ... All the records should have been matches. 

I have had two other libraries (one on the koha mailing list) report this very problem, so I think that the matching rules are being ignored in the marc import tool all of the sudden.
Comment 1 Chris Cormack 2011-08-11 20:28:46 UTC
Have tested in master, works for me

Stage MARC Records For Import

MARC Staging results :

    1 records in file
    0 records not staged because of MARC error
    1 records staged
    1 records with at least one match in catalog per matching rule "ISBN"
    1 item records found and staged
Comment 2 Chris Cormack 2011-08-11 20:30:43 UTC
Stage MARC Records For Import

MARC Staging results :

    100 records in file
    0 records not staged because of MARC error
    100 records staged
    81 records with at least one match in catalog per matching rule "ISBN"
    0 item records found and staged

So I currently can't recreate on master, will check out 3.4.x
Comment 3 Ian Walls 2011-08-11 20:31:53 UTC
I've seen this issue on 3.05.00.004 and current HEAD, but NOT on 3.05.00.005, which indicates to me that this is not strictly a code issue, but some combination of data/Zebra indexing.
Comment 4 Chris Cormack 2011-08-11 20:33:06 UTC
I cant replicate on current head of master, or the head of 3.4.x

It does feel like data
Comment 5 Chris Cormack 2011-08-11 20:35:55 UTC
Nicole, can you give me a copy of your marc file, ill try double loading it and see if that makes any difference
Comment 6 Liz Rea 2011-08-11 20:49:49 UTC
editing the rule, even the description in admin -> matching rules breaks the rule.

http://screencast.com/t/GsxcrPp60 - before rule change
http://screencast.com/t/IfySIxaz - after rule change

Exact same import.
Comment 7 Chris Cormack 2011-08-11 20:51:45 UTC
In the db, before edit

mysql> select * from matchpoints;
+------------+---------------+--------------+-------+
| matcher_id | matchpoint_id | search_index | score |
+------------+---------------+--------------+-------+
|          1 |             1 | isbn         |  1000 |
|          2 |             2 | isbn         |  1000 |
+------------+---------------+--------------+-------+

after edit



mysql> select * from matchpoints;
+------------+---------------+--------------+-------+
| matcher_id | matchpoint_id | search_index | score |
+------------+---------------+--------------+-------+
|          2 |             2 | isbn         |  1000 |
|          1 |             3 | isbn         |  1000 |
|          1 |             4 |              |     0 |
|          1 |             5 |              |     0 |
+------------+---------------+--------------+-------+
Comment 8 Chris Cormack 2011-08-11 21:16:54 UTC
The hunt continues, it seems to be adding match checks, even when not told to
Comment 9 Chris Cormack 2011-08-11 22:25:35 UTC Comment hidden (obsolete)
Comment 10 Ian Walls 2011-08-12 11:53:31 UTC
Most of this patch is template scope changes, which are easy to read and appear quite consistent throughout.

The one change in the perl script is obviously right now that I see it.

Marking as Passed QA
Comment 11 Chris Cormack 2011-08-13 08:10:02 UTC
Pushed please test
Comment 12 David Hartman 2011-10-07 19:54:51 UTC
Created attachment 5791 [details]
My marc records

I cannot get my duplicate records to be imported .. please see attached MARC records...
Comment 13 Paul Poulain 2011-10-26 16:15:38 UTC
mmm... I tried to add a matchpoint and feel i've reproduced a bugguy behaviour : i've just added 010a (isbn in unimarc) as matchpoint, and I get :

kohacommunity>select * from matchpoints;
+------------+---------------+--------------+-------+
| matcher_id | matchpoint_id | search_index | score |
+------------+---------------+--------------+-------+
|          1 |             1 | isbn         |   100 |
|          1 |             2 |              |     0 |
|          1 |             3 |              |     0 |
+------------+---------------+--------------+-------+

aren't the id 2 & 3 buggy ? Note i've edited them and no new lines are added :
portable kohacommunity>select * from matchpoints;
+------------+---------------+--------------+-------+
| matcher_id | matchpoint_id | search_index | score |
+------------+---------------+--------------+-------+
|          1 |             4 | isbn         |   100 |
|          1 |             5 |              |     0 |
|          1 |             6 |              |     0 |
+------------+---------------+--------------+-------+
Comment 14 Joy Nelson 2011-11-16 19:43:36 UTC
Created attachment 6321 [details]
marc files not matching
Comment 15 Joy Nelson 2011-11-16 19:44:09 UTC
I am not having luck either with importing matching bib records.  See attached bib records (Little Fockers).  Import rule should cause match on 024a and 245a fields.  Marc file attached.
Comment 16 laurenthdl 2012-02-08 17:11:52 UTC
Created attachment 7510 [details]
data to test : ? and !

biblio records with strings containing ! or ? are not matched.
Comment 17 laurenthdl 2012-02-08 17:27:19 UTC
Created attachment 7511 [details] [review]
Followup proposed.
Comment 18 laurenthdl 2012-02-08 17:30:40 UTC
To Test, use a UNIMARC Koha, and add some rule to check on Title 200$a and ISBN 010$a.
Import the file http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=7510 once, index. and then try to stage the same records with that rule.
Comment 19 Paul Poulain 2012-02-17 14:39:11 UTC
The script devel/misc/testbugzillapatches.pl has detected that this bug does not apply anymore, so i'm marking this bug as "does not apply".

The script is supposed to handle conflicts related to updatedatabase, but no other cases of merge conflict. So the conflict may be trivial to fix.

If it happens that the patch attached does apply cleanly, please send me an email, there's something wrong in the script. It's possible, as it's a recent script.
If the patch applies does not apply as I think, please rebase and resubmit, thanks
Comment 20 Nicole C. Engard 2012-05-15 12:12:42 UTC
Somehow I missed that I was asked for files ... do you still need my files? Looks like you've moved on and found the problem since I reported this. 

Sorry I missed this!!
Comment 21 Katrin Fischer 2012-06-17 07:45:18 UTC
I used sandbox 3 with current master, to do a first test on the matching. I followed hdl's instructions on comment 18. 

I don't see problems with the matching algorithm, the results for my second import were perfect:

627 records in file
0 records not staged because of MARC error
627 records staged
627 records with at least one match in catalog per matching rule "ISBN"

Not sure how to move on from here?
Comment 22 Katrin Fischer 2012-08-14 20:48:41 UTC
Chris, Jared and me have tested it and couldn't reproduce the problem mentioned here. I am closing this as RESOLVED WORKSFORME. Please reopen and update the bug, if there is still a problem.