Bug 14762 - Importing records doesn't add missing subelements
Summary: Importing records doesn't add missing subelements
Status: CLOSED INVALID
Alias: None
Product: Koha
Classification: Unclassified
Component: Tools (show other bugs)
Version: Main
Hardware: All All
: P5 - low minor (vote)
Assignee: Galen Charlton
QA Contact: Testopia
URL:
Keywords:
Depends on:
Blocks: 14755
  Show dependency treegraph
 
Reported: 2015-08-31 15:24 UTC by Joonas Kylmälä
Modified: 2016-06-21 21:36 UTC (History)
3 users (show)

See Also:
Change sponsored?: ---
Patch complexity: ---
Documentation contact:
Documentation submission:
Text to go in the release notes:
Version(s) released in:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Joonas Kylmälä 2015-08-31 15:24:58 UTC
If using stage-marc-import.pl it doesn't generate the missing data to bibliographic records (subelements, updating some data, etc.).
Comment 1 Katrin Fischer 2015-08-31 21:05:36 UTC
Can you describe the problem a bit more? What should be added/is missing?
Comment 2 Joonas Kylmälä 2015-09-01 05:08:34 UTC
Steps to reproduce the problem:

1. Have an imported biblio record (from maybe some other version of koha / with some other library's details) which you haven't edited yet but only created a new item for it.
2. Go to the biblio records detail view
3. Save the record by clicking Save->MARCXML
4. Click Edit->Record
5. Click Save
6. Save the record by clicking Save->MARCXML
7. Compare the output of the two files, for example with diff -u <file>.
8. Notice that the latter file has changed its info / there is more info.

So the above info adding doesn't happen in the stage-marc-import.pl script and it is wasted time to go each and one thousand records by hand and just click save on them (you need to add this info or otherwise the bug 14755 appears).
Comment 3 Katrin Fischer 2015-09-01 05:59:28 UTC
Joonas, this is rather confusing me :) We use the staged imports a lot and have never run into problems like you describe. Could you show which elements were missing in your record?
Comment 4 Joonas Kylmälä 2015-09-01 06:24:32 UTC
(In reply to Katrin Fischer from comment #3)
> Joonas, this is rather confusing me :) We use the staged imports a lot and
> have never run into problems like you describe. Could you show which
> elements were missing in your record?

Output from diff of MARCXML files:

...

-  <leader>00977cam a22003374a 4500</leader>
+  <leader>00961cam a22003374a 4500</leader>

...

-  <controlfield tag="005">20150831135648.0</controlfield>
+  <controlfield tag="005">20150831142850.0</controlfield>

...

-    <subfield code="c">KI</subfield>
-    <subfield code="1">2013-08-26 00:00:00</subfield>
+    <subfield code="c">BK</subfield>
+    <subfield code="2">ddc</subfield>

After those changes were made the bug 14755 didn't show up anymore.
Comment 5 Katrin Fischer 2015-09-01 06:58:29 UTC
Hm, this looks to me like the bug is not so much in the staged import, but in the data - it looks like you might be importing with an itemtype code that is not defined in your database (942$c). Once you open and save the record again, the itemtype will probably be set to the first in the pull down of available codes... and that's why the checkout works after. Could this be possible?
Comment 6 Joonas Kylmälä 2015-09-01 07:13:54 UTC
(In reply to Katrin Fischer from comment #5)
> Hm, this looks to me like the bug is not so much in the staged import, but
> in the data - it looks like you might be importing with an itemtype code
> that is not defined in your database (942$c). Once you open and save the
> record again, the itemtype will probably be set to the first in the pull
> down of available codes... and that's why the checkout works after. Could
> this be possible?

Adding an item type entry in Koha for code KI solved the problem indeed.

But I think the feature Edit biblio adds still some other data. So why not do that already in import stage, because some data might be vital(?).

-  <leader>00977cam a22003374a 4500</leader>
+  <leader>00961cam a22003374a 4500</leader>

For example, should the line above be like that or is it enough that there is the old line / is this some info which we really don't need but we just happen to have?

And I got the picture from that Edit record feature, by quickly looking at the code, that it also adds some missing subfields. So what if these imported records don't have those subfields and it will cause some problems... (and then the only fix would be go one by one through the imported records or fixing the data some how)
Comment 7 Katrin Fischer 2015-09-01 07:28:22 UTC
005 is a timestamp of the last change and the first positions of the leader that change are the record length - this change being caused by the other changes. 
So I think Koha's current behaviour is correct and nothing needs to be changed on import. I think what we could do is catch an invalid itemtype code better, so it's easier to locate the problem.
Comment 8 paxed 2015-09-01 09:57:16 UTC
(In reply to Joonas Kylmälä from comment #6)
> 
> -  <leader>00977cam a22003374a 4500</leader>
> +  <leader>00961cam a22003374a 4500</leader>
> 
> For example, should the line above be like that or is it enough that there
> is the old line / is this some info which we really don't need but we just
> happen to have?
> 

vink vink, Kansalliskirjastolta löytyy MARC21 suomeksi:
http://www.kansalliskirjasto.fi/extra/marc21/bib/000.htm
Comment 9 Joonas Kylmälä 2015-09-02 07:29:32 UTC
(In reply to Katrin Fischer from comment #7)
> 005 is a timestamp of the last change and the first positions of the leader
> that change are the record length - this change being caused by the other
> changes. 
> So I think Koha's current behaviour is correct and nothing needs to be
> changed on import. I think what we could do is catch an invalid itemtype
> code better, so it's easier to locate the problem.

What about the missing subfields/something? I don't know how much they will have effect on how things work but I'll try to look into the code of the Edit record feature and see if we should add those kind of things to this import tool, too.
Comment 10 Katrin Fischer 2015-09-02 08:07:04 UTC
Which missing subfields? 999 information with bilbionumber/biblioitemnumber etc. is added correctly (or nothing would work). I can't think of any other fields right now that the import would change in the record.
Comment 11 Joonas Kylmälä 2015-09-02 08:26:27 UTC
(In reply to Katrin Fischer from comment #10)
> etc. is added correctly (or nothing would work). I can't think of any other
> fields right now that the import would change in the record.

Not change, but add. But I will look into the code soon.. and then we can be sure.
Comment 12 Joonas Kylmälä 2015-09-02 08:59:13 UTC
(In reply to Joonas Kylmälä from comment #11)
> (In reply to Katrin Fischer from comment #10)
> > etc. is added correctly (or nothing would work). I can't think of any other
> > fields right now that the import would change in the record.
> 
> Not change, but add. But I will look into the code soon.. and then we can be
> sure.

Ok. authorities.pl and addbiblio.pl does the missing subfield adding as I understood by reading the code quickly. The code starts at line 590 in addbiblio.pl. Katrin, do you know by that info if that is something we need? (if not, why do we have it in addbiblio.pl and authorities.pl?).
Comment 13 Katrin Fischer 2015-09-02 09:22:38 UTC
To me the code looks to be mostly about building the cataloguing form correctly - I can't spot any elements being added. I feel a bit like we are hunting 'ghosts' here, as there is no known problem to work from and reading all the code will always turn up some/lots of question marks.
Comment 14 Joonas Kylmälä 2015-09-02 10:40:24 UTC
Looks like this is then resolved. I'll open a new report for the invalid item type thingie.
Comment 15 Tomás Cohen Arazi 2015-09-02 10:59:21 UTC
(In reply to Joonas Kylmälä from comment #2)
> So the above info adding doesn't happen in the stage-marc-import.pl script
> and it is wasted time to go each and one thousand records by hand and just
> click save on them

Just for yupi to notice, Joonas, that's what touch_all_biblios.pl is for ☺