Bug 5858 - Update default MARC21 framework to Update No. 16 (April 2013)
Summary: Update default MARC21 framework to Update No. 16 (April 2013)
Status: CLOSED FIXED
Alias: None
Product: Koha
Classification: Unclassified
Component: MARC Bibliographic data support (show other bugs)
Version: master
Hardware: All All
: P5 - low enhancement (vote)
Target Milestone: ---
Assignee: Bernardo Gonzalez Kriegel
QA Contact: Marcel de Rooy
URL:
Whiteboard:
Keywords:
: 3684 5587 (view as bug list)
Depends on:
Blocks:
 
Reported: 2011-03-13 15:08 UTC by Jared Camins-Esakov
Modified: 2014-12-07 20:03 UTC (History)
13 users (show)

See Also:
Change sponsored?: ---
Patch complexity: ---
Bot Control: ---
When did the bot last check this:
Who signed the patch off:
Text to go in the release notes:
Version(s) released in:


Attachments
Bug 5858 - Updating the marc21_framework_DEFAULT.sql to MARC21 version 14 (266.65 KB, patch)
2012-06-20 22:02 UTC, Chris Cormack
Details | Diff | Splinter Review
[SIGNED-OFF] Bug 5858 - Updating the marc21_framework_DEFAULT.sql to MARC21 version 14 (266.77 KB, patch)
2013-02-12 12:26 UTC, Bernardo Gonzalez Kriegel
Details | Diff | Splinter Review
Followup Bug 5858 - Default frameworks missing many MARC21 tags (104.83 KB, patch)
2013-02-12 12:27 UTC, Bernardo Gonzalez Kriegel
Details | Diff | Splinter Review
Bug 5858 - Default frameworks missing many MARC21 tags (157.86 KB, patch)
2013-03-20 00:40 UTC, Bernardo Gonzalez Kriegel
Details | Diff | Splinter Review
Bug 5858 - Default frameworks missing many MARC21 tags (158.50 KB, patch)
2013-04-10 19:36 UTC, Bernardo Gonzalez Kriegel
Details | Diff | Splinter Review
Bug 5858 - Default frameworks missing many MARC21 tags (158.59 KB, patch)
2013-06-01 01:40 UTC, Mason James
Details | Diff | Splinter Review
[FOLLOWUP] Bug 5858 - Default frameworks missing many MARC21 tags (5.91 KB, patch)
2013-06-01 01:57 UTC, Mason James
Details | Diff | Splinter Review
[FOLLOWUP 2] Bug 5858 - Update default MARC21 framework (54.89 KB, patch)
2013-07-02 00:53 UTC, Bernardo Gonzalez Kriegel
Details | Diff | Splinter Review
Bug 5858 - Default frameworks missing many MARC21 tags (160.87 KB, patch)
2013-07-03 14:32 UTC, Marcel de Rooy
Details | Diff | Splinter Review
Bug 5858 - Default frameworks missing many MARC21 tags (5.96 KB, patch)
2013-07-03 14:32 UTC, Marcel de Rooy
Details | Diff | Splinter Review
Bug 5858 - Update default MARC21 framework to Update No. 16 (April 2013) (91.89 KB, patch)
2013-07-03 14:32 UTC, Marcel de Rooy
Details | Diff | Splinter Review
Bug 5858 - Default frameworks missing many MARC21 tags (202.62 KB, patch)
2013-07-07 11:57 UTC, Marcel de Rooy
Details | Diff | Splinter Review
Bug 5858: Followup for INSERT IGNOREs in marc21_framework_DEFAULT.sql (7.27 KB, patch)
2013-07-07 12:48 UTC, Marcel de Rooy
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description Jared Camins-Esakov 2011-03-13 15:08:34 UTC
The default frameworks for MARC21 are missing many tags, seemingly all introduced after 2006. This bug does not address missing subfields or redefinitions. The missing tags are:
083 - Additional Dewey Decimal Classification Number
085 - Synthesized Classification Number Components
336 - Content Type
337 - Media Type
338 - Carrier Type
363 - Normalized date and sequential designation
380 - Form of Work
381 - Other Distinguishing Characteristics of Work or Expression
382 - Medium of Performance
383 - Numeric Designation of Musical Work
384 - Key
542 - Information Relating to Copyright Status
545 - Biographical or historical data
588 - Source of description note
751 - Added entry - geographic name
882 - Replacement record information

For summaries of changes to the MARC21 standard starting 2007 see:
http://replay.waybackmachine.org/20080915133838/http://www.loc.gov/marc/bibliographic/bdapndxg.html
http://replay.waybackmachine.org/20090907173218/http://www.loc.gov/marc/bibliographic/bdapndxg.html
http://replay.waybackmachine.org/20100311061810/http://www.loc.gov/marc/bibliographic/bdapndxg.html
http://www.loc.gov/marc/bibliographic/bdapndxg.html
Comment 1 Jared Camins-Esakov 2011-03-13 15:09:34 UTC
*** Bug 5587 has been marked as a duplicate of this bug. ***
Comment 2 Jared Camins-Esakov 2011-03-13 15:29:09 UTC
Subfields that are probably missing (I did not check all these subfields in the default frameworks, but I thought the list of new subfields might be of use to someone, even unproofed):

015 $z - Canceled/Invalid national bibliography number (National bibliography number)
017 $z - Canceled/Invalid Copyright or Legal Deposit Number (Copyright or Legal Deposit Number)
022 $l - ISSN-L (ISSN)
022 $m - Canceled ISSN-L (ISSN)
033 $0 - Record control number (R) (Date/Time and Place of an Event)
033 $2 - Source of term (R) (Date/Time and Place of an Event)
033 $p - Place of event (R) (Date/Time and Place of an Event)
034 $3 - Materials specified (NR) (Coded Cartographic Mathematical Data)
041 $j - Language code of subtitles or captions (Language code)
082 $m - Standard or Optional Designation (Dewey Decimal Classification Number)
082 $q - Assigning agency (Dewey Decimal Classification Number)
100 $0 - Authority record control number (Main entry -- Personal name)
110 $0 - Authority record control number (Main entry -- Corporate name)
111 $0 - Authority record control number (Main entry -- Meeting name)
130 $0 - Authority record control number (Main entry -- Uniform title)
240 $0 - Authority record control number (Uniform title)
257 $2 - Source (NR) (Country of Producing Entity)
440 $0 - Authority record control number (Series statement/Added entry -- Title)
440 $w - Bibliographic record control number (Series statement/Added entry -- Title)
490 $3 - Materials Specified (Series Statement)
502 $b - Degree Type (Dissertation Note)
502 $c - Name of Granting Institution (Dissertation Note)
502 $d - Year of Degree Granted (Dissertation Note)
502 $g - Miscellaneous Information (Dissertation Note)
502 $o - Dissertation Identifier (Dissertation Note)
510 $u - Uniform Resource Identifier (R) (Citation/References Note)
518 $0 - Record control number (R) (Date/Time and Place of an Event Note)
518 $2 - Source of term (R) (Date/Time and Place of an Event Note)
518 $d - Date of event (R) (Date/Time and Place of an Event Note)
518 $o - Other event information (R) (Date/Time and Place of an Event Note)
518 $p - Place of event (R) (Date/Time and Place of an Event Note)
520 $2 - Source (Summary, etc.)
520 $c - Assigning agency (Summary, etc.)
533 $5 - Institution to which field applies (Reproduction note)
534 $3 - Materials specified (NR) (Original Version Note)
534 $o - Other Resource Identifier (Original Version Note)
538 $5 - Institution to which field applies (System details note)
561 $u - Uniform Resource Identifier (R) (Ownership and Custodial History)
600 $0 - Authority record control number (Subject added entry -- Personal name)
610 $0 - Authority record control number (Subject added entry -- Corporate name)
611 $0 - Authority record control number (Subject added entry -- Meeting name)
630 $0 - Authority record control number (Subject added entry -- Uniform title)
648 $0 - Authority record control number (Subject added entry -- Chronological term)
650 $0 - Authority record control number (Subject added entry -- Topical term)
651 $0 - Authority record control number (Subject added entry -- Geographic name)
654 $0 - Authority record control number (Subject added entry -- Faceted topical terms)
655 $0 - Authority record control number (Index term -- Genre/form)
656 $0 - Authority record control number (Index term -- Occupation)
657 $0 - Authority record control number (Index term -- Function)
662 $0 - Authority record control number (Subject added entry -- Hierarchical place name)
700 $0 - Authority record control number (Added entry -- Personal name)
700 $i - Relationship information (R) (Added Entry-Personal Name)
710 $0 - Authority record control number (Added entry -- Corporate name)
710 $i - Relationship information (R) (Added Entry-Corporate Name)
711 $0 - Authority record control number (Added entry -- Meeting name)
711 $i - Relationship information (R) (Added Entry-Meeting Name)
730 $0 - Authority record control number (Added entry -- Uniform title)
730 $i - Relationship information (R) (Added Entry-Uniform Title)
752 $0 - Authority record control number (Added entry -- Hierarchical place name)
754 $0 - Authority record control number (Added entry -- Taxonomic identification)
760-787 $4 - Relationship code (R) (Linking Entry fields)
800 $0 - Authority record control number (Series added entry -- Personal name)
800 $3 - Materials Specified (Series Added Entry-Personal Name)
800 $5 - Institution to which field applies (R) (Series Added Entry-Personal Name)
800 $x - International Standard Serial Number (Series Added Entry-Personal Name)
810 $0 - Authority record control number (Series added entry -- Corporate name)
810 $3 - Materials Specified (Series Added Entry-Corporate Name)
810 $5 - Institution to which field applies (R) (Series Added Entry-Corporate Name)
810 $x - International Standard Serial Number (Series Added Entry-Corporate Name)
811 $0 - Authority record control number (Series added entry -- Meeting name)
811 $3 - Materials Specified (Series Added Entry-Meeting Name)
811 $5 - Institution to which field applies (R) (Series Added Entry-Meeting Name)
811 $x - International Standard Serial Number (Series Added Entry-Meeting Name)
830 $0 - Authority record control number (Series added entry -- Uniform title)
830 $3 - Materials Specified (Series Added Entry-Uniform Title)
830 $5 - Institution to which field applies (R) (Series Added Entry-Uniform Title)
830 $x - International Standard Serial Number (Series Added Entry-Uniform Title)
852 $d - Former shelving location (Location)
Comment 3 Chris Cormack 2012-06-20 22:02:51 UTC Comment hidden (obsolete)
Comment 4 Chris Cormack 2012-06-20 22:03:46 UTC
This is currently english only.
Comment 5 Frédéric Demians 2012-07-10 16:50:20 UTC
What prevents this patch to be signed-off? It seems all good.
Comment 6 Jared Camins-Esakov 2012-07-10 16:53:25 UTC
(In reply to comment #5)
> What prevents this patch to be signed-off? It seems all good.

I thought I had signed it off, actually. I tested it and was quite pleased with how it worked on a new install.
Comment 7 Katrin Fischer 2012-07-10 17:19:52 UTC
It seems it's not only adding, but also deleting a lot of fields? I did only look at the patch, maybe I am missing something.
Comment 8 Chris Cormack 2012-07-11 10:37:30 UTC
(In reply to comment #7)
> It seems it's not only adding, but also deleting a lot of fields? I did only
> look at the patch, maybe I am missing something.

Yeah its getting rid of all the deprecated fields.

This is why its only good for a new install and not in updatedatabase.pl
Comment 9 Katrin Fischer 2012-07-11 11:12:03 UTC
I am not sure deleting them is a good idea - I would have to test it, but is there a chance we loose data if someone imports data that uses fields that are not defined in the default framework? I think I would prefer to see them hidden and marked [OBSOLETE] to be sure no data can be lost.
Comment 10 Magnus Enger 2012-08-10 09:42:41 UTC
(In reply to comment #9)
> I am not sure deleting them is a good idea - I would have to test it, but is
> there a chance we loose data if someone imports data that uses fields that
> are not defined in the default framework? I think I would prefer to see them
> hidden and marked [OBSOLETE] to be sure no data can be lost.

Should this be marked as "in discussion"?
Comment 11 Chris Cormack 2012-10-17 06:50:55 UTC
*** Bug 3684 has been marked as a duplicate of this bug. ***
Comment 12 Bernardo Gonzalez Kriegel 2013-02-09 12:13:04 UTC
(In reply to comment #9)
> I am not sure deleting them is a good idea - I would have to test it, but is
> there a chance we loose data if someone imports data that uses fields that
> are not defined in the default framework? I think I would prefer to see them
> hidden and marked [OBSOLETE] to be sure no data can be lost.

I did some tests. 
Data is not lost when you import, in case some field is not present in frameworks.
But if you edit and save the record, then it's lost.

So, to prevent data loss, it would be convenient to hide the fields.

I volunteer to write a followup to do that.
Is that enough to proceed? 
The missing fields are real.

Regards
Comment 13 Katrin Fischer 2013-02-10 21:27:26 UTC
I confirm the data loss upon edit/save. I recently ran into it again...
I think adding the missing fields and hiding unused/deprecated fields would be great.
Comment 14 Bernardo Gonzalez Kriegel 2013-02-12 12:26:45 UTC Comment hidden (obsolete)
Comment 15 Bernardo Gonzalez Kriegel 2013-02-12 12:27:25 UTC Comment hidden (obsolete)
Comment 16 Owen Leonard 2013-03-19 16:47:51 UTC
I still see many instances of fields which are removed altogether. Should there be nothing removed?
Comment 17 Bernardo Gonzalez Kriegel 2013-03-19 16:50:48 UTC
I will check and resubmit.
Comment 18 Bernardo Gonzalez Kriegel 2013-03-20 00:40:17 UTC Comment hidden (obsolete)
Comment 19 Bernardo Gonzalez Kriegel 2013-04-10 19:36:54 UTC Comment hidden (obsolete)
Comment 20 Mason James 2013-06-01 01:40:42 UTC Comment hidden (obsolete)
Comment 21 Mason James 2013-06-01 01:57:09 UTC Comment hidden (obsolete)
Comment 22 Mason James 2013-06-01 03:33:39 UTC
(In reply to comment #9)
> I am not sure deleting them is a good idea - I would have to test it, but is
> there a chance we loose data if someone imports data that uses fields that
> are not defined in the default framework? I think I would prefer to see them
> hidden and marked [OBSOLETE] to be sure no data can be lost.

it looks like the official MARC21 spec does this too
-> http://www.loc.gov/marc/bibliographic/ecbdlist.html

ie: it does not delete old fields/subfields, but just marks them [OBSOLETE]
Comment 23 Mason James 2013-06-01 03:35:44 UTC
(In reply to comment #20)
> Created attachment 18584 [details] [review] [review]
> Bug 5858 - Default frameworks missing many MARC21 tags
>
> Signed-off-by: Mason James <mtj@kohaaloha.com>

i've taken a look a Bernardo's patch, and tested it

it looks good
Comment 24 Mason James 2013-06-11 03:43:42 UTC
(In reply to comment #23)
> (In reply to comment #20)
> > Created attachment 18584 [details] [review] [review] [review]
> > Bug 5858 - Default frameworks missing many MARC21 tags
> >
> > Signed-off-by: Mason James <mtj@kohaaloha.com>
> 
> i've taken a look a Bernardo's patch, and tested it
> 
> it looks good

i've signed-off on Bernardo's patch

just awaiting sign-off on my follow-up patch...
Comment 25 Marcel de Rooy 2013-07-01 12:23:15 UTC
Looking at this now..
Comment 26 Marcel de Rooy 2013-07-01 12:34:39 UTC
Mason:
Something is wrong with your followup. 
Look here:

        ('383', 'NUMERIC DESIGNATION OF MUSICAL WORK', 'NUMERIC DESIGNATION OF MUSICAL WORK', 0, 1, NULL, ''),
 0, 1, NULL, ''),
UDIENCE CHARACTERISTICS', 'AUDIENCE CHARACTERISTICS', 1, 0, NULL, ''),

Bernardo or Mason: please correct! 

Failed QA
Comment 27 Bernardo Gonzalez Kriegel 2013-07-02 00:53:49 UTC Comment hidden (obsolete)
Comment 28 Marcel de Rooy 2013-07-03 13:28:39 UTC
fatal: sha1 information is lacking or useless (installer/data/mysql/en/marcflavour/marc21/mandatory/marc21_framework_DEFAULT.sql).
Repository lacks necessary blobs to fall back on 3-way merge.
Cannot fall back to three-way merge.
Patch failed at 0001 Bug 5858 - Update default MARC21 framework to Update No. 16 (April 2013)
Comment 29 Bernardo Gonzalez Kriegel 2013-07-03 14:06:31 UTC
(In reply to M. de Rooy from comment #28)
> fatal: sha1 information is lacking or useless
> (installer/data/mysql/en/marcflavour/marc21/mandatory/
> marc21_framework_DEFAULT.sql).
> Repository lacks necessary blobs to fall back on 3-way merge.
> Cannot fall back to three-way merge.
> Patch failed at 0001 Bug 5858 - Update default MARC21 framework to Update
> No. 16 (April 2013)

Strange

~/kohaclone$ git pull
Already up-to-date.
~/kohaclone$ git checkout bug_5858-B
Switched to branch 'bug_5858-B'
Your branch is behind 'origin/master' by 11 commits, and can be fast-forwarded.
~/kohaclone$ git rebase master
First, rewinding head to replay your work on top of it...
Fast-forwarded bug_5858-B to master.
~/kohaclone$ git bz apply 5858
Bug 5858 - Update default MARC21 framework to Update No. 16 (April 2013)

Bug 5858 - Default frameworks missing many MARC21 tags
Apply? [yn] y

Applying: Bug 5858 - Default frameworks missing many MARC21 tags
[FOLLOWUP] Bug 5858 - Default frameworks missing many MARC21 tags
Apply? [yn] y

Applying: Bug 5858 - Default frameworks missing many MARC21 tags
[FOLLOWUP 2] Bug 5858 - Update default MARC21 framework
Apply? [yn] y

Applying: Bug 5858 - Update default MARC21 framework to Update No. 16 (April 2013)
~/kohaclone$

For me it's applying without problem, should I switch to Needs Singoff?
Comment 30 Marcel de Rooy 2013-07-03 14:07:48 UTC
Bernardo:
Try to apply them on top of current master (with git-bz) ?
Comment 31 Marcel de Rooy 2013-07-03 14:29:21 UTC
Bernardo:
I am getting further with Mason's patch when I use --ignore-whitespace on the patch file.
Shall I continue?
Comment 32 Marcel de Rooy 2013-07-03 14:32:04 UTC Comment hidden (obsolete)
Comment 33 Marcel de Rooy 2013-07-03 14:32:08 UTC Comment hidden (obsolete)
Comment 34 Marcel de Rooy 2013-07-03 14:32:13 UTC Comment hidden (obsolete)
Comment 35 Marcel de Rooy 2013-07-03 14:33:23 UTC
Sorry, couldn't wait. Yes, we are back in the air. Three patches applied. Will have a look again :)
Comment 36 Marcel de Rooy 2013-07-03 15:16:19 UTC
I still need some time for these patches.
Will hopefully finish this tomorrow.
When squashing these patches, I again had the whitespace trouble. So we need some care with this set of patches.

This is an impressive amount of work, Bernardo!
Comment 37 Marcel de Rooy 2013-07-04 09:49:39 UTC
QA Comment:
Good work! 
The first patch contains a lot of mixups with repeatable and mandatory flag for the MARC tags. (I see that you resolve them in the third patch.)
The second patch contains a subfield 046 $3. It is not in the list of LOC. So I think that it should be removed until we are sure about it.
The second (and third?) patch also has some problem with whitespace. I tried several things, but it keeps coming back in the lines around tag 082 - 086.

Tag 260: You remove (IMPRINT), I think it should stay. See LOC.

I checked all tags with LOC, but focused on mandatory/repeatable for the subfields.

Rest looks good to me. Please make final corrections for:

a) 046 $3 and 260 [You can probably amend Mason's patch, plz mention in commit]
b) whitespace trouble

Changing status to reflect need for minor adjustments.
Comment 38 Bernardo Gonzalez Kriegel 2013-07-04 10:55:04 UTC
(In reply to M. de Rooy from comment #37)
> 
> Rest looks good to me. Please make final corrections for:
> 
> a) 046 $3 and 260 [You can probably amend Mason's patch, plz mention in
> commit]
> b) whitespace trouble
> 
> Changing status to reflect need for minor adjustments.

Thanks for testing!!
I'll ammend and fix spaces and tags.
Comment 39 Marcel de Rooy 2013-07-07 11:54:20 UTC
Bernardo,
I received your new file. And will recreate your patch now. Thanks.
Comment 40 Marcel de Rooy 2013-07-07 11:57:10 UTC
Created attachment 19454 [details] [review]
Bug 5858 - Default frameworks missing many MARC21 tags

This patch combines original patch and two followups (with thanks to Mason).
Patch has been recreated to recover from the whitespace errors coming back in
the followups.
The patch adds missing tags/subtags, or updates descriptions.

Does only apply to new installs. We will still find a way to have current
installs also/easier benefit from these changes too.

To test:
1) Delete MARC21 default framework
2) Apply patch
3) Test that new default framework loads without problem

Signed-off-by: Mason James <mtj@kohaaloha.com>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Run this file with and without this patch.
This patch adds 24 tags (338 vs 314) and 323 subfields (3951 vs 3628).
Verified that last QA comments were incorporated.
Comment 41 Marcel de Rooy 2013-07-07 12:26:42 UTC
Final(?:) QA comment:
Looks good to me now, Bernardo. Good work again.
Refer to my former comment. Points have been incorporated.

Passed QA

The following points still need attention:

1. How do we translate? My proposal is here: do this via bug 10509; the i18n proposal there will make life easier for the non-English installs.
BTW I waited for this patch to include the marc21 changes of this report.

2. How will current installs benefit?
Thinking out loud: Would it be an idea if we could Upgrade on the MARC structure administration form? Upgrade would just run this file and try to insert the new ones. (That could include translation with the i18n approach. Run the file via load_sql.)
Will add a followup to replace INSERT by INSERT IGNORE. This allows you to run the file on an existing install.

RM: With respect to the second question, it might be good to include a dbrev that prints a message about the possibility to add new tags when upgrading by running the script manually.
Comment 42 Marcel de Rooy 2013-07-07 12:48:32 UTC
Created attachment 19455 [details] [review]
Bug 5858: Followup for INSERT IGNOREs in marc21_framework_DEFAULT.sql

Changes 8 INSERTs into INSERT IGNOREs.
This allows current installs to benefit easier from the new tags/subfields.
Note that running with --force will not achieve the same! The multi-value
inserts will still be aborted, though execution continues.

Test plan:
Run the file or do a new install.

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Ran the file on a current install manually.
Comment 43 Marcel de Rooy 2013-07-07 12:49:52 UTC
Last thought on the Upgrade idea:
Would be nice to do something like that. Since we still have the non-default frameworks here too (with their redundancy..)
Comment 44 Bernardo Gonzalez Kriegel 2013-07-07 13:06:48 UTC
My idea is rewrite marc21_simple_bib_frameworks.sql to create
bib frameworks like authtypes in Bug 10488.
Comment 45 Galen Charlton 2013-07-08 15:01:48 UTC
Pushed to master.  Thanks, Bernardo and Marcel!
Comment 46 Tomás Cohen Arazi 2013-07-16 17:24:46 UTC
This patch has been pushed to 3.12.x, will be in 3.12.2.

Thanks Bernardo and Marcel!

Note: I added an entry in updatedatabase.pl for 3.12.x, that tells the user about this, just in case they want to apply it.
Comment 47 Bernardo Gonzalez Kriegel 2013-08-04 13:43:22 UTC
Pushed to 3.10.x, will be in 3.10.10
Comment 48 Chris Hall 2013-09-16 08:36:34 UTC
Pushed to 3.8.x, will be in 3.8.18