The default frameworks for MARC21 are missing many tags, seemingly all introduced after 2006. This bug does not address missing subfields or redefinitions. The missing tags are: 083 - Additional Dewey Decimal Classification Number 085 - Synthesized Classification Number Components 336 - Content Type 337 - Media Type 338 - Carrier Type 363 - Normalized date and sequential designation 380 - Form of Work 381 - Other Distinguishing Characteristics of Work or Expression 382 - Medium of Performance 383 - Numeric Designation of Musical Work 384 - Key 542 - Information Relating to Copyright Status 545 - Biographical or historical data 588 - Source of description note 751 - Added entry - geographic name 882 - Replacement record information For summaries of changes to the MARC21 standard starting 2007 see: http://replay.waybackmachine.org/20080915133838/http://www.loc.gov/marc/bibliographic/bdapndxg.html http://replay.waybackmachine.org/20090907173218/http://www.loc.gov/marc/bibliographic/bdapndxg.html http://replay.waybackmachine.org/20100311061810/http://www.loc.gov/marc/bibliographic/bdapndxg.html http://www.loc.gov/marc/bibliographic/bdapndxg.html
*** Bug 5587 has been marked as a duplicate of this bug. ***
Subfields that are probably missing (I did not check all these subfields in the default frameworks, but I thought the list of new subfields might be of use to someone, even unproofed): 015 $z - Canceled/Invalid national bibliography number (National bibliography number) 017 $z - Canceled/Invalid Copyright or Legal Deposit Number (Copyright or Legal Deposit Number) 022 $l - ISSN-L (ISSN) 022 $m - Canceled ISSN-L (ISSN) 033 $0 - Record control number (R) (Date/Time and Place of an Event) 033 $2 - Source of term (R) (Date/Time and Place of an Event) 033 $p - Place of event (R) (Date/Time and Place of an Event) 034 $3 - Materials specified (NR) (Coded Cartographic Mathematical Data) 041 $j - Language code of subtitles or captions (Language code) 082 $m - Standard or Optional Designation (Dewey Decimal Classification Number) 082 $q - Assigning agency (Dewey Decimal Classification Number) 100 $0 - Authority record control number (Main entry -- Personal name) 110 $0 - Authority record control number (Main entry -- Corporate name) 111 $0 - Authority record control number (Main entry -- Meeting name) 130 $0 - Authority record control number (Main entry -- Uniform title) 240 $0 - Authority record control number (Uniform title) 257 $2 - Source (NR) (Country of Producing Entity) 440 $0 - Authority record control number (Series statement/Added entry -- Title) 440 $w - Bibliographic record control number (Series statement/Added entry -- Title) 490 $3 - Materials Specified (Series Statement) 502 $b - Degree Type (Dissertation Note) 502 $c - Name of Granting Institution (Dissertation Note) 502 $d - Year of Degree Granted (Dissertation Note) 502 $g - Miscellaneous Information (Dissertation Note) 502 $o - Dissertation Identifier (Dissertation Note) 510 $u - Uniform Resource Identifier (R) (Citation/References Note) 518 $0 - Record control number (R) (Date/Time and Place of an Event Note) 518 $2 - Source of term (R) (Date/Time and Place of an Event Note) 518 $d - Date of event (R) (Date/Time and Place of an Event Note) 518 $o - Other event information (R) (Date/Time and Place of an Event Note) 518 $p - Place of event (R) (Date/Time and Place of an Event Note) 520 $2 - Source (Summary, etc.) 520 $c - Assigning agency (Summary, etc.) 533 $5 - Institution to which field applies (Reproduction note) 534 $3 - Materials specified (NR) (Original Version Note) 534 $o - Other Resource Identifier (Original Version Note) 538 $5 - Institution to which field applies (System details note) 561 $u - Uniform Resource Identifier (R) (Ownership and Custodial History) 600 $0 - Authority record control number (Subject added entry -- Personal name) 610 $0 - Authority record control number (Subject added entry -- Corporate name) 611 $0 - Authority record control number (Subject added entry -- Meeting name) 630 $0 - Authority record control number (Subject added entry -- Uniform title) 648 $0 - Authority record control number (Subject added entry -- Chronological term) 650 $0 - Authority record control number (Subject added entry -- Topical term) 651 $0 - Authority record control number (Subject added entry -- Geographic name) 654 $0 - Authority record control number (Subject added entry -- Faceted topical terms) 655 $0 - Authority record control number (Index term -- Genre/form) 656 $0 - Authority record control number (Index term -- Occupation) 657 $0 - Authority record control number (Index term -- Function) 662 $0 - Authority record control number (Subject added entry -- Hierarchical place name) 700 $0 - Authority record control number (Added entry -- Personal name) 700 $i - Relationship information (R) (Added Entry-Personal Name) 710 $0 - Authority record control number (Added entry -- Corporate name) 710 $i - Relationship information (R) (Added Entry-Corporate Name) 711 $0 - Authority record control number (Added entry -- Meeting name) 711 $i - Relationship information (R) (Added Entry-Meeting Name) 730 $0 - Authority record control number (Added entry -- Uniform title) 730 $i - Relationship information (R) (Added Entry-Uniform Title) 752 $0 - Authority record control number (Added entry -- Hierarchical place name) 754 $0 - Authority record control number (Added entry -- Taxonomic identification) 760-787 $4 - Relationship code (R) (Linking Entry fields) 800 $0 - Authority record control number (Series added entry -- Personal name) 800 $3 - Materials Specified (Series Added Entry-Personal Name) 800 $5 - Institution to which field applies (R) (Series Added Entry-Personal Name) 800 $x - International Standard Serial Number (Series Added Entry-Personal Name) 810 $0 - Authority record control number (Series added entry -- Corporate name) 810 $3 - Materials Specified (Series Added Entry-Corporate Name) 810 $5 - Institution to which field applies (R) (Series Added Entry-Corporate Name) 810 $x - International Standard Serial Number (Series Added Entry-Corporate Name) 811 $0 - Authority record control number (Series added entry -- Meeting name) 811 $3 - Materials Specified (Series Added Entry-Meeting Name) 811 $5 - Institution to which field applies (R) (Series Added Entry-Meeting Name) 811 $x - International Standard Serial Number (Series Added Entry-Meeting Name) 830 $0 - Authority record control number (Series added entry -- Uniform title) 830 $3 - Materials Specified (Series Added Entry-Uniform Title) 830 $5 - Institution to which field applies (R) (Series Added Entry-Uniform Title) 830 $x - International Standard Serial Number (Series Added Entry-Uniform Title) 852 $d - Former shelving location (Location)
Created attachment 10393 [details] [review] Bug 5858 - Updating the marc21_framework_DEFAULT.sql to MARC21 version 14 Based on work in the Koha Kobli fork
This is currently english only.
What prevents this patch to be signed-off? It seems all good.
(In reply to comment #5) > What prevents this patch to be signed-off? It seems all good. I thought I had signed it off, actually. I tested it and was quite pleased with how it worked on a new install.
It seems it's not only adding, but also deleting a lot of fields? I did only look at the patch, maybe I am missing something.
(In reply to comment #7) > It seems it's not only adding, but also deleting a lot of fields? I did only > look at the patch, maybe I am missing something. Yeah its getting rid of all the deprecated fields. This is why its only good for a new install and not in updatedatabase.pl
I am not sure deleting them is a good idea - I would have to test it, but is there a chance we loose data if someone imports data that uses fields that are not defined in the default framework? I think I would prefer to see them hidden and marked [OBSOLETE] to be sure no data can be lost.
(In reply to comment #9) > I am not sure deleting them is a good idea - I would have to test it, but is > there a chance we loose data if someone imports data that uses fields that > are not defined in the default framework? I think I would prefer to see them > hidden and marked [OBSOLETE] to be sure no data can be lost. Should this be marked as "in discussion"?
*** Bug 3684 has been marked as a duplicate of this bug. ***
(In reply to comment #9) > I am not sure deleting them is a good idea - I would have to test it, but is > there a chance we loose data if someone imports data that uses fields that > are not defined in the default framework? I think I would prefer to see them > hidden and marked [OBSOLETE] to be sure no data can be lost. I did some tests. Data is not lost when you import, in case some field is not present in frameworks. But if you edit and save the record, then it's lost. So, to prevent data loss, it would be convenient to hide the fields. I volunteer to write a followup to do that. Is that enough to proceed? The missing fields are real. Regards
I confirm the data loss upon edit/save. I recently ran into it again... I think adding the missing fields and hiding unused/deprecated fields would be great.
Created attachment 15272 [details] [review] [SIGNED-OFF] Bug 5858 - Updating the marc21_framework_DEFAULT.sql to MARC21 version 14 Based on work in the Koha Kobli fork Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com> Comment: deleted fields added in a followup.
Created attachment 15273 [details] [review] Followup Bug 5858 - Default frameworks missing many MARC21 tags It's a pity, but this patch re-instates deleted fields to prevent data loss. Also correct minor errors, and add field 883.
I still see many instances of fields which are removed altogether. Should there be nothing removed?
I will check and resubmit.
Created attachment 16445 [details] [review] Bug 5858 - Default frameworks missing many MARC21 tags This patch replaces previous ones. Only adds missing tags/subtags or update descriptions. Only to new installs To test: 1) Delete MARC21 default framework 2) Apply patch 3) Test that new default framework loads without problem
Created attachment 17352 [details] [review] Bug 5858 - Default frameworks missing many MARC21 tags This patch replaces previous ones. Only adds missing tags/subtags or update descriptions. Only to new installs. To test: 1) Delete MARC21 default framework 2) Apply patch 3) Test that new default framework loads without problem Rebased to master.
Created attachment 18584 [details] [review] Bug 5858 - Default frameworks missing many MARC21 tags This patch replaces previous ones. Only adds missing tags/subtags or update descriptions. Only to new installs. To test: 1) Delete MARC21 default framework 2) Apply patch 3) Test that new default framework loads without problem Rebased to master. Signed-off-by: Mason James <mtj@kohaaloha.com>
Created attachment 18585 [details] [review] [FOLLOWUP] Bug 5858 - Default frameworks missing many MARC21 tags This patch is a *followup* to the previous patch Update MARC21 fields/subfields to version 16 http://www.loc.gov/marc/bibliographic/bdapndxg.html Only to new installs. To test: 1) Delete MARC21 default framework 2) Apply patch 3) Test that new default framework loads without problem Rebased to master.
(In reply to comment #9) > I am not sure deleting them is a good idea - I would have to test it, but is > there a chance we loose data if someone imports data that uses fields that > are not defined in the default framework? I think I would prefer to see them > hidden and marked [OBSOLETE] to be sure no data can be lost. it looks like the official MARC21 spec does this too -> http://www.loc.gov/marc/bibliographic/ecbdlist.html ie: it does not delete old fields/subfields, but just marks them [OBSOLETE]
(In reply to comment #20) > Created attachment 18584 [details] [review] [review] > Bug 5858 - Default frameworks missing many MARC21 tags > > Signed-off-by: Mason James <mtj@kohaaloha.com> i've taken a look a Bernardo's patch, and tested it it looks good
(In reply to comment #23) > (In reply to comment #20) > > Created attachment 18584 [details] [review] [review] [review] > > Bug 5858 - Default frameworks missing many MARC21 tags > > > > Signed-off-by: Mason James <mtj@kohaaloha.com> > > i've taken a look a Bernardo's patch, and tested it > > it looks good i've signed-off on Bernardo's patch just awaiting sign-off on my follow-up patch...
Looking at this now..
Mason: Something is wrong with your followup. Look here: ('383', 'NUMERIC DESIGNATION OF MUSICAL WORK', 'NUMERIC DESIGNATION OF MUSICAL WORK', 0, 1, NULL, ''), 0, 1, NULL, ''), UDIENCE CHARACTERISTICS', 'AUDIENCE CHARACTERISTICS', 1, 0, NULL, ''), Bernardo or Mason: please correct! Failed QA
Created attachment 19330 [details] [review] [FOLLOWUP 2] Bug 5858 - Update default MARC21 framework This is another followup. It updates some missing tags/subtags, also corrects and verify repeatable value for tags/subtags Also rearrange some out of order entries(referenced on comment #26) I think that now the update is correct and complete
fatal: sha1 information is lacking or useless (installer/data/mysql/en/marcflavour/marc21/mandatory/marc21_framework_DEFAULT.sql). Repository lacks necessary blobs to fall back on 3-way merge. Cannot fall back to three-way merge. Patch failed at 0001 Bug 5858 - Update default MARC21 framework to Update No. 16 (April 2013)
(In reply to M. de Rooy from comment #28) > fatal: sha1 information is lacking or useless > (installer/data/mysql/en/marcflavour/marc21/mandatory/ > marc21_framework_DEFAULT.sql). > Repository lacks necessary blobs to fall back on 3-way merge. > Cannot fall back to three-way merge. > Patch failed at 0001 Bug 5858 - Update default MARC21 framework to Update > No. 16 (April 2013) Strange ~/kohaclone$ git pull Already up-to-date. ~/kohaclone$ git checkout bug_5858-B Switched to branch 'bug_5858-B' Your branch is behind 'origin/master' by 11 commits, and can be fast-forwarded. ~/kohaclone$ git rebase master First, rewinding head to replay your work on top of it... Fast-forwarded bug_5858-B to master. ~/kohaclone$ git bz apply 5858 Bug 5858 - Update default MARC21 framework to Update No. 16 (April 2013) Bug 5858 - Default frameworks missing many MARC21 tags Apply? [yn] y Applying: Bug 5858 - Default frameworks missing many MARC21 tags [FOLLOWUP] Bug 5858 - Default frameworks missing many MARC21 tags Apply? [yn] y Applying: Bug 5858 - Default frameworks missing many MARC21 tags [FOLLOWUP 2] Bug 5858 - Update default MARC21 framework Apply? [yn] y Applying: Bug 5858 - Update default MARC21 framework to Update No. 16 (April 2013) ~/kohaclone$ For me it's applying without problem, should I switch to Needs Singoff?
Bernardo: Try to apply them on top of current master (with git-bz) ?
Bernardo: I am getting further with Mason's patch when I use --ignore-whitespace on the patch file. Shall I continue?
Created attachment 19371 [details] [review] Bug 5858 - Default frameworks missing many MARC21 tags This patch replaces previous ones. Only adds missing tags/subtags or update descriptions. Only to new installs. To test: 1) Delete MARC21 default framework 2) Apply patch 3) Test that new default framework loads without problem Rebased to master. Signed-off-by: Mason James <mtj@kohaaloha.com>
Created attachment 19372 [details] [review] Bug 5858 - Default frameworks missing many MARC21 tags This patch is a *followup* to the previous patch Update MARC21 fields/subfields to version 16 http://www.loc.gov/marc/bibliographic/bdapndxg.html Only to new installs. To test: 1) Delete MARC21 default framework 2) Apply patch 3) Test that new default framework loads without problem Rebased to master.
Created attachment 19373 [details] [review] Bug 5858 - Update default MARC21 framework to Update No. 16 (April 2013) This is another followup. It updates some missing tags/subtags, also corrects and verify repeatable value for tags/subtags
Sorry, couldn't wait. Yes, we are back in the air. Three patches applied. Will have a look again :)
I still need some time for these patches. Will hopefully finish this tomorrow. When squashing these patches, I again had the whitespace trouble. So we need some care with this set of patches. This is an impressive amount of work, Bernardo!
QA Comment: Good work! The first patch contains a lot of mixups with repeatable and mandatory flag for the MARC tags. (I see that you resolve them in the third patch.) The second patch contains a subfield 046 $3. It is not in the list of LOC. So I think that it should be removed until we are sure about it. The second (and third?) patch also has some problem with whitespace. I tried several things, but it keeps coming back in the lines around tag 082 - 086. Tag 260: You remove (IMPRINT), I think it should stay. See LOC. I checked all tags with LOC, but focused on mandatory/repeatable for the subfields. Rest looks good to me. Please make final corrections for: a) 046 $3 and 260 [You can probably amend Mason's patch, plz mention in commit] b) whitespace trouble Changing status to reflect need for minor adjustments.
(In reply to M. de Rooy from comment #37) > > Rest looks good to me. Please make final corrections for: > > a) 046 $3 and 260 [You can probably amend Mason's patch, plz mention in > commit] > b) whitespace trouble > > Changing status to reflect need for minor adjustments. Thanks for testing!! I'll ammend and fix spaces and tags.
Bernardo, I received your new file. And will recreate your patch now. Thanks.
Created attachment 19454 [details] [review] Bug 5858 - Default frameworks missing many MARC21 tags This patch combines original patch and two followups (with thanks to Mason). Patch has been recreated to recover from the whitespace errors coming back in the followups. The patch adds missing tags/subtags, or updates descriptions. Does only apply to new installs. We will still find a way to have current installs also/easier benefit from these changes too. To test: 1) Delete MARC21 default framework 2) Apply patch 3) Test that new default framework loads without problem Signed-off-by: Mason James <mtj@kohaaloha.com> Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl> Run this file with and without this patch. This patch adds 24 tags (338 vs 314) and 323 subfields (3951 vs 3628). Verified that last QA comments were incorporated.
Final(?:) QA comment: Looks good to me now, Bernardo. Good work again. Refer to my former comment. Points have been incorporated. Passed QA The following points still need attention: 1. How do we translate? My proposal is here: do this via bug 10509; the i18n proposal there will make life easier for the non-English installs. BTW I waited for this patch to include the marc21 changes of this report. 2. How will current installs benefit? Thinking out loud: Would it be an idea if we could Upgrade on the MARC structure administration form? Upgrade would just run this file and try to insert the new ones. (That could include translation with the i18n approach. Run the file via load_sql.) Will add a followup to replace INSERT by INSERT IGNORE. This allows you to run the file on an existing install. RM: With respect to the second question, it might be good to include a dbrev that prints a message about the possibility to add new tags when upgrading by running the script manually.
Created attachment 19455 [details] [review] Bug 5858: Followup for INSERT IGNOREs in marc21_framework_DEFAULT.sql Changes 8 INSERTs into INSERT IGNOREs. This allows current installs to benefit easier from the new tags/subfields. Note that running with --force will not achieve the same! The multi-value inserts will still be aborted, though execution continues. Test plan: Run the file or do a new install. Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl> Ran the file on a current install manually.
Last thought on the Upgrade idea: Would be nice to do something like that. Since we still have the non-default frameworks here too (with their redundancy..)
My idea is rewrite marc21_simple_bib_frameworks.sql to create bib frameworks like authtypes in Bug 10488.
Pushed to master. Thanks, Bernardo and Marcel!
This patch has been pushed to 3.12.x, will be in 3.12.2. Thanks Bernardo and Marcel! Note: I added an entry in updatedatabase.pl for 3.12.x, that tells the user about this, just in case they want to apply it.
Pushed to 3.10.x, will be in 3.10.10
Pushed to 3.8.x, will be in 3.8.18