Bug 37133

Summary: Update MARC21 authority frameworks to Update 36
Product: Koha Reporter: Caroline Cyr La Rose <caroline.cyr-la-rose>
Component: MARC Authority data supportAssignee: Caroline Cyr La Rose <caroline.cyr-la-rose>
Status: RESOLVED FIXED QA Contact: Testopia <testopia>
Severity: enhancement    
Priority: P5 - low CC: david, lucas, phil, philippe.blouin
Version: Main   
Hardware: All   
OS: All   
Change sponsored?: --- Patch complexity: ---
Documentation contact: Documentation submission:
Text to go in the release notes:
This enhancement updates the MARC21 authority frameworks for new installations to reflect the changes from Update 36 (June 2023). NOTES: - For existing installations, manually updating the default and other frameworks with the changes is required.
 Version(s) released in:
24.11.00
Circulation function:
Attachments: Bug 37133: Update MARC21 authority frameworks to Update 36
Bug 37133: Update MARC21 authority frameworks to Update 36
Bug 37133: Update MARC21 authority frameworks to Update 36
Bug 37133: Update MARC21 authority frameworks to Update 36
Bug 37133: (follow-up) Fix tagfield - 856$6 should be 857$6

Description Caroline Cyr La Rose 2024-06-19 19:37:10 UTC 
https://www.loc.gov/marc/up36authority/adapndxf.html

■ New content designators:
Fields
361Structured Ownership and Custodial History (R)
857Electronic Archive Location and Access (R)

■ Changes in content designator names and definitions:
Subfield
$dTitle of person redescribed in 368 (Other Attributes of Person or Corporate Body)
Comment 1 Caroline Cyr La Rose 2024-06-19 20:09:10 UTC 
Created attachment 167898 [details] [review]
Bug 37133: Update MARC21 authority frameworks to Update 36

The patch updates the MARC21 authority frameworks to Update 36

To test:
1. Apply patch and reset_all
2. Go to Administration > Authority types
3. Click Actions next to the Default framework and choose MARC structure
4. Check for the changes detailed in the update
   https://www.loc.gov/marc/up36authority/adapndxf.html

- There should be a new field 361 named STRUCTURED OWNERSHIP AND CUSTODIAL
  HISTORY with the following subfields
  https://www.loc.gov/marc/authority/ad361.html
  - 0 - Authority record control number or standard number (R)
  - 1 - Real World Object URI (R)
  - 3 - Materials specified (NR)
  - 5 - Institution to which field applies (NR)
  - 6 - Linkage (NR)
  - 7 - Data provenance (R)
  - 8 - Field link and sequence number (R)
  - a - Name (NR)
  - f - Ownership and custodial history evidence term (R)
  - k - Formatted date (NR)
  - l - Date (NR)
  - o - Type of ownership and custodial history information (R)
  - s - Shelf mark of copy described (NR)
  - u - Uniform Resource Identifier (R)
  - x - Nonpublic note (R)
  - y - Identifier of the copy described (NR)
  - z - Public note (R)

- No changes to subfield 368$d, changes only affect subfield documentation

- There should be a new field 857 named ELECTRONIC ARCHIVE LOCATION AND ACCESS
  with the following subfields
  https://www.loc.gov/marc/authority/ad857.html
  - 2 - Access method (NR)
  - 3 - Materials specified (NR)
  - 5 - Institution to which field applies (NR)
  - 6 - Linkage (NR)
  - 7 - Access status (NR)
  - 8 - Field link and sequence number (R)
  - b - Name of archiving agency (NR)
  - c - Name of Web archive or digital archive repository (NR)
  - d - Date range of archived material (NR)
  - e - Data provenance (R)
  - f - Archive completeness (NR)
  - g - Persistent identifier (R)
  - h - Non-functioning Uniform Resource Identifier (R)
  - l - Standardized information governing access (R)
  - m - Contact for access assistance (R)
  - n - Terms governing access (R)
  - q - Electronic format type (R)
  - r - Standardized information governing use and reproduction (R)
  - s - File size (R)
  - t - Terms governing use and reproduction (R)
  - u - Uniform Resource Identifier (R)
  - x - Nonpublic note (R)
  - y - Link text (R)
  - z - Public note (R)

5. Optional: Check other authority frameworks, they should all be changed
Comment 2 Phil Ringnalda 2024-06-19 21:25:53 UTC 
Created attachment 167903 [details] [review]
Bug 37133: Update MARC21 authority frameworks to Update 36

The patch updates the MARC21 authority frameworks to Update 36

To test:
1. Apply patch and reset_all
2. Go to Administration > Authority types
3. Click Actions next to the Default framework and choose MARC structure
4. Check for the changes detailed in the update
   https://www.loc.gov/marc/up36authority/adapndxf.html

- There should be a new field 361 named STRUCTURED OWNERSHIP AND CUSTODIAL
  HISTORY with the following subfields
  https://www.loc.gov/marc/authority/ad361.html
  - 0 - Authority record control number or standard number (R)
  - 1 - Real World Object URI (R)
  - 3 - Materials specified (NR)
  - 5 - Institution to which field applies (NR)
  - 6 - Linkage (NR)
  - 7 - Data provenance (R)
  - 8 - Field link and sequence number (R)
  - a - Name (NR)
  - f - Ownership and custodial history evidence term (R)
  - k - Formatted date (NR)
  - l - Date (NR)
  - o - Type of ownership and custodial history information (R)
  - s - Shelf mark of copy described (NR)
  - u - Uniform Resource Identifier (R)
  - x - Nonpublic note (R)
  - y - Identifier of the copy described (NR)
  - z - Public note (R)

- No changes to subfield 368$d, changes only affect subfield documentation

- There should be a new field 857 named ELECTRONIC ARCHIVE LOCATION AND ACCESS
  with the following subfields
  https://www.loc.gov/marc/authority/ad857.html
  - 2 - Access method (NR)
  - 3 - Materials specified (NR)
  - 5 - Institution to which field applies (NR)
  - 6 - Linkage (NR)
  - 7 - Access status (NR)
  - 8 - Field link and sequence number (R)
  - b - Name of archiving agency (NR)
  - c - Name of Web archive or digital archive repository (NR)
  - d - Date range of archived material (NR)
  - e - Data provenance (R)
  - f - Archive completeness (NR)
  - g - Persistent identifier (R)
  - h - Non-functioning Uniform Resource Identifier (R)
  - l - Standardized information governing access (R)
  - m - Contact for access assistance (R)
  - n - Terms governing access (R)
  - q - Electronic format type (R)
  - r - Standardized information governing use and reproduction (R)
  - s - File size (R)
  - t - Terms governing use and reproduction (R)
  - u - Uniform Resource Identifier (R)
  - x - Nonpublic note (R)
  - y - Link text (R)
  - z - Public note (R)

5. Optional: Check other authority frameworks, they should all be changed

Signed-off-by: Phil Ringnalda <phil@chetcolibrary.org>
Comment 3 Katrin Fischer 2024-06-28 12:48:22 UTC 
I have applied all the other ones up to this one and now have a conflict I don't quite understand. Can you please check and resolve, Caroline?
Comment 4 Caroline Cyr La Rose 2024-06-28 14:16:37 UTC 
Created attachment 168243 [details] [review]
Bug 37133: Update MARC21 authority frameworks to Update 36

The patch updates the MARC21 authority frameworks to Update 36

To test:
1. Apply patch and reset_all
2. Go to Administration > Authority types
3. Click Actions next to the Default framework and choose MARC structure
4. Check for the changes detailed in the update
   https://www.loc.gov/marc/up36authority/adapndxf.html

- There should be a new field 361 named STRUCTURED OWNERSHIP AND CUSTODIAL
  HISTORY with the following subfields
  https://www.loc.gov/marc/authority/ad361.html
  - 0 - Authority record control number or standard number (R)
  - 1 - Real World Object URI (R)
  - 3 - Materials specified (NR)
  - 5 - Institution to which field applies (NR)
  - 6 - Linkage (NR)
  - 7 - Data provenance (R)
  - 8 - Field link and sequence number (R)
  - a - Name (NR)
  - f - Ownership and custodial history evidence term (R)
  - k - Formatted date (NR)
  - l - Date (NR)
  - o - Type of ownership and custodial history information (R)
  - s - Shelf mark of copy described (NR)
  - u - Uniform Resource Identifier (R)
  - x - Nonpublic note (R)
  - y - Identifier of the copy described (NR)
  - z - Public note (R)

- No changes to subfield 368$d, changes only affect subfield documentation

- There should be a new field 857 named ELECTRONIC ARCHIVE LOCATION AND ACCESS
  with the following subfields
  https://www.loc.gov/marc/authority/ad857.html
  - 2 - Access method (NR)
  - 3 - Materials specified (NR)
  - 5 - Institution to which field applies (NR)
  - 6 - Linkage (NR)
  - 7 - Access status (NR)
  - 8 - Field link and sequence number (R)
  - b - Name of archiving agency (NR)
  - c - Name of Web archive or digital archive repository (NR)
  - d - Date range of archived material (NR)
  - e - Data provenance (R)
  - f - Archive completeness (NR)
  - g - Persistent identifier (R)
  - h - Non-functioning Uniform Resource Identifier (R)
  - l - Standardized information governing access (R)
  - m - Contact for access assistance (R)
  - n - Terms governing access (R)
  - q - Electronic format type (R)
  - r - Standardized information governing use and reproduction (R)
  - s - File size (R)
  - t - Terms governing use and reproduction (R)
  - u - Uniform Resource Identifier (R)
  - x - Nonpublic note (R)
  - y - Link text (R)
  - z - Public note (R)

5. Optional: Check other authority frameworks, they should all be changed
Comment 5 Caroline Cyr La Rose 2024-06-28 14:17:59 UTC 
I admit I don't really understand what is happening. So I redid the patch and obsoleted the old one. Hopefully the diff is easier to understand/apply...

I put it back to NSO since this is basically a new patch
Comment 6 Phil Ringnalda 2024-06-28 19:06:37 UTC 
I don't really understand what's happening even more, but with this new patch applied ktd up dies at

koha-1       | Inserting /kohadevbox/koha/installer/data/mysql/en/marcflavour/marc21/mandatory/authorities_normal_marc21.yml...
koha-1 exited with code 1
Comment 7 Caroline Cyr La Rose 2024-06-28 20:03:20 UTC 
Ok I probably made an error in the yaml... I will look at it next week.
Comment 8 Katrin Fischer 2024-07-01 09:36:40 UTC 
(In reply to Caroline Cyr La Rose from comment #7)
> Ok I probably made an error in the yaml... I will look at it next week.

Maybe the files are already broken in main... I am checking.
Comment 9 Katrin Fischer 2024-07-01 11:36:45 UTC 
Hm ktd up and reset_all work ok for me on main. So that seems OK.
rest_all with the patch seems also ok.
Manually running the web installer... also ok and no SQL errors with the patch.

Phil, could you double-check you were testing on latest main?
Comment 10 Phil Ringnalda 2024-07-01 21:36:46 UTC 
I had a fresh pull last Friday when it failed for me, and I just freshly pulled again and had the same failed result. Don't know why.
Comment 11 David Nind 2024-07-02 00:13:10 UTC 
Created attachment 168332 [details] [review]
Bug 37133: Update MARC21 authority frameworks to Update 36

The patch updates the MARC21 authority frameworks to Update 36

To test:
1. Apply patch and reset_all
2. Go to Administration > Authority types
3. Click Actions next to the Default framework and choose MARC structure
4. Check for the changes detailed in the update
   https://www.loc.gov/marc/up36authority/adapndxf.html

- There should be a new field 361 named STRUCTURED OWNERSHIP AND CUSTODIAL
  HISTORY with the following subfields
  https://www.loc.gov/marc/authority/ad361.html
  - 0 - Authority record control number or standard number (R)
  - 1 - Real World Object URI (R)
  - 3 - Materials specified (NR)
  - 5 - Institution to which field applies (NR)
  - 6 - Linkage (NR)
  - 7 - Data provenance (R)
  - 8 - Field link and sequence number (R)
  - a - Name (NR)
  - f - Ownership and custodial history evidence term (R)
  - k - Formatted date (NR)
  - l - Date (NR)
  - o - Type of ownership and custodial history information (R)
  - s - Shelf mark of copy described (NR)
  - u - Uniform Resource Identifier (R)
  - x - Nonpublic note (R)
  - y - Identifier of the copy described (NR)
  - z - Public note (R)

- No changes to subfield 368$d, changes only affect subfield documentation

- There should be a new field 857 named ELECTRONIC ARCHIVE LOCATION AND ACCESS
  with the following subfields
  https://www.loc.gov/marc/authority/ad857.html
  - 2 - Access method (NR)
  - 3 - Materials specified (NR)
  - 5 - Institution to which field applies (NR)
  - 6 - Linkage (NR)
  - 7 - Access status (NR)
  - 8 - Field link and sequence number (R)
  - b - Name of archiving agency (NR)
  - c - Name of Web archive or digital archive repository (NR)
  - d - Date range of archived material (NR)
  - e - Data provenance (R)
  - f - Archive completeness (NR)
  - g - Persistent identifier (R)
  - h - Non-functioning Uniform Resource Identifier (R)
  - l - Standardized information governing access (R)
  - m - Contact for access assistance (R)
  - n - Terms governing access (R)
  - q - Electronic format type (R)
  - r - Standardized information governing use and reproduction (R)
  - s - File size (R)
  - t - Terms governing use and reproduction (R)
  - u - Uniform Resource Identifier (R)
  - x - Nonpublic note (R)
  - y - Link text (R)
  - z - Public note (R)

5. Optional: Check other authority frameworks, they should all be changed

Signed-off-by: David Nind <david@davidnind.com>
Comment 12 David Nind 2024-07-02 00:13:13 UTC 
Created attachment 168333 [details] [review]
Bug 37133: (follow-up) Fix tagfield - 856$6 should be 857$6

Signed-off-by: David Nind <david@davidnind.com>
Comment 13 David Nind 2024-07-02 00:17:39 UTC 
Testing notes (using KTD):

1. I applied the patch and then did a reset_all. There was an error generated with "Duplicate entry '856-6'"

reset_all
This action will erase all your data. Are you sure? (y/N) y
Running [sudo koha-shell kohadev -p -c 'PERL5LIB=/kohadevbox/koha:/kohadevbox/koha/lib:/kohadevbox/qa-test-tools perl /kohadevbox/misc4dev/populate_db.pl -v --opac-base-url http://kohadev.koha-testing:8080 --intranet-base-url http://kohadev-intra.koha-testing:8081 --marcflavour MARC21']...
C4::Installer::load_sql returned the following errors while attempting to load /kohadevbox/koha/installer/data/mysql/en/marcflavour/marc21/mandatory/authorities_normal_marc21.yml:
DBD::mysql::db do failed: Duplicate entry '856-6' for key 'PRIMARY' at /kohadevbox/koha/C4/Installer.pm line 577.
Inserting koha db structure...
Inserting /kohadevbox/koha/installer/data/mysql/mandatory/sysprefs.sql...
.....

2. I added a follow-up patch to change 856$6 to 857$7.

3. Everything else looked OK to me - as per the Library of Congress update.

4. Note: all the previous updates (up to update 35) were in main.
Comment 14 Katrin Fischer 2024-07-02 08:09:40 UTC 
Thanks for the team work!
Comment 15 Katrin Fischer 2024-07-02 08:21:34 UTC 
Pushed for 24.11!

Well done everyone, thank you!
Comment 16 Caroline Cyr La Rose 2024-07-02 13:24:48 UTC 
Thanks for the follow-up patch David!