Bug 35380

Summary: PUT /biblios/:biblio_id doesn't respect overlay rules
Product: Koha Reporter: Tomás Cohen Arazi (tcohen) <tomascohen>
Component: REST APIAssignee: Tomás Cohen Arazi (tcohen) <tomascohen>
Status: NEW --- QA Contact:
Severity: major    
Priority: P5 - low CC: bibliothek, Laura.escamilla, mnero, mspinney, tomascohen
Version: unspecified   
Hardware: All   
OS: All   
Change sponsored?: --- Patch complexity: ---
Documentation contact: Documentation submission:
Text to go in the release notes:
Version(s) released in:
Circulation function:
Bug Depends on: 31801    
Bug Blocks:    
Attachments: Bug 35380: Make updateBiblio-API respect Record overlay rules
Bug 35380: [DO NOT PUSH] dbic
Bug 35380: Make record sources available in overlay rules
Bug 35380: Make updateBiblio-API respect Record overlay rules

Description Tomás Cohen Arazi (tcohen) 2023-11-21 22:04:19 UTC
Based on bug 31801#c5, this endpoint should, at least, respect the any default rule as defined in the MARC overlay rules.
Comment 1 Jan Kissig 2024-09-05 12:09:24 UTC
I would like to move this forward and try to write a patch. 

Do I understand it right, that f.e. I have a record source (f.e. union catalogue) defined in admin -> record sources. Then I should pass the source id as header param and add it to the ModBiblio call like

ModBiblio( $record, $biblio->id, $frameworkcode, { overlay_context => { source => RECORD_SOURCE_ID } } );

I remember we had a discussion about it but could not built the pieces together..
Comment 2 Tomás Cohen Arazi (tcohen) 2024-09-05 13:07:55 UTC
Jan, thanks stepping up.

The MARC overlay rules' sources are not linked to the 'Record sources' feature (yet). That work still needs to be done.

That said, my understanding is that there are a few options here:

(a) Add an 'API' source to the existing MARC overlay rules valid sources.
(b) Add a way to pick 'Record sources' within the MARC overlay rules feature.
(c) Move the 'MARC overlay rules sources' to the 'Record sources' handling. Having them be 'system defined' and untouchable, and then make the 'MARC overlay rules' feature pick all the options from the 'Record sources'.
(d) Make the API controller at least pass a context to `ModBiblio` so *at least* a fallback overlay rule (*) would be applied.


I think the compromise solution is (d), and the ideal is (c). (a) and (b) would be transition steps towards (c), and I'd avoid them as much as possible.
Comment 3 Jan Kissig 2024-12-19 19:10:42 UTC
Created attachment 175800 [details] [review]
Bug 35380: Make updateBiblio-API respect Record overlay rules

This patch moves hardcoded source filter from 'MARC overlay rules' (batchmod, intranet, z3950, staged import, bulk import, import_lexile) to 'Record Sources' and makes them untouchable. They cannot be renamed or modified.
New record sources can be added and modified though.
All default and newly created record sources are available as filter for the module "Source" in 'MARC overlay rules'.
The API route for PUT /biblios/:biblio_id now checks for a given header x-record-source-id.
If an id is given the name of the rule is retrieved and forwarded as an overlay_context to ModBiblio.
If no id is given the fallback '*' is added as overlay_context.
Also the record source id is getting saved.

The patch is still missing some tests but I would like to get some feedback if this solution meets the intentional usecase for record sources and the use of them in the updateBiblio API.



Test plan:
pre) run perl installer/data/mysql/updatedatabase.pl to add new is_system column
pre) run dbic to build Koha::Schema::Result::RecordSource as it gets a new columns is_system
pre) run 'redocly bundle --ext json api/v1/swagger/swagger.yaml --output api/v1/swagger/swagger_bundle.json' or 'yarn build'

a) enable system preference MARCOverlayRules and RESTBasicAuth
b) create new record source in /cgi-bin/koha/admin/record_sources and recognize its <NAME> and <ID>
c) create overlay rules in /cgi-bin/koha/admin/marc-overlay-rules.pl
f.e.
+----+-----+--------+----------+-----------+
| id | tag | module | filter   | preset    |
+----+-----+--------+----------+-----------+
|  1 | 245 | source | *        | protect   |
+----+-----+--------+----------+-----------+

This will make the 245 field protected from change, unless the record source matches
d) go to /cgi-bin/koha/catalogue/detail.pl?biblionumber=262 and change 245 (title)  via edit -> edit record
e) save and take a look the title is still the same
f) run the updateBiblio-Request from below with a API-testing client like REST Client
g) check that title gets overwritten even though it is protected. This is due the fact the updateBiblio is not setting an overlay context.

h) apply patch
redo d) and e)
i) add another overlay rule allowing your record source to change 245:
+----+-----+--------+----------+-----------+
| id | tag | module | filter   | preset    |
+----+-----+--------+----------+-----------+
|  1 | 245 | source | *        | protect   |
|  2 | 245 | source | <NAME>   | overwrite |
+----+-----+--------+----------+-----------+

j) run the updateBiblio-Request from below, be sure to fill out x-record-source-id:with your ID from step b)
k) check that title gets overwritten
l) run the updateBiblio-Request from below, but omit x-record-source-id or set it to another value than <ID>
m) check that title is not overwritten



PUT http://main-intra.localhost/api/v1/biblios/262
Authorization: Basic koha:koha
Content-Type: application/marcxml+xml
x-record-source-id: <ID>

<?xml version="1.0" encoding="UTF-8"?>
<record
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://www.loc.gov/MARC21/slim http://www.loc.gov/standards/marcxml/schema/MARC21slim.xsd"
    xmlns="http://www.loc.gov/MARC21/slim">

  <leader>01075cam a2200313 a 4500</leader>
  <controlfield tag="001">foo</controlfield>
  <controlfield tag="005">20200421093825.0</controlfield>
  <controlfield tag="008">120417s2012    ch a     b    001 0 eng  </controlfield>
  <datafield tag="040" ind1=" " ind2=" ">
    <subfield code="c">482</subfield>
  </datafield>
  <datafield tag="100" ind1="1" ind2=" ">
    <subfield code="a">Christiansen, Tom.</subfield>
    <subfield code="9">482</subfield>
  </datafield>
  <datafield tag="245" ind1="1" ind2="0">
    <subfield code="a">New Title</subfield>
    <subfield code="c">Tom Christiansen, Brian D. Foy &amp; Larry Wall.</subfield>
  </datafield>
  <datafield tag="260" ind1=" " ind2=" ">
    <subfield code="a">City</subfield>
  </datafield>
  <datafield tag="300" ind1=" " ind2=" ">
    <subfield code="a">Pages and more</subfield>
  </datafield>
  <datafield tag="942" ind1=" " ind2=" ">
    <subfield code="c">BKS</subfield>
  </datafield>
</record>
Comment 4 Jan Kissig 2024-12-19 19:10:44 UTC
Created attachment 175801 [details] [review]
Bug 35380: [DO NOT PUSH] dbic
Comment 5 Jan Kissig 2024-12-19 19:10:47 UTC
Created attachment 175802 [details] [review]
Bug 35380: Make record sources available in overlay rules

This patch enhances the filters in record overlay rules when using the "Source" module. The available filter values are now system defined in Administration -> Record sources and can be extended by own record souces. The former hard coded values (f.e. batchmod, intranet) are added to the database and cannot be deleted.
Comment 6 Jan Kissig 2024-12-19 19:10:49 UTC
Created attachment 175803 [details] [review]
Bug 35380: Make updateBiblio-API respect Record overlay rules

This patch moves hardcoded source filter from 'MARC overlay rules' (batchmod, intranet, z3950, staged import, bulk import, import_lexile) to 'Record Sources' and makes them untouchable. They cannot be renamed or modified.
New record sources can be added and modified though.
All default and newly created record sources are available as filter for the module "Source" in 'MARC overlay rules'.
The API route for PUT /biblios/:biblio_id now checks for a given header x-record-source-id.
If an id is given the name of the rule is retrieved and forwarded as an overlay_context to ModBiblio.
If no id is given the fallback '*' is added as overlay_context.
Also the record source id is getting saved.

The patch is still missing some tests but I would like to get some feedback if this solution meets the intentional usecase for record sources and the use of them in the updateBiblio API.

Test plan:
pre) run perl installer/data/mysql/updatedatabase.pl to add new is_system column
pre) run dbic to build Koha::Schema::Result::RecordSource as it gets a new columns is_system
pre) run 'redocly bundle --ext json api/v1/swagger/swagger.yaml --output api/v1/swagger/swagger_bundle.json' or 'yarn build'

a) enable system preference MARCOverlayRules and RESTBasicAuth
b) create new record source in /cgi-bin/koha/admin/record_sources and recognize its <NAME> and <ID>
c) create overlay rules in /cgi-bin/koha/admin/marc-overlay-rules.pl
f.e.
+----+-----+--------+----------+-----------+
| id | tag | module | filter   | preset    |
+----+-----+--------+----------+-----------+
|  1 | 245 | source | *        | protect   |
+----+-----+--------+----------+-----------+

This will make the 245 field protected from change, unless the record source matches
d) go to /cgi-bin/koha/catalogue/detail.pl?biblionumber=262 and change 245 (title)  via edit -> edit record
e) save and take a look the title is still the same
f) run the updateBiblio-Request from below with a API-testing client like REST Client
g) check that title gets overwritten even though it is protected. This is due the fact the updateBiblio is not setting an overlay context.

h) apply patch
redo d) and e)
i) add another overlay rule allowing your record source to change 245:
+----+-----+--------+----------+-----------+
| id | tag | module | filter   | preset    |
+----+-----+--------+----------+-----------+
|  1 | 245 | source | *        | protect   |
|  2 | 245 | source | <NAME>   | overwrite |
+----+-----+--------+----------+-----------+

j) run the updateBiblio-Request from below, be sure to fill out x-record-source-id:with your ID from step b)
k) check that title gets overwritten
l) run the updateBiblio-Request from below, but omit x-record-source-id or set it to another value than <ID>
m) check that title is not overwritten

PUT http://main-intra.localhost/api/v1/biblios/262
Authorization: Basic koha:koha
Content-Type: application/marcxml+xml
x-record-source-id: <ID>

<?xml version="1.0" encoding="UTF-8"?>
<record
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://www.loc.gov/MARC21/slim http://www.loc.gov/standards/marcxml/schema/MARC21slim.xsd"
    xmlns="http://www.loc.gov/MARC21/slim">

  <leader>01075cam a2200313 a 4500</leader>
  <controlfield tag="001">foo</controlfield>
  <controlfield tag="005">20200421093825.0</controlfield>
  <controlfield tag="008">120417s2012    ch a     b    001 0 eng  </controlfield>
  <datafield tag="040" ind1=" " ind2=" ">
    <subfield code="c">482</subfield>
  </datafield>
  <datafield tag="100" ind1="1" ind2=" ">
    <subfield code="a">Christiansen, Tom.</subfield>
    <subfield code="9">482</subfield>
  </datafield>
  <datafield tag="245" ind1="1" ind2="0">
    <subfield code="a">New Title</subfield>
    <subfield code="c">Tom Christiansen, Brian D. Foy &amp; Larry Wall.</subfield>
  </datafield>
  <datafield tag="260" ind1=" " ind2=" ">
    <subfield code="a">City</subfield>
  </datafield>
  <datafield tag="300" ind1=" " ind2=" ">
    <subfield code="a">Pages and more</subfield>
  </datafield>
  <datafield tag="942" ind1=" " ind2=" ">
    <subfield code="c">BKS</subfield>
  </datafield>
</record>