I'd suggest the following url

PUT: /biblios/{biblio_id}/merge/{biblio_id_to_merge}

where biblio_id is the biblio_id of the record to be kept

additionally we need a list (json, csv ..) of the records that should be taken from the biblio_id_to_merge record

output should be a list from the "report_records" from the preference('MergeReportFields')

ideally the whole logic of "merge" from 
/cataloguing/merge.pl

starting with
86     # Rewriting the leader

should go into the storage class
package Koha::Biblio;

By "additionally we need a list (json, csv ..) of the records that should be taken from the biblio_id_to_merge record" did you mean a list of fields from the biblio_id_to_merge record?

Created attachment 152436 [details] [review]
Bug 33036: REST API: Merge biblio records implements merging of records + attached items, subscriptions etc via the API as an alternative to the web interface: cgi-bin/koha/cataloguing/merge.pl

To Test:
1) you need an API user with the permissions "editcatalogue"
2) two records: one to be merged into (with biblio_id, eg 262) and another one from
   which to merge (with biblio_id_to_merge, eg 9) which will be deleted!
   both records may/should have items, subscription, subscriptionhistory, serial, suggestions
   orders and holds
3) check both records via the web
4) Apply patch
5) execute API calls eg like
   PUT /biblios/{biblio_id}/merge/{biblio_id_to_merge}
   eg:
   curl  -s -u koha:koha -X PUT  "http://127.0.0.1:8081/api/v1/biblios/262/merge/9"
6) the record with the id <biblio_id_to_merge> is deleted now, the record with <biblio_id>
   has all items, etc attached, return code is 200, with the message {"message":"Successfuly merged 9 into 262"}
7) optionally a full MARCXML record may be sent as body of the API call
   curl  -s -u koha:koha -X PUT  "http://127.0.0.1:8081/api/v1/biblios/262/merge/2" -d @marcfile.xml
8) now also the content of the record with <biblio_id> is replaced with the content of the MARCXML file
9) Sign off.
10) Thx

Sponsored-by: Technische Hochschule Wildau
Co-authored-by: domm@plix.at

Created attachment 152437 [details] [review]
Bug 33036: REST API: Merge biblio records implements merging of records + attached items, subscriptions etc via the API as an alternative to the web interface: cgi-bin/koha/cataloguing/merge.pl

To Test:
1) you need an API user with the permissions "editcatalogue"
2) two records: one to be merged into (with biblio_id, eg 262) and another one from
   which to merge (with biblio_id_to_merge, eg 9) which will be deleted!
   both records may/should have items, subscription, subscriptionhistory, serial, suggestions
   orders and holds
3) check both records via the web
4) Apply patch
5) execute API calls eg like
   PUT /biblios/{biblio_id}/merge/{biblio_id_to_merge}
   eg:
   curl  -s -u koha:koha -X PUT  "http://127.0.0.1:8081/api/v1/biblios/262/merge/9"
6) the record with the id <biblio_id_to_merge> is deleted now, the record with <biblio_id>
   has all items, etc attached, return code is 200, with the message {"message":"Successfuly merged 9 into 262"}
7) optionally a full MARCXML record may be sent as body of the API call
   curl  -s -u koha:koha -X PUT  "http://127.0.0.1:8081/api/v1/biblios/262/merge/2" -d @marcfile.xml
8) now also the content of the record with <biblio_id> is replaced with the content of the MARCXML file
9) Sign off.
10) Thx

Sponsored-by: Technische Hochschule Wildau
Co-authored-by: domm@plix.at

Created attachment 152438 [details] [review]
Bug 33036: REST API: Merge biblio records

implements merging of records + attached items, subscriptions etc via the API
as an alternative to the web interface: cgi-bin/koha/cataloguing/merge.pl

To Test:
1) you need an API user with the permissions "editcatalogue"
2) two records: one to be merged into (with biblio_id, eg 262) and another one from
   which to merge (with biblio_id_to_merge, eg 9) which will be deleted!
   both records may/should have items, subscription, subscriptionhistory, serial, suggestions
   orders and holds
3) check both records via the web
4) Apply patch
5) execute API calls eg like
   PUT /biblios/{biblio_id}/merge/{biblio_id_to_merge}
   eg:
   curl  -s -u koha:koha -X PUT  "http://127.0.0.1:8081/api/v1/biblios/262/merge/9"
6) the record with the id <biblio_id_to_merge> is deleted now, the record with <biblio_id>
   has all items, etc attached, return code is 200, with the message {"message":"Successfuly merged 9 into 262"}
7) optionally a full MARCXML record may be sent as body of the API call
   curl  -s -u koha:koha -X PUT  "http://127.0.0.1:8081/api/v1/biblios/262/merge/2" -d @marcfile.xml
8) now also the content of the record with <biblio_id> is replaced with the content of the MARCXML file
9) Sign off.
10) Thx

Sponsored-by: Technische Hochschule Wildau
Co-authored-by: domm@plix.at

Created attachment 152439 [details] [review]
Bug 33036: REST API: Merge biblio records implements merging of records + attached items, subscriptions etc via the API as an alternative to the web interface: cgi-bin/koha/cataloguing/merge.pl

To Test:
1) you need an API user with the permissions "editcatalogue"
2) two records: one to be merged into (with biblio_id, eg 262) and another one from
   which to merge (with biblio_id_to_merge, eg 9) which will be deleted!
   both records may/should have items, subscription, subscriptionhistory, serial, suggestions
   orders and holds
3) check both records via the web
4) Apply patch
5) execute API calls eg like
   PUT /biblios/{biblio_id}/merge/{biblio_id_to_merge}
   eg:
   curl  -s -u koha:koha -X PUT  "http://127.0.0.1:8081/api/v1/biblios/262/merge/9"
6) the record with the id <biblio_id_to_merge> is deleted now, the record with <biblio_id>
   has all items, etc attached, return code is 200, with the message {"message":"Successfuly merged 9 into 262"}
7) optionally a full MARCXML record may be sent as body of the API call
   curl  -s -u koha:koha -X PUT  "http://127.0.0.1:8081/api/v1/biblios/262/merge/2" -d @marcfile.xml
8) now also the content of the record with <biblio_id> is replaced with the content of the MARCXML file
9) Sign off.
10) Thx

Sponsored-by: Technische Hochschule Wildau
Co-authored-by: domm@plix.at

Created attachment 152440 [details] [review]
Bug 33036: REST API: Merge biblio records implements merging of records + attached items, subscriptions etc via the API as an alternative to the web interface: cgi-bin/koha/cataloguing/merge.pl

To Test:
1) you need an API user with the permissions "editcatalogue"
2) two records: one to be merged into (with biblio_id, eg 262) and another one from
   which to merge (with biblio_id_to_merge, eg 9) which will be deleted!
   both records may/should have items, subscription, subscriptionhistory, serial, suggestions
   orders and holds
3) check both records via the web
4) Apply patch
5) execute API calls eg like
   PUT /biblios/{biblio_id}/merge/{biblio_id_to_merge}
   eg:
   curl  -s -u koha:koha -X PUT  "http://127.0.0.1:8081/api/v1/biblios/262/merge/9"
6) the record with the id <biblio_id_to_merge> is deleted now, the record with <biblio_id>
   has all items, etc attached, return code is 200, with the message {"message":"Successfuly merged 9 into 262"}
7) optionally a full MARCXML record may be sent as body of the API call
   curl  -s -u koha:koha -X PUT  "http://127.0.0.1:8081/api/v1/biblios/262/merge/2" -d @marcfile.xml
8) now also the content of the record with <biblio_id> is replaced with the content of the MARCXML file
9) Sign off.
10) Thx

Sponsored-by: Technische Hochschule Wildau
Co-authored-by: domm@plix.at

Created attachment 152447 [details] [review]
Bug 33036: REST API: Merge biblio records implements merging of records + attached items, subscriptions etc via the API as an alternative to the web interface: cgi-bin/koha/cataloguing/merge.pl

To Test:
1) you need an API user with the permissions "editcatalogue"
2) two records: one to be merged into (with biblio_id, eg 262) and another one from
   which to merge (with biblio_id_to_merge, eg 9) which will be deleted!
   both records may/should have items, subscription, subscriptionhistory, serial, suggestions
   orders and holds
3) check both records via the web
4) Apply patch
5) execute API calls eg like
   PUT /biblios/{biblio_id}/merge/{biblio_id_to_merge}
   eg:
   curl  -s -u koha:koha -X PUT  "http://127.0.0.1:8081/api/v1/biblios/262/merge/9"
6) the record with the id <biblio_id_to_merge> is deleted now, the record with <biblio_id>
   has all items, etc attached, return code is 200, with the message {"message":"Successfuly merged 9 into 262"}
7) optionally a full MARCXML record may be sent as body of the API call
   curl  -s -u koha:koha -X PUT  "http://127.0.0.1:8081/api/v1/biblios/262/merge/2" -d @marcfile.xml
8) now also the content of the record with <biblio_id> is replaced with the content of the MARCXML file
9) Sign off.
10) Thx

Sponsored-by: Technische Hochschule Wildau
Co-authored-by: domm@plix.at

Created attachment 152472 [details] [review]
Bug 33036: REST API: Merge biblio records implements merging of records + attached items, subscriptions etc via the API as an alternative to the web interface: cgi-bin/koha/cataloguing/merge.pl

To Test:
1) you need an API user with the permissions "editcatalogue"
2) two records: one to be merged into (with biblio_id, eg 262) and another one from
   which to merge (with biblio_id_to_merge, eg 9) which will be deleted!
   both records may/should have items, subscription, subscriptionhistory, serial, suggestions
   orders and holds
3) check both records via the web
4) Apply patch
5) execute API calls eg like
   PUT /biblios/{biblio_id}/merge/{biblio_id_to_merge}
   eg:
   curl  -s -u koha:koha -X PUT  "http://127.0.0.1:8081/api/v1/biblios/262/merge/9"
6) the record with the id <biblio_id_to_merge> is deleted now, the record with <biblio_id>
   has all items, etc attached, return code is 200, with the message {"message":"Successfuly merged 9 into 262"}
7) optionally a full MARCXML record may be sent as body of the API call
   curl  -s -u koha:koha -X PUT  "http://127.0.0.1:8081/api/v1/biblios/262/merge/2" -d @marcfile.xml
8) now also the content of the record with <biblio_id> is replaced with the content of the MARCXML file
9) Sign off.
10) Thx

Sponsored-by: Technische Hochschule Wildau
Co-authored-by: domm@plix.at
Signed-off-by: Jan Kissig <jkissig@th-wildau.de>

When using Postman instead of curl to send HTTP-request (body:raw, type:xml) with a body like this:

<?xml version="1.0" encoding="UTF-8"?>
<record
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://www.loc.gov/MARC21/slim http://www.loc.gov/standards/marcxml/schema/MARC21slim.xsd"
    xmlns="http://www.loc.gov/MARC21/slim">
  <leader>00224nam a22001097a 4500</leader>
  <controlfield tag="003">OSt</controlfield>
  <controlfield tag="005">20230412064312.0</controlfield>
  <controlfield tag="008">230412b        |||||||| |||| 00| 0 eng d</controlfield>
  <datafield tag="040" ind1=" " ind2=" ">
    <subfield code="c">Ost</subfield>
  </datafield>
  <datafield tag="245" ind1=" " ind2=" ">
    <subfield code="a">NewTitle</subfield>
  </datafield>
</record>

f.e. PUT http://localhost:8081/api/v1/biblios/262/merge/6 I get the error

{
    "errors": [
        {
            "message": "Expected object - got string.",
            "path": "/body"
        }
    ],
    "status": 400
}

The proposed way just works fine:
 7) optionally a full MARCXML record may be sent as body of the API call
   curl  -s -u koha:koha -X PUT  "http://127.0.0.1:8081/api/v1/biblios/262/merge/2" -d @marcfile.xml

Created attachment 152659 [details] [review]
Bug 33036: REST API: Merge biblio records implements merging of records + attached items, subscriptions etc via the API as an alternative to the web interface: cgi-bin/koha/cataloguing/merge.pl

To Test:
1) you need an API user with the permissions "editcatalogue"
2) two records: one to be merged into (with biblio_id, eg 262) and another one from
   which to merge (with biblio_id_to_merge, eg 9) which will be deleted!
   both records may/should have items, subscription, subscriptionhistory, serial, suggestions
   orders and holds
3) check both records via the web
4) Apply patch
5) execute API calls eg like
   PUT /biblios/{biblio_id}/merge/{biblio_id_to_merge}
   eg:
   curl  -s -u koha:koha -X PUT  "http://127.0.0.1:8081/api/v1/biblios/262/merge/9"
6) the record with the id <biblio_id_to_merge> is deleted now, the record with <biblio_id>
   has all items, etc attached, return code is 200, with the message {"message":"Successfuly merged 9 into 262"}
7) optionally a full MARCXML record may be sent as body of the API call
   curl  -s -u koha:koha -X PUT  "http://127.0.0.1:8081/api/v1/biblios/262/merge/2" -d @marcfile.xml
8) now also the content of the record with <biblio_id> is replaced with the content of the MARCXML file
9) Sign off.
10) Thx

Sponsored-by: Technische Hochschule Wildau
Co-authored-by: domm@plix.at

Created attachment 152678 [details] [review]
Bug 33036: REST API: Merge biblio records implements merging of records + attached items, subscriptions etc via the API as an alternative to the web interface: cgi-bin/koha/cataloguing/merge.pl

To Test:
1) you need an API user with the permissions "editcatalogue"
2) two records: one to be merged into (with biblio_id, eg 262) and another one from
   which to merge (with biblio_id_to_merge, eg 9) which will be deleted!
   both records may/should have items, subscription, subscriptionhistory, serial, suggestions
   orders and holds
3) check both records via the web
4) Apply patch
5) execute API calls eg like
   PUT /biblios/{biblio_id}/merge/{biblio_id_to_merge}
   eg:
   curl  -s -u koha:koha -X PUT  "http://127.0.0.1:8081/api/v1/biblios/262/merge/9"
6) the record with the id <biblio_id_to_merge> is deleted now, the record with <biblio_id>
   has all items, etc attached, return code is 200, with the message {"message":"Successfuly merged 9 into 262"}
7) optionally a full MARCXML record may be sent as body of the API call
   curl  -s -u koha:koha -X PUT  "http://127.0.0.1:8081/api/v1/biblios/262/merge/2" -d @marcfile.xml
8) now also the content of the record with <biblio_id> is replaced with the content of the MARCXML file
9) Sign off.
10) Thx

Sponsored-by: Technische Hochschule Wildau
Co-authored-by: domm@plix.at
Signed-off-by: Jan Kissig <jkissig@th-wildau.de>

"From: Tomas Cohen Arazi <tomascohen@theke.io>"

This does not seem correct.

1. Tests are missing

2. You are duplicating code. The correct way to do this is to first move the code to a module, provide test, then call that method from merge.pl and your Mojo controller.

Also don't forget to run the QA script:

 FAIL   Koha/REST/V1/Biblios.pm
   FAIL   forbidden patterns
                forbidden pattern: Use of $c->validation in API controllers should be avoided (bug 33556) (line 876)
                forbidden pattern: Use of $c->validation in API controllers should be avoided (bug 33556) (line 877)
   WARN   tidiness
                The file is less tidy than before (bad/messy lines before: 138, now: 175)

Hi Jonathan, I'm trying to develop this patch so I ask same explanation about your requests:
1)'first move the code to a module'.
In this patch the new verb 'merge' is inside an existing module, 'Koha/REST/V1/Biblios.pm' and it is also a Mojo controller.
So the code is in the correct place.

2)'provide test'
OK, to do.

3)'then call that method from merge.pl and your Mojo controller'
Yes, I need to change the merge.pl and use the REST API instead of the present code.

Is it a correct work plan?

(In reply to Zeno Tajoli from comment #16)
> Hi Jonathan, I'm trying to develop this patch so I ask same explanation
> about your requests:
> 1)'first move the code to a module'.
> In this patch the new verb 'merge' is inside an existing module,
> 'Koha/REST/V1/Biblios.pm' and it is also a Mojo controller.
> So the code is in the correct place.

To make it reusable from another place than the REST API.
It needs to be placed into Koha::Biblio (see Koha::Patron::merge_with)

> 3)'then call that method from merge.pl and your Mojo controller'
> Yes, I need to change the merge.pl and use the REST API instead of the
> present code.

No, you need to call Koha::Biblio::merge_with from merge.pl and the REST API controller.

Created attachment 157481 [details]
A MARC21 record in MARCXML format

You can use MARCXML file for tests

Created attachment 157482 [details] [review]
patch to implement the feature

This is a rewrite of the feature with the use of a module, tests, and perltidy.
Now merge.pl and the REST API controller use Koha::Biblio::merge_with.

Test plan:
    1) you need an API user with the permissions "editcatalogue"
    2) two records: one to be merged into (with biblio_id, eg 262) and another one from
       which to merge (with biblio_id_to_merge, eg 9) which will be deleted!
       both records may/should have items, subscription, subscriptionhistory, serial, suggestions
       orders and holds
    3) check both records via the web
    4) Apply patch
    5) execute API calls eg like
       PUT /biblios/{biblio_id}/merge/{biblio_id_to_merge}
       eg:
       curl  -s -u koha:koha -X POST  "http://127.0.0.1:8081/api/v1/biblios/262/merge/9"
    6) the record with the id <biblio_id_to_merge> is deleted now, the record with <biblio_id>
       has all items, etc attached, return code is 200, with the message {"message":"Successfuly merged 9 into 262"}
    7) optionally a full MARCXML record may be sent as body of the API call
       curl  -s -u koha:koha -X POST  "http://127.0.0.1:8081/api/v1/biblios/262/merge/2" -d @marcfile.xml
    8) now also the content of the record with <biblio_id> is replaced with the content of the MARCXML file
    9) Go into intramet and do a search. Select two or (better) more record.
    10) Merge them; merge must be a success.
    11) Test with prove -v t/db_dependent/Koha/Biblio.t

Created attachment 157731 [details] [review]
Bug 33036: REST API: Merge biblio records implements merging of records + attached items, subscriptions etc via the API as an alternative to the web interface: cgi-bin/koha/cataloguing/merge.pl

To test:
    1) you need an API user with the permissions "editcatalogue"
    2) two records: one to be merged into (with biblio_id, eg 262) and another one from
       which to merge (with biblio_id_to_merge, eg 9) which will be deleted!
       both records may/should have items, subscription, subscriptionhistory, serial, suggestions
       orders and holds
    3) check both records via the web
    4) Apply patch
    5) execute API calls eg like
       POST /biblios/{biblio_id}/merge/{biblio_id_to_merge}
       eg:
       curl  -s -u koha:koha -X POST  "http://127.0.0.1:8081/api/v1/biblios/262/merge/9"
    6) the record with the id <biblio_id_to_merge> is deleted now, the record with <biblio_id>
       has all items, etc attached, return code is 200, with the message {"message":"Successfuly merged 9 into 262"}
    7) optionally a full MARCXML record may be sent as body of the API call
       curl  -s -u koha:koha -X POST  "http://127.0.0.1:8081/api/v1/biblios/262/merge/2" -d @marcfile.xml
    8) now also the content of the record with <biblio_id> is replaced with the content of the MARCXML file
    9) Go into intramet and do a search. Select two or (better) more record.
    10) Merge them; merge must be a success.
    11) Test with prove -v t/db_dependent/Koha/Biblio.t

To do test with curl you canuse the MARCXML (MARC21) file attached in bugzilla.

Sponsored-by: Technische Hochschule Wildau
Co-authored-by: domm@plix.at
Signed-off-by: Jan Kissig <jkissig@th-wildau.de>

Tested merge with
- attached holds
- attached items
- attached subscription
- attached order
- POSTing XML

Hi Zeno,

Some QA feedbacks:

1. Please tidy new code (using perltidy https://wiki.koha-community.org/wiki/Perltidy)

2. Those lines are non needed and must be removed:

+use C4::Serials qw( CountSubscriptionFromBiblionumber ); (duplicate)

+    #my $biblio = Koha::Biblios->find($self->biblionumber);

+    #say Dumper $results;

3. New REST API endpoint needs to be covered by tests

4.
+      - $ref: "../swagger.yaml#/parameters/marc_schema_header"
=> You are not using it (x-record-schema) in the controller code

5. The error handling is wrong:
  a. You should raise an exception instead of returning
  b. You should run the whole merge code in a transaction

6. Not sure about the response of the endpoint, IMO it should return the biblio generated by the merge.
Also the response contains "merged" and "kept" with the IDs we passed, it does not seem needed.

Note that Tomas suggested on the list:
POST /biblios/123/merges
{
    "biblio_id": 456,
    "rules": ?
}

We will need his feedback here.

(In reply to Jonathan Druart from comment #23)
> 
> 6. Not sure about the response of the endpoint, IMO it should return the
> biblio generated by the merge.
> Also the response contains "merged" and "kept" with the IDs we passed, it
> does not seem needed.

I'd say it needs to return the resulting object, with the Location header pointing to the /biblios/:biblio_id endpoint (instead of the /merge)

> Note that Tomas suggested on the list:
> POST /biblios/123/merges
> {
>     "biblio_id": 456,
>     "rules": ?
> }

Yes, we certainly need a way to specify what to pick from which record... An initial implementation could just implement a full overwrite (i.e. effectively moving the items and removing the biblio) but it needs to allow, by design, for future enhancements like this, so it is more useful.

Hi Jonathan Druart,

are you sure that there are problems with tidying for this patch?
I use perltidy conf that are in the git archive and I do a check with 
koha-qa.pl -v 2 -c 1 and I pass the test.

(In reply to Zeno Tajoli from comment #25)
> Hi Jonathan Druart,
> 
> are you sure that there are problems with tidying for this patch?
> I use perltidy conf that are in the git archive and I do a check with 
> koha-qa.pl -v 2 -c 1 and I pass the test.

+    if($@)
+    {   push @errors, $@;
+    }   else
+    {   if ($rmerge->{'status'}==0)
+        {   push @errors, $rmerge->{'error'};
         }

This looks wrong, we have 

  8 # } else {
  9 --cuddled-else

in our .perltidyrc (at the root of Koha src)

See here too: https://wiki.koha-community.org/wiki/Perltidy

Hi Thomas,
could you write here a full example with curl on how do you want the call of the API ?
I need parmas of curl and headers, json file fields and how to pass the MARC21/UNIMARC data (a field of the json file?).
To undestand better how to do the work.

Created attachment 159011 [details]
An example of JSON file to use in testing

Created attachment 159012 [details] [review]
Patch that implemnts the end point

A new version that follow the requests given.

To test:
    1) you need an API user with the permissions "editcatalogue"
    2) two records: one to be merged into (with biblio_id, eg 262) and another 
       one from which to merge (with biblio_id_to_merge, eg 9) which will be
       deleted! both records may/should have items, subscription, 
       subscriptionhistory, serial, suggestions, orders and holds
    3) check both records via the web
    4) Apply patch
    5) Write a JSON file with inside the field 'biblio_id_to_merge' and the 
      biblionumber from wihich to merge.
       As example:
       {
         "biblio_id_to_merge" : 9
       }
     6) Execute an API call with correct headers and location. For example:
        curl -s -u koha:koha --header "Content-Type: application/json" --header "Accept: application/marc-in-json" --request POST "http://127.0.0.1:8080/api/v1/biblios/262/merge" -d @file.json
        You must to setup the headers and to use a json file with parameters
     7) The record with the id 9 is deleted now, the record with 262 has all
        items, etc attached, the return is: return code 200 and the changed
        record 262 in marc-in-json format
     8) It is possible to override biblio data with an external bib record. You
        need to put external bib record into the json file in marc-in-json 
        format. To write use the json file uploaded as example
        You need to fill the fields 'rules' and 'datarecord'. The field 'rules' 
        must contains 'override_ext'
        To do the call:
         curl -s -u koha:koha --header "Content-Type: application/json" --header 
 "Accept: application/marc-in-json" --request POST "http://127.0.0.1:8080/api/v1/biblios/XXX/merge" -d @file_with_recod.json
      9) The record in 'biblio_id_to_merge' is deleted now, in biblio XXX now 
         there are the bibliographic data of field 'datarecord' of json file, 
         the return is: return code 200 and the changed record XXX in 
         marc-in-json format
      10) Go into intranet and do a search. Select two or (better) more record.
      11) Merge them; merge must be a success.
      12) Test with prove -v t/db_dependent/Koha/Biblio.t
      13) Test with prove -v t/db_dependent/api/v1/biblios.t

To test with curl the step 8 you can customize the json file attached in bugzilla.
The marc-in-json record inside follows the MAR21 standard

Created attachment 159103 [details] [review]
Bug 33036: REST API: Merge biblio records implements merging of records + attached items, subscriptions etc via the API as an alternative to the web interface: cgi-bin/koha/cataloguing/merge.pl

This is a slightly improved version of Zenos patch: I (domm) have converted the code in Koha::Biblio to a more DBICy style and packed it into a transaction (as requested in Comment 23)

The QA script complains about files being less tidy, but if we add the perltidy changes to this commit I'm afraid the actual change will be much less clear to review. So I did not run perltidy on the changed files.

To test:
    1) you need an API user with the permissions "editcatalogue"
    2) two records: one to be merged into (with biblio_id, eg 262) and another one from
       which to merge (with biblio_id_to_merge, eg 9) which will be deleted!
       both records may/should have items, subscription, subscriptionhistory, serial, suggestions
       orders and holds
    3) check both records via the web
    4) Apply patch
    5) Write a JSON file with inside the field 'biblio_id_to_merge' and the biblionumber from wihich to merge.
       As example:
       {
         "biblio_id_to_merge" : 9
       }
     6) Execute an API call with correct headers and location. For example:
        curl -s -u koha:koha --header "Content-Type: application/json" --header "Accept: application/marc-in-json"
                  --request POST "http://127.0.0.1:8080/api/v1/biblios/262/merge" -d @file.json
        You must to setup the headers and to use a json file with parameters
     7) The record with the id 9 is deleted now, the record with 262 has all items, etc attached,
        the return is: return code 200 and the changed record 262 in marc-in-json format
     8) It is possible to override biblio data with an external bib record. You need to put external bib record
        into the json file in marc-in-json format. To write use the json file uploaded as example
        You need to fill the fields 'rules' and 'datarecord'. The field 'rules' must contains 'override_ext'
        To do the call:
         curl -s -u koha:koha --header "Content-Type: application/json" --header "Accept: application/marc-in-json"
                  --request POST "http://127.0.0.1:8080/api/v1/biblios/XXX/merge" -d @file_with_recod.json
      9) The record in 'biblio_id_to_merge' is deleted now, in biblio XXX now there are the bibliographic data
         of field 'datarecord' of json file, the return is: return code 200 and the changed record XXX in marc-in-json format
      10) Go into intranet and do a search. Select two or (better) more record.
      11) Merge them; merge must be a success.
      12) Test with prove -v t/db_dependent/Koha/Biblio.t
      13) Test with prove -v t/db_dependent/api/v1/biblios.t

To test with curl the step 8 you can customize the json file attached in bugzilla.
The marc-in-json record inside follows the MAR21 standard

Sponsored-by: Technische Hochschule Wildau
Co-authored-by: Zeno Tajoli <ztajoli@gmail.com>
Co-authored-by: Thomas Klausner <domm@plix.at>
Co-authored-by: Mark Hofstetter <<mark@hofstetter.at>>
Signed-off-by: Jan Kissig <jkissig@th-wildau.de>

Created attachment 159120 [details] [review]
Bug 33036: REST API: Merge biblio records implements merging of records + attached items, subscriptions etc via the API as an alternative to the web interface: cgi-bin/koha/cataloguing/merge.pl

This is a slightly improved version of Zenos patch: I (domm) have converted the code in Koha::Biblio to a more DBICy style and packed it into a transaction (as requested in Comment 23)

Even the QA script is happy now!

To test:
    1) you need an API user with the permissions "editcatalogue"
    2) two records: one to be merged into (with biblio_id, eg 262) and another one from
       which to merge (with biblio_id_to_merge, eg 9) which will be deleted!
       both records may/should have items, subscription, subscriptionhistory, serial, suggestions
       orders and holds
    3) check both records via the web
    4) Apply patch
    5) Write a JSON file with inside the field 'biblio_id_to_merge' and the biblionumber from wihich to merge.
       As example:
       {
         "biblio_id_to_merge" : 9
       }
     6) Execute an API call with correct headers and location. For example:
        curl -s -u koha:koha --header "Content-Type: application/json" --header "Accept: application/marc-in-json"
                  --request POST "http://127.0.0.1:8080/api/v1/biblios/262/merge" -d @file.json
        You must to setup the headers and to use a json file with parameters
     7) The record with the id 9 is deleted now, the record with 262 has all items, etc attached,
        the return is: return code 200 and the changed record 262 in marc-in-json format
     8) It is possible to override biblio data with an external bib record. You need to put external bib record
        into the json file in marc-in-json format. To write use the json file uploaded as example
        You need to fill the fields 'rules' and 'datarecord'. The field 'rules' must contains 'override_ext'
        To do the call:
         curl -s -u koha:koha --header "Content-Type: application/json" --header "Accept: application/marc-in-json"
                  --request POST "http://127.0.0.1:8080/api/v1/biblios/XXX/merge" -d @file_with_recod.json
      9) The record in 'biblio_id_to_merge' is deleted now, in biblio XXX now there are the bibliographic data
         of field 'datarecord' of json file, the return is: return code 200 and the changed record XXX in marc-in-json format
      10) Go into intranet and do a search. Select two or (better) more record.
      11) Merge them; merge must be a success.
      12) Test with prove -v t/db_dependent/Koha/Biblio.t
      13) Test with prove -v t/db_dependent/api/v1/biblios.t

To test with curl the step 8 you can customize the json file attached in bugzilla.
The marc-in-json record inside follows the MAR21 standard

Sponsored-by: Technische Hochschule Wildau
Co-authored-by: Zeno Tajoli <ztajoli@gmail.com>
Co-authored-by: Thomas Klausner <domm@plix.at>
Co-authored-by: Mark Hofstetter <<mark@hofstetter.at>>
Signed-off-by: Jan Kissig <jkissig@th-wildau.de>

Created attachment 159191 [details] [review]
Bug 33036: REST API: Merge biblio records implements merging of records + attached items, subscriptions etc via the API as an alternative to the web interface: cgi-bin/koha/cataloguing/merge.pl

This is a slightly improved version of Zenos patch: I (domm) have converted the code in Koha::Biblio to a more DBICy style and packed it into a transaction (as requested in Comment 23)

Even the QA script is happy now!

To test:
    1) you need an API user with the permissions "editcatalogue"
    2) two records: one to be merged into (with biblio_id, eg 262) and another one from
       which to merge (with biblio_id_to_merge, eg 9) which will be deleted!
       both records may/should have items, subscription, subscriptionhistory, serial, suggestions
       orders and holds
    3) check both records via the web
    4) Apply patch
    5) Write a JSON file with inside the field 'biblio_id_to_merge' and the biblionumber from wihich to merge.
       As example:
       {
         "biblio_id_to_merge" : 9
       }
     6) Execute an API call with correct headers and location. For example:
        curl -s -u koha:koha --header "Content-Type: application/json" --header "Accept: application/marc-in-json"
                  --request POST "http://127.0.0.1:8080/api/v1/biblios/262/merge" -d @file.json
        You must to setup the headers and to use a json file with parameters
     7) The record with the id 9 is deleted now, the record with 262 has all items, etc attached,
        the return is: return code 200 and the changed record 262 in marc-in-json format
     8) It is possible to override biblio data with an external bib record. You need to put external bib record
        into the json file in marc-in-json format. To write use the json file uploaded as example
        You need to fill the fields 'rules' and 'datarecord'. The field 'rules' must contains 'override_ext'
        To do the call:
         curl -s -u koha:koha --header "Content-Type: application/json" --header "Accept: application/marc-in-json"
                  --request POST "http://127.0.0.1:8080/api/v1/biblios/XXX/merge" -d @file_with_recod.json
      9) The record in 'biblio_id_to_merge' is deleted now, in biblio XXX now there are the bibliographic data
         of field 'datarecord' of json file, the return is: return code 200 and the changed record XXX in marc-in-json format
      10) Go into intranet and do a search. Select two or (better) more record.
      11) Merge them; merge must be a success.
      12) Test with prove -v t/db_dependent/Koha/Biblio.t
      13) Test with prove -v t/db_dependent/api/v1/biblios.t

To test with curl the step 8 you can customize the json file attached in bugzilla.
The marc-in-json record inside follows the MAR21 standard

Sponsored-by: Technische Hochschule Wildau
Co-authored-by: Zeno Tajoli <ztajoli@gmail.com>
Co-authored-by: Thomas Klausner <domm@plix.at>
Co-authored-by: Mark Hofstetter <<mark@hofstetter.at>>
Signed-off-by: Jan Kissig <jkissig@th-wildau.de>

Sign off done by Jan Kissig as I see.

Created attachment 159239 [details]
An example of JSON file to use in testing

Created attachment 159826 [details] [review]
Bug 33036: Improve merge_with tests

Created attachment 159827 [details] [review]
Bug 33036: Add more tests

Created attachment 159828 [details] [review]
Bug 33036: Use Koha::Objects

Better to use Koha::Objects everywhere and have the same pattern for the
different related objects.

Created attachment 159829 [details] [review]
Bug 33036: Improve REST API controller

1. Added some improvements and missing unit tests

2. Wondering if we should not support more than json for the marc record. Not considering blocker however (can be done later if needed).

3. Error handling is not the best, but it was bad already (ModBiblio and DelBiblio erk!) but we improved it.

IMO This can go now, but I would like a last round of test (+ review) from Mark, domm or Zeno on my follow-ups.

looks ok to me

for testing/using please don't forget to activate RESTBasicAuth

/cgi-bin/koha/admin/preferences.pl?op=search&searchfield=RESTBasicAuth

cheers and thx to all

Created attachment 160302 [details] [review]
Update of test number

I check the patches of Jonathan and for me are OK.

I add a new little patch because there is a new test inside t/db_dependent/API/v1/biblios.t (bug 35574).

Now there is only JSON for inputting a MARC record because inside JSON I also insert other parameters ["biblio_id_to_merge", rules", "framework_to_use"]

I don't know how to pass them without using JSON.

Created attachment 160468 [details] [review]
Bug 33036: REST API: Merge biblio records implements merging of records

+ attached items, subscriptions etc via the API as an alternative to the web interface: cgi-bin/koha/cataloguing/merge.pl

This is a slightly improved version of Zenos patch: I (domm) have converted the code in Koha::Biblio to a more DBICy style and packed it into a transaction (as requested in Comment 23)

Even the QA script is happy now!

To test:
    1) you need an API user with the permissions "editcatalogue"
    2) two records: one to be merged into (with biblio_id, eg 262) and another one from
       which to merge (with biblio_id_to_merge, eg 9) which will be deleted!
       both records may/should have items, subscription, subscriptionhistory, serial, suggestions
       orders and holds
    3) check both records via the web
    4) Apply patch
    5) Write a JSON file with inside the field 'biblio_id_to_merge' and the biblionumber from wihich to merge.
       As example:
       {
         "biblio_id_to_merge" : 9
       }
     6) Execute an API call with correct headers and location. For example:
        curl -s -u koha:koha --header "Content-Type: application/json" --header "Accept: application/marc-in-json"
                  --request POST "http://127.0.0.1:8080/api/v1/biblios/262/merge" -d @file.json
        You must to setup the headers and to use a json file with parameters
     7) The record with the id 9 is deleted now, the record with 262 has all items, etc attached,
        the return is: return code 200 and the changed record 262 in marc-in-json format
     8) It is possible to override biblio data with an external bib record. You need to put external bib record
        into the json file in marc-in-json format. To write use the json file uploaded as example
        You need to fill the fields 'rules' and 'datarecord'. The field 'rules' must contains 'override_ext'
        To do the call:
         curl -s -u koha:koha --header "Content-Type: application/json" --header "Accept: application/marc-in-json"
                  --request POST "http://127.0.0.1:8080/api/v1/biblios/XXX/merge" -d @file_with_recod.json
      9) The record in 'biblio_id_to_merge' is deleted now, in biblio XXX now there are the bibliographic data
         of field 'datarecord' of json file, the return is: return code 200 and the changed record XXX in marc-in-json format
      10) Go into intranet and do a search. Select two or (better) more record.
      11) Merge them; merge must be a success.
      12) Test with prove -v t/db_dependent/Koha/Biblio.t
      13) Test with prove -v t/db_dependent/api/v1/biblios.t

To test with curl the step 8 you can customize the json file attached in bugzilla.
The marc-in-json record inside follows the MAR21 standard

Sponsored-by: Technische Hochschule Wildau
Co-authored-by: Zeno Tajoli <ztajoli@gmail.com>
Co-authored-by: Thomas Klausner <domm@plix.at>
Co-authored-by: Mark Hofstetter <<mark@hofstetter.at>>
Signed-off-by: Jan Kissig <jkissig@th-wildau.de>

Bug 33036: Update of test number.

File ../biblios.t was update with a new subutest.
So we need this update to have a 'OK' after test running.

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>

Created attachment 160469 [details] [review]
Bug 33036: Improve merge_with tests

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>

Created attachment 160470 [details] [review]
Bug 33036: Add more tests

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>

Created attachment 160471 [details] [review]
Bug 33036: Use Koha::Objects

Better to use Koha::Objects everywhere and have the same pattern for the
different related objects.

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>

Created attachment 160472 [details] [review]
Bug 33036: Improve REST API controller

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>

Fixed some small conflicts in Biblio.t caused by the adding of other unit tests meanwhile. 
Easy to verify for unit tests that you got things right at least :)

I notice this adds a lot of dependencies on C4 modules to Koha/Biblio.pm. I wonder if not some of those could have been replaced easily, like "CountSubscriptionFromBiblioNumber". 

Maybe something for later follow-ups?

Pushed for 24.05!

Well done everyone, thank you!

Test suite is happy, I backport

Pushed to 23.11.x for 23.11.04

Enhancement will not be included in 23.05.x

API route change, nothing to add/edit in the Koha manual. I assume API documentation is done automatically?