Bug 33418 - Allow setting overlay_framework for connexion imports
Summary: Allow setting overlay_framework for connexion imports
Status: Pushed to main
Alias: None
Product: Koha
Classification: Unclassified
Component: MARC Bibliographic record staging/import (show other bugs)
Version: Main
Hardware: All All
: P5 - low enhancement (vote)
Assignee: Nick Clemens (kidclamp)
QA Contact: Testopia
URL:
Keywords: release-notes-needed
Depends on:
Blocks:
 
Reported: 2023-04-05 12:49 UTC by Nick Clemens (kidclamp)
Modified: 2024-05-10 15:10 UTC (History)
4 users (show)

See Also:
Change sponsored?: ---
Patch complexity: Trivial patch
Documentation contact:
Documentation submission:
Text to go in the release notes:
Version(s) released in:
24.05.00


Attachments
Bug 33419: POC untested (4.14 KB, patch)
2023-04-05 12:54 UTC, Nick Clemens (kidclamp)
Details | Diff | Splinter Review
Bug 33418: Add overlay_framework option to connexion scripts (11.57 KB, patch)
2024-02-22 16:39 UTC, Nick Clemens (kidclamp)
Details | Diff | Splinter Review
Test records for Connexion import (596 bytes, text/plain)
2024-02-22 16:41 UTC, Nick Clemens (kidclamp)
Details
Bug 33418: Add overlay_framework option to connexion scripts (11.61 KB, patch)
2024-02-26 19:26 UTC, Brendan Lawlor
Details | Diff | Splinter Review
Bug 33418: Add overlay_framework option to connexion scripts (11.71 KB, patch)
2024-05-10 09:40 UTC, Marcel de Rooy
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description Nick Clemens (kidclamp) 2023-04-05 12:49:50 UTC
Bug 15869 added the ability to specify frameworks when overlaying while importing, we should extend this to allow updating the framework for records imported from OCLC connexion
Comment 1 Nick Clemens (kidclamp) 2023-04-05 12:54:39 UTC
Created attachment 149203 [details] [review]
Bug 33419: POC untested

This will require testing, and better instructions on how to match etc.

To test:
1 - Supply a value for 'framework' and 'overlay_framework' in the connxion.conf file
2 - Setup connexion, there is some help here on testing values:
    https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=15720
3 - Test importing bibs that match, and that don't match
4 - Confirm the framework value is used for new records
5 - Confirm if framework is blank the default framework is used for new records
6 - Confirm if framework is not supplied the default framework is used for new records
7 - Confirm a matching record that overlays obeys the overlay_framework param
8 - Confirm overlayed record uses default framework if the param is blank
9 - Confirm the original records framework is retained if the param is not passed
Comment 2 Nick Clemens (kidclamp) 2023-04-24 18:30:56 UTC
TO DO: 
cronjobs/import_webservice_batch.pl
Comment 3 Nick Clemens (kidclamp) 2024-02-22 16:39:31 UTC
Created attachment 162338 [details] [review]
Bug 33418: Add overlay_framework option to connexion scripts

This patch adds the option to specify a framework to be used when overlaying records from webservices/connexion

To test:
1 - vim /etc/koha/sites/kohadev/connexion.cnf
2 - Set content:
    host:
    port: 8888
    koha:http://localhost:8081
    log:/var/log/koha/kohadev/connexion.log
    match:ISBN
    user:kohauser
    password:kohapass
    overlay_action:replace
    nomatch_action:create_new
    item_action:always_add
    import_mode:direct
    framework:BKS
    overlay_framework:
    debug:1
3 - Save the sample file from this bug into your kohaclone (or copy and paste into a file your koha test site can reach)
4 - On the command line:
    perl misc/bin/connexion_import_daemon.pl -c /etc/koha/sites/kohadev/connexion.cnf
5 - In another terminal:
    cat bug_33418.test  | nc -v localhost 8888
6 - It should report success and a biblionumber
7 - In Koha: Cataloguing->Manage staged MARC record
8 - View the most recent batch with file name (webservice)
9 - Confirm it was imported, no match
10 - Click 'View' under the 'Record' column
11 - Confirm record loads correctly and 'MARC framework' detail is 'Books, Booklets, Workbooks'
12 - On the terminal repeat the command:
    cat bug_33418.test  | nc -v localhost 8888
13 - It should succed
14 - View the new batch, confirm the record matched this time
15 - View the record details, confirm framework is now 'default'
16 - On the first terminal hit Ctrl+C to stop the daemon
16 - Edit connexion.cnf and change:
    import_mode:stage
    framework:ACQ
    overlay_framework:IR
17 - Restart daemon:
     perl misc/bin/connexion_import_daemon.pl -c /etc/koha/sites/kohadev/connexion.cnf
18 - Delete the record created above
19 - On the second terminal repeat the command:
    cat bug_33418.test  | nc -v localhost 8888
20 - Confirm the batch is created, but not imported
21 - In terminal:
     perl misc/cronjobs/import_webservice_batch.pl --framework=ACQ --overlay_framework=BKS
22 - Confirm batch imported, and record in ACQ framework
23 - In terminal:
     cat bug_33418.test  | nc -v localhost 8888
     perl misc/cronjobs/import_webservice_batch.pl --framework=ACQ --overlay_framework=BKS
24 - Confirm batch added, record matched, record imported, and record now in Books framework
25 - Stop the deamon, edit connexion.cnf:
     import_mode:direct
26 - Start the daemon, and on other terminal repeat:
     cat bug_33418.test  | nc -v localhost 8888
27 - Confirm record in Binders framework
28 - Set record framework to Books
29 - Stop daemon, edit cnf and remove 'overlay_framework' setting
30 - Start daemon and cat the file again
31 - Confirm the record remains in Books framework
Comment 4 Nick Clemens (kidclamp) 2024-02-22 16:41:53 UTC
Created attachment 162339 [details]
Test records for Connexion import
Comment 5 Brendan Lawlor 2024-02-26 19:26:52 UTC
Created attachment 162471 [details] [review]
Bug 33418: Add overlay_framework option to connexion scripts

This patch adds the option to specify a framework to be used when overlaying records from webservices/connexion

To test:
1 - vim /etc/koha/sites/kohadev/connexion.cnf
2 - Set content:
    host:
    port: 8888
    koha:http://localhost:8081
    log:/var/log/koha/kohadev/connexion.log
    match:ISBN
    user:kohauser
    password:kohapass
    overlay_action:replace
    nomatch_action:create_new
    item_action:always_add
    import_mode:direct
    framework:BKS
    overlay_framework:
    debug:1
3 - Save the sample file from this bug into your kohaclone (or copy and paste into a file your koha test site can reach)
4 - On the command line:
    perl misc/bin/connexion_import_daemon.pl -c /etc/koha/sites/kohadev/connexion.cnf
5 - In another terminal:
    cat bug_33418.test  | nc -v localhost 8888
6 - It should report success and a biblionumber
7 - In Koha: Cataloguing->Manage staged MARC record
8 - View the most recent batch with file name (webservice)
9 - Confirm it was imported, no match
10 - Click 'View' under the 'Record' column
11 - Confirm record loads correctly and 'MARC framework' detail is 'Books, Booklets, Workbooks'
12 - On the terminal repeat the command:
    cat bug_33418.test  | nc -v localhost 8888
13 - It should succed
14 - View the new batch, confirm the record matched this time
15 - View the record details, confirm framework is now 'default'
16 - On the first terminal hit Ctrl+C to stop the daemon
16 - Edit connexion.cnf and change:
    import_mode:stage
    framework:ACQ
    overlay_framework:IR
17 - Restart daemon:
     perl misc/bin/connexion_import_daemon.pl -c /etc/koha/sites/kohadev/connexion.cnf
18 - Delete the record created above
19 - On the second terminal repeat the command:
    cat bug_33418.test  | nc -v localhost 8888
20 - Confirm the batch is created, but not imported
21 - In terminal:
     perl misc/cronjobs/import_webservice_batch.pl --framework=ACQ --overlay_framework=BKS
22 - Confirm batch imported, and record in ACQ framework
23 - In terminal:
     cat bug_33418.test  | nc -v localhost 8888
     perl misc/cronjobs/import_webservice_batch.pl --framework=ACQ --overlay_framework=BKS
24 - Confirm batch added, record matched, record imported, and record now in Books framework
25 - Stop the deamon, edit connexion.cnf:
     import_mode:direct
26 - Start the daemon, and on other terminal repeat:
     cat bug_33418.test  | nc -v localhost 8888
27 - Confirm record in Binders framework
28 - Set record framework to Books
29 - Stop daemon, edit cnf and remove 'overlay_framework' setting
30 - Start daemon and cat the file again
31 - Confirm the record remains in Books framework

Signed-off-by: Brendan Lawlor <blawlor@clamsnet.org>
Comment 6 Brendan Lawlor 2024-02-26 19:42:01 UTC
Thanks to your detailed instructions it was fun testing this bug.

Test notes:
Update system prefs to allow weak passwords
Set up kohauser with Cataloging permissions, thanks for link to bug
Follow step 18 to delete the record to avoid confusion\

Sometimes when trying to restart the daemon I would get an error like
perl misc/bin/connexion_import_daemon.pl -c /etc/koha/sites/kohadev/connexion.cnf
Couldn't be a tcp server on port 8888: Address already in use IO::Socket::INET: Address already in use at misc/bin/connexion_import_daemon.pl line 209.

but I found you can resolve this issue with sudo fuser 8888/tcp -k

When I first tried the last step it overlaid the record with  the default framework, but I realized that was because I did not remove the whole line overlay_framework. It took me a couple tries to realize I needed to remove the whole line, but I eventually got it.
Comment 7 Marcel de Rooy 2024-05-10 09:40:28 UTC
Created attachment 166518 [details] [review]
Bug 33418: Add overlay_framework option to connexion scripts

This patch adds the option to specify a framework to be used when overlaying records from webservices/connexion

To test:
1 - vim /etc/koha/sites/kohadev/connexion.cnf
2 - Set content:
    host:
    port: 8888
    koha:http://localhost:8081
    log:/var/log/koha/kohadev/connexion.log
    match:ISBN
    user:kohauser
    password:kohapass
    overlay_action:replace
    nomatch_action:create_new
    item_action:always_add
    import_mode:direct
    framework:BKS
    overlay_framework:
    debug:1
3 - Save the sample file from this bug into your kohaclone (or copy and paste into a file your koha test site can reach)
4 - On the command line:
    perl misc/bin/connexion_import_daemon.pl -c /etc/koha/sites/kohadev/connexion.cnf
5 - In another terminal:
    cat bug_33418.test  | nc -v localhost 8888
6 - It should report success and a biblionumber
7 - In Koha: Cataloguing->Manage staged MARC record
8 - View the most recent batch with file name (webservice)
9 - Confirm it was imported, no match
10 - Click 'View' under the 'Record' column
11 - Confirm record loads correctly and 'MARC framework' detail is 'Books, Booklets, Workbooks'
12 - On the terminal repeat the command:
    cat bug_33418.test  | nc -v localhost 8888
13 - It should succed
14 - View the new batch, confirm the record matched this time
15 - View the record details, confirm framework is now 'default'
16 - On the first terminal hit Ctrl+C to stop the daemon
16 - Edit connexion.cnf and change:
    import_mode:stage
    framework:ACQ
    overlay_framework:IR
17 - Restart daemon:
     perl misc/bin/connexion_import_daemon.pl -c /etc/koha/sites/kohadev/connexion.cnf
18 - Delete the record created above
19 - On the second terminal repeat the command:
    cat bug_33418.test  | nc -v localhost 8888
20 - Confirm the batch is created, but not imported
21 - In terminal:
     perl misc/cronjobs/import_webservice_batch.pl --framework=ACQ --overlay_framework=BKS
22 - Confirm batch imported, and record in ACQ framework
23 - In terminal:
     cat bug_33418.test  | nc -v localhost 8888
     perl misc/cronjobs/import_webservice_batch.pl --framework=ACQ --overlay_framework=BKS
24 - Confirm batch added, record matched, record imported, and record now in Books framework
25 - Stop the deamon, edit connexion.cnf:
     import_mode:direct
26 - Start the daemon, and on other terminal repeat:
     cat bug_33418.test  | nc -v localhost 8888
27 - Confirm record in Binders framework
28 - Set record framework to Books
29 - Stop daemon, edit cnf and remove 'overlay_framework' setting
30 - Start daemon and cat the file again
31 - Confirm the record remains in Books framework

Signed-off-by: Brendan Lawlor <blawlor@clamsnet.org>

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Comment 8 Marcel de Rooy 2024-05-10 09:41:07 UTC
Essentially only adding the overlay framework parameter. Not tested but patch looks good to me.
Comment 9 Katrin Fischer 2024-05-10 14:46:51 UTC
Pushed for 24.05!

Well done everyone, thank you!