Since the changes mentioned at https://wiki.koha-community.org/wiki/Koha_/svc/_HTTP_API#Changes_coming_in_Koha_24.05 in version 24.05, the `connexion_import_daemon.pl` no longer works. Related discussions: - https://chat.koha-community.org/koha-community/pl/o8yasuqrwpf8upynroed5fqz1o - https://chat.koha-community.org/koha-community/pl/kyszm1fmr3dp5gd75demen3fpe I'll try to fix it, but I'm very inexperienced with perl, although I know lots of other languages.
Created attachment 169926 [details] WorldShare User Preferences for TCP/IP export If we were to write automated tests for this, we need to do it without an OCLC WorldShare subscription. I could not find docs online showing exactly how their "Send to local system via TCP/IP" protocol works, but I can reverse engineer the spec. It seems to be raw text send over TCP (no HTTP protocol). What they send depends on exactly how the options in this screenshot are set up.
When not using authentication (a dumb idea, because then anyone can send stuff into your Koha instance), it's simply the raw MARC record being sent: 00595cam a2200205Ma 4500001001300000003000600013005001700019008004100036040002500077035002100102049000900123110003700132245003700169260003400206300005400240336002600294337002800320338002700348994001400375ocn863323135OCoLC20151201051249.5131120s1986 enka 000 0 eng d aEQObengcEQOdOCLCQ a(OCoLC)863323135 aRSCM1 aMicroprocessors and peripherals.10aMicroprocessors and peripherals. aLondon :bMullard Ltd,c1986. av, c700 pages in various pagings :billustrations atextbtxt2rdacontent aunmediatedbn2rdamedia avolumebnc2rdacarrier aC0bZARSC When choosing the "Login ID" Authentication option, the MARC record is prefixed with something like A10my_user_idP11my_password where 10 and 11 seem to be the number of characters to consume for the username (A) and password (P) respectively: A10my_user_idP11my_password00595cam a2200205Ma 4500001001300000003000600013005001700019008004100036040002500077035002100102049000900123110003700132245003700169260003400206300005400240336002600294337002800320338002700348994001400375ocn863323135OCoLC20151201051249.5131120s1986 enka 000 0 eng d aEQObengcEQOdOCLCQ a(OCoLC)863323135 aRSCM1 aMicroprocessors and peripherals.10aMicroprocessors and peripherals. aLondon :bMullard Ltd,c1986. av, c700 pages in various pagings :billustrations atextbtxt2rdacontent aunmediatedbn2rdamedia avolumebnc2rdacarrier aC0bZARSC When choosing the "User ID" Authentication option, the MARC record is prefixed with something like @10my_user_id where 10 seems to be the number of characters to consume for the username, this time designated by @ instead of A: @10my_user_id00595cam a2200205Ma 4500001001300000003000600013005001700019008004100036040002500077035002100102049000900123110003700132245003700169260003400206300005400240336002600294337002800320338002700348994001400375ocn863323135OCoLC20151201051249.5131120s1986 enka 000 0 eng d aEQObengcEQOdOCLCQ a(OCoLC)863323135 aRSCM1 aMicroprocessors and peripherals.10aMicroprocessors and peripherals. aLondon :bMullard Ltd,c1986. av, c700 pages in various pagings :billustrations atextbtxt2rdacontent aunmediatedbn2rdamedia avolumebnc2rdacarrier aC0bZARSC It would be great to link to some official docs for this, but I could not find any.
Created attachment 169958 [details] [review] Use CSRF tokens in connexion_import_daemon.pl I wanted to write unit tests for this, but I need a few more days to get my perl skills up to date and read more developer wiki pages on how to do that... So for now I wrote a manual test plan.
Apologies for the delay on this one. I am getting here! (Although not today [U+1F605])
*** Bug 37318 has been marked as a duplicate of this bug. ***
(In reply to David Cook from comment #4) > Apologies for the delay on this one. I am getting here! (Although not today > [U+1F605]) See bug 33418 for test plan on setting up and testing connexion - just ignore all the bits about frameworks and basically check that things work (In reply to Rudolf Byker from comment #3) > Created attachment 169958 [details] [review] [review] > Use CSRF tokens in connexion_import_daemon.pl > > I wanted to write unit tests for this, but I need a few more days to get my > perl skills up to date and read more developer wiki pages on how to do > that... So for now I wrote a manual test plan. We don't generally require unit tests for things outside of modules, it would be lovely to have some here - they may take a little more infrastructure to make happen though, so I would file that as another bug
(In reply to Nick Clemens (kidclamp) from comment #6) > (In reply to David Cook from comment #4) > > Apologies for the delay on this one. I am getting here! (Although not today > > [U+1F605]) > > See bug 33418 for test plan on setting up and testing connexion - just > ignore all the bits about frameworks and basically check that things work > > (In reply to Rudolf Byker from comment #3) > > Created attachment 169958 [details] [review] [review] [review] > > Use CSRF tokens in connexion_import_daemon.pl > > > > I wanted to write unit tests for this, but I need a few more days to get my > > perl skills up to date and read more developer wiki pages on how to do > > that... So for now I wrote a manual test plan. > > We don't generally require unit tests for things outside of modules, it > would be lovely to have some here - they may take a little more > infrastructure to make happen though, so I would file that as another bug +1
Created attachment 170177 [details] Test records for importing
Created attachment 170178 [details] [review] Bug 37543: (follow-up) Tidy Tidy the whole thing Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Created attachment 170179 [details] [review] Bug 37543: Use CSRF tokens in connexion_import_daemon.pl Since version 24.05, due to the changes mentioned at https://wiki.koha-community.org/wiki/Koha_/svc/_HTTP_API#Changes_coming_in_Koha_24.05 , the `connexion_import_daemon.pl` stopped working. The reason for this is that it did not use CSRF tokens. To test: 1. Get a Koha instance on 24.05+, before applying the patch. 2. Create a plain text file somewhere on the server containing a raw MARC record (not XML). Let's call it `marc.txt`. 3. On the server, create a config file like this: ``` host: 0.0.0.0 port: 5500 koha: http://localhost:82 # Where 82 is the port of the Koha staff interface. user: foo # A Koha staff user. password: Fooo1234 # The Koha staff user's password. import_mode: stage ``` 4. Run `./connexion_import_daemon.pl --config the-config-file-path` 5. In another terminal on the same server (or from anywhere that can reach the port opened by the `connexion_import_daemon.pl` script, run `nc localhost 5500 < marc.txt` 6. Observe in the stderr of the daemon script: `Response: Unsuccessful request` 7. Stop the daemon script. 8. Apply the patch and repeat steps 4 and 5. 9. Observe in the stderr of the daemon script: `Response: Success. Batch number ... - biblio record number HASH(...) added to Koha` 10. Check at /cgi-bin/koha/tools/manage-marc-import.pl for a batch named `(webservice)`. It should contain one record now. This is how we know that authentication between the daemon and Koha worked, which is what this patch tries to address. Thanks-to: David Cook <dcook@prosentient.com.au> Sponsored-by: Reformational Study Centre <www.refstudycentre.com> Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Created attachment 170180 [details] [review] Bug 37543: (follow-up) Tidy Tidy the whole thing Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
The test plan is missing that these three fields need values set: - overlay_action - nomatch_action - item_action Otherwise you'll get a: C4::ImportBatch::AddImportBatch(): DBI Exception: DBD::mysql::db do failed: Data truncated for column '<one of the above>' at row 1 at /kohadevbox/koha/C4/ImportBatch.pm line 173
Created attachment 170339 [details] [review] Bug 37543: Use CSRF tokens in connexion_import_daemon.pl Since version 24.05, due to the changes mentioned at https://wiki.koha-community.org/wiki/Koha_/svc/_HTTP_API#Changes_coming_in_Koha_24.05 , the `connexion_import_daemon.pl` stopped working. The reason for this is that it did not use CSRF tokens. To test: 1. Get a Koha instance on 24.05, before applying the patch. 2. Create a plain text file somewhere on the server containing a raw MARC record (not XML). Let's call it `marc.txt`. 3. On the server, create a config file like this: ``` host: 0.0.0.0 port: 5500 koha: http://localhost:82 # Where 82 is the port of the Koha staff interface. user: foo # A Koha staff user. password: Fooo1234 # The Koha staff user's password. import_mode: stage ``` 4. Run `./connexion_import_daemon.pl --config the-config-file-path` 5. In another terminal on the same server (or from anywhere that can reach the port opened by the `connexion_import_daemon.pl` script, run `nc localhost 5500 < marc.txt` 6. Observe in the stderr of the daemon script: `Response: Unsuccessful request` 7. Stop the daemon script. 8. Apply the patch and repeat steps 4 and 5. 9. Observe in the stderr of the daemon script: `Response: Success. Batch number ... - biblio record number HASH(...) added to Koha` 10. Check at /cgi-bin/koha/tools/manage-marc-import.pl for a batch named `(webservice)`. It should contain one record now. This is how we know that authentication between the daemon and Koha worked, which is what this patch tries to address. Thanks-to: David Cook <dcook@prosentient.com.au> Sponsored-by: Reformational Study Centre <www.refstudycentre.com> Signed-off-by: Nick Clemens <nick@bywatersolutions.com> Signed-off-by: Paul Derscheid <paul.derscheid@lmscloud.de>
Created attachment 170340 [details] [review] Bug 37543: (follow-up) Tidy Tidy the whole thing Signed-off-by: Nick Clemens <nick@bywatersolutions.com> Signed-off-by: Paul Derscheid <paul.derscheid@lmscloud.de>
I added a new bug for the three fields I mentioned in comment #12. I think these should have default values (ignore) and warn about them being set automatically. Check out bug 37648.
> The test plan is missing that these three fields need values set Strange... I am 99% certain that I tested without setting those three fields.
I doesn't impact that the fix works, therefore I passed QA but please take another look.
Will do.
Congrats on your first QA, Paul :)(In reply to Rudolf Byker from comment #18) > Will do. Please open a new bug if necessary. Congrats on your first QA, Paul!
Pushed for 24.11! Well done everyone, thank you!
Thank to you all for fixing this. Any chance it will be backported to 24.05? We use this feature in Koha regularly, and our lone cataloger is much bereaved over the loss of it.
Backported to 24.05.x for upcoming 24.05.04
Not for 23.11.x
Thanks, Lucas, for the backport and all for the work to fix this.
I *think* I may have this fix (on 24.05.03) and for one record that I sent from OCLC Connexion Client, I got, "Success. Batch number 8989 - biblio record number HASH(0x5571396a8fd8) added to Koha." This wasn't helpful at all--there was no record number mentioned, let alone a URL of the record so that I could look at it quickly. I had to go to Manage Staged Records to look at the record, and it had imported that single OCLC record *twice,* i.e. created two records from my one record--I had one record with the items on it, and a dup bib (with no items, of course, because they would have been dup barcodes in the items and my Koha won't allow it). I know this is all "fixed," so is it a separate bug to stop it from giving a message like, "Success. Batch number 8989 - biblio record number HASH(0x5571396a8fd8) added to Koha" when exporting *ONE* record, and have it go back to only creating ONE Koha record per ONE bib from OCLC, with a URL with the record number in it like it used to? Or should that be a new bug? Thank you!!!
Hi Heather, it probably only needs a small fix to get you back the right success message, but yes, as this has already been released: please file a new bug and use the depends on field to link to this one.
Okey-dokey! Will do! Thank you, Katrin!!! --h2
Bug 38230 filed to fix the "success" message!