Bug 13930 - Add supporting service svc/ncip for xnciptoolkit driver
Summary: Add supporting service svc/ncip for xnciptoolkit driver
Status: CLOSED WONTFIX
Alias: None
Product: Koha
Classification: Unclassified
Component: Web services (show other bugs)
Version: Main
Hardware: All All
: P3 enhancement (vote)
Assignee: Jiri Kozlovsky
QA Contact: Testopia
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2015-03-30 19:19 UTC by Josef Moravec
Modified: 2017-06-14 22:10 UTC (History)
9 users (show)

See Also:
Change sponsored?: Sponsored
Patch complexity: Small patch
Documentation contact:
Documentation submission:
Text to go in the release notes:
Version(s) released in:


Attachments
Bug 13930 - Created NCIP svc to effectively parse data from database to JSON in order to query those from xcncip2toolkit koha's connector (46.38 KB, patch)
2015-04-13 10:57 UTC, Jiri Kozlovsky
Details | Diff | Splinter Review
Bug 13931 - Fixed Licence & Copyright statements & refactored the code (24.97 KB, patch)
2015-04-17 12:56 UTC, Jiri Kozlovsky
Details | Diff | Splinter Review
Bug 13930 Followup: fixing 'undefined subroutine' errors (1.61 KB, patch)
2015-06-22 16:37 UTC, Mirko Tietgen
Details | Diff | Splinter Review
Bug 13930 Followup: fix subroutine call (981 bytes, patch)
2015-06-25 12:39 UTC, Mirko Tietgen
Details | Diff | Splinter Review
Bug 13930 - Created NCIP svc to effectively parse data from database to JSON in order to query those from xcncip2toolkit koha's connector (46.43 KB, patch)
2015-06-25 12:45 UTC, Mirko Tietgen
Details | Diff | Splinter Review
Bug 13931 - Fixed Licence & Copyright statements & refactored the code (25.02 KB, patch)
2015-06-25 12:45 UTC, Mirko Tietgen
Details | Diff | Splinter Review
Bug 13930 Followup: fixing 'undefined subroutine' errors (1.61 KB, patch)
2015-06-25 12:45 UTC, Mirko Tietgen
Details | Diff | Splinter Review
Bug 13930 Followup: fix subroutine call (981 bytes, patch)
2015-06-25 12:45 UTC, Mirko Tietgen
Details | Diff | Splinter Review
Bug 13930 - Created NCIP svc to effectively parse data from database to JSON in order to query those from xcncip2toolkit koha's connector (45.99 KB, patch)
2015-11-10 12:44 UTC, Josef Moravec
Details | Diff | Splinter Review
Bug 13931 - Fixed Licence & Copyright statements & refactored the code (25.02 KB, patch)
2015-11-10 12:44 UTC, Josef Moravec
Details | Diff | Splinter Review
Bug 13930 Followup: fixing 'undefined subroutine' errors (1.59 KB, patch)
2015-11-10 12:44 UTC, Josef Moravec
Details | Diff | Splinter Review
Bug 13930 Followup: fix subroutine call (975 bytes, patch)
2015-11-10 12:45 UTC, Josef Moravec
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description Josef Moravec 2015-03-30 19:19:54 UTC
We need ncip service for Koha, because it will the main communication protocol for libraries portal here in Czech Republic.

The work is almost done and is sponsored by Municipal Library Ceska Trebova and by grant from Ministry of Culture of Czech Republic.
Comment 1 Chris Cormack 2015-03-30 19:49:46 UTC
Oh, have you seen? 

http://git.evergreen-ils.org/?p=working/NCIPServer.git;a=summary

Looks like Koha will have 2 NCIP servers possible :)

It will be good to see the supporting services, I could change the NCIPServer to use them too. Then people could chose the pure perl one, or if they want a java stack, xnciptoolkit.
Comment 2 Josef Moravec 2015-03-30 20:10:42 UTC
Yes, 

I've seen it before, but we have to use xcncip2toolkit (it was somewhat consensus made by libraries and library system vendors here in Czech Republic)

You can see the work-in-progress in the repository of our developer Jiří Kozlovský: 
https://github.com/jirislav/Koha/

look at svc/ncip and C4/NCIP/* for the Koha service

and here is the xcncip2toolkit connector for Koha:
https://github.com/moravianlibrary/xcncip2toolkit/tree/master/connectors/koha/3.xx/trunk

We are open to any opinion and suggestions!

Thanks!
Comment 3 Jiri Kozlovsky 2015-04-10 14:36:31 UTC
Supported services are: 
LookupUser
LookupItem
LookupItemSet
LookupRequest
RequestItem
CancelRequestItem
RenewItem

This module was basically written in order to parse only relevant data from Koha, respecting good manners like doing C4::Reserves::CanBookBeReserved or CanItemBeReserved before the RequestItem service is done and so on..

Output format is JSON if there was success. Else proprietary status code is returned with a little text/plain response.

We are using xcncip2toolkit because it is being maintained by eXtensible Catalog as open source & any changes made to NCIP schema protocol are afterwards commited to git available to public (https://code.google.com/p/xcncip2toolkit/source/browse/) .. Then it is as easy to implement changes made to protocol as pulling changes from git :)

Also a good reason is that in xcncip2toolkit are all values parsed in proprietary standardized classes, easy to read & work with .. also it has built-in schema validator & in a way it is all prepared for a programmer to focus on the goal he needs to achieve, as is writing an connector for Koha ..
Comment 4 Jiri Kozlovsky 2015-04-10 16:14:46 UTC
To apply this patch, you have to create NCIP folder inside C4 with these modules:
https://github.com/jirislav/Koha/tree/master/C4/NCIP
then add two new subroutines into C4::Reserves (GetReserveCountFromItemnumber & GetReserveFromBorrowernumberAndItemnumber): 
https://github.com/jirislav/Koha/blob/master/C4/Reserves.pm
then finally create svc/ncip:
https://github.com/jirislav/Koha/blob/master/svc/ncip 


Test plan is as follows:
1. Apply the patch
2. Use json validator on all responses (except when status code isn't 200 OK) (e.g http://jsonlint.com/) to check the json is valid
3. Try this URLs with your own parameters, check the response is appropriate to your data :
    Get patron:
    http://intranet:8080/cgi-bin/koha/svc/ncip?service=lookup_user&userId=<your_patron_id> (returns basic user data)
    http://intranet:8080/cgi-bin/koha/svc/ncip?service=lookup_user&userId=<your_patron_id>&loanedItemsDesired&notUserInfo (returns only items issued by user)
    http://intranet:8080/cgi-bin/koha/svc/ncip?service=lookup_user&userId=<your_patron_id>&requestedItemsDesired&notUserInfo (returns only items requested by user)
    http://intranet:8080/cgi-bin/koha/svc/ncip?service=lookup_user&userId=<your_patron_id>&userFiscalAccountDesired&notUserInfo (returns only user's transactions - fees etc)
        - notUserInfo says to omit user's info & all parameters can be joined together as follows:
    http://intranet:8080/cgi-bin/koha/svc/ncip?service=lookup_user&userId=<your_patron_id>&loanedItemsDesired&requestedItemsDesired&userFiscalAccountDesired&notUserInfo

4. repeat 2.-3. for these URLs:
    Get an item:
    http://intranet:8080/cgi-bin/koha/svc/ncip?service=lookup_item&itemId=<item_number> (returns basic item data)
    http://intranet:8080/cgi-bin/koha/svc/ncip?service=lookup_item&itemId=<item_number>&holdQueueLengthDesired&notItemInfo (returns hold queue length - number of reserves placed on item)
    http://intranet:8080/cgi-bin/koha/svc/ncip?service=lookup_item&itemId=<item_number>&circulationStatusDesired&notItemInfo (returns NCIP formatted circulation status - can be 'Available On Shelf' etc.. all possible values are specified here: https://github.com/moravianlibrary/xcncip2toolkit/blob/master/core/trunk/service/src/main/java/org/extensiblecatalog/ncip/v2/service/Version1CirculationStatus.java)
        - notItemInfo again says to omit item's info & all params can be joined:
    http://intranet:8080/cgi-bin/koha/svc/ncip?service=lookup_item&itemId=<item_id>&circulationStatusDesired&holdQueueLengthDesired&notItemInfo

5. repeat 2.-3. for these URLs:
    Get biblio with items:
    http://intranet:8080/cgi-bin/koha/svc/ncip?service=lookup_item_set&bibId=<biblio_id> (returns bibliographic information & items belonging to this biblio)
    http://intranet:8080/cgi-bin/koha/svc/ncip?service=lookup_item_set&bibId=<biblio_id>&holdQueueLengthDesired&notBibInfo (returns items belonging to specified biblio & each item now has element 'holdQueueLength' with number of reserves placed on it)
    http://intranet:8080/cgi-bin/koha/svc/ncip?service=lookup_item_set&bibId=<biblio_id>&circulationStatusDesired&notBibInfo (returns items of biblio & each item has additional 'circulationStatus' defined before)
        - notBibInfo again says to omit biblio's info & all params can be joined:
    http://intranet:8080/cgi-bin/koha/svc/ncip?service=lookup_item_set&bibId=<biblio_id>&holdQueueLengthDesired&circulationStatusDesired&notBibInfo

6. repeat 2.-3. for these URLs:
    Place reserve as patron on item & write down it's 'reserve_id':
    http://intranet:8080/cgi-bin/koha/svc/ncip?service=request_item&userId=<your_patron_id>&itemId=<item_number>&pickupExpiryDate=<date_in_format:DD/MM/YYYY>&earliestDateNeeded=<date_in_format:DD/MM/YYYY>
        - please note that pickupExpiryDate & earliestDateNeeded are optional params .. but should be checked if were properly processed

    Place reserve as patron on biblio & write down it's 'reserve_id':
    http://intranet:8080/cgi-bin/koha/svc/ncip?service=request_item&userId=<your_patron_id>&bibId=<biblio_id>
        -  pickupExpiryDate & earliestDateNeeded could be specified here too

    Lookup those reserves using reserve ids you have written down:
    http://intranet:8080/cgi-bin/koha/svc/ncip?service=lookup_request&requestId=<reserve_id>

    Get reserve from patron & item number:
    http://intranet:8080/cgi-bin/koha/svc/ncip?service=lookup_request&userId=<your_patron_id>&itemId=<item_number>

    Cancel reserve by reserve_id & user_id:
    http://intranet:8080/cgi-bin/koha/svc/ncip?service=cancel_request_item&requestId=<request_number>&userId=<your_patron_id>

    Cancel reserve by item_id & user_id:
    http://intranet:8080/cgi-bin/koha/svc/ncip?service=cancel_request_item&itemId=<item_number>&userId=<your_patron_id>

    Now check both reserves were properly cancelled ..

7. repeat 2.-3. for these URLs:
    Renew patron's loan (first manually do a checkout to a user ..):
    http://intranet:8080/cgi-bin/koha/svc/ncip?service=renew_item&itemId=<item_number>&userId=<your_patron_id>&desiredDateDue=<date_in_format:DD/MM/YYYY>
        - desiredDateDue param is optional & if it is bigger than the date generated from koha's config, than is used the generated one

That should be all .. If you experience any problems don't hesitate to contact me at mail@jkozlovsky.cz
Comment 5 Magnus Enger 2015-04-13 07:14:13 UTC
Hi! This is really interesting, but could you please submit a patch as described here:

http://wiki.koha-community.org/wiki/Version_Control_Using_Git#Share_with_the_rest_of_the_world

The preferred way to submit the actual patch these days is to attach it to the bug here on Bugzilla, and many people use Git bz for that:
http://wiki.koha-community.org/wiki/Git_bz_configuration#Submitting_a_patch
Comment 6 Jiri Kozlovsky 2015-04-13 10:57:09 UTC Comment hidden (obsolete)
Comment 7 Chris Cormack 2015-04-15 00:55:28 UTC
(In reply to Jiří Kozlovský from comment #6)
> Created attachment 37719 [details] [review] [review]
> Bug 13930 - Created NCIP svc to effectively parse data from database to JSON
> in order to query those from xcncip2toolkit koha's connector

A quick fix needed, the copyright statements are wrong (looks like a cut and paste error?) Also Koha is GPLv3 or later not 2 anymore.

I'll try to do some functional testing, but if you could fix those, they won't get caught at QA :)
Comment 8 Jiri Kozlovsky 2015-04-17 12:56:40 UTC Comment hidden (obsolete)
Comment 9 Jiri Kozlovsky 2015-04-17 12:58:00 UTC
(In reply to Chris Cormack from comment #7)
> A quick fix needed, the copyright statements are wrong (looks like a cut and
> paste error?) Also Koha is GPLv3 or later not 2 anymore.
> 
> I'll try to do some functional testing, but if you could fix those, they
> won't get caught at QA :)

Thanks! Now I have licence & copyright repaired so there shouldn't be any issues with it anymore :) .. I also did a little refactoring
Comment 10 Mirko Tietgen 2015-06-22 16:37:10 UTC Comment hidden (obsolete)
Comment 11 Mirko Tietgen 2015-06-22 16:38:24 UTC
I tried the test plam up to (including) 5 and found some errors, added a patch.
Comment 12 Mirko Tietgen 2015-06-25 12:39:27 UTC Comment hidden (obsolete)
Comment 13 Mirko Tietgen 2015-06-25 12:45:03 UTC Comment hidden (obsolete)
Comment 14 Mirko Tietgen 2015-06-25 12:45:17 UTC Comment hidden (obsolete)
Comment 15 Mirko Tietgen 2015-06-25 12:45:23 UTC Comment hidden (obsolete)
Comment 16 Mirko Tietgen 2015-06-25 12:45:27 UTC Comment hidden (obsolete)
Comment 17 Mirko Tietgen 2015-06-25 12:47:01 UTC
I added another small fix. Apart from that, it works for me so far.
Comment 18 Jonathan Druart 2015-08-21 10:36:59 UTC
Be care, AddReserve prototype changed by bug 14526 (contrainst param removed).
Comment 19 Jonathan Druart 2015-08-21 10:40:48 UTC
I meant bug 9809.
Comment 20 Josef Moravec 2015-11-10 12:44:39 UTC
Created attachment 44695 [details] [review]
Bug 13930 - Created NCIP svc to effectively parse data from database to JSON in order to query those from xcncip2toolkit koha's connector

Signed-off-by: Mirko Tietgen <mirko@abunchofthings.net>
Comment 21 Josef Moravec 2015-11-10 12:44:47 UTC
Created attachment 44696 [details] [review]
Bug 13931 - Fixed Licence & Copyright statements & refactored the code

http://bugs.koha-community.org/show_bug.cgi?id=13930
Signed-off-by: Mirko Tietgen <mirko@abunchofthings.net>
Comment 22 Josef Moravec 2015-11-10 12:44:54 UTC
Created attachment 44697 [details] [review]
Bug 13930 Followup: fixing 'undefined subroutine' errors

Fixing the errors

Undefined subroutine &C4::Reserves::GetReserveCountFromItemnumber
called at <…>/koha/C4/NCIP/LookupItemSet.pm line 90.

Undefined subroutine &C4::Reserves::GetReservesFromBorrowernumber
called at <…>/koha/C4/NCIP/LookupUser.pm line 80

Undefined subroutine &C4::Items::GetItem
called at <…>/koha/C4/NCIP/LookupItem.pm line 77.
Comment 23 Josef Moravec 2015-11-10 12:45:00 UTC
Created attachment 44698 [details] [review]
Bug 13930 Followup: fix subroutine call

Fixing error 'Undefined subroutine &C4::NCIP::RequestItem::print409
called at /home/mirko/koha/C4/NCIP/RequestItem.pm line 86.'
Comment 24 Josef Moravec 2015-11-10 12:46:29 UTC
I've rebased the patches on top of current master
Comment 25 Kyle M Hall 2015-12-22 13:46:57 UTC
Is there any chance you could write up a quick and dirty test plan that includes using these scripts *with* xnciptoolkit. I would really like to be able to test the full end-to-end functionality of this!
Comment 26 Kyle M Hall 2015-12-23 13:57:20 UTC
FYI, I tried to install xncip2toolkit via your dockerfile ( https://github.com/moravianlibrary/xcncip2toolkit/tree/master/dockerfile ) and it failed:

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:2.0.2:compile (default-compile) on project koha-jar: Compilation failure
[ERROR] /home/xcncip2toolkit/xcncip2toolkit/connectors/koha/3.xx/trunk/jar/src/main/java/org/extensiblecatalog/ncip/v2/koha/KohaLookupItemService.java:[93,59] error: unreported exception ParseException; must be caught or declared to be thrown
[ERROR] -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn <goals> -rf :koha-jar
Comment 27 Jiri Kozlovsky 2015-12-24 21:51:19 UTC
(In reply to Kyle M Hall from comment #26)
> FYI, I tried to install xncip2toolkit via your dockerfile (
> https://github.com/moravianlibrary/xcncip2toolkit/tree/master/dockerfile )
> and it failed:
> 
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-compiler-plugin:2.0.2:compile
> (default-compile) on project koha-jar: Compilation failure
> [ERROR]
> /home/xcncip2toolkit/xcncip2toolkit/connectors/koha/3.xx/trunk/jar/src/main/
> java/org/extensiblecatalog/ncip/v2/koha/KohaLookupItemService.java:[93,59]
> error: unreported exception ParseException; must be caught or declared to be
> thrown
> [ERROR] -> [Help 1]
> [ERROR]
> [ERROR] To see the full stack trace of the errors, re-run Maven with the -e
> switch.
> [ERROR] Re-run Maven using the -X switch to enable full debug logging.
> [ERROR]
> [ERROR] For more information about the errors and possible solutions, please
> read the following articles:
> [ERROR] [Help 1]
> http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
> [ERROR]
> [ERROR] After correcting the problems, you can resume the build with the
> command
> [ERROR]   mvn <goals> -rf :koha-jar

Oh, it's been a while since I've updated the dockerfile .. the fact is I've moved the newest code to the official GitHub repository of eXtensibleCatalog, where I've just updated the docker build with installation instruction here: https://github.com/eXtensibleCatalog/xcncip2toolkit/tree/master/dockerfile

I've tested the docker build now & it was built successfully - I've basically added an option to specify a port in dockerfile/connectors/koha/configure.sh as the Intranet could run on any other port than just 8080 .. 

So, to install the xcncip2toolkit for your Koha, just run ./install-koha.sh after you have install the Docker, which is available to any OS.

After the installation is done, you should be able of running the ./start.sh bash script which will start the tomcat servlet within the docker with an web archive of the Koha connector on port 8080 - the exposed port could be modified in start.sh on line 4 is:
docker run -d -p 8080:8080 xcncip2toolkit/xcncip2toolkit

So if you would like to have it exposed to port 14014, you would write this:
docker run -d -p 8080:14014 xcncip2toolkit/xcncip2toolkit 

Then just visit http://localhost:14014/koha-web & try to send LookupItem, or LookupAgency (that one service does not communicate with Koha at all), LookupItemSet, CanBeRequested etc .. the only NCIP message being now in development state & not working yet is only "Lookup User Loaned History", which will now always fail as it is not implemented yet.
Comment 28 Jiri Kozlovsky 2016-06-29 07:42:42 UTC
xcncip2toolkit will be using new REST API, see bug #16652, thus this won't be needed soon.

you can watch the progress at https://github.com/eXtensibleCatalog/xcncip2toolkit/tree/master/connectors/koha/3.xx/trunk/jar/src/main/java/org/extensiblecatalog/ncip/v2/koha