Bug 13937 - Add an Elasticsearch-compatible Z39.50/SRU daemon that can inject item status MARC subfields
Summary: Add an Elasticsearch-compatible Z39.50/SRU daemon that can inject item status...
Status: Signed Off
Alias: None
Product: Koha
Classification: Unclassified
Component: Architecture, internals, and plumbing (show other bugs)
Version: master
Hardware: All All
: P5 - low new feature (vote)
Assignee: Ere Maijala
QA Contact: Testopia
URL:
Keywords: dependency, release-notes-needed
: 21210 (view as bug list)
Depends on:
Blocks: 21210
  Show dependency treegraph
 
Reported: 2015-04-01 12:46 UTC by Kyle M Hall
Modified: 2019-07-23 12:31 UTC (History)
24 users (show)

See Also:
Change sponsored?: Sponsored
Patch complexity: Medium patch
Who signed the patch off:
Text to go in the release notes:
Version(s) released in:


Attachments
Bug 13937 - Add a Z39.50 daemon that can inject item status MARC subfields (18.51 KB, patch)
2016-01-06 22:12 UTC, Jesse Weaver
Details | Diff | Splinter Review
Bug 13937 - Add a Z39.50 daemon that can inject item status MARC subfields (19.48 KB, patch)
2016-01-28 22:12 UTC, Jesse Weaver
Details | Diff | Splinter Review
Bug 13937 - Add a Z39.50 daemon that can inject item status MARC subfields (19.61 KB, patch)
2016-01-28 22:15 UTC, Jesse Weaver
Details | Diff | Splinter Review
Bug 13937 - Add a Z39.50 daemon that can inject item status MARC subfields (19.69 KB, patch)
2016-01-28 22:34 UTC, Mirko Tietgen
Details | Diff | Splinter Review
Bug 13937 - Add a Z39.50 daemon that can inject item status MARC subfields (22.14 KB, patch)
2016-07-19 21:36 UTC, Jesse Weaver
Details | Diff | Splinter Review
Bug 13937 (Followup) - Remove reference to sub _prefetch_records (981 bytes, patch)
2016-12-12 14:22 UTC, Nick Clemens
Details | Diff | Splinter Review
Bug 13937 - Add a Z39.50 daemon that can inject item status MARC subfields (22.16 KB, patch)
2016-12-12 14:23 UTC, Nick Clemens
Details | Diff | Splinter Review
Bug 13937 (Followup) - Remove reference to sub _prefetch_records (981 bytes, patch)
2016-12-12 14:23 UTC, Nick Clemens
Details | Diff | Splinter Review
Bug 13937 - Add a Z39.50 daemon that can inject item status MARC subfields (22.24 KB, patch)
2017-01-23 17:13 UTC, Nick Clemens
Details | Diff | Splinter Review
Bug 13937 (Followup) - Remove reference to sub _prefetch_records (981 bytes, patch)
2017-01-23 17:13 UTC, Nick Clemens
Details | Diff | Splinter Review
Bug 13937 - Add a Z39.50 daemon that can inject item status MARC subfields (22.26 KB, patch)
2017-02-28 19:36 UTC, Nick Clemens
Details | Diff | Splinter Review
Bug 13937 (Followup) - Remove reference to sub _prefetch_records (981 bytes, patch)
2017-02-28 19:36 UTC, Nick Clemens
Details | Diff | Splinter Review
Bug 13937 - Add a Z39.50 daemon that can inject item status MARC subfields (22.31 KB, patch)
2017-02-28 20:01 UTC, Barton Chittenden
Details | Diff | Splinter Review
Bug 13937 (Followup) - Remove reference to sub _prefetch_records (1.01 KB, patch)
2017-02-28 20:02 UTC, Barton Chittenden
Details | Diff | Splinter Review
Bug 13937 - Add a Z39.50 daemon that can inject item status MARC subfields (22.31 KB, patch)
2017-09-07 15:53 UTC, Nick Clemens
Details | Diff | Splinter Review
Bug 13937 (Followup) - Remove reference to sub _prefetch_records (1.01 KB, patch)
2017-09-07 15:53 UTC, Nick Clemens
Details | Diff | Splinter Review
Bug 13937 - (Followup) Correct error call to use self (749 bytes, patch)
2017-09-07 15:53 UTC, Nick Clemens
Details | Diff | Splinter Review
Bug 13937 - Unit tests (7.19 KB, patch)
2017-09-07 15:53 UTC, Nick Clemens
Details | Diff | Splinter Review
Bug 13937: Followup - pass through yaz switches and don't ignore case (1.60 KB, patch)
2017-11-14 12:30 UTC, Nick Clemens
Details | Diff | Splinter Review
Bug 13937: Add a Z39.50 daemon that can inject item status MARC subfields (22.30 KB, patch)
2018-11-21 14:26 UTC, Ere Maijala
Details | Diff | Splinter Review
Bug 13937: (follow-up) Remove reference to sub _prefetch_records (1.01 KB, patch)
2018-11-21 14:26 UTC, Ere Maijala
Details | Diff | Splinter Review
Bug 13937: (follow-up) Correct error call to use self (749 bytes, patch)
2018-11-21 14:26 UTC, Ere Maijala
Details | Diff | Splinter Review
Bug 13937: Add unit tests (7.19 KB, patch)
2018-11-21 14:26 UTC, Ere Maijala
Details | Diff | Splinter Review
Bug 13937: (follow-up) Pass through yaz switches and don't ignore case (1.60 KB, patch)
2018-11-21 14:26 UTC, Ere Maijala
Details | Diff | Splinter Review
Bug 13937: Fix issues found in QA (9.17 KB, patch)
2018-11-21 14:26 UTC, Ere Maijala
Details | Diff | Splinter Review
Bug 13937: Add a Z39.50 daemon that can inject item status MARC subfields (22.30 KB, patch)
2018-11-23 07:30 UTC, Ere Maijala
Details | Diff | Splinter Review
Bug 13937: (follow-up) Remove reference to sub _prefetch_records (1.01 KB, patch)
2018-11-23 07:30 UTC, Ere Maijala
Details | Diff | Splinter Review
Bug 13937: (follow-up) Correct error call to use self (749 bytes, patch)
2018-11-23 07:30 UTC, Ere Maijala
Details | Diff | Splinter Review
Bug 13937: Add unit tests (7.19 KB, patch)
2018-11-23 07:30 UTC, Ere Maijala
Details | Diff | Splinter Review
Bug 13937: (follow-up) Pass through yaz switches and don't ignore case (1.60 KB, patch)
2018-11-23 07:30 UTC, Ere Maijala
Details | Diff | Splinter Review
Bug 13937: Fix issues found in QA (9.17 KB, patch)
2018-11-23 07:30 UTC, Ere Maijala
Details | Diff | Splinter Review
Bug 13937: Add support for SRU and Elasticsearch (34.48 KB, patch)
2018-11-23 07:30 UTC, Ere Maijala
Details | Diff | Splinter Review
Bug 13937: Add a Z39.50 daemon that can inject item status MARC subfields (22.30 KB, patch)
2018-11-27 11:45 UTC, Ere Maijala
Details | Diff | Splinter Review
Bug 13937: (follow-up) Remove reference to sub _prefetch_records (1.01 KB, patch)
2018-11-27 11:45 UTC, Ere Maijala
Details | Diff | Splinter Review
Bug 13937: (follow-up) Correct error call to use self (749 bytes, patch)
2018-11-27 11:45 UTC, Ere Maijala
Details | Diff | Splinter Review
Bug 13937: Add unit tests (7.19 KB, patch)
2018-11-27 11:45 UTC, Ere Maijala
Details | Diff | Splinter Review
Bug 13937: (follow-up) Pass through yaz switches and don't ignore case (1.60 KB, patch)
2018-11-27 11:45 UTC, Ere Maijala
Details | Diff | Splinter Review
Bug 13937: Fix issues found in QA (9.17 KB, patch)
2018-11-27 11:45 UTC, Ere Maijala
Details | Diff | Splinter Review
Bug 13937: Add support for SRU and Elasticsearch (43.60 KB, patch)
2018-11-27 11:45 UTC, Ere Maijala
Details | Diff | Splinter Review
Bug 13937: Add tests for search and retrieval (9.11 KB, patch)
2018-11-27 11:45 UTC, Ere Maijala
Details | Diff | Splinter Review
Bug 13937: Add a Z39.50 daemon that can inject item status MARC subfields (22.32 KB, patch)
2019-01-11 12:59 UTC, Ere Maijala
Details | Diff | Splinter Review
Bug 13937: Add a Z39.50 daemon that can inject item status MARC subfields (22.32 KB, patch)
2019-01-11 12:59 UTC, Ere Maijala
Details | Diff | Splinter Review
Bug 13937: (follow-up) Remove reference to sub _prefetch_records (1.02 KB, patch)
2019-01-11 13:00 UTC, Ere Maijala
Details | Diff | Splinter Review
Bug 13937: (follow-up) Correct error call to use self (756 bytes, patch)
2019-01-11 13:00 UTC, Ere Maijala
Details | Diff | Splinter Review
Bug 13937: Add unit tests (7.19 KB, patch)
2019-01-11 13:00 UTC, Ere Maijala
Details | Diff | Splinter Review
Bug 13937: (follow-up) Pass through yaz switches and don't ignore case (1.61 KB, patch)
2019-01-11 13:00 UTC, Ere Maijala
Details | Diff | Splinter Review
Bug 13937: Fix issues found in QA (9.19 KB, patch)
2019-01-11 13:00 UTC, Ere Maijala
Details | Diff | Splinter Review
Bug 13937: Add support for SRU and Elasticsearch (43.61 KB, patch)
2019-01-11 13:00 UTC, Ere Maijala
Details | Diff | Splinter Review
Bug 13937: Add tests for search and retrieval (9.10 KB, patch)
2019-01-11 13:00 UTC, Ere Maijala
Details | Diff | Splinter Review
Bug 13937: Add support for Z39.50 responder to the Debian scripts (18.30 KB, patch)
2019-01-11 13:00 UTC, Ere Maijala
Details | Diff | Splinter Review
Bug 13937: Add a Z39.50 daemon that can inject item status MARC subfields (22.32 KB, patch)
2019-01-15 11:55 UTC, Ere Maijala
Details | Diff | Splinter Review
Bug 13937: (follow-up) Remove reference to sub _prefetch_records (1.02 KB, patch)
2019-01-15 11:56 UTC, Ere Maijala
Details | Diff | Splinter Review
Bug 13937: (follow-up) Correct error call to use self (756 bytes, patch)
2019-01-15 11:56 UTC, Ere Maijala
Details | Diff | Splinter Review
Bug 13937: Add unit tests (7.19 KB, patch)
2019-01-15 11:56 UTC, Ere Maijala
Details | Diff | Splinter Review
Bug 13937: (follow-up) Pass through yaz switches and don't ignore case (1.61 KB, patch)
2019-01-15 11:56 UTC, Ere Maijala
Details | Diff | Splinter Review
Bug 13937: Fix issues found in QA (9.19 KB, patch)
2019-01-15 11:56 UTC, Ere Maijala
Details | Diff | Splinter Review
Bug 13937: Add support for SRU and Elasticsearch (43.63 KB, patch)
2019-01-15 11:56 UTC, Ere Maijala
Details | Diff | Splinter Review
Bug 13937: Add tests for search and retrieval (9.11 KB, patch)
2019-01-15 11:56 UTC, Ere Maijala
Details | Diff | Splinter Review
Bug 13937: Fix RPN conversion (3.20 KB, patch)
2019-01-15 11:56 UTC, Ere Maijala
Details | Diff | Splinter Review
Bug 13937: Add support for Z39.50 responder to the Debian scripts (18.30 KB, patch)
2019-01-16 12:05 UTC, Ere Maijala
Details | Diff | Splinter Review
Bug 13937: Add a Z39.50 daemon that can inject item status MARC subfields (22.32 KB, patch)
2019-01-16 12:08 UTC, Ere Maijala
Details | Diff | Splinter Review
Bug 13937: (follow-up) Remove reference to sub _prefetch_records (1.02 KB, patch)
2019-01-16 12:08 UTC, Ere Maijala
Details | Diff | Splinter Review
Bug 13937: (follow-up) Correct error call to use self (756 bytes, patch)
2019-01-16 12:08 UTC, Ere Maijala
Details | Diff | Splinter Review
Bug 13937: Add unit tests (7.19 KB, patch)
2019-01-16 12:09 UTC, Ere Maijala
Details | Diff | Splinter Review
Bug 13937: (follow-up) Pass through yaz switches and don't ignore case (1.61 KB, patch)
2019-01-16 12:09 UTC, Ere Maijala
Details | Diff | Splinter Review
Bug 13937: Fix issues found in QA (9.19 KB, patch)
2019-01-16 12:10 UTC, Ere Maijala
Details | Diff | Splinter Review
Bug 13937: Add support for SRU and Elasticsearch (43.61 KB, patch)
2019-01-16 12:11 UTC, Ere Maijala
Details | Diff | Splinter Review
Bug 13937: Add tests for search and retrieval (9.10 KB, patch)
2019-01-16 12:11 UTC, Ere Maijala
Details | Diff | Splinter Review
Bug 13937: Add support for Z39.50 responder to the Debian scripts (18.30 KB, patch)
2019-01-16 12:12 UTC, Ere Maijala
Details | Diff | Splinter Review
Bug 13937: Add a Z39.50 daemon that can inject item status MARC subfields (22.32 KB, patch)
2019-01-16 13:21 UTC, Ere Maijala
Details | Diff | Splinter Review
Bug 13937: (follow-up) Remove reference to sub _prefetch_records (1.02 KB, patch)
2019-01-16 13:22 UTC, Ere Maijala
Details | Diff | Splinter Review
Bug 13937: (follow-up) Correct error call to use self (756 bytes, patch)
2019-01-16 13:22 UTC, Ere Maijala
Details | Diff | Splinter Review
Bug 13937: Add unit tests (7.19 KB, patch)
2019-01-16 13:22 UTC, Ere Maijala
Details | Diff | Splinter Review
Bug 13937: (follow-up) Pass through yaz switches and don't ignore case (1.61 KB, patch)
2019-01-16 13:23 UTC, Ere Maijala
Details | Diff | Splinter Review
Bug 13937: Fix issues found in QA (9.19 KB, patch)
2019-01-16 13:24 UTC, Ere Maijala
Details | Diff | Splinter Review
Bug 13937: Add support for SRU and Elasticsearch (43.63 KB, patch)
2019-01-16 13:24 UTC, Ere Maijala
Details | Diff | Splinter Review
Bug 13937: Add tests for search and retrieval (9.11 KB, patch)
2019-01-16 13:25 UTC, Ere Maijala
Details | Diff | Splinter Review
Bug 13937: Fix RPN conversion (3.75 KB, patch)
2019-01-16 13:25 UTC, Ere Maijala
Details | Diff | Splinter Review
Bug 13937: Add support for Z39.50 responder to the Debian scripts (18.30 KB, patch)
2019-01-16 13:26 UTC, Ere Maijala
Details | Diff | Splinter Review
11 - Bug 13937: Add a Z39.50 daemon that can inject item status MARC subfields. Fix log path. (1.03 KB, patch)
2019-02-14 13:17 UTC, Olli-Antti Kivilahti
Details | Diff | Splinter Review
Bug 13937: Add a Z39.50 daemon that can inject item status MARC subfields (22.32 KB, patch)
2019-03-08 10:10 UTC, Ere Maijala
Details | Diff | Splinter Review
Bug 13937: (follow-up) Remove reference to sub _prefetch_records (1.02 KB, patch)
2019-03-08 10:10 UTC, Ere Maijala
Details | Diff | Splinter Review
Bug 13937: (follow-up) Correct error call to use self (756 bytes, patch)
2019-03-08 10:10 UTC, Ere Maijala
Details | Diff | Splinter Review
Bug 13937: Add unit tests (7.19 KB, patch)
2019-03-08 10:11 UTC, Ere Maijala
Details | Diff | Splinter Review
Bug 13937: (follow-up) Pass through yaz switches and don't ignore case (1.61 KB, patch)
2019-03-08 10:11 UTC, Ere Maijala
Details | Diff | Splinter Review
Bug 13937: Fix issues found in QA (6.05 KB, patch)
2019-03-08 10:11 UTC, Ere Maijala
Details | Diff | Splinter Review
Bug 13937: Add support for SRU and Elasticsearch (40.46 KB, patch)
2019-03-08 10:11 UTC, Ere Maijala
Details | Diff | Splinter Review
Bug 13937: Add tests for search and retrieval (9.10 KB, patch)
2019-03-08 10:11 UTC, Ere Maijala
Details | Diff | Splinter Review
Bug 13937: Fix RPN conversion (3.74 KB, patch)
2019-03-08 10:11 UTC, Ere Maijala
Details | Diff | Splinter Review
Bug 13937: Add support for Z39.50 responder to the Debian scripts (18.30 KB, patch)
2019-03-08 10:11 UTC, Ere Maijala
Details | Diff | Splinter Review
Bug 13937: Fix the daemon log path. (1.00 KB, patch)
2019-03-08 10:11 UTC, Ere Maijala
Details | Diff | Splinter Review
Bug 13937: Switch from GetItem() to Koha::Items->find() (1.65 KB, patch)
2019-03-08 10:11 UTC, Ere Maijala
Details | Diff | Splinter Review
Bug 13937: Add a Z39.50 daemon that can inject item status MARC subfields (22.38 KB, patch)
2019-04-26 12:29 UTC, Stefan Berndtsson
Details | Diff | Splinter Review
Bug 13937: (follow-up) Remove reference to sub _prefetch_records (1.08 KB, patch)
2019-04-26 12:29 UTC, Stefan Berndtsson
Details | Diff | Splinter Review
Bug 13937: (follow-up) Correct error call to use self (818 bytes, patch)
2019-04-26 12:29 UTC, Stefan Berndtsson
Details | Diff | Splinter Review
Bug 13937: Add unit tests (7.25 KB, patch)
2019-04-26 12:29 UTC, Stefan Berndtsson
Details | Diff | Splinter Review
Bug 13937: (follow-up) Pass through yaz switches and don't ignore case (1.67 KB, patch)
2019-04-26 12:29 UTC, Stefan Berndtsson
Details | Diff | Splinter Review
Bug 13937: Fix issues found in QA (6.11 KB, patch)
2019-04-26 12:30 UTC, Stefan Berndtsson
Details | Diff | Splinter Review
Bug 13937: Add support for SRU and Elasticsearch (40.52 KB, patch)
2019-04-26 12:30 UTC, Stefan Berndtsson
Details | Diff | Splinter Review
Bug 13937: Add tests for search and retrieval (9.16 KB, patch)
2019-04-26 12:30 UTC, Stefan Berndtsson
Details | Diff | Splinter Review
Bug 13937: Fix RPN conversion (3.80 KB, patch)
2019-04-26 12:30 UTC, Stefan Berndtsson
Details | Diff | Splinter Review
Bug 13937: Add support for Z39.50 responder to the Debian scripts (18.36 KB, patch)
2019-04-26 12:30 UTC, Stefan Berndtsson
Details | Diff | Splinter Review
Bug 13937: Fix the daemon log path. (1.07 KB, patch)
2019-04-26 12:30 UTC, Stefan Berndtsson
Details | Diff | Splinter Review
Bug 13937: Switch from GetItem() to Koha::Items->find() (1.71 KB, patch)
2019-04-26 12:34 UTC, Stefan Berndtsson
Details | Diff | Splinter Review
Bug 13937: Add a Z39.50 daemon that can inject item status MARC subfields (22.38 KB, patch)
2019-07-23 12:29 UTC, Ere Maijala
Details | Diff | Splinter Review
Bug 13937: (follow-up) Remove reference to sub _prefetch_records (1.08 KB, patch)
2019-07-23 12:29 UTC, Ere Maijala
Details | Diff | Splinter Review
Bug 13937: (follow-up) Correct error call to use self (818 bytes, patch)
2019-07-23 12:29 UTC, Ere Maijala
Details | Diff | Splinter Review
Bug 13937: Add unit tests (7.25 KB, patch)
2019-07-23 12:29 UTC, Ere Maijala
Details | Diff | Splinter Review
Bug 13937: (follow-up) Pass through yaz switches and don't ignore case (1.67 KB, patch)
2019-07-23 12:29 UTC, Ere Maijala
Details | Diff | Splinter Review
Bug 13937: Fix issues found in QA (6.11 KB, patch)
2019-07-23 12:30 UTC, Ere Maijala
Details | Diff | Splinter Review
Bug 13937: Add support for SRU and Elasticsearch (40.52 KB, patch)
2019-07-23 12:30 UTC, Ere Maijala
Details | Diff | Splinter Review
Bug 13937: Add tests for search and retrieval (9.16 KB, patch)
2019-07-23 12:30 UTC, Ere Maijala
Details | Diff | Splinter Review
Bug 13937: Fix RPN conversion (3.80 KB, patch)
2019-07-23 12:30 UTC, Ere Maijala
Details | Diff | Splinter Review
Bug 13937: Add support for Z39.50 responder to the Debian scripts (18.36 KB, patch)
2019-07-23 12:30 UTC, Ere Maijala
Details | Diff | Splinter Review
Bug 13937: Fix the daemon log path. (1.07 KB, patch)
2019-07-23 12:30 UTC, Ere Maijala
Details | Diff | Splinter Review
Bug 13937: Switch from GetItem() to Koha::Items->find() (1.71 KB, patch)
2019-07-23 12:30 UTC, Ere Maijala
Details | Diff | Splinter Review
Bug 13937: Add SRU tests (6.19 KB, patch)
2019-07-23 12:30 UTC, Ere Maijala
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description Kyle M Hall 2015-04-01 12:46:22 UTC
Some libraries have found that Koha doesn't work well with some external services like SHAREit that utilize z39.50 to get information about a record and its' items.

a) We propose to add field 'status' to items table ( this field can be mapped to whatever MARC tag you wish )

b) Engineer a subroutine that would generate the item's status

The precedence for status would be as follows:
1) Checked Out
2) Lost
3) Not for Loan
4) Damaged
5) On Hold ( In Transit )
6) In Transit
7) On Hold

This status could eventually be used within Koha to simplify our status display logic, but that is outside the immediate scope of this enhancement.
Comment 1 Katrin Fischer 2015-04-01 12:56:01 UTC
I am not sure about this idea - do we really need to store it in the database as it is a calculated value? Or could it be something that is added on export? 
Some libraries allow check-out of damaged items depending on severity - not sure if the sequence is something that should/could be configurable.
Comment 2 Kyle M Hall 2015-04-01 13:18:00 UTC
(In reply to Katrin Fischer from comment #1)
> I am not sure about this idea - do we really need to store it in the
> database as it is a calculated value? Or could it be something that is added
> on export? 
> Some libraries allow check-out of damaged items depending on severity - not
> sure if the sequence is something that should/could be configurable.

It must have the ability to be stored in the MARC, otherwise it won't be useful. The problem is that external software utilizing z39.50 cannot determine the true status of an item based on the fields returned.
Comment 3 Kyle M Hall 2015-04-01 13:21:36 UTC
I would also like the emphasize that this will be a completely optional component and will not be a required feature in Koha. As I wrote, we *could* use this feature within Koha, but I don't see that happening any time soon.
Comment 4 Galen Charlton 2015-04-01 14:41:34 UTC
(In reply to Kyle M Hall from comment #2)
> (In reply to Katrin Fischer from comment #1)
> > I am not sure about this idea - do we really need to store it in the
> > database as it is a calculated value? Or could it be something that is added
> > on export? 
> > Some libraries allow check-out of damaged items depending on severity - not
> > sure if the sequence is something that should/could be configurable.
> 
> It must have the ability to be stored in the MARC, otherwise it won't be
> useful. The problem is that external software utilizing z39.50 cannot
> determine the true status of an item based on the fields returned.

I'm +1 for the general idea of providing a unified item status value to consumers of Koha APIs, Z39.50 clients, and metadata exports.

But responding to your response to Katrin's point: for the purpose of a Z39.50 client or a consumer of a MARC export, it doesn't need to be _stored_ in the MARC record or in the items table; it only needs to be emitted when called for. This could, for example, be invoked by EmbedItemsInMarcBiblio().  Another option would be to have it be a derived DBIC virtual field.

Why not store it in an items.status column?  Because doing so automatically creates more work: it would HAVE to be kept up to date whenever an item gets modified or participates in a circulation transaction.  Why? You can't stop folks from writing SQL queries on it, and you can't stop developers who are not intimately familiar with Koha's notion of item status from trying to use it. To put it another way, *storing* the calculated unified item status value in the database means that you couldn't evade the scope creep of fixing all the item status things in Koha.
Comment 5 Katrin Fischer 2015-04-01 14:50:36 UTC
Hm, it would also be very annoying to correct in case of changing the logic or adding to it. Calculating on the fly might allow for more flexibility and configuration options here.
Comment 6 Chris Cormack 2015-04-01 16:52:03 UTC
To echo Galen, the items are not stored in MARC .. we got rid of that in Koha 3.4  it's only exported to MARC when we are generating the MARC for zebra or an export. So yeah, adding it to the EmbedItemsInMarcBiblio() subroutine would make a lot of sense to me. Since that is the single point in Koha where items data is converted to MARC
Comment 7 HB-NEKLS 2015-04-14 16:30:34 UTC
I'm not quite following the comments being made here -- forgive my ignorance on how MARC works with the Koha items table. 

TLDR version: Regardless of how this is developed and coded, will the final data field result be mappable to a MARC 952 subfield within Koha? It also is a status that needs to be constantly updated, in order for it to be used successfully by an external system.

Long version: 

Background: Right now the not for loan, the lost, the damaged, and the withdrawn fields are separate items table subfields and mapped to the 952 field within MARC. Additionally, the in transit/on hold, waiting for pickup status of an item is stored elsewhere. 

NEKLS, SEKLS, and CKLS would like to develop an additional Koha field that is then possible to be mapped to a MARC 952 subfield so external systems are able to tell if an item is TRULY available for ILL (in our specific case); I'm sure there are other uses we haven't thought about. This new field would be able to aggregate the separate onloan, notforloan, withdrawn, lost, and damaged item fields, as well as pull in the separate data from the branchtransfers and reserves tables that indicate if an item is in transit and/or on hold waiting for pickup; it would need to be updated automatically -- not only on exports or when scripts are run.

When we first looked into this last fall, we discovered that other ILSes typically have one MARC subfield where availability data is stored. Koha currently has the data that determines an item's availability stored all over the place. Right now, this external system profiles our systems using the 952$q field (items.onloan). That field isn't enough, obviously, to determine true availability; items that are on hold, in transit, lost, damaged, regularly appear on ILL request lists. Years ago, we asked about how to fix this problem, but were told a major rewrite of Koha would have to happen. Obviously, we've lived with the situation since. But Jason Robb at SEKLS started asking questions last fall, and we were told that this problem is possible to be fixed now.  

It was originally explained to us that this development could have greater impact on Koha in other places, too, but Kyle can explain that part much better. 

As long as we can map the outcome of where ever this data combining all of these statuses is stored to a MARC 952 item field, I don't care how it's developed :) But if the end result is that this new field is not going to be constantly updated and it can't be mapped to a MARC subfield, we may have to abandon this development.
Comment 8 Kyle M Hall 2015-04-15 14:07:54 UTC
Thanks Galen! That's a capital idea! I imagine then I would need to add a system preference to decide which field/subfield that status value should go into, right?

(In reply to Galen Charlton from comment #4)
> (In reply to Kyle M Hall from comment #2)
> > (In reply to Katrin Fischer from comment #1)
> > > I am not sure about this idea - do we really need to store it in the
> > > database as it is a calculated value? Or could it be something that is added
> > > on export? 
> > > Some libraries allow check-out of damaged items depending on severity - not
> > > sure if the sequence is something that should/could be configurable.
> > 
> > It must have the ability to be stored in the MARC, otherwise it won't be
> > useful. The problem is that external software utilizing z39.50 cannot
> > determine the true status of an item based on the fields returned.
> 
> I'm +1 for the general idea of providing a unified item status value to
> consumers of Koha APIs, Z39.50 clients, and metadata exports.
> 
> But responding to your response to Katrin's point: for the purpose of a
> Z39.50 client or a consumer of a MARC export, it doesn't need to be _stored_
> in the MARC record or in the items table; it only needs to be emitted when
> called for. This could, for example, be invoked by EmbedItemsInMarcBiblio().
> Another option would be to have it be a derived DBIC virtual field.
> 
> Why not store it in an items.status column?  Because doing so automatically
> creates more work: it would HAVE to be kept up to date whenever an item gets
> modified or participates in a circulation transaction.  Why? You can't stop
> folks from writing SQL queries on it, and you can't stop developers who are
> not intimately familiar with Koha's notion of item status from trying to use
> it. To put it another way, *storing* the calculated unified item status
> value in the database means that you couldn't evade the scope creep of
> fixing all the item status things in Koha.
Comment 9 Jesse Weaver 2015-12-11 16:52:21 UTC
I've stolen this development from Kyle, and I have an idea for implementing it that could get us other benefits down the road. Instead of shoving this new status subfield into the MARC upon export to Zebra, why not just make a simple Z39.50 proxy using https://metacpan.org/pod/Net::Z3950::SimpleServer that will query Zebra and insert the item status information on its way back? This means that the item status information will always be up to date and the export remains simple.

Also, once we bring in ElasticSearch, we won't have to keep Zebra around purely for Z39.50 (though we'll have to add some kind of PQF -> ElasticSearch translation layer to make this work).

Thoughts?
Comment 10 Kyle M Hall 2015-12-11 17:56:04 UTC
I'm for this idea. I think a path to z39.50 without Zebra is very good and useful, aside from the benefit of simplifying the feature in a way, and having it be always up to date.
Comment 11 HB-NEKLS 2015-12-11 18:01:50 UTC
I added a version of this comment privately where we're tracking the development, but I'll go ahead and add it here publicly. 

As one of the development sponsors, if this feature is not going to be coded so it can be profiled to a MARC 952 item field, I can no longer support this development. I can't speak for the development co-sponsors, but we're all in the same boat with why we pursued and are funding this idea in the first place.

We are specifically needing this new field to be able to be profiled to a MARC 952 item field to meet our needs (communicating accurate available/not available item status to an external ILL system that can only profile one MARC subfield using z3950 protocol). If that option is not feasible or supported, I'd consider this enhancement dead. 

If what Jesse is proposing still is able to be profiled to a MARC field in the end, then the above can be completely disregarded.
Comment 12 HB-NEKLS 2015-12-11 18:13:31 UTC
And all is good now -- Jesse has said that the MARC subfield creation is still possible with the procedure of what he's suggesting. So my concerns are satisfied.
Comment 13 Jesse Weaver 2016-01-06 22:12:17 UTC Comment hidden (obsolete)
Comment 14 Mirko Tietgen 2016-01-28 20:45:34 UTC
It works very well for me, except for step 8. I still get the default descriptions.
Comment 15 Jesse Weaver 2016-01-28 22:12:35 UTC Comment hidden (obsolete)
Comment 16 Jesse Weaver 2016-01-28 22:15:29 UTC Comment hidden (obsolete)
Comment 17 Jesse Weaver 2016-01-28 22:16:37 UTC
Sorry for the spam; needed to note a new option in the test plan and show our sponsors.

Mirko, could you try clearing memcached? Can't duplicate your issue.
Comment 18 Mirko Tietgen 2016-01-28 22:34:08 UTC Comment hidden (obsolete)
Comment 19 Mirko Tietgen 2016-01-28 22:35:18 UTC
There is no memcached running on the VM I am testing on. Anyway it worked now with the latest patch. I fixed a conflict in C4/Installer/PerlDependencies.pm.
Comment 20 Jonathan Druart 2016-01-29 10:34:38 UTC
Jesse,
I would love to see this code moved to a module covered by tests.
Do you think it's conceivable?
Comment 21 Jesse Weaver 2016-02-12 23:04:21 UTC
I think that's reasonable, depending on what you have in mind testing-wise. Full round-trip testing of Z39.50 could be difficult, but unit tests for the item-status-adding would be quite doable.
Comment 22 Jonathan Druart 2016-02-15 15:51:23 UTC
Something like start the server (make sure it starts correctly), request it, fetch records.
Comment 23 Jonathan Druart 2016-03-11 10:42:39 UTC
Waiting for test coverage.
Comment 24 Jesse Weaver 2016-07-19 21:36:15 UTC
Created attachment 53503 [details] [review]
Bug 13937 - Add a Z39.50 daemon that can inject item status MARC subfields

This creates a new daemon, misc/z3950_responder.pl, which can respond to
Z39.50 requests. By default, it just proxies searches to Zebra.

If desired, however, it can also add a subfield to the item tags on
outgoing records with a textual description of the item's status
(checked out, lost, etc.). This is useful for certain ILL systems. These
strings can be translated using the 'Z3950_STATUS' authorized value.

Test plan:
  1) Start the Z39.50 server using `perl misc/z3950_responder.pl`.
  2) Connect to the server using `yaz-client 127.0.0.1:9999/biblios`.
  3) Run a search, such as `find @attr 1=1016 book`.
  4) Fetch the results both one at a time with `show 1` and in a batch
     using `show 1+5`.
  5) Turn on MARCXML using `format xml` and `elements marcxml`, and
     verify that the records are still correctly fetched.
  6) Enable the item status subfield by restarting the server with the
     option `--add-item-status=k`.
  7) Search for and fetch records, and verify that a $k subfield is
     added to the item tags as appropriate. It should show some
     combination of "Checked Out", "Lost", "Not For Loan", "Damaged",
     "Withdrawn", "In Transit", or "On Hold" as appropriate, or
     "Available".
  8) Add an authorized value named "Z3950_STATUS" with any of the keys
     "AVAILABLE", "CHECKED_OUT", "LOST", "NOT_FOR_LOAN", "DAMAGED",
     "WITHDRAWN", "IN_TRANSIT" or "ON_HOLD", and verify that their
     descriptions are used instead of the default values above.

This version of the patch moves the server out to a module but still needs unit tests.
Comment 25 Nick Clemens 2016-12-12 14:22:22 UTC
Created attachment 58122 [details] [review]
Bug 13937 (Followup) - Remove reference to sub _prefetch_records

Remove reference to sub _prefetch_records

It appears this subis built into fetch and this call is a duplicate
Comment 26 Nick Clemens 2016-12-12 14:23:20 UTC
Created attachment 58125 [details] [review]
Bug 13937 - Add a Z39.50 daemon that can inject item status MARC subfields

This creates a new daemon, misc/z3950_responder.pl, which can respond to
Z39.50 requests. By default, it just proxies searches to Zebra.

If desired, however, it can also add a subfield to the item tags on
outgoing records with a textual description of the item's status
(checked out, lost, etc.). This is useful for certain ILL systems. These
strings can be translated using the 'Z3950_STATUS' authorized value.

Test plan:
  1) Start the Z39.50 server using `perl misc/z3950_responder.pl`.
  2) Connect to the server using `yaz-client 127.0.0.1:9999/biblios`.
  3) Run a search, such as `find @attr 1=1016 book`.
  4) Fetch the results both one at a time with `show 1` and in a batch
     using `show 1+5`.
  5) Turn on MARCXML using `format xml` and `elements marcxml`, and
     verify that the records are still correctly fetched.
  6) Enable the item status subfield by restarting the server with the
     option `--add-item-status=k`.
  7) Search for and fetch records, and verify that a $k subfield is
     added to the item tags as appropriate. It should show some
     combination of "Checked Out", "Lost", "Not For Loan", "Damaged",
     "Withdrawn", "In Transit", or "On Hold" as appropriate, or
     "Available".
  8) Add an authorized value named "Z3950_STATUS" with any of the keys
     "AVAILABLE", "CHECKED_OUT", "LOST", "NOT_FOR_LOAN", "DAMAGED",
     "WITHDRAWN", "IN_TRANSIT" or "ON_HOLD", and verify that their
     descriptions are used instead of the default values above.
Comment 27 Nick Clemens 2016-12-12 14:23:27 UTC
Created attachment 58126 [details] [review]
Bug 13937 (Followup) - Remove reference to sub _prefetch_records

Remove reference to sub _prefetch_records

It appears this subis built into fetch and this call is a duplicate
Comment 28 Olli-Antti Kivilahti 2017-01-05 11:39:06 UTC
Maybe this Bug can help with availability calculations?

Bug 17712
Comment 29 Nick Clemens 2017-01-23 17:13:03 UTC
Created attachment 59443 [details] [review]
Bug 13937 - Add a Z39.50 daemon that can inject item status MARC subfields

This creates a new daemon, misc/z3950_responder.pl, which can respond to
Z39.50 requests. By default, it just proxies searches to Zebra.

If desired, however, it can also add a subfield to the item tags on
outgoing records with a textual description of the item's status
(checked out, lost, etc.). This is useful for certain ILL systems. These
strings can be translated using the 'Z3950_STATUS' authorized value.

Test plan:
  1) Start the Z39.50 server using `perl misc/z3950_responder.pl`.
  2) Connect to the server using `yaz-client 127.0.0.1:9999/biblios`.
  3) Run a search, such as `find @attr 1=1016 book`.
  4) Fetch the results both one at a time with `show 1` and in a batch
     using `show 1+5`.
  5) Turn on MARCXML using `format xml` and `elements marcxml`, and
     verify that the records are still correctly fetched.
  6) Enable the item status subfield by restarting the server with the
     option `--add-item-status=k`.
  7) Search for and fetch records, and verify that a $k subfield is
     added to the item tags as appropriate. It should show some
     combination of "Checked Out", "Lost", "Not For Loan", "Damaged",
     "Withdrawn", "In Transit", or "On Hold" as appropriate, or
     "Available".
  8) Add an authorized value named "Z3950_STATUS" with any of the keys
     "AVAILABLE", "CHECKED_OUT", "LOST", "NOT_FOR_LOAN", "DAMAGED",
     "WITHDRAWN", "IN_TRANSIT" or "ON_HOLD", and verify that their
     descriptions are used instead of the default values above.
Comment 30 Nick Clemens 2017-01-23 17:13:12 UTC
Created attachment 59444 [details] [review]
Bug 13937 (Followup) - Remove reference to sub _prefetch_records

Remove reference to sub _prefetch_records

It appears this subis built into fetch and this call is a duplicate
Comment 31 Nick Clemens 2017-02-28 19:36:48 UTC
Created attachment 60735 [details] [review]
Bug 13937 - Add a Z39.50 daemon that can inject item status MARC subfields

This creates a new daemon, misc/z3950_responder.pl, which can respond to
Z39.50 requests. By default, it just proxies searches to Zebra.

If desired, however, it can also add a subfield to the item tags on
outgoing records with a textual description of the item's status
(checked out, lost, etc.). This is useful for certain ILL systems. These
strings can be translated using the 'Z3950_STATUS' authorized value.

Test plan:
  1) Start the Z39.50 server using `perl misc/z3950_responder.pl`.
  2) Connect to the server using `yaz-client 127.0.0.1:9999/biblios`.
  3) Run a search, such as `find @attr 1=1016 book`.
  4) Fetch the results both one at a time with `show 1` and in a batch
     using `show 1+5`.
  5) Turn on MARCXML using `format xml` and `elements marcxml`, and
     verify that the records are still correctly fetched.
  6) Enable the item status subfield by restarting the server with the
     option `--add-item-status=k`.
  7) Search for and fetch records, and verify that a $k subfield is
     added to the item tags as appropriate. It should show some
     combination of "Checked Out", "Lost", "Not For Loan", "Damaged",
     "Withdrawn", "In Transit", or "On Hold" as appropriate, or
     "Available".
  8) Add an authorized value named "Z3950_STATUS" with any of the keys
     "AVAILABLE", "CHECKED_OUT", "LOST", "NOT_FOR_LOAN", "DAMAGED",
     "WITHDRAWN", "IN_TRANSIT" or "ON_HOLD", and verify that their
     descriptions are used instead of the default values above.
Comment 32 Nick Clemens 2017-02-28 19:36:56 UTC
Created attachment 60736 [details] [review]
Bug 13937 (Followup) - Remove reference to sub _prefetch_records

Remove reference to sub _prefetch_records

It appears this subis built into fetch and this call is a duplicate
Comment 33 Barton Chittenden 2017-02-28 20:01:50 UTC
Created attachment 60737 [details] [review]
Bug 13937 - Add a Z39.50 daemon that can inject item status MARC subfields

This creates a new daemon, misc/z3950_responder.pl, which can respond to
Z39.50 requests. By default, it just proxies searches to Zebra.

If desired, however, it can also add a subfield to the item tags on
outgoing records with a textual description of the item's status
(checked out, lost, etc.). This is useful for certain ILL systems. These
strings can be translated using the 'Z3950_STATUS' authorized value.

Test plan:
  1) Start the Z39.50 server using `perl misc/z3950_responder.pl`.
  2) Connect to the server using `yaz-client 127.0.0.1:9999/biblios`.
  3) Run a search, such as `find @attr 1=1016 book`.
  4) Fetch the results both one at a time with `show 1` and in a batch
     using `show 1+5`.
  5) Turn on MARCXML using `format xml` and `elements marcxml`, and
     verify that the records are still correctly fetched.
  6) Enable the item status subfield by restarting the server with the
     option `--add-item-status=k`.
  7) Search for and fetch records, and verify that a $k subfield is
     added to the item tags as appropriate. It should show some
     combination of "Checked Out", "Lost", "Not For Loan", "Damaged",
     "Withdrawn", "In Transit", or "On Hold" as appropriate, or
     "Available".
  8) Add an authorized value named "Z3950_STATUS" with any of the keys
     "AVAILABLE", "CHECKED_OUT", "LOST", "NOT_FOR_LOAN", "DAMAGED",
     "WITHDRAWN", "IN_TRANSIT" or "ON_HOLD", and verify that their
     descriptions are used instead of the default values above.

Signed-off-by: George Williams <george@nekls.org>
Comment 34 Barton Chittenden 2017-02-28 20:02:04 UTC
Created attachment 60738 [details] [review]
Bug 13937 (Followup) - Remove reference to sub _prefetch_records

Remove reference to sub _prefetch_records

It appears this subis built into fetch and this call is a duplicate

Signed-off-by: George Williams <george@nekls.org>
Comment 35 Jonathan Druart 2017-03-07 19:10:05 UTC
(In reply to Jonathan Druart from comment #23)
> Waiting for test coverage.

Hum?
Comment 36 Nick Clemens 2017-09-07 15:53:42 UTC
Created attachment 66943 [details] [review]
Bug 13937 - Add a Z39.50 daemon that can inject item status MARC subfields

This creates a new daemon, misc/z3950_responder.pl, which can respond to
Z39.50 requests. By default, it just proxies searches to Zebra.

If desired, however, it can also add a subfield to the item tags on
outgoing records with a textual description of the item's status
(checked out, lost, etc.). This is useful for certain ILL systems. These
strings can be translated using the 'Z3950_STATUS' authorized value.

Test plan:
  1) Start the Z39.50 server using `perl misc/z3950_responder.pl`.
  2) Connect to the server using `yaz-client 127.0.0.1:9999/biblios`.
  3) Run a search, such as `find @attr 1=1016 book`.
  4) Fetch the results both one at a time with `show 1` and in a batch
     using `show 1+5`.
  5) Turn on MARCXML using `format xml` and `elements marcxml`, and
     verify that the records are still correctly fetched.
  6) Enable the item status subfield by restarting the server with the
     option `--add-item-status=k`.
  7) Search for and fetch records, and verify that a $k subfield is
     added to the item tags as appropriate. It should show some
     combination of "Checked Out", "Lost", "Not For Loan", "Damaged",
     "Withdrawn", "In Transit", or "On Hold" as appropriate, or
     "Available".
  8) Add an authorized value named "Z3950_STATUS" with any of the keys
     "AVAILABLE", "CHECKED_OUT", "LOST", "NOT_FOR_LOAN", "DAMAGED",
     "WITHDRAWN", "IN_TRANSIT" or "ON_HOLD", and verify that their
     descriptions are used instead of the default values above.

Signed-off-by: George Williams <george@nekls.org>
Comment 37 Nick Clemens 2017-09-07 15:53:46 UTC
Created attachment 66944 [details] [review]
Bug 13937 (Followup) - Remove reference to sub _prefetch_records

Remove reference to sub _prefetch_records

It appears this subis built into fetch and this call is a duplicate

Signed-off-by: George Williams <george@nekls.org>
Comment 38 Nick Clemens 2017-09-07 15:53:49 UTC
Created attachment 66945 [details] [review]
Bug 13937 - (Followup) Correct error call to use self
Comment 39 Nick Clemens 2017-09-07 15:53:53 UTC
Created attachment 66946 [details] [review]
Bug 13937 - Unit tests
Comment 40 Nick Clemens 2017-11-14 12:30:42 UTC
Created attachment 69139 [details] [review]
Bug 13937: Followup - pass through yaz switches and don't ignore case
Comment 41 Marcel de Rooy 2018-04-20 08:56:35 UTC
QA: Looking here
Comment 42 Marcel de Rooy 2018-04-20 10:29:22 UTC
=== General
The idea is interesting!

It works but I have the impression that it still needs some fine-tuning and further testing. Crashes on a show 0 for instance. Tests need attention.
As an optional feature, it will not harm people not using it. 

On the other hand, it adds 600 lines of code for an additional item status via a Z39.50 daemon (including overlap with C4/Search Zebra code). We need to maintain that. Food for discussion?

The feature does not support showing this new status via SRU/SRW in analogy with Zebra? (Only see YAZ responding with 404 on 9999 now in the browser.)

In terms of new development, I wonder if it would not be simpler to provide an item status via REST API, and not invest in Zebra/Z39.50 (not looking at specific sponsor specs here)? Further thoughts ?
Comment 43 Marcel de Rooy 2018-04-20 10:29:53 UTC
=== Details
Net::Z3950::SimpleServer
Just wondering why we need a new dep. SIPServer uses Net::Server::PreFork ?

t/Koha/Z3950responder.t
This test fails with me. Please check.
not ok 3 - We didn't start server , should fail
#   Failed test 'We didn't start server , should fail'
#   at t/Koha/Z3950responder.t line 24.
#          got: undef
#     expected: '2'
not ok 4 - We didn't start server, should fail because it cannot connect
#   Failed test 'We didn't start server, should fail because it cannot connect'
#   at t/Koha/Z3950responder.t line 25.
#          got: undef
#     expected: 'Cannot connect to upstream server'
not ok 5 - There is no handler as we aren't connected
#   Failed test 'There is no handler as we aren't connected'
#   at t/Koha/Z3950responder.t line 27.
#          got: '13'
#     expected: '30'
not ok 6 - We don't have a handler, should fail because we don't
#   Failed test 'We don't have a handler, should fail because we don't'
#   at t/Koha/Z3950responder.t line 28.
#          got: 'Fetch request out of range'
#     expected: 'No such resultset'
Use of uninitialized value $num_to_prefetch in division (/) at /usr/share/koha/devclone/Koha/Z3950Responder/Session.pm line 158.
Illegal division by zero at /usr/share/koha/devclone/Koha/Z3950Responder/Session.pm line 158.

t/db_dependent/Koha/Z3950Responder/Session.t
t/db_dependent/Koha/Z3950Responder/Session2.t
The test coverage only focuses on the add item status routine. We are not testing the server itself. Would it be too hard to try?

Commit message stuff from qa tools
        * Commit title does not contain 'follow-up' correctly spelt - 8e0da10
        * Commit title does not start with 'Bug XXXXX: ' - 31c0f7b
        * Commit title does not start with 'Bug XXXXX: ' - cf4f62b
        * Commit title does not contain 'follow-up' correctly spelt - cf4f62b
        * Commit title does not start with 'Bug XXXXX: ' - f592650
        * Commit title does not contain 'follow-up' correctly spelt - f592650
        * Commit title does not start with 'Bug XXXXX: ' - c24c232

Path::Tiny
Why is this dependency added? It is not used somewhere.

Log::Log4perl::Appender
Might be a new enhancement on itself?? 

my ($item_tag, $itemnumber_subfield) = GetMarcFromKohaField( "items.itemnumber", '' );
Does not need the default framework parameter anymore. See changed API.

Testing misc/z3950_responder.pl
Crashes on show 0
Can't call method "raw" on an undefined value at /usr/share/koha/devclone/Koha/Z3950Responder/Session.pm line 169.
Target closed connection

+    warn "name: $opt_name and value: $opt_value";
Debug leftover?

Configuration
Defaults to port 9999 (yaz default). Note that we may use port 9999 too for Zebra authorities.. Passing a listener-address worked though.
Should we add our own default?
Comment 44 Ere Maijala 2018-11-21 12:28:08 UTC
I'd suggest making the default port 210 since that's the registered port for Z39.50.
Comment 45 Ere Maijala 2018-11-21 13:14:46 UTC
Then again, 210 might be difficult since the user needs sufficient privileges to bind to it. Maybe 2100 is better.
Comment 46 Ere Maijala 2018-11-21 14:26:07 UTC
Created attachment 82562 [details] [review]
Bug 13937: Add a Z39.50 daemon that can inject item status MARC subfields

This creates a new daemon, misc/z3950_responder.pl, which can respond to
Z39.50 requests. By default, it just proxies searches to Zebra.

If desired, however, it can also add a subfield to the item tags on
outgoing records with a textual description of the item's status
(checked out, lost, etc.). This is useful for certain ILL systems. These
strings can be translated using the 'Z3950_STATUS' authorized value.

Test plan:
  1) Start the Z39.50 server using `perl misc/z3950_responder.pl`.
  2) Connect to the server using `yaz-client 127.0.0.1:9999/biblios`.
  3) Run a search, such as `find @attr 1=1016 book`.
  4) Fetch the results both one at a time with `show 1` and in a batch
     using `show 1+5`.
  5) Turn on MARCXML using `format xml` and `elements marcxml`, and
     verify that the records are still correctly fetched.
  6) Enable the item status subfield by restarting the server with the
     option `--add-item-status=k`.
  7) Search for and fetch records, and verify that a $k subfield is
     added to the item tags as appropriate. It should show some
     combination of "Checked Out", "Lost", "Not For Loan", "Damaged",
     "Withdrawn", "In Transit", or "On Hold" as appropriate, or
     "Available".
  8) Add an authorized value named "Z3950_STATUS" with any of the keys
     "AVAILABLE", "CHECKED_OUT", "LOST", "NOT_FOR_LOAN", "DAMAGED",
     "WITHDRAWN", "IN_TRANSIT" or "ON_HOLD", and verify that their
     descriptions are used instead of the default values above.

Signed-off-by: George Williams <george@nekls.org>
Comment 47 Ere Maijala 2018-11-21 14:26:13 UTC
Created attachment 82563 [details] [review]
Bug 13937: (follow-up) Remove reference to sub _prefetch_records

Remove reference to sub _prefetch_records

It appears this sub is built into fetch and this call is a duplicate

Signed-off-by: George Williams <george@nekls.org>
Comment 48 Ere Maijala 2018-11-21 14:26:19 UTC
Created attachment 82564 [details] [review]
Bug 13937: (follow-up) Correct error call to use self
Comment 49 Ere Maijala 2018-11-21 14:26:24 UTC
Created attachment 82565 [details] [review]
Bug 13937: Add unit tests
Comment 50 Ere Maijala 2018-11-21 14:26:29 UTC
Created attachment 82566 [details] [review]
Bug 13937: (follow-up) Pass through yaz switches and don't ignore case
Comment 51 Ere Maijala 2018-11-21 14:26:34 UTC
Created attachment 82567 [details] [review]
Bug 13937: Fix issues found in QA
Comment 52 Ere Maijala 2018-11-21 14:28:18 UTC
I've addressed many of the QA issues, but I believe that it doesn't make much sense to push this forward until it works with both Zebra and Elasticsearch. That's what I'm going to work on next.
Comment 53 Marcel de Rooy 2018-11-21 14:32:28 UTC
(In reply to Ere Maijala from comment #52)
> I've addressed many of the QA issues, but I believe that it doesn't make
> much sense to push this forward until it works with both Zebra and
> Elasticsearch. That's what I'm going to work on next.

Great !
Comment 54 David Cook 2018-11-21 23:38:40 UTC
(In reply to Ere Maijala from comment #45)
> Then again, 210 might be difficult since the user needs sufficient
> privileges to bind to it. Maybe 2100 is better.

Another common Z39.50 port is 7090.
Comment 55 Ere Maijala 2018-11-22 08:11:08 UTC
It'll still be configurable.
Comment 56 Ere Maijala 2018-11-23 07:30:21 UTC
Created attachment 82589 [details] [review]
Bug 13937: Add a Z39.50 daemon that can inject item status MARC subfields

This creates a new daemon, misc/z3950_responder.pl, which can respond to
Z39.50 requests. By default, it just proxies searches to Zebra.

If desired, however, it can also add a subfield to the item tags on
outgoing records with a textual description of the item's status
(checked out, lost, etc.). This is useful for certain ILL systems. These
strings can be translated using the 'Z3950_STATUS' authorized value.

Test plan:
  1) Start the Z39.50 server using `perl misc/z3950_responder.pl`.
  2) Connect to the server using `yaz-client 127.0.0.1:9999/biblios`.
  3) Run a search, such as `find @attr 1=1016 book`.
  4) Fetch the results both one at a time with `show 1` and in a batch
     using `show 1+5`.
  5) Turn on MARCXML using `format xml` and `elements marcxml`, and
     verify that the records are still correctly fetched.
  6) Enable the item status subfield by restarting the server with the
     option `--add-item-status=k`.
  7) Search for and fetch records, and verify that a $k subfield is
     added to the item tags as appropriate. It should show some
     combination of "Checked Out", "Lost", "Not For Loan", "Damaged",
     "Withdrawn", "In Transit", or "On Hold" as appropriate, or
     "Available".
  8) Add an authorized value named "Z3950_STATUS" with any of the keys
     "AVAILABLE", "CHECKED_OUT", "LOST", "NOT_FOR_LOAN", "DAMAGED",
     "WITHDRAWN", "IN_TRANSIT" or "ON_HOLD", and verify that their
     descriptions are used instead of the default values above.

Signed-off-by: George Williams <george@nekls.org>
Comment 57 Ere Maijala 2018-11-23 07:30:27 UTC
Created attachment 82590 [details] [review]
Bug 13937: (follow-up) Remove reference to sub _prefetch_records

Remove reference to sub _prefetch_records

It appears this sub is built into fetch and this call is a duplicate

Signed-off-by: George Williams <george@nekls.org>
Comment 58 Ere Maijala 2018-11-23 07:30:33 UTC
Created attachment 82591 [details] [review]
Bug 13937: (follow-up) Correct error call to use self
Comment 59 Ere Maijala 2018-11-23 07:30:38 UTC
Created attachment 82592 [details] [review]
Bug 13937: Add unit tests
Comment 60 Ere Maijala 2018-11-23 07:30:43 UTC
Created attachment 82593 [details] [review]
Bug 13937: (follow-up) Pass through yaz switches and don't ignore case
Comment 61 Ere Maijala 2018-11-23 07:30:49 UTC
Created attachment 82594 [details] [review]
Bug 13937: Fix issues found in QA
Comment 62 Ere Maijala 2018-11-23 07:30:54 UTC
Created attachment 82595 [details] [review]
Bug 13937: Add support for SRU and Elasticsearch

Splits Session to GenericSession and ZebraSession where Generic supports any
search backend via the SearchEngine classes and Zebra maintains the direct
channel to the Zebra server.

Adds config files required for mapping BIB-1 attributes to Koha search fields
and SRU indexes to BIB-1 attributes.

Sponsored-by: National Library of Finland
Comment 63 Ere Maijala 2018-11-23 07:33:08 UTC
Ok, so, the latest version is pretty much feature-complete with support for Elasticsearch as well as Zebra. And SRU is working too. I still need to add more PODs and tests and verify that the config files work with packaging etc.
Comment 64 Ere Maijala 2018-11-23 07:52:07 UTC
*** Bug 21210 has been marked as a duplicate of this bug. ***
Comment 65 Ere Maijala 2018-11-27 11:45:06 UTC
Created attachment 82660 [details] [review]
Bug 13937: Add a Z39.50 daemon that can inject item status MARC subfields

This creates a new daemon, misc/z3950_responder.pl, which can respond to
Z39.50 requests. By default, it just proxies searches to Zebra.

If desired, however, it can also add a subfield to the item tags on
outgoing records with a textual description of the item's status
(checked out, lost, etc.). This is useful for certain ILL systems. These
strings can be translated using the 'Z3950_STATUS' authorized value.

Test plan:
  1) Start the Z39.50 server using `perl misc/z3950_responder.pl`.
  2) Connect to the server using `yaz-client 127.0.0.1:9999/biblios`.
  3) Run a search, such as `find @attr 1=1016 book`.
  4) Fetch the results both one at a time with `show 1` and in a batch
     using `show 1+5`.
  5) Turn on MARCXML using `format xml` and `elements marcxml`, and
     verify that the records are still correctly fetched.
  6) Enable the item status subfield by restarting the server with the
     option `--add-item-status=k`.
  7) Search for and fetch records, and verify that a $k subfield is
     added to the item tags as appropriate. It should show some
     combination of "Checked Out", "Lost", "Not For Loan", "Damaged",
     "Withdrawn", "In Transit", or "On Hold" as appropriate, or
     "Available".
  8) Add an authorized value named "Z3950_STATUS" with any of the keys
     "AVAILABLE", "CHECKED_OUT", "LOST", "NOT_FOR_LOAN", "DAMAGED",
     "WITHDRAWN", "IN_TRANSIT" or "ON_HOLD", and verify that their
     descriptions are used instead of the default values above.

Signed-off-by: George Williams <george@nekls.org>
Comment 66 Ere Maijala 2018-11-27 11:45:13 UTC
Created attachment 82661 [details] [review]
Bug 13937: (follow-up) Remove reference to sub _prefetch_records

Remove reference to sub _prefetch_records

It appears this sub is built into fetch and this call is a duplicate

Signed-off-by: George Williams <george@nekls.org>
Comment 67 Ere Maijala 2018-11-27 11:45:19 UTC
Created attachment 82662 [details] [review]
Bug 13937: (follow-up) Correct error call to use self
Comment 68 Ere Maijala 2018-11-27 11:45:26 UTC
Created attachment 82663 [details] [review]
Bug 13937: Add unit tests
Comment 69 Ere Maijala 2018-11-27 11:45:32 UTC
Created attachment 82664 [details] [review]
Bug 13937: (follow-up) Pass through yaz switches and don't ignore case
Comment 70 Ere Maijala 2018-11-27 11:45:38 UTC
Created attachment 82665 [details] [review]
Bug 13937: Fix issues found in QA
Comment 71 Ere Maijala 2018-11-27 11:45:44 UTC
Created attachment 82666 [details] [review]
Bug 13937: Add support for SRU and Elasticsearch

Splits Session to GenericSession and ZebraSession where Generic supports any
search backend via the SearchEngine classes and Zebra maintains the direct
channel to the Zebra server.

Adds config files required for mapping BIB-1 attributes to Koha search fields
and SRU indexes to BIB-1 attributes.

Adds PODs.

Sponsored-by: National Library of Finland
Comment 72 Ere Maijala 2018-11-27 11:45:50 UTC
Created attachment 82667 [details] [review]
Bug 13937: Add tests for search and retrieval

Sponsored-by: National Library of Finland
Comment 73 Katrin Fischer 2018-12-15 09:13:21 UTC
I see that George Williams signed off the initial patches. 
George, could you retest with the new additions maybe?
Comment 74 Ere Maijala 2019-01-11 12:59:26 UTC
Created attachment 83780 [details] [review]
Bug 13937: Add a Z39.50 daemon that can inject item status MARC subfields

This creates a new daemon, misc/z3950_responder.pl, which can respond to
Z39.50 requests. By default, it just proxies searches to Zebra.

If desired, however, it can also add a subfield to the item tags on
outgoing records with a textual description of the item's status
(checked out, lost, etc.). This is useful for certain ILL systems. These
strings can be translated using the 'Z3950_STATUS' authorized value.

Test plan:
  1) Start the Z39.50 server using `perl misc/z3950_responder.pl`.
  2) Connect to the server using `yaz-client 127.0.0.1:9999/biblios`.
  3) Run a search, such as `find @attr 1=1016 book`.
  4) Fetch the results both one at a time with `show 1` and in a batch
     using `show 1+5`.
  5) Turn on MARCXML using `format xml` and `elements marcxml`, and
     verify that the records are still correctly fetched.
  6) Enable the item status subfield by restarting the server with the
     option `--add-item-status=k`.
  7) Search for and fetch records, and verify that a $k subfield is
     added to the item tags as appropriate. It should show some
     combination of "Checked Out", "Lost", "Not For Loan", "Damaged",
     "Withdrawn", "In Transit", or "On Hold" as appropriate, or
     "Available".
  8) Add an authorized value named "Z3950_STATUS" with any of the keys
     "AVAILABLE", "CHECKED_OUT", "LOST", "NOT_FOR_LOAN", "DAMAGED",
     "WITHDRAWN", "IN_TRANSIT" or "ON_HOLD", and verify that their
     descriptions are used instead of the default values above.

Signed-off-by: George Williams <george@nekls.org>
Comment 75 Ere Maijala 2019-01-11 12:59:53 UTC
Created attachment 83781 [details] [review]
Bug 13937: Add a Z39.50 daemon that can inject item status MARC subfields

This creates a new daemon, misc/z3950_responder.pl, which can respond to
Z39.50 requests. By default, it just proxies searches to Zebra.

If desired, however, it can also add a subfield to the item tags on
outgoing records with a textual description of the item's status
(checked out, lost, etc.). This is useful for certain ILL systems. These
strings can be translated using the 'Z3950_STATUS' authorized value.

Test plan:
  1) Start the Z39.50 server using `perl misc/z3950_responder.pl`.
  2) Connect to the server using `yaz-client 127.0.0.1:9999/biblios`.
  3) Run a search, such as `find @attr 1=1016 book`.
  4) Fetch the results both one at a time with `show 1` and in a batch
     using `show 1+5`.
  5) Turn on MARCXML using `format xml` and `elements marcxml`, and
     verify that the records are still correctly fetched.
  6) Enable the item status subfield by restarting the server with the
     option `--add-item-status=k`.
  7) Search for and fetch records, and verify that a $k subfield is
     added to the item tags as appropriate. It should show some
     combination of "Checked Out", "Lost", "Not For Loan", "Damaged",
     "Withdrawn", "In Transit", or "On Hold" as appropriate, or
     "Available".
  8) Add an authorized value named "Z3950_STATUS" with any of the keys
     "AVAILABLE", "CHECKED_OUT", "LOST", "NOT_FOR_LOAN", "DAMAGED",
     "WITHDRAWN", "IN_TRANSIT" or "ON_HOLD", and verify that their
     descriptions are used instead of the default values above.

Signed-off-by: George Williams <george@nekls.org>
Comment 76 Ere Maijala 2019-01-11 13:00:01 UTC
Created attachment 83782 [details] [review]
Bug 13937: (follow-up) Remove reference to sub _prefetch_records

Remove reference to sub _prefetch_records

It appears this sub is built into fetch and this call is a duplicate

Signed-off-by: George Williams <george@nekls.org>
Comment 77 Ere Maijala 2019-01-11 13:00:10 UTC
Created attachment 83783 [details] [review]
Bug 13937: (follow-up) Correct error call to use self
Comment 78 Ere Maijala 2019-01-11 13:00:18 UTC
Created attachment 83784 [details] [review]
Bug 13937: Add unit tests
Comment 79 Ere Maijala 2019-01-11 13:00:26 UTC
Created attachment 83785 [details] [review]
Bug 13937: (follow-up) Pass through yaz switches and don't ignore case
Comment 80 Ere Maijala 2019-01-11 13:00:32 UTC
Created attachment 83786 [details] [review]
Bug 13937: Fix issues found in QA
Comment 81 Ere Maijala 2019-01-11 13:00:38 UTC
Created attachment 83787 [details] [review]
Bug 13937: Add support for SRU and Elasticsearch

Splits Session to GenericSession and ZebraSession where Generic supports any
search backend via the SearchEngine classes and Zebra maintains the direct
channel to the Zebra server.

Adds config files required for mapping BIB-1 attributes to Koha search fields
and SRU indexes to BIB-1 attributes.

Adds PODs.

Sponsored-by: National Library of Finland
Comment 82 Ere Maijala 2019-01-11 13:00:44 UTC
Created attachment 83788 [details] [review]
Bug 13937: Add tests for search and retrieval

Sponsored-by: National Library of Finland
Comment 83 Ere Maijala 2019-01-11 13:00:50 UTC
Created attachment 83789 [details] [review]
Bug 13937: Add support for Z39.50 responder to the Debian scripts
Comment 84 Ere Maijala 2019-01-11 13:02:32 UTC
Rebased and added support for managing the Z39.50 responder in the Debian init scripts. Not included in koha-create by default, but I reworded the current options in it to indicate that they apply to Zebra's Z39.50/SRU server only.
Comment 85 Joonas Kylmälä 2019-01-11 13:08:57 UTC
(In reply to Ere Maijala from comment #84)
> Rebased and added support for managing the Z39.50 responder in the Debian
> init scripts.

What did you rebase? The previous patches applied for me just fine with the current master.
Comment 86 Ere Maijala 2019-01-11 13:26:39 UTC
(In reply to Joonas Kylmälä from comment #85)
> (In reply to Ere Maijala from comment #84)
> > Rebased and added support for managing the Z39.50 responder in the Debian
> > init scripts.
> 
> What did you rebase? The previous patches applied for me just fine with the
> current master.

Nothing, actually. Sorry about that. Just verified that the patches still apply cleanly.
Comment 87 Ere Maijala 2019-01-14 15:33:13 UTC
Handling of quotes and parenthesis still needs some work.
Comment 88 Ere Maijala 2019-01-15 11:55:59 UTC
Created attachment 83951 [details] [review]
Bug 13937: Add a Z39.50 daemon that can inject item status MARC subfields

This creates a new daemon, misc/z3950_responder.pl, which can respond to
Z39.50 requests. By default, it just proxies searches to Zebra.

If desired, however, it can also add a subfield to the item tags on
outgoing records with a textual description of the item's status
(checked out, lost, etc.). This is useful for certain ILL systems. These
strings can be translated using the 'Z3950_STATUS' authorized value.

Test plan:
  1) Start the Z39.50 server using `perl misc/z3950_responder.pl`.
  2) Connect to the server using `yaz-client 127.0.0.1:9999/biblios`.
  3) Run a search, such as `find @attr 1=1016 book`.
  4) Fetch the results both one at a time with `show 1` and in a batch
     using `show 1+5`.
  5) Turn on MARCXML using `format xml` and `elements marcxml`, and
     verify that the records are still correctly fetched.
  6) Enable the item status subfield by restarting the server with the
     option `--add-item-status=k`.
  7) Search for and fetch records, and verify that a $k subfield is
     added to the item tags as appropriate. It should show some
     combination of "Checked Out", "Lost", "Not For Loan", "Damaged",
     "Withdrawn", "In Transit", or "On Hold" as appropriate, or
     "Available".
  8) Add an authorized value named "Z3950_STATUS" with any of the keys
     "AVAILABLE", "CHECKED_OUT", "LOST", "NOT_FOR_LOAN", "DAMAGED",
     "WITHDRAWN", "IN_TRANSIT" or "ON_HOLD", and verify that their
     descriptions are used instead of the default values above.

Signed-off-by: George Williams <george@nekls.org>
Comment 89 Ere Maijala 2019-01-15 11:56:04 UTC
Created attachment 83952 [details] [review]
Bug 13937: (follow-up) Remove reference to sub _prefetch_records

Remove reference to sub _prefetch_records

It appears this sub is built into fetch and this call is a duplicate

Signed-off-by: George Williams <george@nekls.org>
Comment 90 Ere Maijala 2019-01-15 11:56:09 UTC
Created attachment 83953 [details] [review]
Bug 13937: (follow-up) Correct error call to use self
Comment 91 Ere Maijala 2019-01-15 11:56:14 UTC
Created attachment 83954 [details] [review]
Bug 13937: Add unit tests
Comment 92 Ere Maijala 2019-01-15 11:56:19 UTC
Created attachment 83955 [details] [review]
Bug 13937: (follow-up) Pass through yaz switches and don't ignore case
Comment 93 Ere Maijala 2019-01-15 11:56:24 UTC
Created attachment 83956 [details] [review]
Bug 13937: Fix issues found in QA
Comment 94 Ere Maijala 2019-01-15 11:56:29 UTC
Created attachment 83957 [details] [review]
Bug 13937: Add support for SRU and Elasticsearch

Splits Session to GenericSession and ZebraSession where Generic supports any
search backend via the SearchEngine classes and Zebra maintains the direct
channel to the Zebra server.

Adds config files required for mapping BIB-1 attributes to Koha search fields
and SRU indexes to BIB-1 attributes.

Adds PODs.

Sponsored-by: National Library of Finland
Comment 95 Ere Maijala 2019-01-15 11:56:36 UTC
Created attachment 83958 [details] [review]
Bug 13937: Add tests for search and retrieval

Sponsored-by: National Library of Finland
Comment 96 Ere Maijala 2019-01-15 11:56:41 UTC
Created attachment 83959 [details] [review]
Bug 13937: Fix RPN conversion

Fixes handling of quotes and parenthesis.
Comment 97 Ere Maijala 2019-01-16 12:05:32 UTC
Created attachment 84050 [details] [review]
Bug 13937: Add support for Z39.50 responder to the Debian scripts
Comment 98 Ere Maijala 2019-01-16 12:08:10 UTC
Created attachment 84051 [details] [review]
Bug 13937: Add a Z39.50 daemon that can inject item status MARC subfields

This creates a new daemon, misc/z3950_responder.pl, which can respond to
Z39.50 requests. By default, it just proxies searches to Zebra.

If desired, however, it can also add a subfield to the item tags on
outgoing records with a textual description of the item's status
(checked out, lost, etc.). This is useful for certain ILL systems. These
strings can be translated using the 'Z3950_STATUS' authorized value.

Test plan:
  1) Start the Z39.50 server using `perl misc/z3950_responder.pl`.
  2) Connect to the server using `yaz-client 127.0.0.1:9999/biblios`.
  3) Run a search, such as `find @attr 1=1016 book`.
  4) Fetch the results both one at a time with `show 1` and in a batch
     using `show 1+5`.
  5) Turn on MARCXML using `format xml` and `elements marcxml`, and
     verify that the records are still correctly fetched.
  6) Enable the item status subfield by restarting the server with the
     option `--add-item-status=k`.
  7) Search for and fetch records, and verify that a $k subfield is
     added to the item tags as appropriate. It should show some
     combination of "Checked Out", "Lost", "Not For Loan", "Damaged",
     "Withdrawn", "In Transit", or "On Hold" as appropriate, or
     "Available".
  8) Add an authorized value named "Z3950_STATUS" with any of the keys
     "AVAILABLE", "CHECKED_OUT", "LOST", "NOT_FOR_LOAN", "DAMAGED",
     "WITHDRAWN", "IN_TRANSIT" or "ON_HOLD", and verify that their
     descriptions are used instead of the default values above.

Signed-off-by: George Williams <george@nekls.org>
Comment 99 Ere Maijala 2019-01-16 12:08:32 UTC
Created attachment 84052 [details] [review]
Bug 13937: (follow-up) Remove reference to sub _prefetch_records

Remove reference to sub _prefetch_records

It appears this sub is built into fetch and this call is a duplicate

Signed-off-by: George Williams <george@nekls.org>
Comment 100 Ere Maijala 2019-01-16 12:08:56 UTC
Created attachment 84053 [details] [review]
Bug 13937: (follow-up) Correct error call to use self
Comment 101 Ere Maijala 2019-01-16 12:09:19 UTC
Created attachment 84054 [details] [review]
Bug 13937: Add unit tests
Comment 102 Ere Maijala 2019-01-16 12:09:50 UTC
Created attachment 84055 [details] [review]
Bug 13937: (follow-up) Pass through yaz switches and don't ignore case
Comment 103 Ere Maijala 2019-01-16 12:10:31 UTC
Created attachment 84056 [details] [review]
Bug 13937: Fix issues found in QA
Comment 104 Ere Maijala 2019-01-16 12:11:03 UTC
Created attachment 84057 [details] [review]
Bug 13937: Add support for SRU and Elasticsearch

Splits Session to GenericSession and ZebraSession where Generic supports any
search backend via the SearchEngine classes and Zebra maintains the direct
channel to the Zebra server.

Adds config files required for mapping BIB-1 attributes to Koha search fields
and SRU indexes to BIB-1 attributes.

Adds PODs.

Sponsored-by: National Library of Finland
Comment 105 Ere Maijala 2019-01-16 12:11:39 UTC
Created attachment 84058 [details] [review]
Bug 13937: Add tests for search and retrieval

Sponsored-by: National Library of Finland
Comment 106 Ere Maijala 2019-01-16 12:12:57 UTC
Created attachment 84059 [details] [review]
Bug 13937: Add support for Z39.50 responder to the Debian scripts
Comment 107 Ere Maijala 2019-01-16 13:03:24 UTC
Comment on attachment 83959 [details] [review]
Bug 13937: Fix RPN conversion

I made a mess somehow, need to clean up the patches.
Comment 108 Ere Maijala 2019-01-16 13:21:40 UTC
Created attachment 84062 [details] [review]
Bug 13937: Add a Z39.50 daemon that can inject item status MARC subfields

This creates a new daemon, misc/z3950_responder.pl, which can respond to
Z39.50 requests. By default, it just proxies searches to Zebra.

If desired, however, it can also add a subfield to the item tags on
outgoing records with a textual description of the item's status
(checked out, lost, etc.). This is useful for certain ILL systems. These
strings can be translated using the 'Z3950_STATUS' authorized value.

Test plan:
  1) Start the Z39.50 server using `perl misc/z3950_responder.pl`.
  2) Connect to the server using `yaz-client 127.0.0.1:9999/biblios`.
  3) Run a search, such as `find @attr 1=1016 book`.
  4) Fetch the results both one at a time with `show 1` and in a batch
     using `show 1+5`.
  5) Turn on MARCXML using `format xml` and `elements marcxml`, and
     verify that the records are still correctly fetched.
  6) Enable the item status subfield by restarting the server with the
     option `--add-item-status=k`.
  7) Search for and fetch records, and verify that a $k subfield is
     added to the item tags as appropriate. It should show some
     combination of "Checked Out", "Lost", "Not For Loan", "Damaged",
     "Withdrawn", "In Transit", or "On Hold" as appropriate, or
     "Available".
  8) Add an authorized value named "Z3950_STATUS" with any of the keys
     "AVAILABLE", "CHECKED_OUT", "LOST", "NOT_FOR_LOAN", "DAMAGED",
     "WITHDRAWN", "IN_TRANSIT" or "ON_HOLD", and verify that their
     descriptions are used instead of the default values above.

Signed-off-by: George Williams <george@nekls.org>
Comment 109 Ere Maijala 2019-01-16 13:22:00 UTC
Created attachment 84063 [details] [review]
Bug 13937: (follow-up) Remove reference to sub _prefetch_records

Remove reference to sub _prefetch_records

It appears this sub is built into fetch and this call is a duplicate

Signed-off-by: George Williams <george@nekls.org>
Comment 110 Ere Maijala 2019-01-16 13:22:28 UTC
Created attachment 84064 [details] [review]
Bug 13937: (follow-up) Correct error call to use self
Comment 111 Ere Maijala 2019-01-16 13:22:50 UTC
Created attachment 84065 [details] [review]
Bug 13937: Add unit tests
Comment 112 Ere Maijala 2019-01-16 13:23:19 UTC
Created attachment 84066 [details] [review]
Bug 13937: (follow-up) Pass through yaz switches and don't ignore case
Comment 113 Ere Maijala 2019-01-16 13:24:07 UTC
Created attachment 84067 [details] [review]
Bug 13937: Fix issues found in QA
Comment 114 Ere Maijala 2019-01-16 13:24:41 UTC
Created attachment 84068 [details] [review]
Bug 13937: Add support for SRU and Elasticsearch

Splits Session to GenericSession and ZebraSession where Generic supports any
search backend via the SearchEngine classes and Zebra maintains the direct
channel to the Zebra server.

Adds config files required for mapping BIB-1 attributes to Koha search fields
and SRU indexes to BIB-1 attributes.

Adds PODs.

Sponsored-by: National Library of Finland
Comment 115 Ere Maijala 2019-01-16 13:25:09 UTC
Created attachment 84069 [details] [review]
Bug 13937: Add tests for search and retrieval

Sponsored-by: National Library of Finland
Comment 116 Ere Maijala 2019-01-16 13:25:50 UTC
Created attachment 84072 [details] [review]
Bug 13937: Fix RPN conversion

Fixes handling of parenthesis and removes unused quote handling.
Comment 117 Ere Maijala 2019-01-16 13:26:21 UTC
Created attachment 84073 [details] [review]
Bug 13937: Add support for Z39.50 responder to the Debian scripts
Comment 118 Katrin Fischer 2019-01-25 10:31:45 UTC
This seems to have moved beyond it's initial purpose with the Elasticsearch additions. Would it make sense to split it up or at least change the title? It might gain more interest that way too.
Comment 119 Olli-Antti Kivilahti 2019-02-14 13:17:50 UTC
Created attachment 85112 [details] [review]
11 - Bug 13937: Add a Z39.50 daemon that can inject item status MARC subfields. Fix log path.
Comment 120 Josef Moravec 2019-03-05 12:11:08 UTC
Just started to testing, but file Koha/Z3950Responder/Session.pm calls C4::Items::GetItem which was removed from master...

Could you add a follow-up for that?
Comment 121 Ere Maijala 2019-03-08 10:10:44 UTC
Created attachment 86353 [details] [review]
Bug 13937: Add a Z39.50 daemon that can inject item status MARC subfields

This creates a new daemon, misc/z3950_responder.pl, which can respond to
Z39.50 requests. By default, it just proxies searches to Zebra.

If desired, however, it can also add a subfield to the item tags on
outgoing records with a textual description of the item's status
(checked out, lost, etc.). This is useful for certain ILL systems. These
strings can be translated using the 'Z3950_STATUS' authorized value.

Test plan:
  1) Start the Z39.50 server using `perl misc/z3950_responder.pl`.
  2) Connect to the server using `yaz-client 127.0.0.1:9999/biblios`.
  3) Run a search, such as `find @attr 1=1016 book`.
  4) Fetch the results both one at a time with `show 1` and in a batch
     using `show 1+5`.
  5) Turn on MARCXML using `format xml` and `elements marcxml`, and
     verify that the records are still correctly fetched.
  6) Enable the item status subfield by restarting the server with the
     option `--add-item-status=k`.
  7) Search for and fetch records, and verify that a $k subfield is
     added to the item tags as appropriate. It should show some
     combination of "Checked Out", "Lost", "Not For Loan", "Damaged",
     "Withdrawn", "In Transit", or "On Hold" as appropriate, or
     "Available".
  8) Add an authorized value named "Z3950_STATUS" with any of the keys
     "AVAILABLE", "CHECKED_OUT", "LOST", "NOT_FOR_LOAN", "DAMAGED",
     "WITHDRAWN", "IN_TRANSIT" or "ON_HOLD", and verify that their
     descriptions are used instead of the default values above.

Signed-off-by: George Williams <george@nekls.org>
Comment 122 Ere Maijala 2019-03-08 10:10:50 UTC
Created attachment 86354 [details] [review]
Bug 13937: (follow-up) Remove reference to sub _prefetch_records

Remove reference to sub _prefetch_records

It appears this sub is built into fetch and this call is a duplicate

Signed-off-by: George Williams <george@nekls.org>
Comment 123 Ere Maijala 2019-03-08 10:10:55 UTC
Created attachment 86355 [details] [review]
Bug 13937: (follow-up) Correct error call to use self
Comment 124 Ere Maijala 2019-03-08 10:11:01 UTC
Created attachment 86356 [details] [review]
Bug 13937: Add unit tests
Comment 125 Ere Maijala 2019-03-08 10:11:07 UTC
Created attachment 86357 [details] [review]
Bug 13937: (follow-up) Pass through yaz switches and don't ignore case
Comment 126 Ere Maijala 2019-03-08 10:11:12 UTC
Created attachment 86358 [details] [review]
Bug 13937: Fix issues found in QA
Comment 127 Ere Maijala 2019-03-08 10:11:18 UTC
Created attachment 86359 [details] [review]
Bug 13937: Add support for SRU and Elasticsearch

Splits Session to GenericSession and ZebraSession where Generic supports any
search backend via the SearchEngine classes and Zebra maintains the direct
channel to the Zebra server.

Adds config files required for mapping BIB-1 attributes to Koha search fields
and SRU indexes to BIB-1 attributes.

Adds PODs.

Sponsored-by: National Library of Finland
Comment 128 Ere Maijala 2019-03-08 10:11:24 UTC
Created attachment 86360 [details] [review]
Bug 13937: Add tests for search and retrieval

Sponsored-by: National Library of Finland
Comment 129 Ere Maijala 2019-03-08 10:11:30 UTC
Created attachment 86361 [details] [review]
Bug 13937: Fix RPN conversion

Fixes handling of parenthesis and removes unused quote handling.
Comment 130 Ere Maijala 2019-03-08 10:11:35 UTC
Created attachment 86362 [details] [review]
Bug 13937: Add support for Z39.50 responder to the Debian scripts
Comment 131 Ere Maijala 2019-03-08 10:11:41 UTC
Created attachment 86363 [details] [review]
Bug 13937: Fix the daemon log path.

directory _LOG_DIR_/logs does't exists.

All other log4perl -logs are out to _LOG_DIR_

Thank you for the good work Ere!
Comment 132 Ere Maijala 2019-03-08 10:11:47 UTC
Created attachment 86364 [details] [review]
Bug 13937: Switch from GetItem() to Koha::Items->find()
Comment 133 Ere Maijala 2019-03-08 10:12:34 UTC
Fixed the GetItem issue and a couple of pod errors.
Comment 134 Stefan Berndtsson 2019-03-11 10:21:27 UTC
This patch does not apply for me, with the following error:

Applying: Bug 13937: Add a Z39.50 daemon that can inject item status MARC subfields. Fix log path.
error: sha1 information is lacking or useless (etc/log4perl.conf).
error: could not build fake ancestor
Patch failed at 0001 Bug 13937: Add a Z39.50 daemon that can inject item status MARC subfields. Fix log path.
Use 'git am --show-current-patch' to see the failed patch
When you have resolved this problem run "git bz apply --continue".
If you would prefer to skip this patch, instead run "git bz apply --skip".
To restore the original branch and stop patching run "git bz apply --abort".
Patch left in /tmp/11---Bug-13937-Add-a-Z3950-daemon-that-can-inject--6jqZ05.patch



Why are both the first and the second to last patch doing the same change?
Comment 135 Ere Maijala 2019-03-11 10:52:09 UTC
Oops, the first patch was accidentally left over from previous iteration. Should be fixed now.
Comment 136 Stefan Berndtsson 2019-03-12 09:01:13 UTC
(In reply to Ere Maijala from comment #135)
> Oops, the first patch was accidentally left over from previous iteration.
> Should be fixed now.

Figured is was something like that. It applies fine now.

A couple of notes. The test plan says port 9999, but the config file has changed that to 2100.

Also, the test plan doesn't mention anything about the location of the config, nor the location of the pqf.properties (referred to relatively in the config.xml).

It wasn't hard to figure out what to do about it, so no worries there, just thought I'd mention it.
Comment 137 Katrin Fischer 2019-04-03 20:58:36 UTC
(In reply to Stefan Berndtsson from comment #136)
> (In reply to Ere Maijala from comment #135)
> > Oops, the first patch was accidentally left over from previous iteration.
> > Should be fixed now.
> 
> Figured is was something like that. It applies fine now.
> 
> A couple of notes. The test plan says port 9999, but the config file has
> changed that to 2100.
> 
> Also, the test plan doesn't mention anything about the location of the
> config, nor the location of the pqf.properties (referred to relatively in
> the config.xml).
> 
> It wasn't hard to figure out what to do about it, so no worries there, just
> thought I'd mention it.

Hi Stefan, any update on this one? I'd love to see it in 19.05
Comment 138 Michal Denar 2019-04-22 19:39:26 UTC
Ere,
I changed status. We're waiting for follow-up from Ere.
Comment 139 Ere Maijala 2019-04-23 08:37:25 UTC
It's applying fine for me. Where did it fail for you?
Comment 140 Ere Maijala 2019-04-25 12:02:05 UTC
I tested again, and applied cleanly on top of current master.
Comment 141 Stefan Berndtsson 2019-04-26 12:29:24 UTC
Created attachment 88907 [details] [review]
Bug 13937: Add a Z39.50 daemon that can inject item status MARC subfields

This creates a new daemon, misc/z3950_responder.pl, which can respond to
Z39.50 requests. By default, it just proxies searches to Zebra.

If desired, however, it can also add a subfield to the item tags on
outgoing records with a textual description of the item's status
(checked out, lost, etc.). This is useful for certain ILL systems. These
strings can be translated using the 'Z3950_STATUS' authorized value.

Test plan:
  1) Start the Z39.50 server using `perl misc/z3950_responder.pl`.
  2) Connect to the server using `yaz-client 127.0.0.1:9999/biblios`.
  3) Run a search, such as `find @attr 1=1016 book`.
  4) Fetch the results both one at a time with `show 1` and in a batch
     using `show 1+5`.
  5) Turn on MARCXML using `format xml` and `elements marcxml`, and
     verify that the records are still correctly fetched.
  6) Enable the item status subfield by restarting the server with the
     option `--add-item-status=k`.
  7) Search for and fetch records, and verify that a $k subfield is
     added to the item tags as appropriate. It should show some
     combination of "Checked Out", "Lost", "Not For Loan", "Damaged",
     "Withdrawn", "In Transit", or "On Hold" as appropriate, or
     "Available".
  8) Add an authorized value named "Z3950_STATUS" with any of the keys
     "AVAILABLE", "CHECKED_OUT", "LOST", "NOT_FOR_LOAN", "DAMAGED",
     "WITHDRAWN", "IN_TRANSIT" or "ON_HOLD", and verify that their
     descriptions are used instead of the default values above.

Signed-off-by: George Williams <george@nekls.org>
Signed-off-by: Stefan Berndtsson <stefan.berndtsson@ub.gu.se>
Comment 142 Stefan Berndtsson 2019-04-26 12:29:34 UTC
Created attachment 88908 [details] [review]
Bug 13937: (follow-up) Remove reference to sub _prefetch_records

Remove reference to sub _prefetch_records

It appears this sub is built into fetch and this call is a duplicate

Signed-off-by: George Williams <george@nekls.org>
Signed-off-by: Stefan Berndtsson <stefan.berndtsson@ub.gu.se>
Comment 143 Stefan Berndtsson 2019-04-26 12:29:42 UTC
Created attachment 88909 [details] [review]
Bug 13937: (follow-up) Correct error call to use self

Signed-off-by: Stefan Berndtsson <stefan.berndtsson@ub.gu.se>
Comment 144 Stefan Berndtsson 2019-04-26 12:29:52 UTC
Created attachment 88910 [details] [review]
Bug 13937: Add unit tests

Signed-off-by: Stefan Berndtsson <stefan.berndtsson@ub.gu.se>
Comment 145 Stefan Berndtsson 2019-04-26 12:29:59 UTC
Created attachment 88911 [details] [review]
Bug 13937: (follow-up) Pass through yaz switches and don't ignore case

Signed-off-by: Stefan Berndtsson <stefan.berndtsson@ub.gu.se>
Comment 146 Stefan Berndtsson 2019-04-26 12:30:08 UTC
Created attachment 88912 [details] [review]
Bug 13937: Fix issues found in QA

Signed-off-by: Stefan Berndtsson <stefan.berndtsson@ub.gu.se>
Comment 147 Stefan Berndtsson 2019-04-26 12:30:15 UTC
Created attachment 88913 [details] [review]
Bug 13937: Add support for SRU and Elasticsearch

Splits Session to GenericSession and ZebraSession where Generic supports any
search backend via the SearchEngine classes and Zebra maintains the direct
channel to the Zebra server.

Adds config files required for mapping BIB-1 attributes to Koha search fields
and SRU indexes to BIB-1 attributes.

Adds PODs.

Sponsored-by: National Library of Finland
Signed-off-by: Stefan Berndtsson <stefan.berndtsson@ub.gu.se>
Comment 148 Stefan Berndtsson 2019-04-26 12:30:22 UTC
Created attachment 88914 [details] [review]
Bug 13937: Add tests for search and retrieval

Sponsored-by: National Library of Finland
Signed-off-by: Stefan Berndtsson <stefan.berndtsson@ub.gu.se>
Comment 149 Stefan Berndtsson 2019-04-26 12:30:29 UTC
Created attachment 88915 [details] [review]
Bug 13937: Fix RPN conversion

Fixes handling of parenthesis and removes unused quote handling.

Signed-off-by: Stefan Berndtsson <stefan.berndtsson@ub.gu.se>
Comment 150 Stefan Berndtsson 2019-04-26 12:30:36 UTC
Created attachment 88916 [details] [review]
Bug 13937: Add support for Z39.50 responder to the Debian scripts

Signed-off-by: Stefan Berndtsson <stefan.berndtsson@ub.gu.se>
Comment 151 Stefan Berndtsson 2019-04-26 12:30:43 UTC
Created attachment 88917 [details] [review]
Bug 13937: Fix the daemon log path.

directory _LOG_DIR_/logs does't exists.

All other log4perl -logs are out to _LOG_DIR_

Thank you for the good work Ere!

Signed-off-by: Stefan Berndtsson <stefan.berndtsson@ub.gu.se>
Comment 152 Stefan Berndtsson 2019-04-26 12:34:14 UTC
Created attachment 88918 [details] [review]
Bug 13937: Switch from GetItem() to Koha::Items->find()

Signed-off-by: Stefan Berndtsson <stefan.berndtsson@ub.gu.se>
Comment 153 Katrin Fischer 2019-04-26 12:53:28 UTC
Woohoooo! Thx, Stefan! Next time icecream is on me :)
Comment 154 Fridolin SOMERS 2019-04-29 07:39:49 UTC
I see there is pubdate in etc/z3950/attribute_mappings.yaml but since Bug 19575 it is date-of-publication, isn't it ?
Maybe check all changes.
Comment 155 Ere Maijala 2019-04-29 07:43:28 UTC
Actually, it's a bit borked. Bug 19575 missed the change to default mappings file. That will be fixed in bug 22524. I'll check the mappings to match what should be correct.
Comment 156 Alex Arnaud 2019-05-10 14:30:41 UTC
It seems that something breaks the misc/z3950_responder.pl script after the first patch

 - apply only first patch (Bug 13937: Add a Z39.50 daemon that can inject item status MARC subfields),
 - run misc/z3950_responder.pl,
 - sudo netstat -tulpn | grep LISTEN
=> tcp6       0      0 :::9999                 :::*                    LISTEN      8918/perl

 - apply all patches,
 - run misc/z3950_responder.pl,
 - sudo netstat -tulpn | grep LISTEN
No listening on 9999
Comment 157 Alex Arnaud 2019-05-10 14:53:12 UTC
Also, a test plan for SRU would be great
Comment 158 Nick Clemens 2019-05-17 12:14:29 UTC
Excited about this, but too late for 19.05
Comment 159 Alex Arnaud 2019-05-17 14:17:56 UTC
(In reply to Nick Clemens from comment #158)
> Excited about this, but too late for 19.05

Nick,

Can you reproduce the issue on comment #156 ?
Comment 160 Ere Maijala 2019-07-23 07:41:36 UTC
Alex, please see comments #43 - #45. The listening port was changed due to conflict with Zebra. I'll provide a test plan for SRU.
Comment 161 Ere Maijala 2019-07-23 12:29:21 UTC
Created attachment 91728 [details] [review]
Bug 13937: Add a Z39.50 daemon that can inject item status MARC subfields

This creates a new daemon, misc/z3950_responder.pl, which can respond to
Z39.50 requests. By default, it just proxies searches to Zebra.

If desired, however, it can also add a subfield to the item tags on
outgoing records with a textual description of the item's status
(checked out, lost, etc.). This is useful for certain ILL systems. These
strings can be translated using the 'Z3950_STATUS' authorized value.

Test plan:
  1) Start the Z39.50 server using `perl misc/z3950_responder.pl`.
  2) Connect to the server using `yaz-client 127.0.0.1:9999/biblios`.
  3) Run a search, such as `find @attr 1=1016 book`.
  4) Fetch the results both one at a time with `show 1` and in a batch
     using `show 1+5`.
  5) Turn on MARCXML using `format xml` and `elements marcxml`, and
     verify that the records are still correctly fetched.
  6) Enable the item status subfield by restarting the server with the
     option `--add-item-status=k`.
  7) Search for and fetch records, and verify that a $k subfield is
     added to the item tags as appropriate. It should show some
     combination of "Checked Out", "Lost", "Not For Loan", "Damaged",
     "Withdrawn", "In Transit", or "On Hold" as appropriate, or
     "Available".
  8) Add an authorized value named "Z3950_STATUS" with any of the keys
     "AVAILABLE", "CHECKED_OUT", "LOST", "NOT_FOR_LOAN", "DAMAGED",
     "WITHDRAWN", "IN_TRANSIT" or "ON_HOLD", and verify that their
     descriptions are used instead of the default values above.

Signed-off-by: George Williams <george@nekls.org>
Signed-off-by: Stefan Berndtsson <stefan.berndtsson@ub.gu.se>
Comment 162 Ere Maijala 2019-07-23 12:29:28 UTC
Created attachment 91729 [details] [review]
Bug 13937: (follow-up) Remove reference to sub _prefetch_records

Remove reference to sub _prefetch_records

It appears this sub is built into fetch and this call is a duplicate

Signed-off-by: George Williams <george@nekls.org>
Signed-off-by: Stefan Berndtsson <stefan.berndtsson@ub.gu.se>
Comment 163 Ere Maijala 2019-07-23 12:29:35 UTC
Created attachment 91730 [details] [review]
Bug 13937: (follow-up) Correct error call to use self

Signed-off-by: Stefan Berndtsson <stefan.berndtsson@ub.gu.se>
Comment 164 Ere Maijala 2019-07-23 12:29:45 UTC
Created attachment 91731 [details] [review]
Bug 13937: Add unit tests

Signed-off-by: Stefan Berndtsson <stefan.berndtsson@ub.gu.se>
Comment 165 Ere Maijala 2019-07-23 12:29:53 UTC
Created attachment 91732 [details] [review]
Bug 13937: (follow-up) Pass through yaz switches and don't ignore case

Signed-off-by: Stefan Berndtsson <stefan.berndtsson@ub.gu.se>
Comment 166 Ere Maijala 2019-07-23 12:30:00 UTC
Created attachment 91733 [details] [review]
Bug 13937: Fix issues found in QA

Signed-off-by: Stefan Berndtsson <stefan.berndtsson@ub.gu.se>
Comment 167 Ere Maijala 2019-07-23 12:30:08 UTC
Created attachment 91734 [details] [review]
Bug 13937: Add support for SRU and Elasticsearch

Splits Session to GenericSession and ZebraSession where Generic supports any
search backend via the SearchEngine classes and Zebra maintains the direct
channel to the Zebra server.

Adds config files required for mapping BIB-1 attributes to Koha search fields
and SRU indexes to BIB-1 attributes.

Adds PODs.

Sponsored-by: National Library of Finland
Signed-off-by: Stefan Berndtsson <stefan.berndtsson@ub.gu.se>
Comment 168 Ere Maijala 2019-07-23 12:30:14 UTC
Created attachment 91735 [details] [review]
Bug 13937: Add tests for search and retrieval

Sponsored-by: National Library of Finland
Signed-off-by: Stefan Berndtsson <stefan.berndtsson@ub.gu.se>
Comment 169 Ere Maijala 2019-07-23 12:30:20 UTC
Created attachment 91736 [details] [review]
Bug 13937: Fix RPN conversion

Fixes handling of parenthesis and removes unused quote handling.

Signed-off-by: Stefan Berndtsson <stefan.berndtsson@ub.gu.se>
Comment 170 Ere Maijala 2019-07-23 12:30:31 UTC
Created attachment 91737 [details] [review]
Bug 13937: Add support for Z39.50 responder to the Debian scripts

Signed-off-by: Stefan Berndtsson <stefan.berndtsson@ub.gu.se>
Comment 171 Ere Maijala 2019-07-23 12:30:40 UTC
Created attachment 91738 [details] [review]
Bug 13937: Fix the daemon log path.

directory _LOG_DIR_/logs does't exists.

All other log4perl -logs are out to _LOG_DIR_

Thank you for the good work Ere!

Signed-off-by: Stefan Berndtsson <stefan.berndtsson@ub.gu.se>
Comment 172 Ere Maijala 2019-07-23 12:30:46 UTC
Created attachment 91739 [details] [review]
Bug 13937: Switch from GetItem() to Koha::Items->find()

Signed-off-by: Stefan Berndtsson <stefan.berndtsson@ub.gu.se>
Comment 173 Ere Maijala 2019-07-23 12:30:53 UTC
Created attachment 91740 [details] [review]
Bug 13937: Add SRU tests
Comment 174 Ere Maijala 2019-07-23 12:31:31 UTC
Test plan for SRU:

1. Make sure tests in t/db_dependent/Koha/Z3950Responder/GenericSession.t pass.
2. Use Elasticsearch as the search engine
3. Start the server: perl misc/z3950_responder.pl --debug
4. Check that you get an explain response with browser from http://localhost:2100/
5. Try some searches to verify that you get results and good search queries in the output of the daemon . Examples:

http://localhost:2100/biblios?operation=searchRetrieve&maximumRecords=10&recordSchema=marcxml&query=dc.author%3dsmith
http://localhost:2100/biblios?operation=searchRetrieve&maximumRecords=10&recordSchema=marcxml&query=dc.title%3dnew AND dc.author%3dsmith
http://localhost:2100/biblios?operation=searchRetrieve&maximumRecords=10&recordSchema=marcxml&query=dc.title+cql.eq+"new world"+AND+dc.creator=smith