Bug 8280

Summary: SRU should be filterable by Koha Item Type
Product: Koha Reporter: Martin Renvoize <martin.renvoize>
Component: Z39.50 / SRU / OpenSearch ServersAssignee: Nick Clemens <nick>
Status: RESOLVED FIXED QA Contact: Marcel de Rooy <m.de.rooy>
Severity: enhancement    
Priority: P5 - low CC: katrin.fischer, kyle, m.de.rooy, magnus, marjorie.barry-vila, nick
Version: master   
Hardware: All   
OS: All   
Change sponsored?: --- Patch complexity: Small patch
Text to go in the release notes:
Version(s) released in:
21.11.00
Attachments: Bug 8280: Add koha set and item fields to pqf.properties
Bug 8280: Add koha set and item fields to pqf.properties
Bug 8280: Add koha set and item fields to pqf.properties
Bug 8280: Add koha set and item fields to pqf.properties
Bug 8280: Sync whitespace differences between pqf config files
Bug 8280: Add koha set and item fields to pqf.properties

Description Martin Renvoize 2012-06-21 12:03:12 UTC
It would be very useful to be able to query the SRU server by the current index's in Koha, including the Item Type Index.

My understanding is the standard CQL queries should work, so:

:9998/biblios?version=1.1&operation=searchRetrieve&query=itype=XXX

should yeild results conatining all records of itype (952$y in Marc21 Koha) XXX, but I have not been able to get this to work.
Comment 1 Marcel de Rooy 2012-06-21 12:24:56 UTC
You are probably right that there is no itype reference in pqf.properties.
Can you still search on itemtype=... ? It could just be referring to 8031 itype..
Note that we have 8031 and 9006 for itype and itemtype in attset.
If itype and itemtype are also in your record.abs, it may just be a matter of defining it too in pqf.properties.

Test it please, and submit a patch!
Comment 2 Martin Renvoize 2012-06-21 12:50:43 UTC
(In reply to comment #1)
> You are probably right that there is no itype reference in pqf.properties.
> Can you still search on itemtype=... ? It could just be referring to 8031
> itype..
> Note that we have 8031 and 9006 for itype and itemtype in attset.
> If itype and itemtype are also in your record.abs, it may just be a matter
> of defining it too in pqf.properties.
> 
> Test it please, and submit a patch!

Magnus over in the IRC channel just took the time to explain how the zebra config all fits together and has solved the problem for me. 

I'll submit the patch once I've worked out where it fits for the installer (as obviously the config files being sued arn't git aware.

Cheers
Comment 3 Magnus Enger 2012-06-21 12:59:04 UTC
Here is a quick writeup of what we figured out on IRC: 
http://wiki.koha-community.org/wiki/Making_more_index_searchable_with_Zebra_SRU
Comment 4 Marcel de Rooy 2012-06-21 13:00:42 UTC
(In reply to comment #3)
> Here is a quick writeup of what we figured out on IRC: 
> http://wiki.koha-community.org/wiki/
> Making_more_index_searchable_with_Zebra_SRU

Great. You also add it in pqf.properties. Mind the typo. I would not put in dublin core context either..
Comment 5 Magnus Enger 2012-06-21 13:11:47 UTC
By all means, please feel free to improve the wiki page!
Comment 6 Martin Renvoize 2012-06-21 13:50:58 UTC
(In reply to comment #4)
> (In reply to comment #3)
> > Here is a quick writeup of what we figured out on IRC: 
> > http://wiki.koha-community.org/wiki/
> > Making_more_index_searchable_with_Zebra_SRU
> 
> Great. You also add it in pqf.properties. Mind the typo. I would not put in
> dublin core context either..

So, what exactly are the contexts and how do they work?

I just tried adding a "local" context by adding the line:

set.local   = info:srw/cql-context-set/1/cql-v1.1

or even

set.local   = info:srw/cql-context-set/1/cql-v1.2

These don't work as expected, but that's not surprising as it was a stab in the dark.  Can you point to any documentation on contexts Marcel?
Comment 7 Martin Renvoize 2012-06-21 13:52:03 UTC
(In reply to comment #6)
> (In reply to comment #4)
> > (In reply to comment #3)
> > > Here is a quick writeup of what we figured out on IRC: 
> > > http://wiki.koha-community.org/wiki/
> > > Making_more_index_searchable_with_Zebra_SRU
> > 
> > Great. You also add it in pqf.properties. Mind the typo. I would not put in
> > dublin core context either..
> 
> So, what exactly are the contexts and how do they work?
> 
> I just tried adding a "local" context by adding the line:
> 
> set.local   = info:srw/cql-context-set/1/cql-v1.1
> 
> or even
> 
> set.local   = info:srw/cql-context-set/1/cql-v1.2
> 
> These don't work as expected, but that's not surprising as it was a stab in
> the dark.  Can you point to any documentation on contexts Marcel?

Not sure Marcel came form there...  ?  M. de Roy was what I meant to say...
Comment 8 Marcel de Rooy 2012-06-22 06:31:07 UTC
(In reply to comment #7)
> Not sure Marcel came form there...  ?  M. de Roy was what I meant to say...
M=Marcel; Roy->Rooy
Comment 9 Marcel de Rooy 2012-06-22 06:38:13 UTC
(In reply to comment #6)
> So, what exactly are the contexts and how do they work?
> 
> I just tried adding a "local" context by adding the line:
> 
> set.local   = info:srw/cql-context-set/1/cql-v1.1
> 
> or even
> 
> set.local   = info:srw/cql-context-set/1/cql-v1.2
> 
> These don't work as expected, but that's not surprising as it was a stab in
> the dark.  Can you point to any documentation on contexts Marcel?

Please see: http://www.indexdata.com/yaz/doc/tools.html
Personally, I would not define a context set named local. (But maybe that's just being too cautious about colliding with reserved words etc.)
Comment 10 Magnus Enger 2017-02-27 12:52:41 UTC
In etc/zebradb/pqf.properties we have:
index.dc.itemtype           = 1=1031

In etc/zebradb/biblios/etc/bib1.att we have: 
att 1031    Material-type

But "Material-type" is only used to index 007 and 502. 

952$y is actually indexed in an inex called itype. In bib1.att this is mapped to: 
att 8031    itype

But 8031 is not mapped to anything in pqf.properties.

So, what to do? To me, the best option looks like indexing 952$y into Material-type, so it will be possible to search on it using dc.itemtype. Opinions?
Comment 11 Marcel de Rooy 2017-02-27 13:16:44 UTC
(In reply to Magnus Enger from comment #10)
> So, what to do? To me, the best option looks like indexing 952$y into
> Material-type, so it will be possible to search on it using dc.itemtype.
> Opinions?
No 952y is no material type.
Is this the case where itype and itemtype collide?
Comment 12 Magnus Enger 2017-02-27 13:28:38 UTC
(In reply to Marcel de Rooy from comment #11)
> No 952y is no material type.

I don't understand why it isn't then. Could you explain? 

> Is this the case where itype and itemtype collide?

Dunno. :-)
Comment 13 Marcel de Rooy 2017-02-27 14:05:59 UTC
(In reply to Magnus Enger from comment #12)
> (In reply to Marcel de Rooy from comment #11)
> > No 952y is no material type.
> 
> I don't understand why it isn't then. Could you explain? 

Hmm. In my configs I always change dc.itemtype from 1031 to 8031.
Material-type is indexed on base of 007 and 502.
Never submitted a patch for that..

Note that you have both itemtype and itype in the index defs (942c vs 952y).
Comment 14 Nick Clemens 2021-06-11 12:22:06 UTC
Created attachment 121822 [details] [review]
Bug 8280: Add koha set and item fields to pqf.properties

This patch adds a 'koha' set to the pqf properties and maps some basic
koha fields to all for searchign

To test:
1 - Apply patch
2 - cp etc/z3950/pqf.properties /etc/koha/sites/kohadev/z3950/pqf.properties
3 - sudo koha-z3950-responder --restart kohadev
4 - Test a search:
    curl -XGET "http://localhost:2100/biblios?version=1.1&operation=searchRetrieve&query=koha.itemtype=BK&maximumRecords=60&recordSchema=marcxml"
5 - Test other fields added:
    koha.withdrawn
    koha.lost
    koha.classification-source
    koha.materials-specified
    koha.damaged
    koha.restricted
    koha.cn-sort
    koha.notforloan
    koha.ccode
    koha.itemnumber
    koha.homebranch
    koha.holdingbranch
    koha.location
    koha.barcode
    koha.onloan
    koha.itemtype
Comment 15 Martin Renvoize 2021-06-15 14:07:44 UTC
Created attachment 122009 [details] [review]
Bug 8280: Add koha set and item fields to pqf.properties

This patch adds a 'koha' set to the pqf properties and maps some basic
koha fields to all for searchign

To test:
1 - Apply patch
2 - cp etc/z3950/pqf.properties /etc/koha/sites/kohadev/z3950/pqf.properties
3 - sudo koha-z3950-responder --restart kohadev
4 - Test a search:
    curl -XGET "http://localhost:2100/biblios?version=1.1&operation=searchRetrieve&query=koha.itemtype=BK&maximumRecords=60&recordSchema=marcxml"
5 - Test other fields added:
    koha.withdrawn
    koha.lost
    koha.classification-source
    koha.materials-specified
    koha.damaged
    koha.restricted
    koha.cn-sort
    koha.notforloan
    koha.ccode
    koha.itemnumber
    koha.homebranch
    koha.holdingbranch
    koha.location
    koha.barcode
    koha.onloan
    koha.itemtype

Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Comment 16 Martin Renvoize 2021-06-15 14:09:21 UTC
This all appears to work as expected, so signing off..

Must admit though.. allot of the knowledge of how this stuff all fits together has now left my brain, so it would be good to get a QA from someone more familiar ;)
Comment 17 Marcel de Rooy 2021-06-17 14:07:11 UTC
Could you cleanup debug remainders from itemsearch or remove it completely from the patch?

+index.koha.itemtype              = 1=8031

Should this be itype instead of itemtype ?
There is a difference: biblioitems.itemtype or items.itype
Comment 18 Nick Clemens 2021-06-17 14:17:51 UTC
Created attachment 122082 [details] [review]
Bug 8280: Add koha set and item fields to pqf.properties

This patch adds a 'koha' set to the pqf properties and maps some basic
koha fields to all for searchign

To test:
1 - Apply patch
2 - cp etc/z3950/pqf.properties /etc/koha/sites/kohadev/z3950/pqf.properties
3 - sudo koha-z3950-responder --restart kohadev
4 - Test a search:
    curl -XGET "http://localhost:2100/biblios?version=1.1&operation=searchRetrieve&query=koha.itemtype=BK&maximumRecords=60&recordSchema=marcxml"
5 - Test other fields added:
    koha.withdrawn
    koha.lost
    koha.classification-source
    koha.materials-specified
    koha.damaged
    koha.restricted
    koha.cn-sort
    koha.notforloan
    koha.ccode
    koha.itemnumber
    koha.homebranch
    koha.holdingbranch
    koha.location
    koha.barcode
    koha.onloan
    koha.itemtype

Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Comment 19 Nick Clemens 2021-06-17 14:22:40 UTC
(In reply to Marcel de Rooy from comment #17)
> Could you cleanup debug remainders from itemsearch or remove it completely
> from the patch?

^^ I did a 'git commit -a' when I meant 'git commit' - extra changes removed

> +index.koha.itemtype              = 1=8031
> 
> Should this be itype instead of itemtype ?
> There is a difference: biblioitems.itemtype or items.itype

I think 'itemtype' makes more sense externally - the 'itype'/8031 index contains both the 942$c and the 952$y, so the results here should be as expected.
Comment 20 Marcel de Rooy 2021-06-18 06:13:29 UTC
diff --git a/catalogue/itemsearch.pl b/catalogue/itemsearch.pl
index 0329f4b1d8..07cc591c93 100755
--- a/catalogue/itemsearch.pl
+++ b/catalogue/itemsearch.pl
@@ -254,10 +254,8 @@ if ( defined $format ) {
         $template->param(sEcho => scalar $cgi->param('sEcho'));
         output_with_http_headers $cgi, $cookie, $template->output, 'json';
     }
-
     exit;
 }
-
 # Display the search form

 my @branches = map { value => $_->branchcode, label => $_->branchname }, Koha::Libraries->search( {}, { order_by => 'branchname' } );
@@ -309,5 +307,4 @@ $template->param(
     items_search_fields => \@items_search_fields,
     authorised_values_json => to_json($authorised_values),
 );
-
 output_html_with_http_headers $cgi, $cookie, $template->output;

Strange tool git, heh ?
Comment 21 Marcel de Rooy 2021-06-18 06:14:32 UTC
Created attachment 122099 [details] [review]
Bug 8280: Add koha set and item fields to pqf.properties

This patch adds a 'koha' set to the pqf properties and maps some basic
koha fields to all for searchign

To test:
1 - Apply patch
2 - cp etc/z3950/pqf.properties /etc/koha/sites/kohadev/z3950/pqf.properties
3 - sudo koha-z3950-responder --restart kohadev
4 - Test a search:
    curl -XGET "http://localhost:2100/biblios?version=1.1&operation=searchRetrieve&query=koha.itemtype=BK&maximumRecords=60&recordSchema=marcxml"
5 - Test other fields added:
    koha.withdrawn
    koha.lost
    koha.classification-source
    koha.materials-specified
    koha.damaged
    koha.restricted
    koha.cn-sort
    koha.notforloan
    koha.ccode
    koha.itemnumber
    koha.homebranch
    koha.holdingbranch
    koha.location
    koha.barcode
    koha.onloan
    koha.itemtype

Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Comment 22 Marcel de Rooy 2021-06-18 06:22:27 UTC
(In reply to Nick Clemens from comment #19)
> I think 'itemtype' makes more sense externally - the 'itype'/8031 index
> contains both the 942$c and the 952$y, so the results here should be as
> expected.

No blocker for me, but it is confusing too to map itemtype to the internal itype index. Confusion creates bugs.
If only we had e.g. defaultitemtype and itemtype and we could get rid of itype ;)
Comment 23 Marcel de Rooy 2021-06-18 06:30:03 UTC
And now the big question: How would you get this to work if you put it in etc/z3950 ?
It should be in etc/zebradb !
Comment 24 Marcel de Rooy 2021-06-18 06:33:39 UTC
(In reply to Marcel de Rooy from comment #23)
> And now the big question: How would you get this to work if you put it in
> etc/z3950 ?
> It should be in etc/zebradb !

Probably both. The new daemon is not that on my radar yet..
Comment 25 Marcel de Rooy 2021-06-18 06:54:51 UTC
Created attachment 122100 [details] [review]
Bug 8280: Sync whitespace differences between pqf config files

Test plan:
Diff etc/zebradb/pqf.properties with the etc/z3950 one.

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Comment 26 Marcel de Rooy 2021-06-18 06:54:54 UTC
Created attachment 122101 [details] [review]
Bug 8280: Add koha set and item fields to pqf.properties

This patch adds a 'koha' set to the pqf properties and maps some basic
koha fields to all for searchign

To test:
1 - Apply patch
2 - cp etc/z3950/pqf.properties /etc/koha/sites/kohadev/z3950/pqf.properties
3 - sudo koha-z3950-responder --restart kohadev
4 - Test a search:
    curl -XGET "http://localhost:2100/biblios?version=1.1&operation=searchRetrieve&query=koha.itemtype=BK&maximumRecords=60&recordSchema=marcxml"
5 - Test other fields added:
    koha.withdrawn
    koha.lost
    koha.classification-source
    koha.materials-specified
    koha.damaged
    koha.restricted
    koha.cn-sort
    koha.notforloan
    koha.ccode
    koha.itemnumber
    koha.homebranch
    koha.holdingbranch
    koha.location
    koha.barcode
    koha.onloan
    koha.itemtype

Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
[EDIT] Copied the changes from z3950 to zebradb too.
Comment 27 Jonathan Druart 2021-06-21 12:03:19 UTC
Pushed to master for 21.11, thanks to everybody involved!
Comment 28 Kyle M Hall 2021-07-02 13:47:02 UTC
Enhancement not backported to 21.05.x