Note that Pazpar2 must be (re)installed using the new configuration for this patch to work successfully.

Applying: Bug 10486 - Allow external Z39.50 targets to be searched from the OPAC (1/2)
fatal: sha1 information is lacking or useless (koha-tmpl/opac-tmpl/prog/en/css/opac.css).
Repository lacks necessary blobs to fall back on 3-way merge.
Cannot fall back to three-way merge.

I'm having trouble applying these patches, I get 

Repository lacks necessary blobs to fall back on 3-way merge.

and I did all of the fetches and other gymnastics that often fix this, with no luck.

Liz

I am sorry, but this patch does no longer apply. Please rebase and switch back to Signed off.

Hi Jesse, any news on this? :)

Phew, new version, updated to apply on master and with a couple small bugfixes.

Patch applied cleanly, go forth and signoff

Trying to test.
Is there any documentation of how to enable/configure pazpar2?
I'm to the point that I need to know where to find "/pazpar2/search.pz2" file
referenced in pazpar2_init on line 72

(In reply to Bernardo Gonzalez Kriegel from comment #15)
> Trying to test.
> Is there any documentation of how to enable/configure pazpar2?
> I'm to the point that I need to know where to find "/pazpar2/search.pz2" file
> referenced in pazpar2_init on line 72

Ah, yes. The installer's template configuration files have been updated, but here's the changes you need to make to an existing install (none of these will break existing functionality):

  * Change the two lines in the Apache .conf that read:
    
    <DirectoryMatch "/home/feoh3/p/koha/koha-tmpl/.*/(modules|xslt|includes)">

    to read:

    <DirectoryMatch "/home/feoh3/p/koha/koha-tmpl/.*/(modules|includes)">

  * Make sure that the proxy and proxy-http modules are enabled in Apache.
  * If you haven't already, install Pazpar2:
    * Get the packages installed.
    * Copy the configuration files from the Koha source to someplace convenient.
    * Modify pazpar2.xml to listen on an open port and point at the directory you just installed the files to.
    * Change __ZEBRA_MARC_FORMAT__ in koha-biblios.xml or generic-settings.xml to marc21 or unimarc as appropriate.
  * Change the first line in koha-biblios.xml file (if you have it) in the Pazpar2 directory to read:
    <settings target="*">
  * Copy the new *-work-groups.xsl files to your Pazpar2 directory.
  * Start pazpar2.
  * Add the following lines to your Apache configuration under your OPAC VirtualHost:

    <Proxy *>
        AddDefaultCharset off
        Order deny,allow
        Allow from all
    </Proxy>

    ProxyRequests off
    ProxyPass /pazpar2/search.pz2 http://localhost:PORT/search.pz2

    replacing PORT with the port you configured Pazpar2 to listen on.
  * Restart Apache.

2nd try

I did a clean install of master+10486.

When starting pazpar2 I get

20131020 19:58:03 koha-pazpar2-ctl.kohadev: client (pid 516) exited with 1 status
20131020 19:58:03 koha-pazpar2-ctl.kohadev: client (pid 517) exited with 1 status
20131020 19:58:03 koha-pazpar2-ctl.kohadev: client (pid 518) exited with 1 status
20131020 19:58:03 koha-pazpar2-ctl.kohadev: client (pid 519) exited with 1 status
20131020 19:58:03 koha-pazpar2-ctl.kohadev: client (pid 520) exited with 1 status
20131020 19:58:03 koha-pazpar2-ctl.kohadev: terminating too quickly, waiting 30 seconds

on koha-pazpar2daemon.err, and

</icu_chain>
19:58:03-20/10 pazpar2 [fatal] ICU chain definition error
19:58:03-20/10 pazpar2 [warn] Element normalize is deprecated. Use transform instead
19:58:03-20/10 pazpar2 [warn] Element normalize is deprecated. Use transform instead
19:58:03-20/10 pazpar2 [warn] Missing attribute rule for element index
19:58:03-20/10 pazpar2 [fatal] Could not parse ICU chain config:
<icu_chain>
 ... 

on koha-pazpar2daemon-output.log

Q: Is there any documentation of how to enable/configure pazpar2?

Applying: Bug 10486 - Allow external Z39.50 targets to be searched from the OPAC
Using index info to reconstruct a base tree...
M	C4/Search.pm
M	Makefile.PL
M	installer/data/mysql/sysprefs.sql
M	installer/data/mysql/updatedatabase.pl
M	koha-tmpl/intranet-tmpl/prog/en/css/staff-global.css
Falling back to patching base and 3-way merge...
Auto-merging koha-tmpl/intranet-tmpl/prog/en/css/staff-global.css
Auto-merging installer/data/mysql/updatedatabase.pl
CONFLICT (content): Merge conflict in installer/data/mysql/updatedatabase.pl
Auto-merging installer/data/mysql/sysprefs.sql
Auto-merging Makefile.PL
Auto-merging C4/Search.pm
Patch failed at 0001 Bug 10486 - Allow external Z39.50 targets to be searched from the OPAC
The copy of the patch that failed is found in:
   /home/christopher/git/koha/.git/rebase-apply/patch
When you have resolved this problem, run "git am --continue".
If you prefer to skip this patch, run "git am --skip" instead.
To restore the original branch and stop patching, run "git am --abort".

Applying: Bug 10486 - Allow external Z39.50 targets to be searched from the OPAC
Using index info to reconstruct a base tree...
M	C4/Search.pm
M	installer/data/mysql/sysprefs.sql
M	installer/data/mysql/updatedatabase.pl
M	koha-tmpl/intranet-tmpl/prog/en/css/staff-global.css
M	koha-tmpl/intranet-tmpl/prog/en/modules/admin/admin-home.tt
Falling back to patching base and 3-way merge...
Auto-merging koha-tmpl/intranet-tmpl/prog/en/modules/admin/admin-home.tt
Auto-merging koha-tmpl/intranet-tmpl/prog/en/css/staff-global.css
Auto-merging installer/data/mysql/updatedatabase.pl
CONFLICT (content): Merge conflict in installer/data/mysql/updatedatabase.pl
Auto-merging installer/data/mysql/sysprefs.sql
Auto-merging C4/Search.pm
Patch failed at 0001 Bug 10486 - Allow external Z39.50 targets to be searched from the OPAC
The copy of the patch that failed is found in:
   /home/christopher/git/koha/.git/rebase-apply/patch
When you have resolved this problem, run "git am --continue".
If you prefer to skip this patch, run "git am --skip" instead.
To restore the original branch and stop patching, run "git am --abort".

Have rebased, and the functionality seems to be working. I have not tested enough to add a sign off. I would love another set of eyes. However I have noticed a few little issues for a follow up.

Will add in the splinter review. 

I have tested that this does not break any existing functionality, so I will sign off when the issues are fixed as that is always the main thing with any new feature.

Comment on attachment 23112 [details] [review]
Bug 10486 - Allow external Z39.50 targets to be searched from the OPAC

Review of attachment 23112 [details] [review]:
-----------------------------------------------------------------

::: C4/XSLT.pm
@@ +42,4 @@
>      $VERSION = 3.07.00.049;
>      @ISA = qw(Exporter);
>      @EXPORT = qw(
> +        &XSLTGetFilename

We should add a unit test for this method

::: koha-tmpl/opac-tmpl/lib/pz2.js
@@ +1,2 @@
> +/*
> + * $Id: 3a9980787bb5d0fe966140b243a4a5eb6768913e $

What license is this under? Where did we get it from?

::: opac/opac-external-search.pl
@@ +14,5 @@
> +# A PARTICULAR PURPOSE.  See the GNU General Public License for more details.
> +#
> +# You should have received a copy of the GNU General Public License along
> +# with Koha; if not, write to the Free Software Foundation, Inc.,
> +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.

License and copyright header, These aren't correct are they?

::: opac/svc/pazpar2_init
@@ +6,5 @@
> +#
> +# Koha is free software; you can redistribute it and/or modify it under the
> +# terms of the GNU General Public License as published by the Free Software
> +# Foundation; either version 2 of the License, or (at your option) any later
> +# version.

We use GPL3+ now

Failing QA to get attention for comment 26

Seems to work fine, last thing needed is the bootstrap templates. Then I'm happy to sign off.

Comment on attachment 24206 [details] [review]
Bug 10486 - Allow external Z39.50 targets to be searched from the OPAC

Review of attachment 24206 [details] [review]:
-----------------------------------------------------------------

Hi Jesse, I got curious and took a look at the code. Looks mostly good (as far as I can tell), but I spotted some translation issues in the templates. Could you take a look please?

::: C4/Search.pm
@@ +72,4 @@
>    &GetDistinctValues
>    &enabled_staff_search_views
>    &PurgeSearchHistory
> +  &GetExternalSearchTargets

New routine - being super strict I would expect some unit tests :)

::: koha-tmpl/intranet-tmpl/prog/en/modules/admin/external-targets.tt
@@ +27,5 @@
> +    $( '#targets tr[data-targetid=[% saved_id %]]' ).addClass( 'updated' );
> +    [% END %]
> +
> +    $( '#targets .delete' ).click( function() {
> +        return confirm( _( 'Are you sure you wish to delete this target?' ) );

Please use "" else a translation using ' can break the Javascript code.

::: koha-tmpl/opac-tmpl/prog/en/modules/opac-external-search.tt
@@ +67,5 @@
> +    } );
> +}
> +
> +function search( offset, reset_search ) {
> +    $( '#pazpar2-status' ).html( 'Searching external targets... <img class="throbber" src="/opac-tmpl/lib/jquery/plugins/themes/classic/throbber.gif" /></span>' );

'Searching external targets' is untranslatable. Marking all spotted translation problems with 'T'.

@@ +77,5 @@
> +    }
> +
> +    function callback( data ) {
> +        if ( data.error ) {
> +            $( '#pazpar2-status' ).html( '<strong class="unavailable">Error searching external targets.</strong>' );

T

@@ +82,5 @@
> +            return;
> +        }
> +
> +        if ( !data.total ) {
> +            $( '#pazpar2-status' ).html( '<strong>No results found in the external targets.</strong>' );

T

@@ +88,5 @@
> +        }
> +
> +        $( '#results tbody' ).empty();
> +
> +        $( '#pazpar2-status' ).html( '<strong>' + _( 'Found ' ) + data.total + _( ' results in ' ) + $( '#targets-facet input:checked' ).length + _( ' external targets.' ) + '</strong>' );

T (single quotes). Also a bit hard to translate, as this will be split up in the translation file.

@@ +104,5 @@
> +
> +            if ( resultRenderCache[ hit.recid[0] ] ) {
> +                results.push( resultRenderCache[ hit.recid[0] ] );
> +            } else {
> +                results.push( hit['md-work-title'] ? hit['md-work-title'][0] : 'Loading...' );

T

@@ +131,5 @@
> +        var cur_page = data.start / results_per_page;
> +        var max_page = Math.floor( data.total / results_per_page );
> +
> +        if ( cur_page != 0 ) {
> +            pages.push( '<a class="nav" href="#" data-offset="' + (offset - results_per_page) + '">&lt;&lt; Previous</a>' );

T

@@ +143,5 @@
> +            }
> +        }
> +
> +        if ( cur_page < max_page ) {
> +            pages.push( ' <a class="nav" href="#" data-offset="' + (offset + results_per_page) + '">Next >></a>' );

T

::: koha-tmpl/opac-tmpl/prog/en/modules/opac-results.tt
@@ +319,4 @@
>              }
> +
> +            first_succeeded = true;
> +            FinishExternalSearch( 'pazpar2', _("Found __LINK__ in ") + num_targets + _(" external targets"), data.total, '/cgi-bin/koha/opac-external-search.pl?q=' + escape( querystring ) );

This looks interesting. Maybe a replacement like __LINK__ could make translating the sentences easier.

::: opac/opac-external-search.pl
@@ +17,5 @@
> +# with Koha; if not, write to the Free Software Foundation, Inc.,
> +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
> +
> +use strict;
> +use warnings;

Could be using Modern Perl instead.

::: opac/opac-search.pl
@@ +884,5 @@
>  
> +$template->{VARS}->{OPACSearchExternalTargets} = C4::Context->preference('OPACSearchExternalTargets');
> +$template->{VARS}->{external_search_targets} = GetExternalSearchTargets( C4::Context->userenv ? C4::Context->userenv->{branch} : '' );
> +
> +$template->{VARS}->{OverDriveLibraryID} = C4::Context->preference('OverDriveLibraryID');

I think some of those lines could now be replaced using the TT plugin for looking up the system preferences from the template. Just noting, it's ok to leave it.

In the patch I don't find a patch for 
 installer/data/mysql/kohastructure.sql, so the feature works only when you update and installation, not with a new install.

And now we need also the DBIC schema classes in Koha/Schema/

It's not mandatory for the developer to generate the Schema right now, but when you do please add it as a separate patch. 
Good catch for kohastructure.sql!

These are some notes cait put together after we got pazpar2 working on her machine, which might be useful for any testers (note, all of these should be done after applying the patches):

Setting up Pazpar2 in Debian
============================

# Add indexdata repository
sudo vi /etc/apt/sources.list.d/indexdata.list 
deb http://ftp.indexdata.dk/debian wheezy main
deb-src http://ftp.indexdata.dk/debian wheezy main

wget http://ftp.indexdata.dk/debian/indexdata.asc
sudo apt-key add indexdata.asc
sudo apt-get update

# Install pazpar2
sudo apt-get install pazpar2

Set up Pazpar2
==============

If you enabled pazpar2 before, just run update_zebra_conf. Otherwise, you can either:

* Rerun perl Makefile.PL (saying yes to the Pazpar2 questions); make; make install
* Or, just copy the etc/pazpar2 directory from your git clone to your Koha install directory (koha-dev/etc) and fix all of the __UPPERCASE_PLACEHOLDERS__.

Either way, use the defaults (host localhost, port 11002).

Reconfigure Apache for Pazpar2
==============================

# Add proxy in OPAC koha-httpd.conf
    <Proxy *>
        AddDefaultCharset off
        Order deny,allow
        Allow from all
    </Proxy>

Change the DirectoryMatch lines, removing `xslt`:
<DirectoryMatch "/home/katrin/kohaclone/koha-tmpl/.*/(modules|includes)">

# Restart apache
sudo service apache2 restart

# Start Pazpar2 (may be at different path)
/usr/sbin/pazpar2 -f koha-dev/etc/pazpar2/pazpar2.xml
# If you get a message about <index>, you did not install the new configuration files.

Testing
=======
# Add external search target in Administration
GUI
SWB: z3950n.bsz-bw.de:20210/swb 
LOC: lx2.loc.gov:210/LCDB

# Activate system preferences
# Run a search on the OPAC

I'm trying to install pazpar2 version 1.6.40 from source, but it looks like it requires YAZ 4.2.68 or later.

It looks like the YAZ package in the Debian Wheezy repo is 4.2.30, and my non-Debian systems are a bit older than that.

Might I be missing something?

(In reply to David Cook from comment #45)
> I'm trying to install pazpar2 version 1.6.40 from source, but it looks like
> it requires YAZ 4.2.68 or later.
> 
> It looks like the YAZ package in the Debian Wheezy repo is 4.2.30, and my
> non-Debian systems are a bit older than that.
> 
> Might I be missing something?

Try using an older version of pazpar2; I believe 1.6.32 should work with older versions.

I think updatedatabase.pl needs an update. The web installer isn't being triggered after applying the patches.

I might take a look at yaz again...

(In reply to David Cook from comment #52)
> I think updatedatabase.pl needs an update. The web installer isn't being
> triggered after applying the patches.
> 
> I might take a look at yaz again...

The web installer won't be triggered after applying patches unless kohaversion.pl is changed, which doesn't happen until the RM pushes the patch. In cases like this, you have to run updatedatabase.pl manually, unfortunately.

Jesse accidentally left in a version number in the updatedatabase. If you fix that to XXX it should be ok. I will give it a try.

Ah ok, misread the commit a bit even - it just needs a nudge from 13 to 15.

Setting up pazpar2 is definitely a bit of a pain if you're not using Debian/Ubuntu :/.

I'm using an old version of yaz (4.2.13 from 2011) on my dev system, so I had to go pretty far back to get something that I could even configure.

I suppose it's a bit of a moot point that Debian's Wheezy repo has yaz 4.2.30 (from 2012) in it, since the pazpar2 packages from IndexData must upgrade one's yaz packages as well.

I suppose the thing to do is just update my yaz, but it would be good to know what version of yaz and pazpar2 a person needs to get this feature to work, since we're not all using Debian/Ubuntu. 

In fact, there probably should be some automated checking to make sure that Koha has the right versions of yaz and pazpar2 available, otherwise this feature will just bust, no?

Do we keep track of version dependencies on anything besides Perl modules? Going forward, we'll probably want to do that with yaz and pazpar2...


pazpar2-1.6.32.tar.gz	14-Jun-2013 09:27	1.0M	 
Requires YAZ 4.2.58 or later

pazpar2-1.6.23.tar.gz	02-Jan-2013 12:09	889K	 
Requires YAZ 4.2.40 or later

pazpar2-1.6.22.tar.gz	11-Oct-2012 08:10	890K	
Requires YAZ 4.2.40 or later

pazpar2-1.6.21.tar.gz	24-Sep-2012 18:08	856K	 
Requires YAZ 4.2.20 or later

pazpar2-1.6.20.tar.gz	21-Sep-2012 13:27	882K
Requires YAZ 4.2.20 or later

pazpar2-1.6.18.tar.gz	17-Sep-2012 11:40	860K	 
Requires YAZ 4.2.20 or later

pazpar2-1.6.15.tar.gz	27-Jun-2012 10:17	842K	 
Requires YAZ 4.2.20 or later

pazpar2-1.6.10.tar.gz	23-Feb-2012 10:52	786K	 
Requires YAZ 4.2.20 or later

pazpar2-1.6.5.tar.gz	02-Nov-2011 10:25	1.5M	 
Requires YAZ 4.2.18 or later

pazpar2-1.6.4.tar.gz	28-Oct-2011 10:31	1.5M	 
Requires YAZ 4.2.17 or later

pazpar2-1.6.3.tar.gz	22-Sep-2011 11:08	1.5M	 
Finally found one that works for 4.2.13

Manually ran updatedatabase.pl which worked, so I was able to add an external search target. Yay!

When I tried searching in the OPAC, nothing happened, which is good, since I haven't set up pazpar2 yet.

I'll have to set up pazpar2 on another dev setup when I have a chance...

2 things found in testing:

- The 856 links on the result page are not working for me. It tries to create a
  Koha internal link (major)
- Taking a look at Firebug when the page loads, I can see that it always
  loads something UNIMARC and NORMARC too - is that on purpose? Should it only
  load the files for your current marcflavour?

Giving an example for the links:

Result list:
Online Access: Inhaltsverzeichnis [Inhaltsverzeichnis = table of contents]
http://localhost/cgi-bin/koha/opac-external-search.pl?q=bodensee#

Detail page:
Online resources: Inhaltsverzeichnis 
http://d-nb.info/1037370899/04

I wonder... the results page does not look quite right - having the different label? An URL in a record is not always online access, like in this case, so I'd prefer the wording from the detail page.

(In reply to Katrin Fischer from comment #59)
> Giving an example for the links:
> 
> Result list:
> Online Access: Inhaltsverzeichnis [Inhaltsverzeichnis = table of contents]
> http://localhost/cgi-bin/koha/opac-external-search.pl?q=bodensee#
> 
> Detail page:
> Online resources: Inhaltsverzeichnis 
> http://d-nb.info/1037370899/04
> 
> I wonder... the results page does not look quite right - having the
> different label? An URL in a record is not always online access, like in
> this case, so I'd prefer the wording from the detail page.

The first was an overly aggressive jQuery selector, which I'll be throwing in a QA followup along with a fix for updatedatabase.pl (sorry dcook). The latter is likely an inconsistency in the underlying XSLT.

Jesse, this is much better! 

But now that I know it should also work with UNIMARC I have added a UNIMARC target, which works perfectly... except for the details page. For exmaple there is always 'publication:', but it never seems to show any information behind the label. Most records don't show any other fields at all.

I have tried with BNF2, your example from below. Also I have some weird diacritics showing up in my result list, not sure if that is the target or a bug.

Also paging behind page 10 only brings up empty pages?

Please check?

Ah, and the styling for the mandatory fields on the cnfiguration page doesn't look quite right yet - compared to administration>libraries. But not a blocker.

(In reply to Katrin Fischer from comment #62)
> Jesse, this is much better! 
> 
> But now that I know it should also work with UNIMARC I have added a UNIMARC
> target, which works perfectly... except for the details page. For exmaple
> there is always 'publication:', but it never seems to show any information
> behind the label. Most records don't show any other fields at all.
> 
> I have tried with BNF2, your example from below. Also I have some weird
> diacritics showing up in my result list, not sure if that is the target or a
> bug.
> 
> Also paging behind page 10 only brings up empty pages?
> 
> Please check?

BNF unfortunately returns a lot of rather paltry records (some with only 3 tags total!), but they were one of the few UNIMARC targets I could find that supported keyword search. This problem is on their end, rather than these patches' or the XSLT's.

Fixed the diacritics in the above patch.

The failing after a certain page is a transient problem that happens after the user goes in a few pages, where requests to Pazpar2 seem to get stuck somewhere in Apache or the browser. It usually only happens after going fairly deep in the search results, and can be often be fixed by reloading. Not sure how to fix this one.

It seemed to constantly happyn for me - page 10 worked, page 11 won't.
Maybe if we can't display huge result sets, we need to limit the result set?

Which wouldn't be a good solution... but I think it's a problem like it is now.

Marking as candidate for 3.16.0

Created attachment 27785 [details] [review]
[SIGNED OFF] Bug 10486 - Allow external Z39.50 targets to be searched from the OPAC

This adds a new page to the OPAC that can search several Z39.50 targets
using Pazpar2.

Test plan:
1) Apply patches.
2) Update database.
3) Add one or more Z39.50 targets, using the Administration -> External
   targets screen. Note that not all targets support keyword search, but
   here are some that do:
     * Library of Congress: lx2.loc.gov, 210, LCDB, none, none, MARC21, UTF-8
     * BNF2: z3950.bnf.fr, 2211, TOUT, Z3950, Z3950_BNF, UNIMARC, UTF-8
4) Perform a search on the OPAC, check to see if the "external targets"
   notice pops up at the top.
5) Click on the results link near that message.
6) This should bring back results from all enabled targets, all of which
   should be clickable to bring up a detailed view of the record.

(This version incorporates previous QA followups and some fixes for
translation and per-branch-limiting.)

Signed-off-by: Katrin Fischer <Katrin.Fischer.83@web.de>
Passes tests, template problem showing up in the QA should be ok.
More verbose notes on the bug report.

Created attachment 27786 [details] [review]
[SIGNED OFF] Bug 10486 followup - add tables to kohastructure.sql

Signed-off-by: Katrin Fischer <Katrin.Fischer.83@web.de>

Created attachment 27787 [details] [review]
[SIGNED OFF] Bug 10486 - QA followup: fix several display issues and rebase errors

Signed-off-by: Katrin Fischer <Katrin.Fischer.83@web.de>

Created attachment 27788 [details] [review]
[SIGNED OFF] Bug 10486 - QA followup: fix links in results and updatedatabase

Signed-off-by: Katrin Fischer <Katrin.Fischer.83@web.de>

Created attachment 27789 [details] [review]
[SIGNED OFF] Bug 10486 - QA followup: incorrect diacritics

Correctly asks for UTF-8 now, fixing diacritics for some targets.

Signed-off-by: Katrin Fischer <Katrin.Fischer.83@web.de>

Created attachment 27790 [details] [review]
[SIGNED OFF] Bug 10486 - Followup: Give more accurate results and pages count

Signed-off-by: Katrin Fischer <Katrin.Fischer.83@web.de>

Below my notes from testing and setting this up. I have kept the
paths as they are in my installation, hopefully I didn't miss a step 
so others can get this working as well.

I am using a newer Ubuntu version, so I cheated a bit with the Apache
configuration (bug 11404).

Adding a first sign off to this - but I think we need a bit more testing
to ensure it works alright with packages.

--

Setup steps

# Add indexdata repository
sudo vi /etc/apt/sources.list.d/indexdata.list 
deb http://ftp.indexdata.dk/debian wheezy main
deb-src http://ftp.indexdata.dk/debian wheezy main

wget http://ftp.indexdata.dk/debian/indexdata.asc
sudo apt-key add indexdata.asc
sudo apt-get update

# Install pazpar2
sudo apt-get purge pazpar2
sudo apt-get install pazpar2

# Activate Pazpar2 in Koha
[copy koha-httpd.conf - keep changes for new Apache version around]

perl Makefile.PL
Install the PazPar2 configuration files? [no] yes
Keep standard settings for Pazpar2, like port 11002

make
sudo make install

[reactivate old koha-httpd.conf

# Add proxy in OPAC koha-httpd.conf
    <Proxy *>
        AddDefaultCharset off
        Order deny,allow
        Allow from all
    </Proxy>
    
# Change, remove xslt
<DirectoryMatch "/home/katrin/kohaclone/koha-tmpl/.*/(modules|includes)">

restart apache]

#start pazpar2 server
$ pazpar2 -f /home/katrin/koha-dev/etc/pazpar2/pazpar2.xml 

# Add external search targets in Koha
SWB: z3950n.bsz-bw.de:20210/swb MARC21 utf-8
LOC: lx2.loc.gov:210/LCDB MARC21 utf-8
BNF2: z3950.bnf.fr:2211/TOUT (Z3950/Z3950_BNF) UNIMARC

Yes, you can mix UNIMARC and MARC21, but BNF records are not always very
pretty in the result list.

# Activate system preference OPACSearchExternalTargets

Testing
- Search terms entered in simple search will be used to search in 
  the external targets as well, a message on the top of the page 
  shows the number of records found. This works
  * from simple search
  * from advanced search, only for search options, not for limits
    Other search options will be turned into a keyword search for the
    external search targets.
  * when no records were found locally
  * with local results
  * if search goes directly to detail page, external targets are not
    searched
- When looking at the search, the targets are shown to the left
  * Unchecking and checking a target will change the result list 
    accordingly
- The result list is ordered by relevance, targets are sorted together
  * Results are sorted together by relevance
  * Only up to 100 results will be retrieved per target
- When clicking on the title a modal with the record details opens
  * Links except for 856 are removed, to avoid non-working links back
  into the catalog.
- Configuration
  * If you add a branch restriction to a target, it's only visible for
    users logged in at the other libraries
  * If you have no restriction by branch, the target will always be 
    searched, even when not logged in
    
Enhancement ideas:
- Add a timeout like for normal targets to avoid a long wait
- Lift limit on max 100 records and make that a per target setting
- Add ability to use separate XSLT files, to adapt for weirdness in 
  MARC records returned by the targets
- Add the ability to generate links to the original target - right
  now the user is trapped in Koha. Something similar to SearchForTitleIn
  would be great, giving an option to define different kinds of 
  links with placeholders.
- Offer sort options other than relevance, like publication date

I am on it but I did not manage to retrieve record from z3950.bnf.fr, I got the following error in the koha log file:
  Pazpar2 init failed: No Host option provided

However, in opac/svc/pazpar2_init, $target_url == "z3950.bnf.fr:2211/TOUT".

(In reply to Jonathan Druart from comment #78)
> I am on it but I did not manage to retrieve record from z3950.bnf.fr, I got
> the following error in the koha log file:
>   Pazpar2 init failed: No Host option provided
> 
> However, in opac/svc/pazpar2_init, $target_url == "z3950.bnf.fr:2211/TOUT".

I got the same for z3950.biblibre.com:9999/biblios

And same for lx2.loc.gov:210/LCDB
It's my first try with pazpar2, it seems to be a fail...

Hm, I haven't run into that problem - we'll probably need to wait for Jesse :( 

Do MARC21 targets work for you? I tested with 3 active targets: LOC, SWB and BNF. The other thing that might be missing is something in the setup of PazPar2 - did you see my notes?

Do you see an error when running the pazpar2 server? 

#start pazpar2 server
$ pazpar2 -f /home/katrin/koha-dev/etc/pazpar2/pazpar2.xml

So, QA comment just looking at the code:

1/ use Modern::Perl in the opac/svc file

2/ warning 
"my" variable $response masks earlier declaration in same scope at /home/koha/src/opac/svc/pazpar2_init line 80

3/ Why do you create a new table and don't use the entries in z3950servers (with a new column)?

4/ Some error raised by qa tools:

 FAIL	koha-tmpl/opac-tmpl/prog/en/modules/opac-results.tt
   FAIL	  forbidden patterns
    forbidden pattern: opac-tmpl should certainly replaced with [% interface %] (line 276)

And some otheres. You should move new js files in koha-tmpl/intranet-tmpl/lib (see bug 12119, bug 12116, etc.)

5/ For your next patch, please split it in several small ones. It is very hard to read a 249.78 KB patch :)

6/ Unit tests are missing for GetExternalSearchTargets

7/ param debug => 1 is useless in get_template_and_user calls

8/ SQL queries in .pl files should be avoided

9/ I didn't understand the new css rules for .highlight

10/ +                            <input type="checkbox" id="branch-[% branch.branchcode %]" name="branch" value="[% branch.branchcode %]" checked />
should be checked="checked"

11/ Please squash changes in opac.css (you completely rewrite the file in the first patch). And now I understand the size of the patch :)

12/ New js lib should be added to the about page, licenses tab.

I don't change the status because I didn't test, but it seems to fail QA.

(In reply to Katrin Fischer from comment #81)
> Do MARC21 targets work for you? I tested with 3 active targets: LOC, SWB and
> BNF. The other thing that might be missing is something in the setup of
> PazPar2 - did you see my notes?

Yes, very useful!

(In reply to Katrin Fischer from comment #82)
> Do you see an error when running the pazpar2 server? 

> #start pazpar2 server
> $ pazpar2 -f /home/katrin/koha-dev/etc/pazpar2/pazpar2.xml

No, nothing here (some warnings), only the startup:
16:12:01-02/05 pazpar2 [warn] Wrapper element <relevance> deprecated
16:12:01-02/05 pazpar2 [log] Use <icu_chain id="relevance">.. only
16:12:01-02/05 pazpar2 [warn] Wrapper element <sort> deprecated
16:12:01-02/05 pazpar2 [log] Use <icu_chain id="sort">.. only
16:12:01-02/05 pazpar2 [warn] Wrapper element <mergekey> deprecated
16:12:01-02/05 pazpar2 [log] Use <icu_chain id="mergekey">.. only
16:12:01-02/05 pazpar2 [log] Pazpar2 start 1.6.41 18f2ca94b3cfb8f77ab8444b5afa1479abd00308
16:12:01-02/05 pazpar2 [log] HTTP listener @:11002

And nothing after.

I can't check with my installation at home now, to check if the warnings show up for me as well :( I remmber seeing those and I think we fixed it, but not entirely sure. Guess we will need Jesse now - I am going to fail it.

Very tangential comment from me...

It would be interesting in the long-term to see this feature integrate with an interlibrary loan feature.

Jesse, could you take a look at the problem Jonathan had (comment 84) please?

*** Bug 3673 has been marked as a duplicate of this bug. ***

What's the status of this?  Is it still being pursued?  I guess its going to miss 3.18?

I put a lot of time into testing this... is there still a plan to make this go into Koha?

Hi Jesse and all,

my boss tells that we need to help as much as possible to insert this feature in master.
Now the code is in  "Failed QA" status.
In which direction do you want to port the code ?
Do you want to drop the use of Pazpar ?

Bye

(In reply to Zeno Tajoli from comment #91)
> Hi Jesse and all,
> 
> my boss tells that we need to help as much as possible to insert this
> feature in master.
> Now the code is in  "Failed QA" status.
> In which direction do you want to port the code ?
> Do you want to drop the use of Pazpar ?
> 
> Bye

Yes, I would. There's a basic metasearch that I added to bug 11559 that, while not perfect, is much more reliable than pazpar2. The one feature that it eliminates is on-the-fly merging; aka, how the current interface shows results as they come in. It also has slightly lower limits on how many records it can fetch, being written in pure Perl.

Are either of these dealbreakers?

>Are either of these dealbreakers?
For us no.
It is ok for us:
-- drop pazpar2
-- drop on-the-fly merging;
-- lower limits

I am a bit sad to see the merging go as this would make it more than just a federated search - how low are the limits?

(In reply to Jesse Weaver from comment #1)
> Created attachment 19142 [details] [review] [review]
> Bug 10486 - Allow external Z39.50 targets to be searched from the OPAC (1/2)
> 
> This adds a new page to the OPAC that can search several Z39.50 targets
> using Pazpar2.
> 
> Test plan:
> 1) Apply patches.
> 2) Update database.
> 3) Add one or more Z39.50 targets, using the Administration -> External
>    targets screen. Note that not all targets support keyword search, but
>    here are some that do:
>      * Library of Congress: lx2.loc.gov, 210, LCDB, none, none, MARC21, UTF-8
>      * BNF2: z3950.bnf.fr, 2211, TOUT, Z3950, Z3950_BNF, UNIMARC, UTF-8
> 4) Perform a search on the OPAC, check to see if the "external targets"
>    notice pops up at the top.
> 5) Click on the results link near that message.
> 6) This should bring back results from all enabled targets, all of which
>    should be clickable to bring up a detailed view of the record.

Can we get an update on this? Also, if we drop pazpar2, is this a duplicate to bug 11361?

Hi Katrin,

in fact I'm working a version of this request.
Pazpar2 is mandatory, I can't drop it. It is much more fast that our z39.50 interface.
In fact this bug and 11361 are very similar.

I try to push a version for 3.22

Bye

Hi Zeno, thx for the update. Is there a possiblity of reusing the code here?

I think yes. Stay tuned !!

Hi Zeno, any news on this?

(In reply to Mirko Tietgen from comment #100)
> Hi Zeno, any news on this?

Hi,
There was a problem on installation on Jessie that block my work on topic:
http://lists.koha-community.org/pipermail/koha-devel/2016-January/042204.html

Now stay tuned !!

This is a great first step in setting up Koha for ILL request in the OPAC!  Looking forward to this one.

Any update on this? I think still a feature a lot of people would like to see!

I wanted to test it at the hackfest in Berlin. Unfortunately Zeno said the code is not available, he has to ask his employer first (If I understood correctly). The bug is still assigned to Jesse, who is also not working on this any longer I guess.

Zeno: could you update on this? If it is a proprietary enhancement you are working on, you should not imply you are working on it for the community. It would be helpful to know if your code will end up here, because otherwise you are keeping people from developing this feature for Koha.

I'm sorry but I need to confirm the words
of Mirko.
There are also problems on installing PazPar on a server
with a previous installation of Koha.
Today my suggestion is to use a different server with PazPar.
I will write more info after 25 of August.

Any update on this?

We have a simpler version (Bug 11361) that hasn't been pushed much because of this, understandbly.  Our client's code has been in production at 20+ colleges for many years now and satisfies their needs.

We can help on getting this through, or just get the word to kill it (proprietary info??) and give love to the other one.  Zeno hasn't followed up on his 2016-08-14 comment.

Hi to all,

I have same update.

1)With new bugfix of bug 17871 now you can install PazPar2 and Zebra on the same server.

2) I have update my howto on the topic:
https://wiki.koha-community.org/wiki/PazPar_to_search_external_Z39_50_on_OPAC

3)In the last year we have many discusion inside my firm in what is proprietary and what not, but now a news: they said me that I will have a definitve answer for 06-March-2017.

Bye
Zeno Tajoli

Hi all,

I have the authorization.
I will publish my code in github (I hope soon).

Hi, 

Can anyone tell me what the current status if this is? A previous LMS had this and it had +3,000 hits the last year we had the system. It is missed by staff any patrons.
Thanks, 
Niamh

As we said 2.5y ago, we have that running *now* for 35 colleges.  Our own version of it, Bug 11361.  We aren't making any more effort to maintain the community patch since this bug here seemed more popular.

If you have interest in it (Bug 11361 again), and get some traction from the QAs, we'll be happy to rebase it for the latest version within days.  Otherwise we stay put, as all we want is the feature, and we don't care which version make it forward.

Without looking at the other bug report closely I can say that Jessie and Zeno are no longer working on Koha, so this might need to be adopted to move forward. If there is a similar bug with people willing to push it forward, this one might stay orphaned.