Bug 6536 - Z3950 Search Enhancements: SRU targets and additional XSLT processing
Summary: Z3950 Search Enhancements: SRU targets and additional XSLT processing
Status: CLOSED FIXED
Alias: None
Product: Koha
Classification: Unclassified
Component: Z39.50 / SRU / OpenSearch Servers (show other bugs)
Version: master
Hardware: All All
: P3 new feature (vote)
Assignee: Marcel de Rooy
QA Contact:
URL:
Whiteboard:
Keywords:
: 6304 11109 12711 (view as bug list)
Depends on: 1561
Blocks: 12594 12823
  Show dependency treegraph
 
Reported: 2011-06-27 13:20 UTC by Marcel de Rooy
Modified: 2017-08-31 08:53 UTC (History)
16 users (show)

See Also:
Change sponsored?: ---
Patch complexity: Medium patch
Bot Control: ---
When did the bot last check this:
Who signed the patch off:
Text to go in the release notes:


Attachments
Patch (functionality) (151.38 KB, patch)
2011-06-30 13:03 UTC, Marcel de Rooy
Details | Diff | Splinter Review
Patch (db revision) (3.78 KB, patch)
2011-06-30 13:04 UTC, Marcel de Rooy
Details | Diff | Splinter Review
Patch (db revision) -- rebased (2.91 KB, patch)
2011-08-11 08:47 UTC, Marcel de Rooy
Details | Diff | Splinter Review
Rebased patch for db rev (3.71 KB, patch)
2011-12-21 12:39 UTC, Marcel de Rooy
Details | Diff | Splinter Review
[SIGNED-OFF] 6536: Z3950 Enhancements (Functionality) (151.43 KB, patch)
2011-12-22 05:39 UTC, Katrin Fischer
Details | Diff | Splinter Review
[SIGNED-OFF] 6536: Z3950 Enhancements (DB Revision) (5.54 KB, patch)
2011-12-22 05:40 UTC, Katrin Fischer
Details | Diff | Splinter Review
Follow up to fix MARC21toUNIMARC.xsl (1.61 KB, patch)
2011-12-22 14:36 UTC, Marcel de Rooy
Details | Diff | Splinter Review
Bug MT6536: Followup: Fix some issues for unimarc (4.50 KB, patch)
2012-11-15 15:00 UTC, Jonathan Druart
Details | Diff | Splinter Review
[TESTPATCH 3.8]8610 Normalize imported Z3950 record (2.85 KB, patch)
2014-03-12 12:39 UTC, Marcel de Rooy
Details | Diff | Splinter Review
[TESTPATCH 3.8] 6536 part 2 (29.96 KB, patch)
2014-03-12 12:40 UTC, Marcel de Rooy
Details | Diff | Splinter Review
[TESTPATCH 3.8] 6536 3 sru (10.12 KB, patch)
2014-03-12 12:40 UTC, Marcel de Rooy
Details | Diff | Splinter Review
[TESTPATCH 3.8] 6536: xslt on v38 (10.97 KB, patch)
2014-03-12 12:40 UTC, Marcel de Rooy
Details | Diff | Splinter Review
Bug 6536: Dbrev for Z3950 search improvements (5.10 KB, patch)
2014-07-16 14:08 UTC, Marcel de Rooy
Details | Diff | Splinter Review
[DO_NOT_PUSH] Bug 6536: Update DBIx schema of z3950servers for testing (4.40 KB, patch)
2014-07-16 14:18 UTC, Marcel de Rooy
Details | Diff | Splinter Review
Bug 6536: Add a form for SRU Field mapping (6.09 KB, patch)
2014-07-16 14:26 UTC, Marcel de Rooy
Details | Diff | Splinter Review
Bug 6536: SRU servers administration (11.05 KB, patch)
2014-07-17 07:46 UTC, Marcel de Rooy
Details | Diff | Splinter Review
Bug 6536: Adjustments for servername and servertype (5.77 KB, patch)
2014-07-17 07:56 UTC, Marcel de Rooy
Details | Diff | Splinter Review
Bug 6536: Include SRU searching in Breeding.pm (14.47 KB, patch)
2014-07-17 09:53 UTC, Marcel de Rooy
Details | Diff | Splinter Review
Bug 6536: Add XSLT transformation on Z3950 search results (8.45 KB, patch)
2014-07-17 10:39 UTC, Marcel de Rooy
Details | Diff | Splinter Review
[SAMPLE XSLT; DO NOT PUSH] Bug 6536: Two simple test XSLT files (2.65 KB, patch)
2014-07-17 10:49 UTC, Marcel de Rooy
Details | Diff | Splinter Review
Bug 6536: Dbrev for Z3950 search improvements (5.13 KB, patch)
2014-07-18 09:41 UTC, sandboxes@biblibre.com
Details | Diff | Splinter Review
Bug 6536: Update DBIx schema of z3950servers for testing (4.41 KB, patch)
2014-07-18 09:41 UTC, sandboxes@biblibre.com
Details | Diff | Splinter Review
Bug 6536: Add a form for SRU Field mapping (6.12 KB, patch)
2014-07-18 09:41 UTC, sandboxes@biblibre.com
Details | Diff | Splinter Review
Bug 6536: SRU servers administration (11.08 KB, patch)
2014-07-18 09:41 UTC, sandboxes@biblibre.com
Details | Diff | Splinter Review
Bug 6536: Adjustments for servername and servertype (5.80 KB, patch)
2014-07-18 09:41 UTC, sandboxes@biblibre.com
Details | Diff | Splinter Review
Bug 6536: Include SRU searching in Breeding.pm (14.50 KB, patch)
2014-07-18 09:41 UTC, sandboxes@biblibre.com
Details | Diff | Splinter Review
Bug 6536: Add XSLT transformation on Z3950 search results (8.48 KB, patch)
2014-07-18 09:41 UTC, sandboxes@biblibre.com
Details | Diff | Splinter Review
[DO_NOT_PUSH] Bug 6536: Two simple test XSLT files (2.64 KB, patch)
2014-07-18 09:41 UTC, sandboxes@biblibre.com
Details | Diff | Splinter Review
Bug 6536: Dbrev for Z3950 search improvements (5.35 KB, patch)
2014-07-28 10:46 UTC, Marcel de Rooy
Details | Diff | Splinter Review
Bug 6536: Update DBIx schema of z3950servers for testing (4.45 KB, patch)
2014-07-28 10:47 UTC, Marcel de Rooy
Details | Diff | Splinter Review
Bug 6536: SRU servers administration (11.12 KB, patch)
2014-07-28 10:47 UTC, Marcel de Rooy
Details | Diff | Splinter Review
Bug 6536: Adjustments for servername and servertype (5.83 KB, patch)
2014-07-28 10:47 UTC, Marcel de Rooy
Details | Diff | Splinter Review
Bug 6536: Include SRU searching in Breeding.pm (14.54 KB, patch)
2014-07-28 10:47 UTC, Marcel de Rooy
Details | Diff | Splinter Review
Bug 6536: Add XSLT transformation on Z3950 search results (8.52 KB, patch)
2014-07-28 10:47 UTC, Marcel de Rooy
Details | Diff | Splinter Review
Bug 6536: Add a form for SRU Field mapping (6.16 KB, patch)
2014-08-18 08:10 UTC, Marcel de Rooy
Details | Diff | Splinter Review
[TESTING_ONLY] Bug 6536: Two simple test XSLT files (2.70 KB, patch)
2014-08-18 08:10 UTC, Marcel de Rooy
Details | Diff | Splinter Review
Bug 6536: Dbrev for Z3950 search improvements (5.35 KB, patch)
2014-08-18 08:10 UTC, Marcel de Rooy
Details | Diff | Splinter Review
[DO_NOT_PUSH] Bug 6536: Update DBIx schema of z3950servers for testing (4.45 KB, patch)
2014-08-18 08:10 UTC, Marcel de Rooy
Details | Diff | Splinter Review
Bug 6536: SRU servers administration (11.31 KB, patch)
2014-08-18 08:10 UTC, Marcel de Rooy
Details | Diff | Splinter Review
Bug 6536: Adjustments for servername and servertype (5.83 KB, patch)
2014-08-18 08:10 UTC, Marcel de Rooy
Details | Diff | Splinter Review
Bug 6536: Include SRU searching in Breeding.pm (14.54 KB, patch)
2014-08-18 08:11 UTC, Marcel de Rooy
Details | Diff | Splinter Review
Bug 6536: Add XSLT transformation on Z3950 search results (8.52 KB, patch)
2014-08-18 08:11 UTC, Marcel de Rooy
Details | Diff | Splinter Review
[PASSED QA] Bug 6536: Add a form for SRU Field mapping (6.18 KB, patch)
2014-08-18 13:47 UTC, Martin Renvoize
Details | Diff | Splinter Review
[DO_NOT_PUSH] Bug 6536: Two simple test XSLT files (2.65 KB, patch)
2014-08-18 13:47 UTC, Martin Renvoize
Details | Diff | Splinter Review
[PASSED QA] Bug 6536: Dbrev for Z3950 search improvements (5.38 KB, patch)
2014-08-18 13:47 UTC, Martin Renvoize
Details | Diff | Splinter Review
[RM NOTE] Bug 6536: Update DBIx schema of z3950servers for testing (4.48 KB, patch)
2014-08-18 13:47 UTC, Martin Renvoize
Details | Diff | Splinter Review
[PASSED QA] Bug 6536: SRU servers administration (11.35 KB, patch)
2014-08-18 13:47 UTC, Martin Renvoize
Details | Diff | Splinter Review
[PASSED QA] Bug 6536: Adjustments for servername and servertype (6.00 KB, patch)
2014-08-18 13:48 UTC, Martin Renvoize
Details | Diff | Splinter Review
[PASSED QA] Bug 6536: Include SRU searching in Breeding.pm (14.58 KB, patch)
2014-08-18 13:48 UTC, Martin Renvoize
Details | Diff | Splinter Review
[PASSED QA] Bug 6536: Add XSLT transformation on Z3950 search results (8.55 KB, patch)
2014-08-18 13:48 UTC, Martin Renvoize
Details | Diff | Splinter Review
[PASSED_QA] Bug 6536: Add XSLT transformation on Z3950 search results (8.65 KB, patch)
2014-08-21 12:53 UTC, Marcel de Rooy
Details | Diff | Splinter Review
Bug 6536: [QA Follow-up] Remove surrounding spaces in sru_fields (1.29 KB, patch)
2014-08-21 12:53 UTC, Marcel de Rooy
Details | Diff | Splinter Review
Bug 6536: QA Follow-up for string changes referring to Z39.50 (14.52 KB, patch)
2014-08-25 13:18 UTC, Marcel de Rooy
Details | Diff | Splinter Review
Bug 6536: QA Follow-up: Add some unit tests for Z3950Search in Breeding.pm (8.92 KB, patch)
2014-08-27 15:04 UTC, Marcel de Rooy
Details | Diff | Splinter Review
Bug 6536: QA Follow-up for removing warnings from QA tools (4.57 KB, patch)
2014-08-27 15:04 UTC, Marcel de Rooy
Details | Diff | Splinter Review
Bug 6536: QA Follow-up for removing warnings from QA tools (1.07 KB, patch)
2014-08-27 15:25 UTC, Marcel de Rooy
Details | Diff | Splinter Review
Bug 6536: QA Follow-up for fixing a unit test under Perl 5.18 (1.51 KB, patch)
2014-08-28 06:58 UTC, Marcel de Rooy
Details | Diff | Splinter Review
Bug 6536: Follow-up: Updating web installer sql files for z3950servers (9.46 KB, patch)
2014-08-28 13:23 UTC, Marcel de Rooy
Details | Diff | Splinter Review
Bug 6536: Follow-up: Updating installer sql files for z3950servers (Part 2) (7.85 KB, patch)
2014-08-28 13:23 UTC, Marcel de Rooy
Details | Diff | Splinter Review
sample MARC21 to UNIMARC xslt (81.93 KB, application/xml)
2015-03-26 11:13 UTC, Fridolin SOMERS
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Marcel de Rooy 2011-06-27 13:20:38 UTC
See also http://wiki.koha-community.org/wiki/Z3950_RFC
Comment 1 Marcel de Rooy 2011-06-30 13:03:02 UTC Comment hidden (obsolete)
Comment 2 Marcel de Rooy 2011-06-30 13:04:24 UTC Comment hidden (obsolete)
Comment 3 Marcel de Rooy 2011-06-30 13:09:07 UTC
Z3950 Enhancements include SRU search targets, MARC conversion, additional XSLT processing and some code refactoring.
The MARC21 to UNIMARC conversion and additional XSLT processing has actually been created by BibLibre and is incorporated into these two patches.

A testplan with 12 steps is included: see http://wiki.koha-community.org/wiki/Z3950_RFC

This patch needs a signoff for MARC21 and another one for UNIMARC.
Comment 4 Magnus Enger 2011-06-30 13:19:04 UTC
*** Bug 6304 has been marked as a duplicate of this bug. ***
Comment 5 Marcel de Rooy 2011-08-11 08:47:35 UTC Comment hidden (obsolete)
Comment 6 Paul Poulain 2011-10-24 11:38:21 UTC
Updating Version : This ENH will be for Koha 3.8
Comment 7 Paul Poulain 2011-10-25 15:05:58 UTC
Bug versionned for master. entries will be made against rel_3_8 once the patch has been applied (see thread about that on koha-devel yesterday)
Comment 8 Katrin Fischer 2011-12-21 06:07:11 UTC
Hi Marcel,

this looks like a very interesting feature. There are some conflicts for the second patch:

Apply? [y]es/[n]o/[e]dit/[v]iew patch/[a]ccept all y
Applying: 6536: Z3950 Enhancements (DB Revision)
Using index info to reconstruct a base tree...
Falling back to patching base and 3-way merge...
Auto-merging installer/data/mysql/kohastructure.sql
CONFLICT (content): Merge conflict in installer/data/mysql/kohastructure.sql
Auto-merging installer/data/mysql/updatedatabase.pl
CONFLICT (content): Merge conflict in installer/data/mysql/updatedatabase.pl
Failed to merge in the changes.
Patch failed at 0001 6536: Z3950 Enhancements (DB Revision)
When you have resolved this problem run "git am --resolved".
If you would prefer to skip this patch, instead run "git am --skip".
To restore the original branch and stop patching run "git am --abort".

Could you perhaps fix and resubmit? I will try to test soon after you did, according to the test plan on the wiki.
Comment 9 Marcel de Rooy 2011-12-21 12:39:47 UTC Comment hidden (obsolete)
Comment 10 Marcel de Rooy 2011-12-21 12:42:16 UTC
(In reply to comment #8)
> Could you perhaps fix and resubmit? I will try to test soon after you did,
> according to the test plan on the wiki.
Resubmitted the dbrev. Hope you can test soon; needs rebasing often..
Comment 11 Katrin Fischer 2011-12-21 21:44:27 UTC
Patch applied, started testing.
Comment 12 Katrin Fischer 2011-12-21 21:52:03 UTC
So far some notes, but nothing that would prevent a sign-off:

1) Information about SRU configuration should be added to the
help file of this page.

2) XSLT and sru field information don't show on administration
page, but I think that's ok. Perhaps heading on this page should be changed to Z39.50/SRU server administration to reflect that both options are available now.
Comment 13 Katrin Fischer 2011-12-22 05:39:55 UTC Comment hidden (obsolete)
Comment 14 Katrin Fischer 2011-12-22 05:40:06 UTC Comment hidden (obsolete)
Comment 15 Katrin Fischer 2011-12-22 05:46:41 UTC
I think a second sign-off from a UNIMARC user would be a good addition but apart from the notes made above no real problems found in my tests.
Comment 16 Marcel de Rooy 2011-12-22 14:36:06 UTC Comment hidden (obsolete)
Comment 17 Jonathan Druart 2011-12-23 08:26:29 UTC
I have an error with a Z3950 server (LOC) :
Field 531 must have indicators (use ' ' for empty indicators) at /usr/share/perl5/MARC/File/SAX.pm line 92

Bu I have no more information to give to you :-/
params : z3950.loc.gov:7090 DB:voyager MARC21 MARC8.

I searched with 'tintin' on this server and I got this error.

With SWB (swb.bsz-bw.de:210 DB:swblite USMARC, MARC8) I have a warning : "Use of uninitialized value $marc in substitution (s///) at /.../C4/Charset.pm line 229." for each result and nothing is substitued.

With klnapp2.passhe.edu, Timeout (10007)
and catalog.nypl.org Connect failed (10000).

Sorry but I currently have no time to debug this.
Comment 18 Marcel de Rooy 2012-01-26 12:49:28 UTC
Changing status to reflect current condition of this report. *sigh ;)
Comment 19 Gaetan Boisson 2012-08-20 12:30:59 UTC
I am reviving this old bug, this would be a very interesting enhancement!
Comment 20 Jonathan Druart 2012-11-15 15:00:38 UTC
Hi Marcel,
The good news is I have a patch for the problems we encountered. The xslt transforms with success some notices from the LOC server.
The bad news is the precedent patches don't apply anymore.

My patch is based on yours patches, I did not get through to rebased them easily :-/
Comment 21 Jonathan Druart 2012-11-15 15:00:54 UTC Comment hidden (obsolete)
Comment 22 Marcel de Rooy 2012-11-19 09:20:41 UTC
(In reply to comment #20)
> Hi Marcel,
> The good news is I have a patch for the problems we encountered. The xslt
> transforms with success some notices from the LOC server.
> The bad news is the precedent patches don't apply anymore.
> 
> My patch is based on yours patches, I did not get through to rebased them
> easily :-/

Thanks for looking into this again. I still have it on my list.
At this time I have a simpler rebased version in production under 3.8 where I removed xslt and marc/unimarc conversion.
My new approach for this report would be:
1 move the code to module
2 add the sru targets
3 add the xslt option
4 (rudimentary) marc conversion (if there is still interest for it)

Will create some new reports to do this gradually. Your patch has to do with the last two steps.
Comment 23 Marcel de Rooy 2012-11-21 14:56:30 UTC
The first step in my previous comment is implemented under report 9105.
Comment 24 claire.hernandez@biblibre.com 2013-04-10 09:24:28 UTC
Hello Marcel, did you plan to do this work soon ? Could we help you to do something ?
Comment 25 Marcel de Rooy 2013-04-10 09:40:07 UTC
(In reply to comment #24)
> Hello Marcel, did you plan to do this work soon ? Could we help you to do
> something ?

Hi Claire: Yes, I will take that up again soon. Your help in signing off would be welcome :)
Comment 26 Marcel de Rooy 2013-06-13 13:27:27 UTC
Bug 10462 paves the way for reviving this development.
Comment 27 David Cook 2013-08-01 07:06:12 UTC
Hi Marcel!

I'm working on a client for OAI-PMH harvesting, so I'm wondering what you're planning in regards to MARC conversion and additional XSLT processing.

My plan (and WIP) at the moment is to point to a base url, harvest the records (likely in Dublin core format), store them in a MySQL table, then transform them into MARC and import into Koha. 

Do you see any problems with this?
Comment 28 Marcel de Rooy 2013-08-01 09:22:43 UTC
(In reply to David Cook from comment #27)
> I'm working on a client for OAI-PMH harvesting, so I'm wondering what you're
> planning in regards to MARC conversion and additional XSLT processing.
> 
> My plan (and WIP) at the moment is to point to a base url, harvest the
> records (likely in Dublin core format), store them in a MySQL table, then
> transform them into MARC and import into Koha. 
> 
> Do you see any problems with this?

Hi David,
Looks like: No problems. Will add XSLT transformations on the returned Z3950 results and will add SRU targets as Z3950 servers. (MARC conversion will be possible via the XSLTs, but I will not include it specifically anymore.)
Comment 29 Magnus Enger 2014-02-07 10:20:31 UTC
Could the "MARC Modification Templates" introduced by Byg 8015 be extended to modify records that are imported via Z39.50 to? 

Docs: http://manual.koha-community.org/3.14/en/catalogtools.html#marcmodtemplates
Comment 30 Marcel de Rooy 2014-02-10 08:52:11 UTC
(In reply to Magnus Enger from comment #29)
> Could the "MARC Modification Templates" introduced by Byg 8015 be extended
> to modify records that are imported via Z39.50 to? 
> 
> Docs:
> http://manual.koha-community.org/3.14/en/catalogtools.html#marcmodtemplates

Yes. That could be an interesting addition. Running an xslt file will allow developers to do more than the modification template allows you, but regular (more experienced) users would probably be happy to use the templates.
I will have a further look at that later.
Comment 31 Marcel de Rooy 2014-02-24 11:19:36 UTC
I added bug 11826 in order to add an object to facilitate xslt handling as a first step to still incorporate xslt to z3950 search results.
Please have a look..
Comment 32 Marcel de Rooy 2014-03-12 12:39:42 UTC Comment hidden (obsolete)
Comment 33 Marcel de Rooy 2014-03-12 12:40:11 UTC Comment hidden (obsolete)
Comment 34 Marcel de Rooy 2014-03-12 12:40:28 UTC Comment hidden (obsolete)
Comment 35 Marcel de Rooy 2014-03-12 12:40:44 UTC Comment hidden (obsolete)
Comment 36 Marcel de Rooy 2014-03-12 12:43:06 UTC
Attached temporarily four 3.8 patches to this report for Paul A.
Paul: Please use dbrev in attachment 4541 [details] [review] (obsolete patch on this report).
This will add srufields and add_xslt to your z3950servers table.
The first attached patch in this series comes from report 8610.
Comment 37 Marcel de Rooy 2014-07-16 14:08:14 UTC Comment hidden (obsolete)
Comment 38 Marcel de Rooy 2014-07-16 14:18:09 UTC Comment hidden (obsolete)
Comment 39 Marcel de Rooy 2014-07-16 14:26:12 UTC Comment hidden (obsolete)
Comment 40 Marcel de Rooy 2014-07-16 14:30:54 UTC
Please wait: Still four patches coming within some days..
Comment 41 Marcel de Rooy 2014-07-17 07:46:39 UTC Comment hidden (obsolete)
Comment 42 Marcel de Rooy 2014-07-17 07:56:06 UTC Comment hidden (obsolete)
Comment 43 Marcel de Rooy 2014-07-17 09:53:48 UTC Comment hidden (obsolete)
Comment 44 Marcel de Rooy 2014-07-17 10:39:43 UTC Comment hidden (obsolete)
Comment 45 Marcel de Rooy 2014-07-17 10:49:34 UTC Comment hidden (obsolete)
Comment 46 Marcel de Rooy 2014-07-17 10:52:28 UTC
This is ready for signoff now/again.
Please let me know if you encounter problems testing this..
Comment 47 sandboxes@biblibre.com 2014-07-18 09:39:40 UTC
Patch tested with a sandbox, by Giuseppe Angilella <giuseppe.angilella@ct.infn.it>
Comment 48 sandboxes@biblibre.com 2014-07-18 09:41:16 UTC Comment hidden (obsolete)
Comment 49 sandboxes@biblibre.com 2014-07-18 09:41:20 UTC Comment hidden (obsolete)
Comment 50 sandboxes@biblibre.com 2014-07-18 09:41:24 UTC Comment hidden (obsolete)
Comment 51 sandboxes@biblibre.com 2014-07-18 09:41:28 UTC Comment hidden (obsolete)
Comment 52 sandboxes@biblibre.com 2014-07-18 09:41:32 UTC Comment hidden (obsolete)
Comment 53 sandboxes@biblibre.com 2014-07-18 09:41:36 UTC Comment hidden (obsolete)
Comment 54 sandboxes@biblibre.com 2014-07-18 09:41:41 UTC Comment hidden (obsolete)
Comment 55 sandboxes@biblibre.com 2014-07-18 09:41:45 UTC Comment hidden (obsolete)
Comment 56 Martin Renvoize 2014-07-24 12:06:47 UTC
*** Bug 11109 has been marked as a duplicate of this bug. ***
Comment 57 Marcel de Rooy 2014-07-28 10:46:57 UTC Comment hidden (obsolete)
Comment 58 Marcel de Rooy 2014-07-28 10:47:02 UTC Comment hidden (obsolete)
Comment 59 Marcel de Rooy 2014-07-28 10:47:07 UTC Comment hidden (obsolete)
Comment 60 Marcel de Rooy 2014-07-28 10:47:11 UTC Comment hidden (obsolete)
Comment 61 Marcel de Rooy 2014-07-28 10:47:16 UTC Comment hidden (obsolete)
Comment 62 Marcel de Rooy 2014-07-28 10:47:21 UTC Comment hidden (obsolete)
Comment 63 Marcel de Rooy 2014-07-28 10:54:23 UTC
Rebased some patches in this series. The order has been slightly adjusted. The first two patches (in the current order) only add new files and are on top now. This should not do any harm imho..
Comment 64 Vitor Fernandes 2014-08-04 11:23:58 UTC
*** Bug 12711 has been marked as a duplicate of this bug. ***
Comment 65 Marcel de Rooy 2014-08-18 08:10:29 UTC Comment hidden (obsolete)
Comment 66 Marcel de Rooy 2014-08-18 08:10:36 UTC Comment hidden (obsolete)
Comment 67 Marcel de Rooy 2014-08-18 08:10:40 UTC Comment hidden (obsolete)
Comment 68 Marcel de Rooy 2014-08-18 08:10:45 UTC Comment hidden (obsolete)
Comment 69 Marcel de Rooy 2014-08-18 08:10:49 UTC Comment hidden (obsolete)
Comment 70 Marcel de Rooy 2014-08-18 08:10:54 UTC Comment hidden (obsolete)
Comment 71 Marcel de Rooy 2014-08-18 08:11:00 UTC Comment hidden (obsolete)
Comment 72 Marcel de Rooy 2014-08-18 08:11:05 UTC Comment hidden (obsolete)
Comment 73 Marcel de Rooy 2014-08-18 08:12:51 UTC
Rebased it again.
Comment 74 Martin Renvoize 2014-08-18 13:23:27 UTC
Some Minor comments to start the ball rolling.. Nothing fail worthy so far.

1. Z3950 references in the staff client. There are references to Z3950 in a number of places in the staff client, for instance admin-home where we go to configure z3950 targets should make it clear that we're now configuruing z3950 OR sru.. and 'Add new from z3950' under cataloguing should also be updated.  Perhaps 'New from external database' or similar?
2. 'Database' on the 'Add Server' page is a required field, but it's also not obviose what it expects
3. I'de like a more visual queue somewhere in the Add SRU server page that one really aught to define the 'SRU Search fields mappings' else all searches will be against the 'any' index on the remote server.  Maybe this is a documentation thing.  Either a stronger hint to fill that stuff in.. or to drop the index specific search boxes in the 'Add from ....' popup when the sru server is selected.. Hard call really.. we want people to map them to get sensible results really.  Maybe set some sensible defaults and make it a required field?

Back to testing
Comment 75 Martin Renvoize 2014-08-18 13:26:19 UTC
Also.. any chance of a documentation patch to go along with the updated functionality?
Comment 76 Martin Renvoize 2014-08-18 13:42:19 UTC
1 Final comment before passing...

We've already discussed that a user accessible way to set the xslt transformations would be a very useful and sensible future enhancement to this, and I believe from our chat that you are intended on implementing this via the marc modification templates framework.  I don't really like how the configuration of SRU is currently spread between Serverside and Clientside actions; requiring templates to be placed on the server, yet the links to said templates to go into the staff client.  Again.. not a fail.. but a comment for future enhancement..
Comment 77 Martin Renvoize 2014-08-18 13:47:05 UTC
Created attachment 30929 [details] [review]
[PASSED QA] Bug 6536: Add a form for SRU Field mapping

This form allows you to enter the match between the standard names of
the Z3950 search fields and the field names for a specific SRU server.
This patch only adds the form; it will be used in a subsequent patch.

Test plan:
Not needed, but you can call the form by constructing a URL like:
    cgi-bin/koha/admin/sru_modmapping.pl?mapping=title=1,author=2
You should see the form with title and author filled.
Note that you cannot save the information yet.

Signed-off-by: Giuseppe Angilella <giuseppe.angilella@ct.infn.it>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Comment 78 Martin Renvoize 2014-08-18 13:47:25 UTC
Created attachment 30930 [details] [review]
[DO_NOT_PUSH] Bug 6536: Two simple test XSLT files

You could use Add901.xsl or Set003ABC.xsl in field add_xslt of table
z3950servers when testing this report.
Set003ABC will change field 003 into ABC.
And Add901 adds a tag 901.

Signed-off-by: Giuseppe Angilella <giuseppe.angilella@ct.infn.it>
Comment 79 Martin Renvoize 2014-08-18 13:47:37 UTC
Created attachment 30931 [details] [review]
[PASSED QA] Bug 6536: Dbrev for Z3950 search improvements

Adjusts table z3950servers:
Drops unused columns icon, description and position.
Moves id column to first position.
Renames name to servername, and type to servertype. (This is not only more
clear but may eliminate some problems too with DBIx.)
Changes recordtype from varchar(45) to enumeration with two members. [The
upgrade replaces unknown record types with biblio, although it is very
unlikely to find such records.]
Adds SRU as servertype enumeration member. Removes opensearch, since it is
not used/supported. [The upgrade replaces unknown server types with zed
(z3950) (in exceptional cases).]
Adds new columns: sru_options, sru_fields, add_xslt.

Test plan:
Run database update via webinstaller.
Check your z3950servers table.

Signed-off-by: Giuseppe Angilella <giuseppe.angilella@ct.infn.it>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Comment 80 Martin Renvoize 2014-08-18 13:47:49 UTC
Created attachment 30932 [details] [review]
[RM NOTE] Bug 6536: Update DBIx schema of z3950servers for testing

This patch adjusts the DBIx schema file for z3950servers table.
This can be used for testing this set of patches.
When the report is pushed, the RM should update the DBIx schema.

Signed-off-by: Giuseppe Angilella <giuseppe.angilella@ct.infn.it>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Comment 81 Martin Renvoize 2014-08-18 13:47:59 UTC
Created attachment 30933 [details] [review]
[PASSED QA] Bug 6536: SRU servers administration

This patch makes adjustments in admin/z3950servers in order to insert,
edit and delete SRU server records.

Test plan:
Add, edit, copy and delete SRU server.
Check if adding, editing, copying and deleting Z3950 servers still works.

Signed-off-by: Giuseppe Angilella <giuseppe.angilella@ct.infn.it>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Comment 82 Martin Renvoize 2014-08-18 13:48:10 UTC
Created attachment 30934 [details] [review]
[PASSED QA] Bug 6536: Adjustments for servername and servertype

Replaces name by servername, type by servertype for running Z3950 search.
Limit search scripts to zed (z3950) servers until sru is supported.

Test plan:
Perform a Z3950 search in Cataloguing and Acquisition.
Verify that it still works as it did.

Signed-off-by: Giuseppe Angilella <giuseppe.angilella@ct.infn.it>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Comment 83 Martin Renvoize 2014-08-18 13:48:21 UTC
Created attachment 30935 [details] [review]
[PASSED QA] Bug 6536: Include SRU searching in Breeding.pm

This patch makes it possible to include SRU servers in Z3950 search.
It adjusts the Z3950Search routine in Breeding module.
It also replaces SQL code with DBIx statements in Breeding.pm/Z3950Search
and the associated scripts z3950search.pl in cataloguing and acqui.

Test plan:
Verify if a normal Z3950 search still works in cataloging/acqui.
Add a SRU target. (You could just use Koha's port 9998.)
Define sru_options like sru=get.
Use that target in a Z3950 search in cataloging and acqui. (Import.)
Test sru_fields translation by comparing search results between various
settings for some of the fields. For instance, leave title empty and
after that set it to the title field of your SRU target.

Signed-off-by: Giuseppe Angilella <giuseppe.angilella@ct.infn.it>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Comment 84 Martin Renvoize 2014-08-18 13:48:32 UTC Comment hidden (obsolete)
Comment 85 Martin Renvoize 2014-08-18 13:54:52 UTC
Passing QA

Nice work marcel, this adds a useful and usable enhancement.  As commented above, I see a number of further follow-ups, but would be happy to see them as separate follow-up bugs once this piece has made it's way in and a few other areas of code have settled enough for the follow-ups to be based upon them.

The main one I would like to see is a clear documentation patch to show exactly how one should configure all this stuff as is. It currently lies firmly in the system admin field for the xslt transformations stuff.. but even the SRU pieces need 'extra bits' that a normal library admin may not fully understand without a little more guidance.

I can't find any regressions and the code all checks out through the qa scripts.

SRU wise.. I tested mostly against the LOC SRU gateway.. but I also did some quick tests against a koha instance and a few other places. To note, if the gateway only outputs dublin core, then this it will not work with koha.. another future enhancement maybe.

To test against LOC:

Host: z3950.loc.gov
Port: 7090
Database: voyager
Comment 86 Marcel de Rooy 2014-08-21 11:30:46 UTC
Have to fix another merge conflict again..
Comment 87 Marcel de Rooy 2014-08-21 12:53:27 UTC
Created attachment 31046 [details] [review]
[PASSED_QA] Bug 6536: Add XSLT transformation on Z3950 search results

Use the stylesheets listed in field add_xslt of z3950servers to transform
search results of Z3950/SRU search.
Additionally, the template has been changed to make more error messages (or
warnings) visible when displaying results. Until now, error message were
shown in the results table and when connection errors occurred, no results
were displayed at all.

Test plan:
Create some stylesheets (or see the sample patch on bug 6536).
Add these stylesheets to some Z3950/SRU servers.
Do Z3950 search and verify the transformations.
Do a search with 2 targets; make one target fail (by manipulating its server
data). Do you see the connection error and the results for the other target?
Generate a XSLT error by modifying one stylesheet. Check search results. You
should see warnings.

Signed-off-by: Giuseppe Angilella <giuseppe.angilella@ct.infn.it>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Comment 88 Marcel de Rooy 2014-08-21 12:53:36 UTC
Created attachment 31048 [details] [review]
Bug 6536: [QA Follow-up] Remove surrounding spaces in sru_fields

This patch only removes surrouding spaces at comma and equals-sign while
passing the options in sru_fields to the ZOOM object.

Test plan:
If you add spaces between options in sru_fields, searching should still work.
E.g. sru_fields= sru = get , sru_version = 1.1

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Comment 89 Marcel de Rooy 2014-08-21 12:58:32 UTC
Rebased a patch, adding it on top of bug 1561 (pushed partially at this moment).
Added a small follow-up removing spaces when parsing zoom options from sru_fields (just extends usability, reducing errors).
Comment 90 Marcel de Rooy 2014-08-21 13:32:11 UTC
Martin: Thanks for QAing.
Tomas: I will first put a note on this report how I will implement some of the documentation issues Martin mentioned before. New reports will deal with functionality changes.
Comment 91 Marcel de Rooy 2014-08-25 13:18:36 UTC
Created attachment 31137 [details] [review]
Bug 6536: QA Follow-up for string changes referring to Z39.50

Z39.50 search now includes SRU targets.
This patch only touches strings referring to it.
No changes in behavior.
It replaces Z39.50 with Z39.50/SRU or external.

Deletes obsolete template z3950/searchresult.tt: not in use for some time.

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Comment 92 Marcel de Rooy 2014-08-25 13:19:52 UTC
TO update the status of this report:
Follow-up sent for: 1. Z3950 references in the staff client. 

Will still look at QA points: 
2. Database field not obvious
3. Hint to define the SRU search field mappings
4. Documentation patch
5. Unit tests (request Tomas)

FUTURE WORK (on new reports)
Integration with marc mod templates
Comment 93 Marcel de Rooy 2014-08-26 08:38:56 UTC
(In reply to M. de Rooy from comment #92)
> 2. Database field not obvious

I submitted a patch under bug 12823 for this point, to be handled as a separate follow-up report. (No need to push them together..)
Comment 94 Marcel de Rooy 2014-08-26 09:20:17 UTC
(In reply to M. de Rooy from comment #92)
> 3. Hint to define the SRU search field mappings
I extended the solution for point 2 to include this point. So see also bug 12823.
Comment 95 Marcel de Rooy 2014-08-26 16:17:44 UTC
(In reply to M. de Rooy from comment #92)
> 4. Documentation patch
The online Help is updated via bug 12828. Can be handled separately.
Comment 96 Marcel de Rooy 2014-08-27 15:04:51 UTC
Created attachment 31193 [details] [review]
Bug 6536: QA Follow-up: Add some unit tests for Z3950Search in Breeding.pm

This patch may serve as a starting point for more unit tests in this part
of the code.
In this patch most internal subroutines of Z3950Search are covered.
In the XSLT part we reuse a XSL template from the XSLT_Handler.t test.
It needed a little tweaking for namespaces. And added the indicators on
the 990 field.

Test plan:
Run both Breeding.t and XSLT_Handler.t.

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Comment 97 Marcel de Rooy 2014-08-27 15:04:58 UTC Comment hidden (obsolete)
Comment 98 Marcel de Rooy 2014-08-27 15:06:21 UTC
(In reply to M. de Rooy from comment #92)
> 5. Unit tests (request Tomas)
Done.
Comment 99 Marcel de Rooy 2014-08-27 15:21:24 UTC
Comment on attachment 31194 [details] [review]
Bug 6536: QA Follow-up for removing warnings from QA tools

Sorry, this implies the separate follow-up reports too. Will revise.
Comment 100 Marcel de Rooy 2014-08-27 15:25:14 UTC
Created attachment 31195 [details] [review]
Bug 6536: QA Follow-up for removing warnings from QA tools

Resolved:
[1]  FAIL   C4/Breeding.pm
FAIL   critic ControlStructures::ProhibitMutatingListFunctions

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
No warnings anymore.
Comment 101 Tomás Cohen Arazi 2014-08-27 16:21:36 UTC
Marcel, last test on the first series, fails randomly on Perl 5.18:

    not ok 12 - Third query makes no difference
    
    #   Failed test 'Third query makes no difference'
    #   at t/db_dependent/Breeding.t line 104.
    #          got: ''
    #     expected: '1'
    # Looks like you failed 1 test of 12.
not ok 1 - _build_query
Comment 102 Tomás Cohen Arazi 2014-08-27 16:24:18 UTC
This change makes tests pass. Please consider if this needs to be fixed (i.e. sort order matters) or the test needs to be rewritten:

tomas@zoe:koha-community-src(clean_master)$ git diff HEAD
diff --git a/C4/Breeding.pm b/C4/Breeding.pm
index fa3af82..21a12c1 100644
--- a/C4/Breeding.pm
+++ b/C4/Breeding.pm
@@ -249,7 +249,7 @@ sub _build_query {
     my $zquery='';
     my $squery='';
     my $nterms=0;
-    foreach my $k ( keys %$pars ) {
+    foreach my $k ( sort keys %$pars ) {
         if( ( my $val=$pars->{$k} ) && $qry_build->{$k} ) {
             $qry_build->{$k} =~ s/#term/$val/g;
             $zquery .= $qry_build->{$k};
tomas@zoe:koha-community-src(clean_master)$
Comment 103 Marcel de Rooy 2014-08-28 06:34:12 UTC
(In reply to Tomás Cohen Arazi from comment #101)
> Marcel, last test on the first series, fails randomly on Perl 5.18:
> 
Good catch, Tomas. It did not fail with me under Perl 5.14.
I will sign off your suggested change.
Thanks.
Comment 104 Marcel de Rooy 2014-08-28 06:58:47 UTC
Created attachment 31227 [details] [review]
Bug 6536: QA Follow-up for fixing a unit test under Perl 5.18

The last test on the first series, fails randomly on Perl 5.18:
    not ok 12 - Third query makes no difference
    #   Failed test 'Third query makes no difference'
    #   at t/db_dependent/Breeding.t line 104.
    #          got: ''
    #     expected: '1'
    # Looks like you failed 1 test of 12.
not ok 1 - _build_query

This change makes tests pass. Please consider if this needs to be fixed
(i.e. sort order matters) or the test needs to be rewritten.

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
I agree with adding the sort. (The need for doing this in Perl 5.18 is another
topic.)
Comment 105 Marcel de Rooy 2014-08-28 12:17:26 UTC
Must still add a patch for the two renamed cols in sql files of new install.
Development never stops ;)
Comment 106 Marcel de Rooy 2014-08-28 13:23:14 UTC
Created attachment 31247 [details] [review]
Bug 6536: Follow-up: Updating web installer sql files for z3950servers

Removing icon, description and position from installer sql files.
Not all files contained them.

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
I have run all sql files on an empty table by script.
Comment 107 Marcel de Rooy 2014-08-28 13:23:18 UTC
Created attachment 31248 [details] [review]
Bug 6536: Follow-up: Updating installer sql files for z3950servers (Part 2)

This patch renames name to servername and type to servertype.

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Tested all sql files by script.
Comment 108 Marcel de Rooy 2014-08-28 13:24:55 UTC
Tomas: if these last two patches present a problem for pushing them, I could also move them to a separate report for additional sign off. Just let me know.
Comment 109 Tomás Cohen Arazi 2014-09-01 13:53:54 UTC
New feature pushed to master!

Thanks Marcel!
Comment 110 Marcel de Rooy 2014-09-01 19:54:11 UTC
(In reply to Tomás Cohen Arazi from comment #109)
> New feature pushed to master!
> 
> Thanks Marcel!

Thanks, Tomas.
Comment 111 Dobrica Pavlinusic 2014-09-10 11:32:07 UTC
This bug introduced regression fixed in Bug 12898 - Z39.50 title search doesn't work with multiple words
Comment 112 Fridolin SOMERS 2015-03-26 11:13:05 UTC
Created attachment 37264 [details]
sample MARC21 to UNIMARC xslt

I attach our actual XSLT as example
Comment 113 Manos PETRIDIS 2015-11-23 08:04:25 UTC
Is there any functional UNIMARC to MARC21 MARC conversion xslt available? 
In other words, how may one search UNIMARC z39.50 targets in order to populate a MARC21 library?