|
Description
Marcel de Rooy
2011-06-27 13:20:38 UTC
Created attachment 4540 [details] [review] Patch (functionality) Created attachment 4541 [details] [review] Patch (db revision) Z3950 Enhancements include SRU search targets, MARC conversion, additional XSLT processing and some code refactoring. The MARC21 to UNIMARC conversion and additional XSLT processing has actually been created by BibLibre and is incorporated into these two patches. A testplan with 12 steps is included: see http://wiki.koha-community.org/wiki/Z3950_RFC This patch needs a signoff for MARC21 and another one for UNIMARC. *** Bug 6304 has been marked as a duplicate of this bug. *** Created attachment 4959 [details] [review] Patch (db revision) -- rebased Rebased. Functional changes from updatedatabase moved to bug 6700. Updating Version : This ENH will be for Koha 3.8 Bug versionned for master. entries will be made against rel_3_8 once the patch has been applied (see thread about that on koha-devel yesterday) Hi Marcel, this looks like a very interesting feature. There are some conflicts for the second patch: Apply? [y]es/[n]o/[e]dit/[v]iew patch/[a]ccept all y Applying: 6536: Z3950 Enhancements (DB Revision) Using index info to reconstruct a base tree... Falling back to patching base and 3-way merge... Auto-merging installer/data/mysql/kohastructure.sql CONFLICT (content): Merge conflict in installer/data/mysql/kohastructure.sql Auto-merging installer/data/mysql/updatedatabase.pl CONFLICT (content): Merge conflict in installer/data/mysql/updatedatabase.pl Failed to merge in the changes. Patch failed at 0001 6536: Z3950 Enhancements (DB Revision) When you have resolved this problem run "git am --resolved". If you would prefer to skip this patch, instead run "git am --skip". To restore the original branch and stop patching run "git am --abort". Could you perhaps fix and resubmit? I will try to test soon after you did, according to the test plan on the wiki. Created attachment 6901 [details] [review] Rebased patch for db rev (In reply to comment #8) > Could you perhaps fix and resubmit? I will try to test soon after you did, > according to the test plan on the wiki. Resubmitted the dbrev. Hope you can test soon; needs rebasing often.. Patch applied, started testing. So far some notes, but nothing that would prevent a sign-off: 1) Information about SRU configuration should be added to the help file of this page. 2) XSLT and sru field information don't show on administration page, but I think that's ok. Perhaps heading on this page should be changed to Z39.50/SRU server administration to reflect that both options are available now. Created attachment 6919 [details] [review] [SIGNED-OFF] 6536: Z3950 Enhancements (Functionality) Z3950 Enhancements: SRU search targets, MARC conversion and additional XSLT processing Testplan: see http://wiki.koha-community.org/wiki/Z3950_RFC Signed-off-by: Katrin Fischer <Katrin.Fischer.83@web.de> Comments on second patch. Created attachment 6920 [details] [review] [SIGNED-OFF] 6536: Z3950 Enhancements (DB Revision) This patch contains db revision for Z3950 enhancements: SRU search targets, MARC conversion and additional XSLT processing. August 11, 2011: Rebased. Removed changes for different XXX handling (see Bugzilla 6700). December 21, 2011: Rebased. Signed-off-by: Katrin Fischer <Katrin.Fischer.83@web.de> Followed test plan on the wiki: http://wiki.koha-community.org/wiki/Z3950_RFC 1) Patches apply. 2) Database update was stuck in an endless loop. I removed the comment from the dbversion update in updatedatabase.pl to make it work. I have not added this change to the patch, leaving it for RM. 3) + 12) I added the SRU configuration suggested. Newly added fields save, can be edited and deleted. Note: Information about SRU configuration should be added to the help file of this page. Note: XSLT and sru field information don't show on administration page. Perhaps heading should be changed here to Z39.50/SRU? 4) Successfully performed searches in cataloguing - result list looks ok - marc and isbd preview work - import works 5) Successfully performed searches in acquisitions see above 6) + 8) Searched a UNIMARC target from a MARC21 installation - result list looks ok - marc and isbd preview work - import works, field mapping looks ok Note: isbd view has a little display problem with non filing characters 7) Should be tested by a UNIMARC user 8) Custom XSLT transformation Works great with the example provided. I saved it as MARC21toFunny.xsl into the xslt folder and used a relative url. Result list, previews and imported records show the changes. Note: An option to edit the xsl file online or to upload it here and in other instances would be a nice improvement. 9) Searched 3 targets at once SWB: Z39.50, MARC21 + custom XSLT ABES: Z39.50, UNIMARC Rijksmuseum: SRU Got a mixed result list, result list and previews are all ok. X) Confirmed z39.50 search still works Tried different searches in cataloging and acquisition to confirm there is no regression. All tests passed. I think we need a bit more documentation inside of Koha to make it easier for people to use the new features, but everything I tested works nicely. I think a second sign-off from a UNIMARC user would be a good addition but apart from the notes made above no real problems found in my tests. Created attachment 6924 [details] [review] Follow up to fix MARC21toUNIMARC.xsl BibLibre testing revealed problem in conversion to UNIMARC. Can be resolved by this [simple] follow up on top of other patches. I have an error with a Z3950 server (LOC) : Field 531 must have indicators (use ' ' for empty indicators) at /usr/share/perl5/MARC/File/SAX.pm line 92 Bu I have no more information to give to you :-/ params : z3950.loc.gov:7090 DB:voyager MARC21 MARC8. I searched with 'tintin' on this server and I got this error. With SWB (swb.bsz-bw.de:210 DB:swblite USMARC, MARC8) I have a warning : "Use of uninitialized value $marc in substitution (s///) at /.../C4/Charset.pm line 229." for each result and nothing is substitued. With klnapp2.passhe.edu, Timeout (10007) and catalog.nypl.org Connect failed (10000). Sorry but I currently have no time to debug this. Changing status to reflect current condition of this report. *sigh ;) I am reviving this old bug, this would be a very interesting enhancement! Hi Marcel, The good news is I have a patch for the problems we encountered. The xslt transforms with success some notices from the LOC server. The bad news is the precedent patches don't apply anymore. My patch is based on yours patches, I did not get through to rebased them easily :-/ Created attachment 13462 [details] [review] Bug MT6536: Followup: Fix some issues for unimarc 1/ In the Breeding module, it is useless to parse 2x the xslt transformation. 2/ Fix encoding in the xslt 3/ The as_xml method takes the marcflavour value. (In reply to comment #20) > Hi Marcel, > The good news is I have a patch for the problems we encountered. The xslt > transforms with success some notices from the LOC server. > The bad news is the precedent patches don't apply anymore. > > My patch is based on yours patches, I did not get through to rebased them > easily :-/ Thanks for looking into this again. I still have it on my list. At this time I have a simpler rebased version in production under 3.8 where I removed xslt and marc/unimarc conversion. My new approach for this report would be: 1 move the code to module 2 add the sru targets 3 add the xslt option 4 (rudimentary) marc conversion (if there is still interest for it) Will create some new reports to do this gradually. Your patch has to do with the last two steps. The first step in my previous comment is implemented under report 9105. Hello Marcel, did you plan to do this work soon ? Could we help you to do something ? (In reply to comment #24) > Hello Marcel, did you plan to do this work soon ? Could we help you to do > something ? Hi Claire: Yes, I will take that up again soon. Your help in signing off would be welcome :) Bug 10462 paves the way for reviving this development. Hi Marcel! I'm working on a client for OAI-PMH harvesting, so I'm wondering what you're planning in regards to MARC conversion and additional XSLT processing. My plan (and WIP) at the moment is to point to a base url, harvest the records (likely in Dublin core format), store them in a MySQL table, then transform them into MARC and import into Koha. Do you see any problems with this? (In reply to David Cook from comment #27) > I'm working on a client for OAI-PMH harvesting, so I'm wondering what you're > planning in regards to MARC conversion and additional XSLT processing. > > My plan (and WIP) at the moment is to point to a base url, harvest the > records (likely in Dublin core format), store them in a MySQL table, then > transform them into MARC and import into Koha. > > Do you see any problems with this? Hi David, Looks like: No problems. Will add XSLT transformations on the returned Z3950 results and will add SRU targets as Z3950 servers. (MARC conversion will be possible via the XSLTs, but I will not include it specifically anymore.) Could the "MARC Modification Templates" introduced by Byg 8015 be extended to modify records that are imported via Z39.50 to? Docs: http://manual.koha-community.org/3.14/en/catalogtools.html#marcmodtemplates (In reply to Magnus Enger from comment #29) > Could the "MARC Modification Templates" introduced by Byg 8015 be extended > to modify records that are imported via Z39.50 to? > > Docs: > http://manual.koha-community.org/3.14/en/catalogtools.html#marcmodtemplates Yes. That could be an interesting addition. Running an xslt file will allow developers to do more than the modification template allows you, but regular (more experienced) users would probably be happy to use the templates. I will have a further look at that later. I added bug 11826 in order to add an object to facilitate xslt handling as a first step to still incorporate xslt to z3950 search results. Please have a look.. Created attachment 26169 [details] [review] [TESTPATCH 3.8]8610 Normalize imported Z3950 record Aangepast voor 3.8 Created attachment 26170 [details] [review] [TESTPATCH 3.8] 6536 part 2 Created attachment 26171 [details] [review] [TESTPATCH 3.8] 6536 3 sru Created attachment 26172 [details] [review] [TESTPATCH 3.8] 6536: xslt on v38 Voeg xslt optie toe aan z3950 search Pas templates aan om errors altijd te laten zien ook als er results zijn Bij het testen kwam er gemiddeld 0.005 second per record bij voor xslt slag Attached temporarily four 3.8 patches to this report for Paul A. Paul: Please use dbrev in attachment 4541 [details] [review] (obsolete patch on this report). This will add srufields and add_xslt to your z3950servers table. The first attached patch in this series comes from report 8610. Created attachment 29754 [details] [review] Bug 6536: Dbrev for Z3950 search improvements Adjusts table z3950servers: Drops unused columns icon, description and position. Moves id column to first position. Renames name to servername, and type to servertype. (This is not only more clear but may eliminate some problems too with DBIx.) Changes recordtype from varchar(45) to enumeration with two members. [The upgrade replaces unknown record types with biblio, although it is very unlikely to find such records.] Adds SRU as servertype enumeration member. Removes opensearch, since it is not used/supported. [The upgrade replaces unknown server types with zed (z3950) (in exceptional cases).] Adds new columns: sru_options, sru_fields, add_xslt. Test plan: Run database update via webinstaller. Check your z3950servers table. Created attachment 29755 [details] [review] [DO_NOT_PUSH] Bug 6536: Update DBIx schema of z3950servers for testing This patch adjusts the DBIx schema file for z3950servers table. This can be used for testing this set of patches. When the report is pushed, the RM should update the DBIx schema. Created attachment 29756 [details] [review] Bug 6536: Add a form for SRU Field mapping This form allows you to enter the match between the standard names of the Z3950 search fields and the field names for a specific SRU server. This patch only adds the form; it will be used in a subsequent patch. Test plan: Not needed, but you can call the form by constructing a URL like: cgi-bin/koha/admin/sru_modmapping.pl?mapping=title=1,author=2 You should see the form with title and author filled. Note that you cannot save the information yet. Please wait: Still four patches coming within some days.. Created attachment 29779 [details] [review] Bug 6536: SRU servers administration This patch makes adjustments in admin/z3950servers in order to insert, edit and delete SRU server records. Test plan: Add, edit, copy and delete SRU server. Check if adding, editing, copying and deleting Z3950 servers still works. Created attachment 29780 [details] [review] Bug 6536: Adjustments for servername and servertype Replaces name by servername, type by servertype for running Z3950 search. Limit search scripts to zed (z3950) servers until sru is supported. Test plan: Perform a Z3950 search in Cataloguing and Acquisition. Verify that it still works as it did. Created attachment 29788 [details] [review] Bug 6536: Include SRU searching in Breeding.pm This patch makes it possible to include SRU servers in Z3950 search. It adjusts the Z3950Search routine in Breeding module. It also replaces SQL code with DBIx statements in Breeding.pm/Z3950Search and the associated scripts z3950search.pl in cataloguing and acqui. Test plan: Verify if a normal Z3950 search still works in cataloging/acqui. Add a SRU target. (You could just use Koha's port 9998.) Define sru_options like sru=get. Use that target in a Z3950 search in cataloging and acqui. (Import.) Test sru_fields translation by comparing search results between various settings for some of the fields. For instance, leave title empty and after that set it to the title field of your SRU target. Created attachment 29790 [details] [review] Bug 6536: Add XSLT transformation on Z3950 search results Use the stylesheets listed in field add_xslt of z3950servers to transform search results of Z3950/SRU search. Additionally, the template has been changed to make more error messages (or warnings) visible when displaying results. Until now, error message were shown in the results table and when connection errors occurred, no results were displayed at all. Test plan: Create some stylesheets (or see the sample patch on bug 6536). Add these stylesheets to some Z3950/SRU servers. Do Z3950 search and verify the transformations. Do a search with 2 targets; make one target fail (by manipulating its server data). Do you see the connection error and the results for the other target? Generate a XSLT error by modifying one stylesheet. Check search results. You should see warnings. Created attachment 29791 [details] [review] [SAMPLE XSLT; DO NOT PUSH] Bug 6536: Two simple test XSLT files You could use Add901.xsl or Set003ABC.xsl in field add_xslt of table z3950servers when testing this report. Set003ABC will change field 003 into ABC. And Add901 adds a tag 901. This is ready for signoff now/again. Please let me know if you encounter problems testing this.. Patch tested with a sandbox, by Giuseppe Angilella <giuseppe.angilella@ct.infn.it> Created attachment 29812 [details] [review] Bug 6536: Dbrev for Z3950 search improvements Adjusts table z3950servers: Drops unused columns icon, description and position. Moves id column to first position. Renames name to servername, and type to servertype. (This is not only more clear but may eliminate some problems too with DBIx.) Changes recordtype from varchar(45) to enumeration with two members. [The upgrade replaces unknown record types with biblio, although it is very unlikely to find such records.] Adds SRU as servertype enumeration member. Removes opensearch, since it is not used/supported. [The upgrade replaces unknown server types with zed (z3950) (in exceptional cases).] Adds new columns: sru_options, sru_fields, add_xslt. Test plan: Run database update via webinstaller. Check your z3950servers table. Signed-off-by: Giuseppe Angilella <giuseppe.angilella@ct.infn.it> Created attachment 29813 [details] [review] Bug 6536: Update DBIx schema of z3950servers for testing This patch adjusts the DBIx schema file for z3950servers table. This can be used for testing this set of patches. When the report is pushed, the RM should update the DBIx schema. Signed-off-by: Giuseppe Angilella <giuseppe.angilella@ct.infn.it> Created attachment 29814 [details] [review] Bug 6536: Add a form for SRU Field mapping This form allows you to enter the match between the standard names of the Z3950 search fields and the field names for a specific SRU server. This patch only adds the form; it will be used in a subsequent patch. Test plan: Not needed, but you can call the form by constructing a URL like: cgi-bin/koha/admin/sru_modmapping.pl?mapping=title=1,author=2 You should see the form with title and author filled. Note that you cannot save the information yet. Signed-off-by: Giuseppe Angilella <giuseppe.angilella@ct.infn.it> Created attachment 29815 [details] [review] Bug 6536: SRU servers administration This patch makes adjustments in admin/z3950servers in order to insert, edit and delete SRU server records. Test plan: Add, edit, copy and delete SRU server. Check if adding, editing, copying and deleting Z3950 servers still works. Signed-off-by: Giuseppe Angilella <giuseppe.angilella@ct.infn.it> Created attachment 29816 [details] [review] Bug 6536: Adjustments for servername and servertype Replaces name by servername, type by servertype for running Z3950 search. Limit search scripts to zed (z3950) servers until sru is supported. Test plan: Perform a Z3950 search in Cataloguing and Acquisition. Verify that it still works as it did. Signed-off-by: Giuseppe Angilella <giuseppe.angilella@ct.infn.it> Created attachment 29817 [details] [review] Bug 6536: Include SRU searching in Breeding.pm This patch makes it possible to include SRU servers in Z3950 search. It adjusts the Z3950Search routine in Breeding module. It also replaces SQL code with DBIx statements in Breeding.pm/Z3950Search and the associated scripts z3950search.pl in cataloguing and acqui. Test plan: Verify if a normal Z3950 search still works in cataloging/acqui. Add a SRU target. (You could just use Koha's port 9998.) Define sru_options like sru=get. Use that target in a Z3950 search in cataloging and acqui. (Import.) Test sru_fields translation by comparing search results between various settings for some of the fields. For instance, leave title empty and after that set it to the title field of your SRU target. Signed-off-by: Giuseppe Angilella <giuseppe.angilella@ct.infn.it> Created attachment 29818 [details] [review] Bug 6536: Add XSLT transformation on Z3950 search results Use the stylesheets listed in field add_xslt of z3950servers to transform search results of Z3950/SRU search. Additionally, the template has been changed to make more error messages (or warnings) visible when displaying results. Until now, error message were shown in the results table and when connection errors occurred, no results were displayed at all. Test plan: Create some stylesheets (or see the sample patch on bug 6536). Add these stylesheets to some Z3950/SRU servers. Do Z3950 search and verify the transformations. Do a search with 2 targets; make one target fail (by manipulating its server data). Do you see the connection error and the results for the other target? Generate a XSLT error by modifying one stylesheet. Check search results. You should see warnings. Signed-off-by: Giuseppe Angilella <giuseppe.angilella@ct.infn.it> Created attachment 29819 [details] [review] [DO_NOT_PUSH] Bug 6536: Two simple test XSLT files You could use Add901.xsl or Set003ABC.xsl in field add_xslt of table z3950servers when testing this report. Set003ABC will change field 003 into ABC. And Add901 adds a tag 901. Signed-off-by: Giuseppe Angilella <giuseppe.angilella@ct.infn.it> *** Bug 11109 has been marked as a duplicate of this bug. *** Created attachment 30189 [details] [review] Bug 6536: Dbrev for Z3950 search improvements Adjusts table z3950servers: Drops unused columns icon, description and position. Moves id column to first position. Renames name to servername, and type to servertype. (This is not only more clear but may eliminate some problems too with DBIx.) Changes recordtype from varchar(45) to enumeration with two members. [The upgrade replaces unknown record types with biblio, although it is very unlikely to find such records.] Adds SRU as servertype enumeration member. Removes opensearch, since it is not used/supported. [The upgrade replaces unknown server types with zed (z3950) (in exceptional cases).] Adds new columns: sru_options, sru_fields, add_xslt. Test plan: Run database update via webinstaller. Check your z3950servers table. Signed-off-by: Giuseppe Angilella <giuseppe.angilella@ct.infn.it> Created attachment 30190 [details] [review] Bug 6536: Update DBIx schema of z3950servers for testing This patch adjusts the DBIx schema file for z3950servers table. This can be used for testing this set of patches. When the report is pushed, the RM should update the DBIx schema. Signed-off-by: Giuseppe Angilella <giuseppe.angilella@ct.infn.it> Created attachment 30191 [details] [review] Bug 6536: SRU servers administration This patch makes adjustments in admin/z3950servers in order to insert, edit and delete SRU server records. Test plan: Add, edit, copy and delete SRU server. Check if adding, editing, copying and deleting Z3950 servers still works. Signed-off-by: Giuseppe Angilella <giuseppe.angilella@ct.infn.it> Created attachment 30192 [details] [review] Bug 6536: Adjustments for servername and servertype Replaces name by servername, type by servertype for running Z3950 search. Limit search scripts to zed (z3950) servers until sru is supported. Test plan: Perform a Z3950 search in Cataloguing and Acquisition. Verify that it still works as it did. Signed-off-by: Giuseppe Angilella <giuseppe.angilella@ct.infn.it> Created attachment 30193 [details] [review] Bug 6536: Include SRU searching in Breeding.pm This patch makes it possible to include SRU servers in Z3950 search. It adjusts the Z3950Search routine in Breeding module. It also replaces SQL code with DBIx statements in Breeding.pm/Z3950Search and the associated scripts z3950search.pl in cataloguing and acqui. Test plan: Verify if a normal Z3950 search still works in cataloging/acqui. Add a SRU target. (You could just use Koha's port 9998.) Define sru_options like sru=get. Use that target in a Z3950 search in cataloging and acqui. (Import.) Test sru_fields translation by comparing search results between various settings for some of the fields. For instance, leave title empty and after that set it to the title field of your SRU target. Signed-off-by: Giuseppe Angilella <giuseppe.angilella@ct.infn.it> Created attachment 30194 [details] [review] Bug 6536: Add XSLT transformation on Z3950 search results Use the stylesheets listed in field add_xslt of z3950servers to transform search results of Z3950/SRU search. Additionally, the template has been changed to make more error messages (or warnings) visible when displaying results. Until now, error message were shown in the results table and when connection errors occurred, no results were displayed at all. Test plan: Create some stylesheets (or see the sample patch on bug 6536). Add these stylesheets to some Z3950/SRU servers. Do Z3950 search and verify the transformations. Do a search with 2 targets; make one target fail (by manipulating its server data). Do you see the connection error and the results for the other target? Generate a XSLT error by modifying one stylesheet. Check search results. You should see warnings. Signed-off-by: Giuseppe Angilella <giuseppe.angilella@ct.infn.it> Rebased some patches in this series. The order has been slightly adjusted. The first two patches (in the current order) only add new files and are on top now. This should not do any harm imho.. *** Bug 12711 has been marked as a duplicate of this bug. *** Created attachment 30900 [details] [review] Bug 6536: Add a form for SRU Field mapping This form allows you to enter the match between the standard names of the Z3950 search fields and the field names for a specific SRU server. This patch only adds the form; it will be used in a subsequent patch. Test plan: Not needed, but you can call the form by constructing a URL like: cgi-bin/koha/admin/sru_modmapping.pl?mapping=title=1,author=2 You should see the form with title and author filled. Note that you cannot save the information yet. Signed-off-by: Giuseppe Angilella <giuseppe.angilella@ct.infn.it> Created attachment 30901 [details] [review] [TESTING_ONLY] Bug 6536: Two simple test XSLT files You could use Add901.xsl or Set003ABC.xsl in field add_xslt of table z3950servers when testing this report. Set003ABC will change field 003 into ABC. And Add901 adds a tag 901. Signed-off-by: Giuseppe Angilella <giuseppe.angilella@ct.infn.it> Created attachment 30902 [details] [review] Bug 6536: Dbrev for Z3950 search improvements Adjusts table z3950servers: Drops unused columns icon, description and position. Moves id column to first position. Renames name to servername, and type to servertype. (This is not only more clear but may eliminate some problems too with DBIx.) Changes recordtype from varchar(45) to enumeration with two members. [The upgrade replaces unknown record types with biblio, although it is very unlikely to find such records.] Adds SRU as servertype enumeration member. Removes opensearch, since it is not used/supported. [The upgrade replaces unknown server types with zed (z3950) (in exceptional cases).] Adds new columns: sru_options, sru_fields, add_xslt. Test plan: Run database update via webinstaller. Check your z3950servers table. Signed-off-by: Giuseppe Angilella <giuseppe.angilella@ct.infn.it> Created attachment 30903 [details] [review] [DO_NOT_PUSH] Bug 6536: Update DBIx schema of z3950servers for testing This patch adjusts the DBIx schema file for z3950servers table. This can be used for testing this set of patches. When the report is pushed, the RM should update the DBIx schema. Signed-off-by: Giuseppe Angilella <giuseppe.angilella@ct.infn.it> Created attachment 30904 [details] [review] Bug 6536: SRU servers administration This patch makes adjustments in admin/z3950servers in order to insert, edit and delete SRU server records. Test plan: Add, edit, copy and delete SRU server. Check if adding, editing, copying and deleting Z3950 servers still works. Signed-off-by: Giuseppe Angilella <giuseppe.angilella@ct.infn.it> Created attachment 30905 [details] [review] Bug 6536: Adjustments for servername and servertype Replaces name by servername, type by servertype for running Z3950 search. Limit search scripts to zed (z3950) servers until sru is supported. Test plan: Perform a Z3950 search in Cataloguing and Acquisition. Verify that it still works as it did. Signed-off-by: Giuseppe Angilella <giuseppe.angilella@ct.infn.it> Created attachment 30906 [details] [review] Bug 6536: Include SRU searching in Breeding.pm This patch makes it possible to include SRU servers in Z3950 search. It adjusts the Z3950Search routine in Breeding module. It also replaces SQL code with DBIx statements in Breeding.pm/Z3950Search and the associated scripts z3950search.pl in cataloguing and acqui. Test plan: Verify if a normal Z3950 search still works in cataloging/acqui. Add a SRU target. (You could just use Koha's port 9998.) Define sru_options like sru=get. Use that target in a Z3950 search in cataloging and acqui. (Import.) Test sru_fields translation by comparing search results between various settings for some of the fields. For instance, leave title empty and after that set it to the title field of your SRU target. Signed-off-by: Giuseppe Angilella <giuseppe.angilella@ct.infn.it> Created attachment 30907 [details] [review] Bug 6536: Add XSLT transformation on Z3950 search results Use the stylesheets listed in field add_xslt of z3950servers to transform search results of Z3950/SRU search. Additionally, the template has been changed to make more error messages (or warnings) visible when displaying results. Until now, error message were shown in the results table and when connection errors occurred, no results were displayed at all. Test plan: Create some stylesheets (or see the sample patch on bug 6536). Add these stylesheets to some Z3950/SRU servers. Do Z3950 search and verify the transformations. Do a search with 2 targets; make one target fail (by manipulating its server data). Do you see the connection error and the results for the other target? Generate a XSLT error by modifying one stylesheet. Check search results. You should see warnings. Signed-off-by: Giuseppe Angilella <giuseppe.angilella@ct.infn.it> Rebased it again. Some Minor comments to start the ball rolling.. Nothing fail worthy so far. 1. Z3950 references in the staff client. There are references to Z3950 in a number of places in the staff client, for instance admin-home where we go to configure z3950 targets should make it clear that we're now configuruing z3950 OR sru.. and 'Add new from z3950' under cataloguing should also be updated. Perhaps 'New from external database' or similar? 2. 'Database' on the 'Add Server' page is a required field, but it's also not obviose what it expects 3. I'de like a more visual queue somewhere in the Add SRU server page that one really aught to define the 'SRU Search fields mappings' else all searches will be against the 'any' index on the remote server. Maybe this is a documentation thing. Either a stronger hint to fill that stuff in.. or to drop the index specific search boxes in the 'Add from ....' popup when the sru server is selected.. Hard call really.. we want people to map them to get sensible results really. Maybe set some sensible defaults and make it a required field? Back to testing Also.. any chance of a documentation patch to go along with the updated functionality? 1 Final comment before passing... We've already discussed that a user accessible way to set the xslt transformations would be a very useful and sensible future enhancement to this, and I believe from our chat that you are intended on implementing this via the marc modification templates framework. I don't really like how the configuration of SRU is currently spread between Serverside and Clientside actions; requiring templates to be placed on the server, yet the links to said templates to go into the staff client. Again.. not a fail.. but a comment for future enhancement.. Created attachment 30929 [details] [review] [PASSED QA] Bug 6536: Add a form for SRU Field mapping This form allows you to enter the match between the standard names of the Z3950 search fields and the field names for a specific SRU server. This patch only adds the form; it will be used in a subsequent patch. Test plan: Not needed, but you can call the form by constructing a URL like: cgi-bin/koha/admin/sru_modmapping.pl?mapping=title=1,author=2 You should see the form with title and author filled. Note that you cannot save the information yet. Signed-off-by: Giuseppe Angilella <giuseppe.angilella@ct.infn.it> Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com> Created attachment 30930 [details] [review] [DO_NOT_PUSH] Bug 6536: Two simple test XSLT files You could use Add901.xsl or Set003ABC.xsl in field add_xslt of table z3950servers when testing this report. Set003ABC will change field 003 into ABC. And Add901 adds a tag 901. Signed-off-by: Giuseppe Angilella <giuseppe.angilella@ct.infn.it> Created attachment 30931 [details] [review] [PASSED QA] Bug 6536: Dbrev for Z3950 search improvements Adjusts table z3950servers: Drops unused columns icon, description and position. Moves id column to first position. Renames name to servername, and type to servertype. (This is not only more clear but may eliminate some problems too with DBIx.) Changes recordtype from varchar(45) to enumeration with two members. [The upgrade replaces unknown record types with biblio, although it is very unlikely to find such records.] Adds SRU as servertype enumeration member. Removes opensearch, since it is not used/supported. [The upgrade replaces unknown server types with zed (z3950) (in exceptional cases).] Adds new columns: sru_options, sru_fields, add_xslt. Test plan: Run database update via webinstaller. Check your z3950servers table. Signed-off-by: Giuseppe Angilella <giuseppe.angilella@ct.infn.it> Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com> Created attachment 30932 [details] [review] [RM NOTE] Bug 6536: Update DBIx schema of z3950servers for testing This patch adjusts the DBIx schema file for z3950servers table. This can be used for testing this set of patches. When the report is pushed, the RM should update the DBIx schema. Signed-off-by: Giuseppe Angilella <giuseppe.angilella@ct.infn.it> Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com> Created attachment 30933 [details] [review] [PASSED QA] Bug 6536: SRU servers administration This patch makes adjustments in admin/z3950servers in order to insert, edit and delete SRU server records. Test plan: Add, edit, copy and delete SRU server. Check if adding, editing, copying and deleting Z3950 servers still works. Signed-off-by: Giuseppe Angilella <giuseppe.angilella@ct.infn.it> Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com> Created attachment 30934 [details] [review] [PASSED QA] Bug 6536: Adjustments for servername and servertype Replaces name by servername, type by servertype for running Z3950 search. Limit search scripts to zed (z3950) servers until sru is supported. Test plan: Perform a Z3950 search in Cataloguing and Acquisition. Verify that it still works as it did. Signed-off-by: Giuseppe Angilella <giuseppe.angilella@ct.infn.it> Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com> Created attachment 30935 [details] [review] [PASSED QA] Bug 6536: Include SRU searching in Breeding.pm This patch makes it possible to include SRU servers in Z3950 search. It adjusts the Z3950Search routine in Breeding module. It also replaces SQL code with DBIx statements in Breeding.pm/Z3950Search and the associated scripts z3950search.pl in cataloguing and acqui. Test plan: Verify if a normal Z3950 search still works in cataloging/acqui. Add a SRU target. (You could just use Koha's port 9998.) Define sru_options like sru=get. Use that target in a Z3950 search in cataloging and acqui. (Import.) Test sru_fields translation by comparing search results between various settings for some of the fields. For instance, leave title empty and after that set it to the title field of your SRU target. Signed-off-by: Giuseppe Angilella <giuseppe.angilella@ct.infn.it> Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com> Created attachment 30936 [details] [review] [PASSED QA] Bug 6536: Add XSLT transformation on Z3950 search results Use the stylesheets listed in field add_xslt of z3950servers to transform search results of Z3950/SRU search. Additionally, the template has been changed to make more error messages (or warnings) visible when displaying results. Until now, error message were shown in the results table and when connection errors occurred, no results were displayed at all. Test plan: Create some stylesheets (or see the sample patch on bug 6536). Add these stylesheets to some Z3950/SRU servers. Do Z3950 search and verify the transformations. Do a search with 2 targets; make one target fail (by manipulating its server data). Do you see the connection error and the results for the other target? Generate a XSLT error by modifying one stylesheet. Check search results. You should see warnings. Signed-off-by: Giuseppe Angilella <giuseppe.angilella@ct.infn.it> Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com> Passing QA Nice work marcel, this adds a useful and usable enhancement. As commented above, I see a number of further follow-ups, but would be happy to see them as separate follow-up bugs once this piece has made it's way in and a few other areas of code have settled enough for the follow-ups to be based upon them. The main one I would like to see is a clear documentation patch to show exactly how one should configure all this stuff as is. It currently lies firmly in the system admin field for the xslt transformations stuff.. but even the SRU pieces need 'extra bits' that a normal library admin may not fully understand without a little more guidance. I can't find any regressions and the code all checks out through the qa scripts. SRU wise.. I tested mostly against the LOC SRU gateway.. but I also did some quick tests against a koha instance and a few other places. To note, if the gateway only outputs dublin core, then this it will not work with koha.. another future enhancement maybe. To test against LOC: Host: z3950.loc.gov Port: 7090 Database: voyager Have to fix another merge conflict again.. Created attachment 31046 [details] [review] [PASSED_QA] Bug 6536: Add XSLT transformation on Z3950 search results Use the stylesheets listed in field add_xslt of z3950servers to transform search results of Z3950/SRU search. Additionally, the template has been changed to make more error messages (or warnings) visible when displaying results. Until now, error message were shown in the results table and when connection errors occurred, no results were displayed at all. Test plan: Create some stylesheets (or see the sample patch on bug 6536). Add these stylesheets to some Z3950/SRU servers. Do Z3950 search and verify the transformations. Do a search with 2 targets; make one target fail (by manipulating its server data). Do you see the connection error and the results for the other target? Generate a XSLT error by modifying one stylesheet. Check search results. You should see warnings. Signed-off-by: Giuseppe Angilella <giuseppe.angilella@ct.infn.it> Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com> Created attachment 31048 [details] [review] Bug 6536: [QA Follow-up] Remove surrounding spaces in sru_fields This patch only removes surrouding spaces at comma and equals-sign while passing the options in sru_fields to the ZOOM object. Test plan: If you add spaces between options in sru_fields, searching should still work. E.g. sru_fields= sru = get , sru_version = 1.1 Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl> Rebased a patch, adding it on top of bug 1561 (pushed partially at this moment). Added a small follow-up removing spaces when parsing zoom options from sru_fields (just extends usability, reducing errors). Martin: Thanks for QAing. Tomas: I will first put a note on this report how I will implement some of the documentation issues Martin mentioned before. New reports will deal with functionality changes. Created attachment 31137 [details] [review] Bug 6536: QA Follow-up for string changes referring to Z39.50 Z39.50 search now includes SRU targets. This patch only touches strings referring to it. No changes in behavior. It replaces Z39.50 with Z39.50/SRU or external. Deletes obsolete template z3950/searchresult.tt: not in use for some time. Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl> TO update the status of this report: Follow-up sent for: 1. Z3950 references in the staff client. Will still look at QA points: 2. Database field not obvious 3. Hint to define the SRU search field mappings 4. Documentation patch 5. Unit tests (request Tomas) FUTURE WORK (on new reports) Integration with marc mod templates (In reply to M. de Rooy from comment #92) > 2. Database field not obvious I submitted a patch under bug 12823 for this point, to be handled as a separate follow-up report. (No need to push them together..) (In reply to M. de Rooy from comment #92) > 3. Hint to define the SRU search field mappings I extended the solution for point 2 to include this point. So see also bug 12823. (In reply to M. de Rooy from comment #92) > 4. Documentation patch The online Help is updated via bug 12828. Can be handled separately. Created attachment 31193 [details] [review] Bug 6536: QA Follow-up: Add some unit tests for Z3950Search in Breeding.pm This patch may serve as a starting point for more unit tests in this part of the code. In this patch most internal subroutines of Z3950Search are covered. In the XSLT part we reuse a XSL template from the XSLT_Handler.t test. It needed a little tweaking for namespaces. And added the indicators on the 990 field. Test plan: Run both Breeding.t and XSLT_Handler.t. Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl> Created attachment 31194 [details] [review] Bug 6536: QA Follow-up for removing warnings from QA tools Resolved: [1] FAIL C4/Breeding.pm FAIL critic [2] FAIL koha-tmpl/intranet-tmpl/prog/en/modules/help/admin/z3950servers.tt FAIL forbidden patterns forbidden pattern: tab char (line 15) forbidden pattern: tab char (line 21) forbidden pattern: tab char (line 16) [3] FAIL koha-tmpl/intranet-tmpl/prog/en/modules/admin/z3950servers.tt FAIL forbidden patterns forbidden pattern: trailing space char (line 62) forbidden pattern: trailing space char (line 54) Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl> No warnings anymore. (In reply to M. de Rooy from comment #92) > 5. Unit tests (request Tomas) Done. Comment on attachment 31194 [details] [review] Bug 6536: QA Follow-up for removing warnings from QA tools Sorry, this implies the separate follow-up reports too. Will revise. Created attachment 31195 [details] [review] Bug 6536: QA Follow-up for removing warnings from QA tools Resolved: [1] FAIL C4/Breeding.pm FAIL critic ControlStructures::ProhibitMutatingListFunctions Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl> No warnings anymore. Marcel, last test on the first series, fails randomly on Perl 5.18:
not ok 12 - Third query makes no difference
# Failed test 'Third query makes no difference'
# at t/db_dependent/Breeding.t line 104.
# got: ''
# expected: '1'
# Looks like you failed 1 test of 12.
not ok 1 - _build_query
This change makes tests pass. Please consider if this needs to be fixed (i.e. sort order matters) or the test needs to be rewritten:
tomas@zoe:koha-community-src(clean_master)$ git diff HEAD
diff --git a/C4/Breeding.pm b/C4/Breeding.pm
index fa3af82..21a12c1 100644
--- a/C4/Breeding.pm
+++ b/C4/Breeding.pm
@@ -249,7 +249,7 @@ sub _build_query {
my $zquery='';
my $squery='';
my $nterms=0;
- foreach my $k ( keys %$pars ) {
+ foreach my $k ( sort keys %$pars ) {
if( ( my $val=$pars->{$k} ) && $qry_build->{$k} ) {
$qry_build->{$k} =~ s/#term/$val/g;
$zquery .= $qry_build->{$k};
tomas@zoe:koha-community-src(clean_master)$
(In reply to Tomás Cohen Arazi from comment #101) > Marcel, last test on the first series, fails randomly on Perl 5.18: > Good catch, Tomas. It did not fail with me under Perl 5.14. I will sign off your suggested change. Thanks. Created attachment 31227 [details] [review] Bug 6536: QA Follow-up for fixing a unit test under Perl 5.18 The last test on the first series, fails randomly on Perl 5.18: not ok 12 - Third query makes no difference # Failed test 'Third query makes no difference' # at t/db_dependent/Breeding.t line 104. # got: '' # expected: '1' # Looks like you failed 1 test of 12. not ok 1 - _build_query This change makes tests pass. Please consider if this needs to be fixed (i.e. sort order matters) or the test needs to be rewritten. Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl> I agree with adding the sort. (The need for doing this in Perl 5.18 is another topic.) Must still add a patch for the two renamed cols in sql files of new install. Development never stops ;) Created attachment 31247 [details] [review] Bug 6536: Follow-up: Updating web installer sql files for z3950servers Removing icon, description and position from installer sql files. Not all files contained them. Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl> I have run all sql files on an empty table by script. Created attachment 31248 [details] [review] Bug 6536: Follow-up: Updating installer sql files for z3950servers (Part 2) This patch renames name to servername and type to servertype. Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl> Tested all sql files by script. Tomas: if these last two patches present a problem for pushing them, I could also move them to a separate report for additional sign off. Just let me know. New feature pushed to master! Thanks Marcel! (In reply to Tomás Cohen Arazi from comment #109) > New feature pushed to master! > > Thanks Marcel! Thanks, Tomas. This bug introduced regression fixed in Bug 12898 - Z39.50 title search doesn't work with multiple words Created attachment 37264 [details]
sample MARC21 to UNIMARC xslt
I attach our actual XSLT as example
Is there any functional UNIMARC to MARC21 MARC conversion xslt available? In other words, how may one search UNIMARC z39.50 targets in order to populate a MARC21 library? *** Bug 8075 has been marked as a duplicate of this bug. *** |