Bug 10096

Summary: Add a Z39.50 interface for authority searching
Product: Koha Reporter: David Cook <dcook>
Component: Z39.50 / SRU / OpenSearch ServersAssignee: David Cook <dcook>
Status: CLOSED FIXED QA Contact:
Severity: new feature    
Priority: P5 - low CC: gmcharlt, katrin.fischer, m.de.rooy, mr_megahed, mtj, robin, ztajoli
Version: master   
Hardware: All   
OS: All   
Change sponsored?: --- Patch complexity: Medium patch
Documentation contact: Documentation submission:
Text to go in the release notes:
Version(s) released in:
Attachments: Bug 10096 - Add a Z39.50 interface for authority searching
Bug 10096 - Add a Z39.50 interface for authority searching (2)
Bug 10096 [DB Follow-up] - Add a Z39.50 interface for authority searching
Bug 10096 [FOLLOW-UP] - Add a z39.50 interface for authority searching
Bug 10096 - Add a Z39.50 interface for authority searching
Bug 10096 [FOLLOW-UP] - Add a z39.50 interface for authority searching
Bug 10096 [FOLLOW-UP] - fix tabs/whitespace errors to pass QA
Bug 10096 [FOLLOW-UP] - Add a z39.50 interface for authority searching
Bug 10096 - Add a Z39.50 interface for authority searching
Bug 10096 [FOLLOW-UP] - Add a z39.50 interface for authority searching
Bug 10096 [FOLLOW-UP] - fix tabs/whitespace errors to pass QA
Bug 10096 - [FOLLOW-UP] QA Improvements
Bug 10096 - Add a Z39.50 interface for authority searching
Bug 10096 - [FOLLOW-UP] QA Improvements
Bug 10096 - Add a Z39.50 interface for authority searching
Bug 10096 - [FOLLOW-UP] QA Improvements
Bug 10096 - Add a Z39.50 interface for authority searching
Bug 10096 - [FOLLOW-UP] QA Improvements

Description David Cook 2013-04-23 01:42:03 UTC
Currently, Koha provides Z39.50 searching for bibliographic records but not for authority records.

I propose that we create another Z39.50 search interface specifically targeted at searching authority record databases. While they are far less common, they are around:

http://www.loc.gov/z3950/lcserver.html (I think this is just a test server. Not sure if they provide access to their production server. Some LOC web sites say no but they might be outdated. OCLC says they provide access to LOC authorities via their service.)

http://www.nla.gov.au/librariesaustralia/services/search/z3950/database-connection-details/#la_authorities

OCLC also provides Z39.50 access to authority files, although like LibrariesAustralia you need to have a login to access them (even the test server, although you can apparently get a login by contacting their support people). https://www.oclc.org/support/services/z3950/documentation/config_guide.en.html.

Koha can also be used as a public facing Z39.50 authority server, so the possibility to share authority records is out there!

--

I think this could be a very useful feature, since there are already lots of authority records out there. This will save people manually creating authority records (which can result in duplicated labour in numerous libraries or authority records of varying quality) or from going into another library catalogue (such as LoC) and manually downloading them then getting Koha sysadmins to upload them via the commandline.
Comment 1 David Cook 2013-04-23 02:46:59 UTC
Update: Jcamins has mentioned that it's actually possible to use the "Stage MARC records for import" tool for importing authorities as well starting from 3.10+
Comment 2 David Cook 2013-05-08 00:31:47 UTC Comment hidden (obsolete)
Comment 3 David Cook 2013-05-08 00:38:06 UTC
While I have no doubt about the functionality of this feature (I think it works quite well), I'm sure that the code could benefit from closer scrutiny by eyes other than mine. My code is almost entirely based off of the Z39.50 bibliographic searching, so there might be some vestigial code that I missed which can be removed or perhaps just an optimization of the code that I have included. 

Ideally, in the future, it would be nice if the Bib-1 attributes used for searching could be configurable rather than hard-coded as they are at present. They attributes represent what works for LibrariesAustralia and are a best guess. There are more specific Bib-1 attributes out there for subjects (e.g. geographic terms, topical terms, etc.), but these weren't supported by LibrariesAustralia and they might not be supported by other major organizations like OCLC, so I did not include them. I'm open to changing that though.

Jared also suggested using the QueryParser to allow the creation of configurable attributes/indexes, and this is something I would really be interested in doing in the future, since it would allow for different Z39.50 targets to use different Bib-1 attributes (which are often published by the target providers). 

As the QueryParser becomes more integrated into Koha, I would like it to become more a part of the external Z39.50 searches as well.
Comment 4 David Cook 2013-05-08 02:49:07 UTC
I think it might supposed to be ModAuthorityInBatch not ModAuthInBatch in the ImportBreedingAuth sub...

It never gets called, but worth fixing that mistake...I think this same typo might exist elsewhere in Koha as well...
Comment 5 David Cook 2013-05-08 05:18:27 UTC
Looks like there's lots of trailing whitespace and a typo in ImportBreedingAuth where controlnumber should be control_number...
Comment 6 David Cook 2013-05-08 07:21:20 UTC Comment hidden (obsolete)
Comment 7 David Cook 2013-05-13 02:44:51 UTC Comment hidden (obsolete)
Comment 8 Mason James 2013-06-04 21:13:52 UTC
(In reply to comment #7)
> Created attachment 18108 [details] [review] [review]
> Bug 10096 [DB Follow-up] - Add a Z39.50 interface for authority searching
> 
> This patch adds the "recordtype" column to the "z3950servers" table.
> 
> The value in this column (biblio or authority) then controls whether
> the z3950 server shows up in a bibliographic search (through the
> Acq and Cataloguing modules) or in an authority search (through
> the Authorities module).
> 
> I also edited the z3950 management console to show this value
> and allow users to edit it. The default value is "biblio", since
> the vast majority of z3950 targets will be bibliographic. However,
> there is an option to add/edit a z3950 target as a source of
> authority records.
> 
> Test Plan:
> 
> 1) Apply both patches

hmm, the 2nd patch does not apply to master (b74ac4a)

-------------------------------------------
head$ git reset --hard origin/master
HEAD is now at b74ac4a Bug 10307 - z39.50 server search tab shows z390.50
mason@xen1:~/g/head$ git bz attach 10096

Bug 10096 - Add a Z39.50 interface for authority searching (2)
Apply? [yn] y

Applying: Bug 10096 - Add a Z39.50 interface for authority searching
Bug 10096 [DB Follow-up] - Add a Z39.50 interface for authority searching
Apply? [yn] y

Applying: Bug 10096 [FOLLOW-UP] - Add a z39.50 interface for authority searching
fatal: sha1 information is lacking or useless (koha-tmpl/intranet-tmpl/prog/en/modules/cataloguing/z3950_auth_search.tt).
Repository lacks necessary blobs to fall back on 3-way merge.
Cannot fall back to three-way merge.
Patch failed at 0001 Bug 10096 [FOLLOW-UP] - Add a z39.50 interface for authority searching
-------------------------------------------
Comment 9 Mason James 2013-06-04 23:28:54 UTC Comment hidden (obsolete)
Comment 10 Mason James 2013-06-04 23:30:02 UTC
> > Test Plan:
> > 
> > 1) Apply both patches
> 
> hmm, the 2nd patch does not apply to master (b74ac4a)
> 

i've uploaded a 2nd patch that does apply to master
Comment 11 Mason James 2013-06-10 02:10:10 UTC Comment hidden (obsolete)
Comment 12 Mason James 2013-06-10 02:10:46 UTC Comment hidden (obsolete)
Comment 13 Mason James 2013-06-10 02:11:02 UTC Comment hidden (obsolete)
Comment 14 Mason James 2013-06-10 02:14:21 UTC
(In reply to comment #13)
> Created attachment 18791 [details] [review] [review]
> Bug 10096 [FOLLOW-UP] - fix tabs/whitespace errors to pass QA

i've tested these 2 patches, and added a follow-up patch to fix various whitespace errors
Comment 15 Katrin Fischer 2013-07-12 22:55:11 UTC
David, I am sorry, but the patch doesn't apply any longer, can you please provide a rebased patch?

Apply? [y]es/[n]o/[e]dit/[v]iew patch/[a]ccept all y
Applying: Bug 10096 [FOLLOW-UP] - Add a z39.50 interface for authority searching
fatal: sha1 information is lacking or useless (koha-tmpl/intranet-tmpl/prog/en/modules/cataloguing/z3950_auth_search.tt).
Repository lacks necessary blobs to fall back on 3-way merge.
Cannot fall back to three-way merge.
Comment 16 David Cook 2013-07-15 01:15:20 UTC
(In reply to Katrin Fischer from comment #15)
> David, I am sorry, but the patch doesn't apply any longer, can you please
> provide a rebased patch?
> 
> Apply? [y]es/[n]o/[e]dit/[v]iew patch/[a]ccept all y
> Applying: Bug 10096 [FOLLOW-UP] - Add a z39.50 interface for authority
> searching
> fatal: sha1 information is lacking or useless
> (koha-tmpl/intranet-tmpl/prog/en/modules/cataloguing/z3950_auth_search.tt).
> Repository lacks necessary blobs to fall back on 3-way merge.
> Cannot fall back to three-way merge.

Alas, I think that's Mason's follow-up, so he'll need to provide the rebased version. 

I can rebase my original, but it would be unfortunate to have to lose/redo his follow-ups.
Comment 17 David Cook 2013-07-15 10:50:03 UTC
Just noticed that I need to update the copyright on those new files to include at least a Parts Copyright for Prosentient Systems...
Comment 18 David Cook 2013-07-26 05:59:24 UTC Comment hidden (obsolete)
Comment 19 David Cook 2013-07-26 06:01:22 UTC Comment hidden (obsolete)
Comment 20 David Cook 2013-07-26 06:01:35 UTC Comment hidden (obsolete)
Comment 21 David Cook 2013-07-26 06:01:46 UTC Comment hidden (obsolete)
Comment 22 David Cook 2013-07-26 06:18:23 UTC
Hey Katrin! I've finally rebased these patches! (I feel like I've levelled up my vim and git skills...also this git bz interactive is wild!)

Here is a more up-to-date test plan:

0) Run updatedatabase.pl (to add "recordtype" to the z3950servers table.

1) Set up Z39.50 client targets for authority databases. (I've included
information at the bottom of this message for LibrariesAustralia's
test server for authorities as well as instructions on how to use
your Koha's z39.50 authority server as well. The Library of Congress
also has authority databases available (unsure if these are test or
prod), and you might have access to others through OCLC or RLIN. OCLC
provides login credentials for their test databases.

N.B. Make sure that your Z39.50 client target has a Record Type
of Authority, otherwise it won't display when you're doing a
Z3950 search for authorities.

2) Go to the Authorities module

3) Click on the new "Z39.50 search button"

4) Select your authority search targets from the list 

5) Do a search for an authority you would like using either the "Keyword"
input box or the more specific input boxes for names, subjects, subject
sub divisions, or titles. (I like searching Author (personal): Eric on
the LibrariesAustralia test DB.)

6) You should see a table listing the server, heading, authority type,
and two other columns (MARC and a nameless column). "Authority type"
is the type of authority it will become when imported in to Koha. In
the Eric example, "PERSO_NAME".

7) Click on "MARC" next to the results of interest to review the MARC
authority record.

8) When you're satisfied with a record, click on "Import".

9) The pop-up window will close and your original Koha window will
change to the "Adding authority Personal Name" screen (in the Eric
example).

10) All the relevant fields should be filled out for the record. Review
them and make any changes as necessary. (N.B. The 001 will be cleared
when saved, so if you have a use for the imported control number, move
it to the 010, 016, or 035 as appropriate. If you have a default value
for the 003, this will also likely be overwritten. Move it if necessary.
The 005 will also be updated when saved, so do not worry about that.)

11) When you're satisfied, click save.

12) Presto! You've imported your first authority record via Z39.50!
Comment 23 Katrin Fischer 2013-09-08 23:07:29 UTC
Hi David,

some very first observations from reading the code and the QA script:

The license on cataloguing/z3950_auth_search.pl is GPL2+ - maybe you could consider making it GPL3+?

There are some .pl files with SQL statements in them - I think it would be nicer to move them out to an existing or new module. I know that most of it probably is kind of a preexisting problem stemming from how we handle bibliographic records.

For the changes in Breeding.pm some unit tests should be provided.

In your updatedatabase statement could you specify after which column the new column should be added (little pet peeve of mine I guess :) )?
Comment 24 Katrin Fischer 2013-09-08 23:08:40 UTC
Oh, in Breeding.pm there are some left over TODOs and FIXMEs - maybe take a look at those again?
Comment 25 David Cook 2013-09-10 02:40:31 UTC
(In reply to Katrin Fischer from comment #23)
> Hi David,
> 
> some very first observations from reading the code and the QA script:
> 
> The license on cataloguing/z3950_auth_search.pl is GPL2+ - maybe you could
> consider making it GPL3+?
> 
> There are some .pl files with SQL statements in them - I think it would be
> nicer to move them out to an existing or new module. I know that most of it
> probably is kind of a preexisting problem stemming from how we handle
> bibliographic records.
> 
> For the changes in Breeding.pm some unit tests should be provided.
> 
> In your updatedatabase statement could you specify after which column the
> new column should be added (little pet peeve of mine I guess :) )?

Hi Katrin:

1) Sure. I've changed the license on cataloguing/z3950_auth_search.pl to GPL3+.

2) It is a pre-existing problem, so perhaps a separate bug should be opened instead for removing SQL from the .pl files and putting them in an existing or new module? (It's funny that we removed the defunct Z39.50 module since it wasn't being used and now it seems that we could use a new Z39.50 module :P). 

That'll actually be quite a large re-write, I think. 

3) As for the unit tests...I can't think of any way to really create a unit test for this, since "Z3950SearchAuth" is too tightly tied to the template and an actual logged in user. The same goes for the regular "Z3950Search".

There should probably be a better separation of code and template with the Z3950 searching, but that also seems like a pre-existing problem. I just copied the code and made it work for authorities instead of bibliographic records. 

I imagine fixing that would also be better as a separate bug. 

4) Fixed updatedatabase to specify an "AFTER" column (as well as removing MySQL backticks). 

5) I've removed the FIXMEs and TODOs from the parts of Breeding.pm that I've added. (They weren't "real" FIXMEs.)
Comment 26 David Cook 2013-09-10 02:45:22 UTC Comment hidden (obsolete)
Comment 27 David Cook 2013-09-10 02:46:43 UTC Comment hidden (obsolete)
Comment 28 David Cook 2013-09-10 02:48:11 UTC Comment hidden (obsolete)
Comment 29 Katrin Fischer 2013-09-29 20:55:31 UTC Comment hidden (obsolete)
Comment 30 Katrin Fischer 2013-09-29 20:55:56 UTC Comment hidden (obsolete)
Comment 31 Katrin Fischer 2013-09-29 20:56:31 UTC
Rebased and fixed some tabs the QA script was complaining about.
Comment 32 Kyle M Hall 2013-10-03 13:31:31 UTC
Created attachment 21762 [details] [review]
Bug 10096 - Add a Z39.50 interface for authority searching

This patch introduces a new Z39.50 interface for searching z39.50
compliant databases for MARC authority records.

These databases aren't as common as their bibliographic equivalents,
but they're out there and very useful. I have included info at the
bottom of this messsage for sample authority databases you can try.

To test this patch:

1) Set up Z39.50 client targets for authority databases. (I've included
information at the bottom of this message for LibrariesAustralia's
test server for authorities as well as instructions on how to use
your Koha's z39.50 authority server as well. The Library of Congress
also has authority databases available (unsure if these are test or
prod), and you might have access to others through OCLC or RLIN. OCLC
provides login credentials for their test databases.

2) Go to the Authorities module

3) Click on the new "Z39.50 search button"

4) Select your authoriy search targets from the list (N.B. bibliographic
targets will also display. These are of no use to you.)

5) Do a search for an authority you would like using either the "Raw"
input box or the more specific input boxes for names, subjects, subject
sub divisions, or titles. (I like searching Name (personal): Eric on
the LibrariesAustralia test DB.)

6) You should see a table listing the server, heading, authority type,
and two other columns (MARC and a nameless column). "Authority type"
is the type of authority it will become when imported in to Koha. In
the Eric example, "PERSO_NAME".

7) Click on "MARC" next to the results of interest to review the MARC
authority record.

8) When you're satisfied with a record, click on "Import".

9) The pop-up window will close and your original Koha window will
change to the "Adding authority Personal Name" screen (in the Eric
example).

10) All the relevant fields should be filled out for the record. Review
them and make any changes as necessary. (N.B. The 001 will be cleared
when saved, so if you have a use for the imported control number, move
it to the 010, 016, or 035 as appropriate. If you have a default value
for the 003, this will also likely be overwritten. Move it if necessary.
The 005 will also be updated when saved, so do not worry about that.)

11) When you're satisfied, click save.

12) Presto! You've imported your first authority record via Z39.50!

--

Here is the info for the LibrariesAustralia test Z39.50 authority
database:

Z39.50 server: LibrariesAustralia Authorities
Hostname: z3950-test.librariesaustralia.nla.gov.au
Port: 210
Database: AuthTraining
Userid: ANLEZ
Password: z39.50
Syntax: MARC21/USMARC
Encoding: utf8

-

The U.S.A. Library of Congress also provides Z39.50 access to its Name
and Subject Authorities (http://www.loc.gov/z3950/lcserver.html).

Name Authority:
Z39.50 server: Library of Congress Name Authority File
Hostname: lx2.loc.gov
Port: 210
Database: NAF
Syntax: MARC21/USMARC
Encoding: utf8

Subject Authority:
Z39.50 server: Library of Congress Subject Authority File
Hostname: lx2.loc.gov
Port: 210
Database: SAF
Syntax: MARC21/USMARC
Encoding: utf8

(N.B. Both of these databases also include title authorities.)

-

For testing purposes, you can also set up a Z39.50 client target,
which points at your own Koha instance's Z39.50 authority server.

To find the hostname, go to /etc/koha-conf.xml and find the value for
the <listen id="authorityserver"> element. Depending on your
configuration, this could be something like the following:

unix:/zebra/koha/var/run/zebradb/authoritysocket

(N.B. You might be using a different scheme than unix sockets...)

To find the database, scroll down to the bottom of koha-conf.xml until
you reach the <config> element. Within this, look for the value of the
element <authorityserver>. It should probably be "authorities".

To set up this Z39.50 client target in Koha...

Z39.50 server: my koha authorities
Hostname: unix:/zebra/koha/var/run/zebradb/authoritysocket
Port:
Database: authorities
Userid:
Password:
Syntax: MARC21/USMARC (or whichever flavour you need)
Encoding: utf8

Signed-off-by: Mason James <mtj@kohaaloha.com>

Bug 10096 [FOLLOW-UP] - Add a z39.50 interface for authority searching

This patch adds the "recordtype" column to the "z3950servers" table.

The value in this column (biblio or authority) then controls whether
the z3950 server shows up in a bibliographic search (through the
Acq and Cataloguing modules) or in an authority search (through
the Authorities module).

I also edited the z3950 management console to show this value
and allow users to edit it. The default value is "biblio", since
the vast majority of z3950 targets will be bibliographic. However,
there is an option to add/edit a z3950 target as a source of
authority records.

Test Plan:

1) Apply both patches
2) Run updatedatabase.pl (after setting your KOHA_CONF and PERL5
environmental variables)
3) Use the test plan from the 1st patch

N.B. Make sure that your Z39.50 client target has a Record Type
of Authority, otherwise it won't display when you're doing a
Z3950 search for authorities.

Signed-off-by: Mason James <mtj@kohaaloha.com>

Bug 10096 [FOLLOW-UP] - fix tabs/whitespace errors to pass QA

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Comment 33 Kyle M Hall 2013-10-03 13:31:48 UTC
Created attachment 21763 [details] [review]
Bug 10096 - [FOLLOW-UP] QA Improvements

0) Run updatedatabase.pl (to add "recordtype" to the z3950servers table.

1) Set up Z39.50 client targets for authority databases. (I've included
information at the bottom of this message for LibrariesAustralia's
test server for authorities as well as instructions on how to use
your Koha's z39.50 authority server as well. The Library of Congress
also has authority databases available (unsure if these are test or
prod), and you might have access to others through OCLC or RLIN. OCLC
provides login credentials for their test databases.

N.B. Make sure that your Z39.50 client target has a Record Type
of Authority, otherwise it won't display when you're doing a
Z3950 search for authorities.

2) Go to the Authorities module

3) Click on the new "Z39.50 search button"

4) Select your authority search targets from the list

5) Do a search for an authority you would like using either the "Keyword"
input box or the more specific input boxes for names, subjects, subject
sub divisions, or titles. (I like searching Author (personal): Eric on
the LibrariesAustralia test DB.)

6) You should see a table listing the server, heading, authority type,
and two other columns (MARC and a nameless column). "Authority type"
is the type of authority it will become when imported in to Koha. In
the Eric example, "PERSO_NAME".

7) Click on "MARC" next to the results of interest to review the MARC
authority record.

8) When you're satisfied with a record, click on "Import".

9) The pop-up window will close and your original Koha window will
change to the "Adding authority Personal Name" screen (in the Eric
example).

10) All the relevant fields should be filled out for the record. Review
them and make any changes as necessary. (N.B. The 001 will be cleared
when saved, so if you have a use for the imported control number, move
it to the 010, 016, or 035 as appropriate. If you have a default value
for the 003, this will also likely be overwritten. Move it if necessary.
The 005 will also be updated when saved, so do not worry about that.)

11) When you're satisfied, click save.

12) Presto! You've imported your first authority record via Z39.50!

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Comment 34 Galen Charlton 2013-10-04 15:35:09 UTC
Pushed to master.  Thanks, David!

But also note bug 10998, as the code duplication introduced by this patch concerns me -- not to the point of not pushing the feature, but were it earlier in the 3.14 cycle, I would have asked for some refactoring before this got in.