Bug 7152 - Koha should make it easy for librarians to attach covers using Open Library
Summary: Koha should make it easy for librarians to attach covers using Open Library
Status: NEW
Alias: None
Product: Koha
Classification: Unclassified
Component: Cataloging (show other bugs)
Version: master
Hardware: All All
: P4 enhancement with 1 vote (vote)
Assignee: Bugs List
QA Contact:
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2011-11-04 08:25 UTC by Robin Sheat
Modified: 2019-04-28 22:52 UTC (History)
6 users (show)

See Also:
Change sponsored?: ---
Patch complexity: ---
Who signed the patch off:
Text to go in the release notes:
Version(s) released in:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Robin Sheat 2011-11-04 08:25:26 UTC
While currently Koha can fetch covers from Open Library, it doesn't make it easy for them to be added. We should have adding the OL covers be part of the cataloguing process.

This would also extend to writing the OLID into records, probably using 035. This will require a field with a prefix to be defined by OL. See http://www.loc.gov/marc/organizations/ for info on this.

This will require two APIs to be defined by OL: a search-or-create interface that will find an ID for the current record if it can, and if not, it will create one; and a cover uploader interface that will allow us to upload a cover given an OLID.
Comment 1 Anand 2011-11-05 07:37:10 UTC
Added search API to Open Library. This can be used to find OLID from book metadata. The format of both request and response is JSON.

POST http://openlibrary.org/api/ils_search

Required Headers:
    Content-type: application/json
    Authorization: Basic base64-of-username:password (optional)

Payload:

    {
        "title": "book title",
        "authors": ["author 1", "author 2"],
        "publisher": "Name of the publisher",
        "publish_date": "June 2011",
        "isbns": ["1234567890", "234567890X"]
    }

Response Format:

    {
        "status": "found", 
        "olid": "OL24605334M",
        "key": "/books/OL24605334M", 
        "cover": {
            "small": "http://covers.openlibrary.org/b/id/6685633-S.jpg", 
            "large": "http://covers.openlibrary.org/b/id/6685633-L.jpg", 
            "medium": "http://covers.openlibrary.org/b/id/6685633-M.jpg"
        }
    }

Status can be one of found, created or notfound. The cover property will be available only if the book has a cover.

Currently the OL API tries to match only the ISBNs and doesn't create a new record if no match is found. This will be improved shortly.
Comment 2 Anand 2011-11-05 10:49:23 UTC
Added Cover Upload API to Open Library.

POST http://openlibrary.org/api/ils_cover_upload

Request Format: Following input fields with enctype multipart/form-data

    * authorization: base64 of username:password
    * olid: Key of the edition. e.g. OL12345M
    * file: file
    * redirect_url: URL to redirect after upload

On Success: Redirect to redirect_url?status=ok

On Failure: Redirect to redirect_url?status=error&reason=bad+olid
Comment 3 Robin Sheat 2011-11-06 03:17:03 UTC
The work-in-progress branch can be found here:

http://git.catalyst.net.nz/gw?p=koha.git;a=shortlog;h=refs/heads/bug7152-upload-covers-openlibrary
Comment 4 Robin Sheat 2011-11-06 07:09:45 UTC
Anand: I'm having issues with the API:

I send:
{"authors":["test"],"publish_date":null,"title":"Test 1","publisher":null,"isbns":["9781935928324"]}

And get back:
{"status": "notfound"}

I'm working on the assumption that it's currently only searching the ISBN. If you plug that ISBN into the search form on the webpage, you get a result.
Comment 5 Robin Sheat 2011-11-06 08:26:40 UTC
OL will have to apply for an organisation code. See here for US MARC21:

http://www.loc.gov/marc/organizations/

It would probably be worthwhile getting one (preferably the same) from the other countries' equivalents, too. I expect France has its own UNIMARC one.

For now I'm using 'OpenLib', but this is easy to change.
Comment 6 Robin Sheat 2011-11-06 08:47:08 UTC
Tried sending a more complete record:

{"authors":["Sifry, Micah L."],"publish_date":"2011","title":"WikiLeaks and the Age of Transparency","publisher":"OR Books","isbns":["9781935928324"]}

and still got the 'notfound' response.
Comment 7 Robin Sheat 2011-11-06 10:38:34 UTC
Anand: are there test records out there on OL that I can use to experiment with without messing around with real entries?
Comment 8 Robin Sheat 2011-11-06 11:16:49 UTC
When using the uploader thing with:

                <form action="http://openlibrary.org/api/ils_cover_upload"
                    method="post"
                    name="upload">
                    <input type="hidden" name="authorization" value="Bl4hB1AHblah==" />
                    <input type="hidden" name="olid" value="OL123TEST" />
                    <input type="hidden" name="redirect_url" id="ol_redirect_url" value="http://koha:8080/cgi-bin/koha/cataloguing/openlibrary.pl?biblionumber=8&redir=additem" />
                    <input type="file" name="file" />
                    <input type="submit" name="submit" value="Upload" />
                </form>

I get a document of type application/json with the content "None", rather than a redirect.

Note that the auth values I provided aren't valid currently, nor is the OLID I'm using for testing, so I expect a failure. However, I also hope for a redirect.
Comment 9 Anand 2011-11-06 17:17:32 UTC
Robin, the ils_search API currently works only with the exact matches. Here is an example that works. 

$ curl -H 'content-type: application/json' -d '{"authors":["test"],"publish_date":null,"title":"Test1","publisher":null,"isbn":["978-1-935928-32-4"]}' http://openlibrary.org/api/ils_search
{"status": "found", "cover": {"small": "http://covers.openlibrary.org/b/id/6685633-S.jpg", "large": "http://covers.openlibrary.org/b/id/6685633-L.jpg", "medium": "http://covers.openlibrary.org/b/id/6685633-M.jpg"}, "olid": "OL24605334M", "key": "/books/OL24605334M"}

I'll work on improving it in next couple of days.

And I forgot to deploy the upload API to production. Sorry about it. It is working now.
Comment 10 Anand 2011-11-07 05:45:46 UTC
Small correction to the ils_search API. It accepts "isbn" not "isbns".
Comment 11 Dobrica Pavlinusic 2011-11-24 18:15:49 UTC
Exact ISBN matches are problem for Koha side. We can't re-create exact format since parts of it denote country, publisher, edition and even if we know where to insert dash for country, guessing correct split between country and publisher is impossible.

I noticed that general web search on OpenLibrary site using ISBNs without dashes work, so there is that index somewhere, can we get it for /ils_search ?

Given wiki nature of OpenLibrary (and theoretical possibility of editing ISBN dashes) I think it makes sense :-)

Robin in fact runs following regexp before calling your API:

 s/[^0-9X]//g

just to clean-up data from Koha before submitting.
Comment 12 Anand 2011-11-24 18:30:24 UTC
The ils_search API is already improved to handle hyphens in the ISBNs. It even handles the isbn-10 to isbn-13 (and vice-versa) conversion internally.

We are working on implementing the title/author match when isbn is not provided.
Comment 13 Dobrica Pavlinusic 2011-11-24 20:16:43 UTC
Hm? Example included above works, but when I remove slashes it doesn't:

$ curl -H 'content-type: application/json' -d '{"authors":["test"],"publish_date":null,"title":"Test1","publisher":null,"isbn":["9781935928324"]}' http://openlibrary.org/api/ils_search
{"status": "notfound"}
Comment 14 Dobrica Pavlinusic 2011-11-24 20:42:42 UTC
Test scenario:
1. open OpenLibrary account with name ending in Bot to indicate it's a robot
2. edit OpenLibraryUsername and OpenLibraryPassword system preferences
   (link to create page handy is included in preference description)
3. change UploadOpenLibraryCovers to ask for cover upload
4. ??

It seems that git checkout miss cataloguing/openlibrary.pl which is referenced from cataloguing/addbiblio.pl
Comment 15 Robin Sheat 2011-11-24 20:55:22 UTC
Whoops, the openlibrary.pl file is somewhat important. I'll fix that up soon. Also note that this is still a work in progress, I haven't had a lot of time to finish it up since getting back. Hope to soon though.
Comment 16 Robin Sheat 2011-11-25 02:13:00 UTC
I've added the openlibrary.pl file now.
Comment 17 Anand 2011-11-25 05:44:08 UTC
Fixed ils_search.

$ curl -H 'content-type: application/json' -d '{"authors":["test"],"publish_date":null,"title":"Test1","publisher":null,"isbn":["9781935928-324"]}' http://openlibrary.org/api/ils_search
{"status": "found", "cover": {"small": "http://covers.openlibrary.org/b/id/6685633-S.jpg", "large": "http://covers.openlibrary.org/b/id/6685633-L.jpg", "medium": "http://covers.openlibrary.org/b/id/6685633-M.jpg"}, "olid": "OL24605334M", "key": "/books/OL24605334M"}
Comment 18 Noufal Ibrahim 2011-12-22 10:20:24 UTC
I'm refactoring parts of the OL import system to support this. There's some non trivial work from our end. I'll keep this bug posted with the details.