Bug 35729 - Koha needs to handle ISBNs starting with 979 for cover images
Summary: Koha needs to handle ISBNs starting with 979 for cover images
Status: Needs Signoff
Alias: None
Product: Koha
Classification: Unclassified
Component: Cataloging (show other bugs)
Version: Main
Hardware: All All
: P5 - low normal
Assignee: Lucas Gass (lukeg)
QA Contact: Testopia
URL:
Keywords:
Depends on: 35755
Blocks:
  Show dependency treegraph
 
Reported: 2024-01-09 01:11 UTC by Kal
Modified: 2024-10-27 23:14 UTC (History)
9 users (show)

See Also:
Change sponsored?: ---
Patch complexity: Trivial patch
Documentation contact:
Documentation submission:
Text to go in the release notes:
Version(s) released in:
Circulation function:


Attachments
Bug 35729: Update Business::ISBN mimimum version to 3.008 (759 bytes, patch)
2024-01-09 18:15 UTC, Lucas Gass (lukeg)
Details | Diff | Splinter Review
Bug 35729: Deal with 979 ISBNs for Koha cover images (1.48 KB, patch)
2024-08-30 17:08 UTC, Lucas Gass (lukeg)
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description Kal 2024-01-09 01:11:36 UTC
Hi,

I noticed that there was a cover mismatch with a coce_amazon cover because the new 2024 title has an ISBN of 9798885795692 (using the 979 instead of 978). There is an Italian title with an ISBN of 8885795692 that is showing instead. 

It looks like they have updated the valid ISBN standards: https://isbn.org/about_isbn_standard and it seems like coce should be updated to recognize this new standard and not match 9798885795692 to be the same as 8885795692.

Thanks!
Kal
Comment 1 David Cook 2024-01-09 07:16:45 UTC
Thanks for reporting this one, Kal. I think this is going to have wide-ranging implications across a number of different areas of Koha (and other library systems). 

I haven't done anything with Coce but I'm going to think about some of those other consequences of this new(ish) prefix...
Comment 2 Lucas Gass (lukeg) 2024-01-09 14:45:23 UTC
I think Koha needs an updated Business::ISBN. I recently filed an issue, which is only an issue in older versions:

https://github.com/briandfoy/business-isbn/issues/25

Currently using Business::ISBN: 3.006
Comment 3 Lucas Gass (lukeg) 2024-01-09 18:15:39 UTC
Created attachment 160714 [details] [review]
Bug 35729: Update Business::ISBN mimimum version to 3.008
Comment 4 Lucas Gass (lukeg) 2024-01-09 18:16:29 UTC
The hard part here is packaging and I've CC'd Mason.
Comment 5 David Cook 2024-01-09 22:34:25 UTC
(In reply to Lucas Gass from comment #4)
> The hard part here is packaging and I've CC'd Mason.

It looks like Debian have a package for 3.008 libbusiness-isbn-perl, so we should be able to pluck it and libbusiness-isbn-data-perl off packages.debian.org and pop it into the Koha community apt repo. 

--

I was going to suggest adding unit tests, and that led me down a little rabbit hole. 

I think we will need to review the unit tests, since tests (like t/db_dependent/Koha.t) for things like C4::Koha::NormalizeISBN will probably start returning different values. 

I'm not sure what Business::ISBN does if you request a ISBN-10 from ISBN-13 prefixed with 979... but there are some existing bugs for this as well, so I'll add them as See Alsos...
Comment 6 David Cook 2024-01-09 22:56:35 UTC
(In reply to Kal from comment #0)
> I noticed that there was a cover mismatch with a coce_amazon cover because
> the new 2024 title has an ISBN of 9798885795692 (using the 979 instead of
> 978). There is an Italian title with an ISBN of 8885795692 that is showing
> instead. 

Actually... looking back on Kal's original comment... I'd like to hear some steps on how to reproduce this problem.

Because currently Koha will generate an empty "normalized_isbn" value for a ISBN starting with 979, so Koha shouldn't be trying to display any cover images. I just tested in koha-testing-docker and no cover image is even attempted with a record with the 9798885795692 ISBN. So... I'm not sure how this could be happening in Koha. 

Now I think that we should upgrade Business::ISBN because we do want to support ISBNs starting with 979. I think that will be a bit of work.

However, even if we do that, Amazon is still going to be a problem. From past experience, I don't think that Amazon actually supports ISBN-13 at all when it comes to cover images.

For example, note that all 3 of these URLs returns the same image. 
https://images-na.ssl-images-amazon.com/images/P/9798885795692.01.LZZZZZZZ.jpg
https://images-na.ssl-images-amazon.com/images/P/9788885795692.01.LZZZZZZZ.jpg
https://images-na.ssl-images-amazon.com/images/P/8885795692.01.LZZZZZZZ.jpg

In the past, I think that's why we've used the ISBN-10 version for requesting images from Amazon. 

ISBN-13 with 979 is going to be a problem for Amazon images in particular.
Comment 7 Kal 2024-01-09 23:11:01 UTC
I noticed my cover mismatch in Aspen Discovery, not Koha, but we pull from coce.bywatersolutions.com and the cover source was coce_amazon so it's a bit of a gray area but why I put in the request for coce to be updated. I hope that helps!

Thanks,
Kal
Comment 8 Lucas Gass (lukeg) 2024-01-09 23:20:33 UTC
> However, even if we do that, Amazon is still going to be a problem. From
> past experience, I don't think that Amazon actually supports ISBN-13 at all
> when it comes to cover images.

Well, that is a problem since 979 ISBN's cannot be converted to ISBN10.
Comment 9 David Cook 2024-01-09 23:21:13 UTC
This twigged another memory... 

(Note I'm not including lots of URLs here because Bugzilla always blocks me when I do.)

If you look at the Amazon entry for "Berry Pickers" by Amanda Peters, obviously Amazon do have a cover image for it.

But they're not using an ISBN to retrieve it. They use a different URL format with the key "91Yab+VyPgL._SY342_"

And if you look at the "Product Advertising API" documentation, you can see that the API returns something like this:

So https://m.media-amazon.com/images/I/91Yab+VyPgL._SY342_.jpg will be our book cover.

Note that 91Yab+VyPgL._SY342_ isn't the identifier for the book. It's an image ID. 

The ASIN looks like it's B0BTSC3TN6. 

It looks like you might be able to use an ISBN to lookup "paapi5/documentation/use-cases/search-with-external-identifiers.html"

However... I don't know whether they even support ISBN-13 for this lookup. 

Plus you need to be an Amazon Associate to use this API. 

--

So ultimately... Amazon cover images are going to be a problem whether or not we improve support for ISBN-13 979 prefixes througout Koha.
Comment 10 David Cook 2024-01-09 23:21:56 UTC
(In reply to Lucas Gass from comment #8)
> > However, even if we do that, Amazon is still going to be a problem. From
> > past experience, I don't think that Amazon actually supports ISBN-13 at all
> > when it comes to cover images.
> 
> Well, that is a problem since 979 ISBN's cannot be converted to ISBN10.

Yup :(
Comment 11 David Cook 2024-01-09 23:24:25 UTC
(In reply to Kal from comment #7)
> I noticed my cover mismatch in Aspen Discovery, not Koha, but we pull from
> coce.bywatersolutions.com and the cover source was coce_amazon so it's a bit
> of a gray area but why I put in the request for coce to be updated. I hope
> that helps!
> 
> Thanks,
> Kal

Thanks, Kal.

Unfortunately, I don't know enough about Coce or how Bywater populated it. 

But your report has certainly surfaced an interesting issue!
Comment 12 David Cook 2024-01-09 23:29:15 UTC
I wonder if we should change the name back to "Amazon images don't work for ISBNs starting with 979" and then have a separate report for upgrading Business::ISBN, since related but different?
Comment 13 Lucas Gass (lukeg) 2024-01-09 23:32:19 UTC
(In reply to David Cook from comment #12)
> I wonder if we should change the name back to "Amazon images don't work for
> ISBNs starting with 979" and then have a separate report for upgrading
> Business::ISBN, since related but different?

David, that's fine by me.
Comment 14 Lucas Gass (lukeg) 2024-07-15 16:33:07 UTC
(In reply to David Cook from comment #12)
> I wonder if we should change the name back to "Amazon images don't work for
> ISBNs starting with 979" and then have a separate report for upgrading
> Business::ISBN, since related but different?

We are noticing this problem now with Syndetics also. So maybe this bug report should cover all of these and how Koha is dealing with 979 ISBN's?
Comment 15 David Cook 2024-07-15 23:07:43 UTC
(In reply to Lucas Gass from comment #14)
> (In reply to David Cook from comment #12)
> > I wonder if we should change the name back to "Amazon images don't work for
> > ISBNs starting with 979" and then have a separate report for upgrading
> > Business::ISBN, since related but different?
> 
> We are noticing this problem now with Syndetics also. So maybe this bug
> report should cover all of these and how Koha is dealing with 979 ISBN's?

Yeah, maybe not a bad idea. 

I think this depends on bug 35755 although maybe I'm wrong.
Comment 16 Esther Melander 2024-07-24 15:04:16 UTC
The Coce image service is supposed to search Amazon, Google Books, and Openlibrary. Koha is not finding the 979 book cover image on Amazon because of the way Amazon is structuring the URL's for the 979 numbers (as discussed above). I think the same thing is happening on Google Books and Openlibrary. The URL's for Google Books and Openlibrary do not have the ISBN. The ISBN is in the content of the page.

I did a search for 9798885795777 and found the listings for the title did have a cover image on Google Books and Openlibrary. I am not sure Koha is doing a cascading search in the other services. If the priority is Amazon first and the search fails or times out, the other services should find it, if it exists. I am not sure I have seen an image pulled from Google Books or OpenLibrary in a very long time. The search either returns Amazon or nothing.

So in addition to adding support for the 979 numbers, we may need to take a look at what the Coce image search is doing.
Comment 17 David Cook 2024-07-25 01:06:18 UTC
I also noticed cover images not showing up as much lately (even for non-979 ISBNs) from sources like Google. Haven't looked into it yet.
Comment 18 Mathieu Saby 2024-07-25 07:58:15 UTC
Hi
I don't believe there is something new. 979 ISBN are in use in France since 2007, and since that date we have known that 979 isbn should never be converted to ISNB10 ;-) That was the whole idea behind introducing a new numbering system...

But I have remarked that some international tools, like our discovery tool, cannot handle this reality, probably because the USA only began to use 979 ISBN recently.
Comment 19 Lucas Gass (lukeg) 2024-08-30 14:24:46 UTC
(In reply to David Cook from comment #15)

> I think this depends on bug 35755 although maybe I'm wrong.


I think you're right. And Bug 35755 is in main now so we should be able to continue here now.
Comment 20 Lucas Gass (lukeg) 2024-08-30 16:27:53 UTC
In sub _isbn_cleanup we assume ISBN-10 as the format. This likely worked in the past but won't work with 979 ISBN's.
Comment 21 Lucas Gass (lukeg) 2024-08-30 17:08:58 UTC
Created attachment 170921 [details] [review]
Bug 35729: Deal with 979 ISBNs for Koha cover images

TO TEST:
1. APPLY PATCH and restart_all
2. Have some records with ISBN13's, both 979 and non-979.
3. Have some different ISBN's that are ISBN10.
4. Turn on each of the following cover image services, ensuring that ISBN's are loading on the staff results page, staff detail page. OPAC results page, and OPAC detail page:

- AmazonCoverImages
- OPACAmazonCoverImages
- Coce (all 3 services )
- Babeltheque
- BakerTaylorEnabled
- GoogleJackets
- NovelistSelectEnabled
- OpenLibraryCovers
- SyndeticsCoverImages
Comment 22 Lucas Gass (lukeg) 2024-08-30 17:26:55 UTC
It's going to be a bit difficult to test the cover images services that require a login/password. 

I was able to test with Baker and Taylor, Syndetics, Coce, Google Books, Amazon.

For Coce you can use 'https://coce.bywatersolutions.com' for the CoceHost sys pref.
Comment 23 David Nind 2024-10-27 23:14:07 UTC
I did have a go at testing, but I struggled to find records to add or books with a 979* ISBN and cover images in Amazon, GoogleJackets, and the Open Library.

Any example 979 ISBNs that have a cover image in any of the public services and that you know work would be great to help with testing.