Bug 20783

Summary: Cannot embed some YouTube videos due to 403 errors
Product: Koha Reporter: Juan Romay Sieira <juan.sieira>
Component: OPACAssignee: Jonathan Druart <jonathan.druart>
Status: CLOSED FIXED QA Contact: Katrin Fischer <katrin.fischer>
Severity: normal    
Priority: P5 - low CC: aleisha, dcook, flyingendpaper, hayleypelham, jonathan.druart, kelly, lucas, mirko, tomascohen, victor
Version: Main   
Hardware: All   
OS: All   
Change sponsored?: --- Patch complexity: Small patch
Documentation contact: Documentation submission:
Text to go in the release notes:
Version(s) released in:
20.11.00, 20.05.01, 19.11.11, 19.05.16
Circulation function:
Bug Depends on: 21476    
Bug Blocks:    
Attachments: Patch to embed youtube videos
OPAC css
Patch to embed youtube videos
Bug 20783: Cannot embed some YouTube videos due to 403 errors
Bug 20783: Cannot embed some YouTube videos due to 403 errors
Bug 20783: Use iframe to embed Youtube videos
Bug 20783: Remove WWW::YouTube::Download dependency
youtube tab on OPAC
Bug 20783: Similar changes for OPAC
Bug 20783: Use iframe to embed Youtube videos
Bug 20783: Remove WWW::YouTube::Download dependency
Bug 20783: Similar changes for OPAC
Bug 20783: Similar changes for OPAC
Bug 20783: Similar changes for OPAC
Bug 20783: Add missing url filters
Bug 20783: Use iframe to embed Youtube videos
Bug 20783: Remove WWW::YouTube::Download dependency
Bug 20783: Similar changes for OPAC
Bug 20783: Add missing url filters
[19.11.x] Bug 20783: Use iframe to embed Youtube videos
[19.11.x] Bug 20783: Similar changes for OPAC
[19.11.x] Bug 20783: Add missing url filters

Description Juan Romay Sieira 2018-05-18 18:18:51 UTC
Some YouTube videos cannot be played beacause of a 403 error of the videoplayback URL. 

Video for testing: https://www.youtube.com/watch?v=haT8g7oKnns
Comment 1 Juan Romay Sieira 2018-05-18 18:26:20 UTC Comment hidden (obsolete)
Comment 2 Juan Romay Sieira 2018-05-18 18:33:53 UTC
To test the patch you have to:
* install a new package: HTML::Video::Embed
* overwrite your opac.css with the attached in the bug, or create your opac.css with this command from the opac-tmpl directory: lessc --clean-css="--s0 --advanced --compatibility=ie7" bootstrap/less/opac.less > bootstrap/css/opac.css
* Enable the syspref HTML5MediaEnabled to some interface, and HTML5MediaYouTube to embed videos.
* Edit a biblio and put some youtube video in a 856u field.
* Check it in OPAC or INTRANET.

The video https://www.youtube.com/watch?v=haT8g7oKnns can not be played before the patch, but after it has no problems.
Comment 3 Juan Romay Sieira 2018-05-18 18:35:03 UTC Comment hidden (obsolete)
Comment 4 Juan Romay Sieira 2018-05-18 18:48:38 UTC
Created attachment 75443 [details] [review]
Patch to embed youtube videos

This patch do not include the opac.css file
Comment 5 Owen Leonard 2018-10-02 16:38:52 UTC Comment hidden (obsolete)
Comment 6 Owen Leonard 2018-10-02 16:44:02 UTC
Created attachment 79833 [details] [review]
Bug 20783: Cannot embed some YouTube videos due to 403 errors

YouTube videos are embeded instead of played with its videoplayback
URL, that can throw a 403 error

To test the patch you have to:
    * Install a new package: HTML::Video::Embed
    * Rebuild the OPAC CSS (https://wiki.koha-community.org/wiki/Working_with_SCSS_in_the_OPAC_and_staff_client)
    * Enable the syspref HTML5MediaEnabled to some interface, and
      HTML5MediaYouTube to embed videos.
    * Edit a biblio and put some youtube video in a 856u field.
    * Check it in OPAC or INTRANET.

The video https://www.youtube.com/watch?v=haT8g7oKnns can not be played
before the patch, but after it has no problems.

Sponsored-by: UCA - Universidad de Cádiz
Comment 7 Owen Leonard 2018-10-02 16:45:20 UTC
I rebased this patch and converted the CSS changes to SCSS. I also modified the commit message to include the test plan.

It works well for me but I'm not sure about the explicit width and height for videos in the OPAC. Is it necessary? If so, shouldn't the same CSS change be added to the staff client?
Comment 8 Mirko Tietgen 2018-10-15 10:52:58 UTC
HTML::Video::Embed has not had an update in 4 years (version 0.016000). Have you looked into options that are actively developed?
Comment 9 Jonathan Druart 2018-10-19 13:50:24 UTC
Hi Juan,
I think the problem went from the fact that no video was available in webm (fmt=43) and that we could ask for the format list and get another URL.

The problem is that 403 is returned for other formats as well (mp4) for instance.

So apparently we should use an iframe to embed youtube videos (what does HTML::Video::Embed), and it is what Google suggests to do, but using JS, https://developers.google.com/youtube/iframe_api_reference

Would it work?
Comment 10 Jonathan Druart 2018-10-19 14:18:58 UTC
Additional infos:

The issue has been reported 3 years ago: https://github.com/xaicron/p5-www-youtube-download/issues/36

Does it only happen for audio-only?
Comment 11 Kelly McElligott 2020-04-28 13:51:35 UTC
This is still not working.  In 19.05.09-there is no tab appearing for the video as there was before.  I believe YouTube adjusted their API - this must have caused this.
Comment 12 Jonathan Druart 2020-05-12 10:58:19 UTC
Created attachment 104747 [details] [review]
Bug 20783: Use iframe to embed Youtube videos

WWW::YouTube::Download is broken and not reliable.
Other alternative was to use HTML::Video::Embed but not updated since
years.

The best alternative seems to follow youtube advise and use an iframe
https://developers.google.com/youtube/iframe_api_reference

Test plan:
Put youtube video in 856$u (using different url formats, youtu.be,
youtube.com/embed, etc.)
Enable HTML5MediaEnabled and HTML5MediaYouTube and confirm that the
youtube videos are correctly embeded.
Comment 13 Jonathan Druart 2020-05-12 10:58:22 UTC
Created attachment 104748 [details] [review]
Bug 20783: Remove WWW::YouTube::Download dependency

No longer used.
Comment 14 Jonathan Druart 2020-05-12 10:59:53 UTC
We are now hitting https://github.com/xaicron/p5-www-youtube-download/issues/55 I decided to remove the dependency and use the recommanded way. Please test these patches and confirm it fixes the different issues.
Comment 15 Kelly McElligott 2020-05-12 12:47:59 UTC
Hi,
This does work on the staff side as it once did, however, the OPAC portion is not working.  The play media tab shows on both the staff side and the OPAC, but the OPAC play media tab does not play the youtube video, only shows a black box - attempting to play the video.
Comment 16 Kelly McElligott 2020-05-12 12:48:21 UTC
Created attachment 104771 [details]
youtube tab on OPAC
Comment 17 Jonathan Druart 2020-05-12 13:05:45 UTC
Can you share what you have in 856$u?
Comment 18 Jonathan Druart 2020-05-12 13:11:19 UTC
Ho right, I did not modify the OPAC template!
Comment 19 Jonathan Druart 2020-05-12 13:14:02 UTC
Created attachment 104774 [details] [review]
Bug 20783: Similar changes for OPAC
Comment 20 Kelly McElligott 2020-05-12 13:20:54 UTC
Jonathan:

https://youtu.be/s16RSFQG4T0
Comment 21 ByWater Sandboxes 2020-05-12 13:27:27 UTC
Created attachment 104779 [details] [review]
Bug 20783: Use iframe to embed Youtube videos

WWW::YouTube::Download is broken and not reliable.
Other alternative was to use HTML::Video::Embed but not updated since
years.

The best alternative seems to follow youtube advise and use an iframe
https://developers.google.com/youtube/iframe_api_reference

Test plan:
Put youtube video in 856$u (using different url formats, youtu.be,
youtube.com/embed, etc.)
Enable HTML5MediaEnabled and HTML5MediaYouTube and confirm that the
youtube videos are correctly embeded.

Signed-off-by: Kelly McElligott <kelly@bywatersolutions.com>
Comment 22 ByWater Sandboxes 2020-05-12 13:27:30 UTC
Created attachment 104780 [details] [review]
Bug 20783: Remove WWW::YouTube::Download dependency

No longer used.

Signed-off-by: Kelly McElligott <kelly@bywatersolutions.com>
Comment 23 ByWater Sandboxes 2020-05-12 13:27:33 UTC
Created attachment 104781 [details] [review]
Bug 20783: Similar changes for OPAC

Signed-off-by: Kelly McElligott <kelly@bywatersolutions.com>
Comment 24 Kelly McElligott 2020-05-12 13:27:59 UTC
Working on both Staff and OPAC!
Comment 25 ByWater Sandboxes 2020-05-12 19:19:40 UTC
Created attachment 104799 [details] [review]
Bug 20783: Similar changes for OPAC

Signed-off-by: Kelly McElligott <kelly@bywatersolutions.com>

Signed-off-by: Kelly McElligott <kelly@bywatersolutions.com>
Comment 26 ByWater Sandboxes 2020-05-12 19:27:57 UTC
Created attachment 104800 [details] [review]
Bug 20783: Similar changes for OPAC

Signed-off-by: Kelly McElligott <kelly@bywatersolutions.com>

Signed-off-by: Kelly McElligott <kelly@bywatersolutions.com>

Signed-off-by: Jessica Zairo <jzairo@bywatersolutions.com>
Comment 27 Katrin Fischer 2020-05-21 13:13:05 UTC
Hi Jonathan, this is missing filters - the automatic script adds html, but I think url or uri might be needed?
Comment 28 Jonathan Druart 2020-05-21 13:47:07 UTC
Created attachment 105198 [details] [review]
Bug 20783: Add missing url filters
Comment 29 Katrin Fischer 2020-05-21 14:41:17 UTC
Created attachment 105210 [details] [review]
Bug 20783: Use iframe to embed Youtube videos

WWW::YouTube::Download is broken and not reliable.
Other alternative was to use HTML::Video::Embed but not updated since
years.

The best alternative seems to follow youtube advise and use an iframe
https://developers.google.com/youtube/iframe_api_reference

Test plan:
Put youtube video in 856$u (using different url formats, youtu.be,
youtube.com/embed, etc.)
Enable HTML5MediaEnabled and HTML5MediaYouTube and confirm that the
youtube videos are correctly embeded.

Signed-off-by: Kelly McElligott <kelly@bywatersolutions.com>

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Comment 30 Katrin Fischer 2020-05-21 14:41:21 UTC
Created attachment 105211 [details] [review]
Bug 20783: Remove WWW::YouTube::Download dependency

No longer used.

Signed-off-by: Kelly McElligott <kelly@bywatersolutions.com>

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Comment 31 Katrin Fischer 2020-05-21 14:41:25 UTC
Created attachment 105212 [details] [review]
Bug 20783: Similar changes for OPAC

Signed-off-by: Kelly McElligott <kelly@bywatersolutions.com>

Signed-off-by: Kelly McElligott <kelly@bywatersolutions.com>

Signed-off-by: Jessica Zairo <jzairo@bywatersolutions.com>

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Comment 32 Katrin Fischer 2020-05-21 14:41:29 UTC
Created attachment 105213 [details] [review]
Bug 20783: Add missing url filters

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Comment 33 Jonathan Druart 2020-06-15 09:59:05 UTC
Pushed to master for 20.11, thanks to everybody involved!
Comment 34 Lucas Gass (lukeg) 2020-06-17 22:43:42 UTC
backported to 20.05.x for 20.05.01
Comment 35 Aleisha Amohia 2020-06-22 03:19:27 UTC
this does not work on 19.11.x. please rebase if needed in 19.11.x
Comment 36 Tomás Cohen Arazi (tcohen) 2020-10-02 19:47:50 UTC
Created attachment 111177 [details] [review]
[19.11.x] Bug 20783: Use iframe to embed Youtube videos

WWW::YouTube::Download is broken and not reliable.
Other alternative was to use HTML::Video::Embed but not updated since
years.

The best alternative seems to follow youtube advise and use an iframe
https://developers.google.com/youtube/iframe_api_reference

Test plan:
Put youtube video in 856$u (using different url formats, youtu.be,
youtube.com/embed, etc.)
Enable HTML5MediaEnabled and HTML5MediaYouTube and confirm that the
youtube videos are correctly embeded.

Signed-off-by: Kelly McElligott <kelly@bywatersolutions.com>

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Comment 37 Tomás Cohen Arazi (tcohen) 2020-10-02 19:48:04 UTC
Created attachment 111178 [details] [review]
[19.11.x] Bug 20783: Similar changes for OPAC

Signed-off-by: Kelly McElligott <kelly@bywatersolutions.com>
Signed-off-by: Jessica Zairo <jzairo@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Comment 38 Tomás Cohen Arazi (tcohen) 2020-10-02 19:48:18 UTC
Created attachment 111179 [details] [review]
[19.11.x] Bug 20783: Add missing url filters

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Comment 39 Tomás Cohen Arazi (tcohen) 2020-10-02 19:49:19 UTC
(In reply to Aleisha Amohia from comment #35)
> this does not work on 19.11.x. please rebase if needed in 19.11.x

Please push the 19.11.x. They basically skip the library dependency portion which is not an issue.
Comment 40 Hayley Pelham 2020-10-13 22:19:05 UTC
Backported to 19.11.x for 19.11.11
Comment 41 Victor Grousset/tuxayo 2020-10-15 13:12:20 UTC
Not backported to oldoldstable (19.05.x). Feel free to ask if it's needed.
Comment 42 Jonathan Druart 2020-10-15 13:27:00 UTC
(In reply to Victor Grousset/tuxayo from comment #41)
> Not backported to oldoldstable (19.05.x). Feel free to ask if it's needed.

Yes I think it's needed.
Comment 43 Victor Grousset/tuxayo 2020-10-18 15:40:39 UTC
ok!

Backported to 19.05.x branch for 19.05.16