Bug 14994 - Add RSS/Atom feed proxy
Summary: Add RSS/Atom feed proxy
Status: Failed QA
Alias: None
Product: Koha
Classification: Unclassified
Component: Web services (show other bugs)
Version: Main
Hardware: All All
: P5 - low enhancement (vote)
Assignee: Martin Persson
QA Contact: Testopia
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2015-10-09 13:40 UTC by Martin Persson
Modified: 2017-06-16 10:23 UTC (History)
3 users (show)

See Also:
Change sponsored?: ---
Patch complexity: ---
Documentation contact:
Documentation submission:
Text to go in the release notes:
Version(s) released in:


Attachments
Bug 14994 - Request proxy (added syspref) (3.51 KB, patch)
2015-10-21 12:41 UTC, Martin Persson
Details | Diff | Splinter Review
Bug 14994 - Request proxy (added opac-proxy.pl) (3.89 KB, patch)
2015-10-21 12:42 UTC, Martin Persson
Details | Diff | Splinter Review
Screenshot showing OPAC home page during test (24.22 KB, image/png)
2016-03-23 13:33 UTC, Owen Leonard
Details
[SIGNED-OFF] Bug 14994 - Request proxy (added syspref) (3.55 KB, patch)
2017-04-17 13:34 UTC, Owen Leonard
Details | Diff | Splinter Review
[SIGNED-OFF] Bug 14994 - Request proxy (added opac-proxy.pl) (3.94 KB, patch)
2017-04-17 13:34 UTC, Owen Leonard
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description Martin Persson 2015-10-09 13:40:28 UTC
Some libraries uses a separate blog on an external site such as blogspot.com. We are working on enhancements to make Koha a fully-featured blog in the future, removing that need.

The external blogs publishes the content as RSS/Atom feeds which can then be pulled to the OPAC via some Javascript and/or Yahoo Pipes. Pipes was recently decomissioned and I am now trying to solve this issue. I do not have full information yet.

At present I have created a small web proxy that pulls RSS/Atom from an external source via LWP. This allows us to overcome the same-origin limitation. There are security considerations for this that must be looked at.
Comment 1 Martin Persson 2015-10-21 12:41:51 UTC
Created attachment 43690 [details] [review]
Bug 14994 - Request proxy (added syspref)

Adds a web proxy allowing external RSS and Atom feeds
to be displayed in OPAC and staff interface pages.

This patch adds system preference(s) under the 'Web services' tabs.

Sponsored-By: Halland County Library
Comment 2 Martin Persson 2015-10-21 12:42:14 UTC
Created attachment 43691 [details] [review]
Bug 14994 - Request proxy (added opac-proxy.pl)

This patch adds a new opac page that is used as an endpoint
for external web requests such as RSS feeds. This allows user
scripts to pull data via XHR requests from external resources
without violating the 'same-origin' restriction.

I would appreciate some feedback on security implications of
doing this. Might need to add some DoS protection etc.

The script takes a single parameter; 'id', the number of the
feed to retrieve. This is an offset into an array of URLs
stored in the system preference named 'RequestProxyURL'.
The offset i based on 1 to make it easier for non-technical
people. The first feed is '1', '0' is invalid.

Test plan:

1)  Apply the syspref patch first.
2)  Apply this patch.
3)  Enable the request proxy by going into System Preferences,
    Web Services, set RequestProxyEnabled to 'yes' and
    enter a test URL such as:
    http://hyltebiblioteken.blogspot.se/feeds/posts/default
    and:
    http://bokbastisarna.blogspot.com/feeds/posts/default
    into RequestProxyURL.
4)  Save and go to the new url:
    http://127.0.1.1/cgi-bin/koha/opac-proxy.pl?id=1
    You should get the content of the first URL with 'id' = 1.
    With 'id' = 2 you should get the second URL.
    With 'id' = 3 you should get a 503 error.
5)  Disable the service again, 'RequestProxyEnabled' = no.
6)  Go back to the proxy url, you should now receive a 503 error
    even for valid feed id's AND for invalid ones.

Sponsored-By: Halland County Library
Comment 3 Martin Persson 2015-10-22 12:24:35 UTC
For a working example which puts the first three posts from two separate Atom feeds on the OPAC front page via jQuery, add this code to the system preferences. It is used with the example URLs used in the test plan:

URL1: http://hyltebiblioteken.blogspot.se/feeds/posts/default
URL2: http://bokbastisarna.blogspot.com/feeds/posts/default

OpacNavRight
============
<div class="rss-widget-container">
<div class="rss-widget-head">
Rubriker från bloggen
</div>
<div class="rss-widget-body" id="bloggen">
<ul class="rss-items"></ul>
<div class="loading">
Laddar innehåll från bloggen. Detta kräver Javascript aktiverat i din webbläsare.
</div>
</div>
</div>

<div class="rss-widget-container">
<div class="rss-widget-head">
Bokbästisarna
</div>
<div class="rss-widget-body" id="bokbastisarna">
<ul class="rss-items"></ul>
<div class="loading">
Laddar innehåll från bloggen. Detta kräver Javascript aktiverat i din webbläsare.
</div>
</div>
</div> 
============

OPACUserJS
==========
$('#bloggen').ready(function() {
    var url = 'http://127.0.1.1/cgi-bin/koha/opac-proxy.pl?id=1';
    $.ajax(url, {
        dataType: 'xml',
        success: function(xmldata) {
            var item_html = '';
            var xml = $(xmldata).find('entry').slice(0,3).each(function(index, value) {
                var title = $(this).children('title');
                var link = $(this).children('link');
                item_html += '<li><a href="' + link.text() + '">' + title.text() + "</a></li>\n";
            });
            $('#bloggen div.loading').fadeOut('slow', function () {
                $(this).html(item_html);
                $(this).slideDown('slow');
            });
        }
    });
});


$('#bokbastisarna').ready(function() {
    var url = 'http://127.0.1.1/cgi-bin/koha/opac-proxy.pl?id=2';
    $.ajax(url, {
        dataType: 'xml',
        success: function(xmldata) {
            var item_html = '';
            var xml = $(xmldata).find('entry').slice(0,3).each(function(index, value) {
                var title = $(this).children('title');
                var link = $(this).children('link');
                item_html += '<li><a href="' + link.text() + '">' + title.text() + "</a></li>\n";
            });
            $('#bokbastisarna div.loading').fadeOut('slow', function () {
                $(this).html(item_html);
                $(this).slideDown('slow');
            });
        }
    });
});
==========

These examples are in Swedish, based on real-life usage.
Comment 4 Owen Leonard 2016-03-23 13:33:26 UTC
Created attachment 49439 [details]
Screenshot showing OPAC home page during test

As far as I can tell these patches are working correctly.

Is this what I should expect to see?
Comment 5 Magnus Enger 2017-02-06 15:22:53 UTC
(In reply to Owen Leonard from comment #4)
> Created attachment 49439 [details]
> Screenshot showing OPAC home page during test
> 
> As far as I can tell these patches are working correctly.
> 
> Is this what I should expect to see?

Laddar innehåll från bloggen etc = "Loading content from the blog. You need to have JavaScript activated in your browser for this to work."

So I don't think this is what you would expect to see. Here is an OPAC using something similar to the stuff here: https://hylte.bibkat.se/ (bottom right cordner).
Comment 6 Owen Leonard 2017-04-17 13:34:38 UTC
Created attachment 62214 [details] [review]
[SIGNED-OFF] Bug 14994 - Request proxy (added syspref)

Adds a web proxy allowing external RSS and Atom feeds
to be displayed in OPAC and staff interface pages.

This patch adds system preference(s) under the 'Web services' tabs.

Sponsored-By: Halland County Library

Signed-off-by: Owen Leonard <oleonard@myacpl.org>
Comment 7 Owen Leonard 2017-04-17 13:34:41 UTC
Created attachment 62215 [details] [review]
[SIGNED-OFF] Bug 14994 - Request proxy (added opac-proxy.pl)

This patch adds a new opac page that is used as an endpoint
for external web requests such as RSS feeds. This allows user
scripts to pull data via XHR requests from external resources
without violating the 'same-origin' restriction.

I would appreciate some feedback on security implications of
doing this. Might need to add some DoS protection etc.

The script takes a single parameter; 'id', the number of the
feed to retrieve. This is an offset into an array of URLs
stored in the system preference named 'RequestProxyURL'.
The offset i based on 1 to make it easier for non-technical
people. The first feed is '1', '0' is invalid.

Test plan:

1)  Apply the syspref patch first.
2)  Apply this patch.
3)  Enable the request proxy by going into System Preferences,
    Web Services, set RequestProxyEnabled to 'yes' and
    enter a test URL such as:
    http://hyltebiblioteken.blogspot.se/feeds/posts/default
    and:
    http://bokbastisarna.blogspot.com/feeds/posts/default
    into RequestProxyURL.
4)  Save and go to the new url:
    http://127.0.1.1/cgi-bin/koha/opac-proxy.pl?id=1
    You should get the content of the first URL with 'id' = 1.
    With 'id' = 2 you should get the second URL.
    With 'id' = 3 you should get a 503 error.
5)  Disable the service again, 'RequestProxyEnabled' = no.
6)  Go back to the proxy url, you should now receive a 503 error
    even for valid feed id's AND for invalid ones.

Sponsored-By: Halland County Library

Signed-off-by: Owen Leonard <oleonard@myacpl.org>
Comment 8 Jonathan Druart 2017-04-18 19:11:10 UTC
What about using YAML to give a "name" to urls, instead of depending on the order of the urls listed in the pref?
Comment 9 Marcel de Rooy 2017-06-16 10:23:19 UTC
Changing status. Please give feedback on questions raised.