Bug 29678 - Can't use external URL for XSLT stylesheet
Summary: Can't use external URL for XSLT stylesheet
Status: In Discussion
Alias: None
Product: Koha
Classification: Unclassified
Component: MARC Bibliographic data support (show other bugs)
Version: Main
Hardware: All All
: P5 - low major
Assignee: Bugs List
QA Contact: Testopia
URL:
Keywords:
Depends on: 23290
Blocks:
  Show dependency treegraph
 
Reported: 2021-12-10 12:39 UTC by Owen Leonard
Modified: 2024-12-12 05:13 UTC (History)
6 users (show)

See Also:
Change sponsored?: ---
Patch complexity: ---
Documentation contact:
Documentation submission:
Text to go in the release notes:
Version(s) released in:
Circulation function:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Owen Leonard 2021-12-10 12:39:40 UTC
If I try to specify an external URL as an XSLT stylesheet it doesn't work. In the error logs I see messages like this:

[2021/12/10 12:37:50] [WARN] XSLT::Base: Could not create file parser context for file "https://path/to/MARC21slim2intranetResults.xsl": No such file or directory at /kohadevbox/koha/Koha/XSLT/Base.pm line 330.

I have verified that the .xsl file is accessible and that internal links (to MARC21slimUtils.xsl for instance) are valid too.
Comment 1 Fridolin Somers 2021-12-11 08:19:00 UTC
In my tests I get logs :

204     Exception::Class::Base::throw('Koha::Exceptions::Metadata::Invalid', 'id', 368, 'biblionumber', 369, 'format', 'marcxml', 'schema', 'MARC21', 'decoding    _error', ':8: parser error : PCDATA invalid Char value 31^J  <controlfield tag="001">00^_aD000015937</controlfield>...) called at /kohadevbox/koha/Koha/Bibl    io/Metadata.pm line 73
205     Koha::Biblio::Metadata::record('Koha::Biblio::Metadata=HASH(0x55905c3efc40)') called at /kohadevbox/koha/Koha/Biblio.pm line 943
206     Koha::Biblio::get_marc_notes('Koha::Biblio=HASH(0x55905cbc1820)', 'HASH(0x55905cbbb5b0)') called at /kohadevbox/koha/catalogue/detail.pl line 148
207     eval {...} at /kohadevbox/koha/catalogue/detail.pl line 2

Very strange...

Maybe some impact I can't explain of Bug 28734 adding transformMARCXML4XSLT() to get_marc_notes().
Comment 2 Fridolin Somers 2021-12-11 08:28:39 UTC
(In reply to Fridolin Somers from comment #1)
> In my tests I get logs :
> 
> 204     Exception::Class::Base::throw('Koha::Exceptions::Metadata::Invalid',
> 'id', 368, 'biblionumber', 369, 'format', 'marcxml', 'schema', 'MARC21',

Ah forget that, biblionumber=369 has bad MARCXML on purpose in KTD
Comment 3 Marcel de Rooy 2021-12-12 13:00:04 UTC
(In reply to Owen Leonard from comment #0)
> If I try to specify an external URL as an XSLT stylesheet it doesn't work.
> In the error logs I see messages like this:
> 
> [2021/12/10 12:37:50] [WARN] XSLT::Base: Could not create file parser
> context for file "https://path/to/MARC21slim2intranetResults.xsl": No such
> file or directory at /kohadevbox/koha/Koha/XSLT/Base.pm line 330.
> 
> I have verified that the .xsl file is accessible and that internal links (to
> MARC21slimUtils.xsl for instance) are valid too.

Where did you specify it? Z3950/SRU target?
If you have an external IntranetResults.xsl, how do you get to the slimUtils? I remember to have opened a bug report for includes over https. (Was not possible)
Comment 4 Katrin Fischer 2021-12-12 13:09:37 UTC
I think this was filed for the XSLT system preferences (results and detail pages), from the discussion on IRC. 

I've found bug 25216 -  could this be the reason?
Comment 5 Marcel de Rooy 2021-12-13 07:04:38 UTC
Bug 12758
Comment 6 Fred King 2023-02-09 15:15:08 UTC
I'm having the same problem, but I'm using http, not https. I copied MARC21slim2OPACResults.xsl, MARC21slim2OPACDetail.xsl, and MARC21slimUtils.xsl to http://dokument.space/scratchmonkey/[filename]; the only thing I changed was the pointer to the Utils file. No data appear. 

HOWEVER, when I use a detail file that I created for an instance running on 17.11, http://dokument.space/scratchmonkey/authcatDetail.xsl, it works like a charm.
Comment 7 Christopher Brannon 2023-02-23 00:38:13 UTC
I pulled an XSL from 17.11 repository, and I can't even get that to load.  This really needs to be fixed.  Our hands are tied as long as we can't make customizations.
Comment 8 Katrin Fischer 2023-02-23 09:49:05 UTC
(In reply to Christopher Brannon from comment #7)
> I pulled an XSL from 17.11 repository, and I can't even get that to load. 
> This really needs to be fixed.  Our hands are tied as long as we can't make
> customizations.

Could you check for any included files at the top of the XSLT? Newer XSLT also include one for language descriptions, so it's not only the Utils that needs to be changed/updated.
Comment 9 Katrin Fischer 2023-02-23 09:49:28 UTC
(In reply to Katrin Fischer from comment #8)
> (In reply to Christopher Brannon from comment #7)
> > I pulled an XSL from 17.11 repository, and I can't even get that to load. 
> > This really needs to be fixed.  Our hands are tied as long as we can't make
> > customizations.
> 
> Could you check for any included files at the top of the XSLT? Newer XSLT
> also include one for language descriptions, so it's not only the Utils that
> needs to be changed/updated.

Ah, and we also need to check the Utils itself isn#t asking for other included files.
Comment 10 Christopher Brannon 2023-02-23 18:56:38 UTC
(In reply to Katrin Fischer from comment #9)
> (In reply to Katrin Fischer from comment #8)
> > (In reply to Christopher Brannon from comment #7)
> > > I pulled an XSL from 17.11 repository, and I can't even get that to load. 
> > > This really needs to be fixed.  Our hands are tied as long as we can't make
> > > customizations.
> > 
> > Could you check for any included files at the top of the XSLT? Newer XSLT
> > also include one for language descriptions, so it's not only the Utils that
> > needs to be changed/updated.
> 
> Ah, and we also need to check the Utils itself isn#t asking for other
> included files.

I'm not sure what you are asking for.  I am new in the XSLT realm, so you would have to guide me to what you are looking for.  I also don't have access to the backend, only what is available in the preferences.  So if I have to look at core files, I can't.
Comment 11 Katrin Fischer 2023-02-23 19:05:08 UTC
Hm, it would have been really helpful to see the error from the logs (server side), because it probably has the information we need.

Which files did you use that worked before?
Comment 12 Christopher Brannon 2023-02-23 19:17:47 UTC
(In reply to Katrin Fischer from comment #11)
> Hm, it would have been really helpful to see the error from the logs (server
> side), because it probably has the information we need.
> 
> Which files did you use that worked before?

Personally, we've only used the default settings, so we have not modified them yet.  But if I do anything aside from default in the preferences, nothing works.  It just goes blank.
Comment 13 Katrin Fischer 2023-02-23 19:29:47 UTC
If the XSLT is broken for whatever reason it will blank, but the logs will have the full error.

The XSLT you link or use from elsewhere for sure needs to be able to work on it's own. In Koha's XSLT files there are includes for other files which is a good practice when working with XSLT, but it makes it harder to move them to a different directory. You need to make sure all files needed are in one place and paths corrected etc.

When you can't access the error, this makes it harder of course. I'll try to give it a go and see if I can figure it out, but it will have to wait a little.
Comment 14 Fred King 2023-02-23 19:36:10 UTC
If Christopher's setup is similar to mine, I'm pretty sure the files he's using are 
MARC21slim2OPACDetail.xsl
MARC21slim2OPACResults.xsl
MARCslimUtils.xsl
MARC21Languages.xsl

OPACDetail and OPACResults both point to Utils, and Utils points to Languages.

I can't verify it right now, but I think you might be able to fix this by adding the full external address (i.e. <xsl:include href="cheerfulvalleypl.online/somedirectory/MARC21Languages.xsl"/> at the top of the Utils file.

The default location of the xsl stylesheets is
   /usr/share/koha/opac/htdocs/opac-tmpl/bootstrap/en/xslt
If you can get SFTP root access to that directory, you can make changes. Give the original a different name first. In any case when you're editing xsl files, I highly recommend saving each incremental change with a new name before you try it. That way if you break the file (very easy to do) you revert to the latest stable version. I'm not sure what gets saved in the browser cache, but I've had more luck seeing changes if Koha uses a new xsl filename when it reloads a page.
Comment 15 Fred King 2023-02-24 15:25:04 UTC
I might have figured this out.

I posted this to the Koha list to see if it works for someone other than me:

You need to modify three files. Let's say you put them in in http://dokument.space/anew

In opac_results.xml on line 10, change <xsl:import href="MARC21slimUtils.xsl"/> to    <xsl:import href= http://dokument.space/anew"MARC21slimUtils.xsl"/>
In opac_detail.xml do the same thing.
In MARC21slimUtils.xsl at line 8, change <xsl:include href="MARC21Languages.xsl"/> to <xsl:include href="document.space/anew/MARC21Languages.xsl"/>

You may need to clear your browser's cache or try it on a different computer. I really do have modified xml results and detail files in that location, in case anyone wants to look at the horrible things I've done to them as part of my rewrite of the MedStar Authors catalog.
Comment 16 Christopher Brannon 2023-03-30 21:49:20 UTC
I've essentially done this, although the ones I've tried to modify are MARC21Languages.xsl, MARC21slim2intranetDetail.xsl, and MARC21slimUtils.xsl.   No joy.
Comment 17 Christopher Brannon 2023-03-30 22:12:57 UTC
I just cannot get an external sheet to work at all.
Comment 18 Christopher Brannon 2023-03-30 22:16:14 UTC
(In reply to Katrin Fischer from comment #13)
> If the XSLT is broken for whatever reason it will blank, but the logs will
> have the full error.
> 
> The XSLT you link or use from elsewhere for sure needs to be able to work on
> it's own. In Koha's XSLT files there are includes for other files which is a
> good practice when working with XSLT, but it makes it harder to move them to
> a different directory. You need to make sure all files needed are in one
> place and paths corrected etc.
> 
> When you can't access the error, this makes it harder of course. I'll try to
> give it a go and see if I can figure it out, but it will have to wait a
> little.

It would be helpful to know which logs to look in.  Can you point me to the right one?  Also, I updated the paths Fred mentioned, but no luck.  Is there more than one path in each file to update?
Comment 19 Katrin Fischer 2023-04-08 15:31:18 UTC
> It would be helpful to know which logs to look in.  Can you point me to the
> right one?  Also, I updated the paths Fred mentioned, but no luck.  Is there
> more than one path in each file to update?

You need to check all the <xsl:import statements in all files.
Also make sure that your paths are correct.

I am not sure which log, but if you do something like 
tail /var/log/koha/$KOHA_INSTANCE/*.log with your instance name, that should bring them all up.
Comment 20 Phil Ringnalda 2024-07-06 23:01:34 UTC
It's possible this same thing was causing bug 37252 since we had one thing that's probably very rarely used which included MARC21slimUtils.xsl from the Library of Congress rather than our copy.
Comment 21 Phil Ringnalda 2024-07-06 23:22:37 UTC
Log for that isn't particularly helpful,

[WARN] XSLT::Base: I/O warning : failed to load external entity "https://www.loc.gov/standards/marcxml/xslt/MARC21slimUtils.xsl"
compilation error: file /kohadevbox/koha/koha-tmpl/intranet-tmpl/prog/en/xslt/MARC21slim2MADS.xsl element include
xsl:include : unable to load https://www.loc.gov/standards/marcxml/xslt/MARC21slimUtils.xsl
 at /kohadevbox/koha/Koha/XSLT/Base.pm line 346.

after two equally unhelpful 

[WARN] XSLT::Base: Unknown IO errorI/O warning : failed to load external entity "http://www.loc.gov/marcxml/xslt/MARC21slimUtils.xsl"

and

[WARN] XSLT::Base: Unknown IO errorI/O warning : failed to load external entity "http://www.loc.gov/standards/marcxml/xslt/MARC21slimUtils.xsl"

because the original loc.gov/marcxml returns a 301 to the http loc.gov/standards/marcxml and then that returns a 301 to https. Not getting very helpful debugging info from XSLT/Base.pm if it deals with a 301 by saying "Unknown IO errorI/O warning"
Comment 22 Katrin Fischer 2024-07-07 10:22:40 UTC
Hm, my guess would be that the URL got auto-guessed from the schema as we only have the file name in the include. Could that be?
I tihnk we already had concluded that all includes needed to be changed to have the full URL of the external or file in Koha in order to work.
Comment 23 Phil Ringnalda 2024-07-07 15:00:11 UTC
I didn't explain that very well. In https://git.koha-community.org/Koha-community/Koha/src/commit/f7e1559a7944abee1c1619cc1f1bbbd646622a60/koha-tmpl/intranet-tmpl/prog/en/xslt/MARC21slim2MADS.xsl#L9 we have the full URL to http://www.loc.gov/marcxml/xslt/MARC21slimUtils.xsl in an include.

That fails with [WARN] XSLT::Base: Unknown IO errorI/O warning : failed to load external entity "http://www.loc.gov/marcxml/xslt/MARC21slimUtils.xsl"

That URL actually returns a 301 to http://www.loc.gov/standards/marcxml/xslt/MARC21slimUtils.xsl so if you alter the URL in the include to be that, it again fails with [WARN] XSLT::Base: Unknown IO errorI/O warning : failed to load external entity "http://www.loc.gov/standards/marcxml/xslt/MARC21slimUtils.xsl"

Then when you see that http://www.loc.gov/standards/marcxml/xslt/MARC21slimUtils.xsl also redirects, to https at https://www.loc.gov/standards/marcxml/xslt/MARC21slimUtils.xsl and change the URL in the include to that URL, you still get [WARN] XSLT::Base: I/O warning : failed to load external entity "https://www.loc.gov/standards/marcxml/xslt/MARC21slimUtils.xsl"

It's not a perfect test since the URL isn't available at both http and https, and maybe it falls under one of the cases that bug 12758 didn't cover, not sure.
Comment 24 David Cook 2024-07-08 00:04:49 UTC
I think that bug 23290 prevents external stylesheets from being imported/included in an XSLT.

But I think you should be able to specify an external XSLT via the XSLT system preferences.
Comment 25 Katrin Fischer 2024-07-08 08:28:46 UTC
(In reply to David Cook from comment #24)
> I think that bug 23290 prevents external stylesheets from being
> imported/included in an XSLT.
> 
> But I think you should be able to specify an external XSLT via the XSLT
> system preferences.

I think that's what they are trying to do in the first place, but it's not working.
Comment 26 Katrin Fischer 2024-07-08 08:29:40 UTC
(In reply to Phil Ringnalda from comment #23)
> I didn't explain that very well. In
> https://git.koha-community.org/Koha-community/Koha/src/commit/
> f7e1559a7944abee1c1619cc1f1bbbd646622a60/koha-tmpl/intranet-tmpl/prog/en/
> xslt/MARC21slim2MADS.xsl#L9 we have the full URL to
> http://www.loc.gov/marcxml/xslt/MARC21slimUtils.xsl in an include.
> 
> That fails with [WARN] XSLT::Base: Unknown IO errorI/O warning : failed to
> load external entity "http://www.loc.gov/marcxml/xslt/MARC21slimUtils.xsl"
> 
> That URL actually returns a 301 to
> http://www.loc.gov/standards/marcxml/xslt/MARC21slimUtils.xsl so if you
> alter the URL in the include to be that, it again fails with [WARN]
> XSLT::Base: Unknown IO errorI/O warning : failed to load external entity
> "http://www.loc.gov/standards/marcxml/xslt/MARC21slimUtils.xsl"
> 
> Then when you see that
> http://www.loc.gov/standards/marcxml/xslt/MARC21slimUtils.xsl also
> redirects, to https at
> https://www.loc.gov/standards/marcxml/xslt/MARC21slimUtils.xsl and change
> the URL in the include to that URL, you still get [WARN] XSLT::Base: I/O
> warning : failed to load external entity
> "https://www.loc.gov/standards/marcxml/xslt/MARC21slimUtils.xsl"
> 
> It's not a perfect test since the URL isn't available at both http and
> https, and maybe it falls under one of the cases that bug 12758 didn't
> cover, not sure.

I didn't know about the existence of this file. Is this bug 37252 then?
Comment 27 David Cook 2024-07-08 23:11:23 UTC
(In reply to Katrin Fischer from comment #25)
> I think that's what they are trying to do in the first place, but it's not
> working.

Since I've done a fair bit of work with the XSLTs, I'll take a quick look...

--

http://localhost:8091/opac-tmpl/bootstrap/en/xslt/MARC21slim2OPACResults.xsl

[2024/07/08 23:09:24] [WARN] http error : Operation in progress
http error : Unknown IO error
[2024/07/08 23:09:24] [WARN] XSLT::Base: Could not create file parser context for file "http://localhost:8091/opac-tmpl/bootstrap/en/xslt/MARC21slim2OPACResults.xsl": No such file or directory at /kohadevbox/koha/Koha/XSLT/Base.pm line 3
34.

--

https://catalog.bywatersolutions.com/opac-tmpl/bootstrap/en/xslt/MARC21slim2OPACResults.xsl

[2024/07/08 23:10:20] [WARN] XSLT::Base: I/O warning : failed to load external entity "MARC21slimUtils.xsl"
compilation error: file unknown-5571f4c2a370 element import
xsl:import : unable to load MARC21slimUtils.xsl
 at /kohadevbox/koha/Koha/XSLT/Base.pm line 346.

--

I'll provide the explanations in another comment.
Comment 28 David Cook 2024-07-08 23:19:51 UTC
(In reply to David Cook from comment #27)
> [2024/07/08 23:09:24] [WARN] http error : Operation in progress
> http error : Unknown IO error
> [2024/07/08 23:09:24] [WARN] XSLT::Base: Could not create file parser
> context for file

When using a HTTP URL, the URL is handed directly to the XSLT library, which is constrained by the security restrictions added by bug 23290.

> [2024/07/08 23:10:20] [WARN] XSLT::Base: I/O warning : failed to load
> external entity "MARC21slimUtils.xsl"
> compilation error: file unknown-5571f4c2a370 element import
> xsl:import : unable to load MARC21slimUtils.xsl
>  at /kohadevbox/koha/Koha/XSLT/Base.pm line 346.

When using a HTTPS URL, the URL is first fetched by Koha code, and then presented to Koha as a string. Here it fails, probably because it can't find the file on the system. 

If loading a XSLT via HTTPS with properly set up filepaths, it should probably work. I'll see about doing a quick test over HTTPS...
Comment 29 David Cook 2024-07-08 23:29:41 UTC
(In reply to David Cook from comment #28)
> If loading a XSLT via HTTPS with properly set up filepaths, it should
> probably work. I'll see about doing a quick test over HTTPS...

Indeed it does. I've created a /usr/share/koha/opac/htdocs/opac-tmpl/bootstrap/en/xslt/MARC21slim2OPACResults2.xsl and updated MARC21slimUtils.xsl to /usr/share/koha/opac/htdocs/opac-tmpl/bootstrap/en/xslt/MARC21slimUtils.xsl and it's working fine.

Note that https://real.instance/opac-tmpl/bootstrap/en/xslt/MARC21slimUtils.xsl won't work here, because it's doing network activity, which is prohibited by bug 23290.

--

We could potentially refine the security features from bug 23290, but that's the state of things at the moment.

I think an argument could be made that we should fetch HTTP URLs the same way we do HTTPS URLs in the Koha code instead of the XSLT code, so that the external stylesheet works consistently in that way though.
Comment 30 David Cook 2024-07-08 23:37:44 UTC
Could people share their workflows for this?

Are you uploading your own XSLTs to the same domain as Koha? To other domains you control? Are you referencing someone else's XLSTs?

--

If we look at the "Using XML::LibXSLT::Security" section, we could modify our "read_net" callback to allow for reads if certain URL conditions are met.

For instance, the URL's hostname/domain matches one we expect (probably in koha-conf.xml for security reasons - a global system preference could be manipulated by the same person updating the XSLT system preference which defeats the point of having the security at all). We could also check for file extension, which typically isn't a super robust check, but would help prevent network requests for other things.