Bug 19735 - Move Perl deps definitions into a cpanfile
Summary: Move Perl deps definitions into a cpanfile
Status: CLOSED FIXED
Alias: None
Product: Koha
Classification: Unclassified
Component: Architecture, internals, and plumbing (show other bugs)
Version: Main
Hardware: All All
: P5 - low enhancement
Assignee: Julian Maurice
QA Contact: Testopia
URL:
Keywords: dependency
: 9979 (view as bug list)
Depends on:
Blocks: 24573 25607
  Show dependency tree / graph
 
Reported: 2017-12-02 08:56 UTC by Julian Maurice
Modified: 2021-03-01 13:44 UTC (History)
13 users (show)

See Also:
Change sponsored?: ---
Patch complexity: ---
Documentation contact:
Documentation submission:
Text to go in the release notes:
This enhancement moves us away from custom-built dependency management and to the widely adopted cpanfile format for perl dependency listing. If you are running koha from git for development purposes you can now install perl dependencies using standard perl tooling and the included cpanfile. This patch also introduces the ability to set maximum versions in our dependancy listing (and excluded versions too), which should help us better track our compatibility.
Version(s) released in:
20.05.00
Circulation function:

Attachments
Bug 19735: Script to generate cpanfile (DO NOT PUSH) (1.17 KB, patch)
2017-12-02 08:58 UTC, Julian Maurice 		Details | Diff | Splinter Review
Bug 19735: Move Perl deps definitions into a cpanfile (52.06 KB, patch)
2017-12-02 08:58 UTC, Julian Maurice 		Details | Diff | Splinter Review
Bug 19735: Script to generate cpanfile (DO NOT PUSH) (1.17 KB, patch)
2018-04-08 11:01 UTC, Julian Maurice 		Details | Diff | Splinter Review
Bug 19735: Move Perl deps definitions into a cpanfile (52.82 KB, patch)
2018-04-08 11:01 UTC, Julian Maurice 		Details | Diff | Splinter Review
Bug 19735: Move Perl deps definitions into a cpanfile (53.33 KB, patch)
2018-10-30 08:36 UTC, Julian Maurice 		Details | Diff | Splinter Review
Bug 19735: Script to generate cpanfile (DO NOT PUSH) (1.17 KB, patch)
2020-01-17 10:04 UTC, Julian Maurice 		Details | Diff | Splinter Review
Bug 19735: Move Perl deps definitions into a cpanfile (53.87 KB, patch)
2020-01-17 10:05 UTC, Julian Maurice 		Details | Diff | Splinter Review
Bug 19735: Script to generate cpanfile (DO NOT PUSH) (1.23 KB, patch)
2020-02-07 14:28 UTC, Martin Renvoize (ashimema) 		Details | Diff | Splinter Review
Bug 19735: Move Perl deps definitions into a cpanfile (29.18 KB, patch)
2020-02-07 14:28 UTC, Martin Renvoize (ashimema) 		Details | Diff | Splinter Review
Bug 19735: Remove C4::Installer::PerlDependancies (25.56 KB, patch)
2020-02-07 14:28 UTC, Martin Renvoize (ashimema) 		Details | Diff | Splinter Review
Bug 19735: (QA follow-up) Drop references to 'usage' (2.13 KB, patch)
2020-02-07 17:15 UTC, Martin Renvoize (ashimema) 		Details | Diff | Splinter Review
Bug 19735: Add support for max_ver (5.66 KB, patch)
2020-02-07 17:34 UTC, Martin Renvoize (ashimema) 		Details | Diff | Splinter Review
Bug 19735: Script to generate cpanfile (DO NOT PUSH) (1.23 KB, patch)
2020-02-10 08:48 UTC, Martin Renvoize (ashimema) 		Details | Diff | Splinter Review
Bug 19735: Move Perl deps definitions into a cpanfile (29.18 KB, patch)
2020-02-10 08:48 UTC, Martin Renvoize (ashimema) 		Details | Diff | Splinter Review
Bug 19735: Remove C4::Installer::PerlDependancies (25.43 KB, patch)
2020-02-10 08:48 UTC, Martin Renvoize (ashimema) 		Details | Diff | Splinter Review
Bug 19735: (QA follow-up) Drop references to 'usage' (2.13 KB, patch)
2020-02-10 08:48 UTC, Martin Renvoize (ashimema) 		Details | Diff | Splinter Review
Bug 19735: Add support for max_ver (7.48 KB, patch)
2020-02-10 08:48 UTC, Martin Renvoize (ashimema) 		Details | Diff | Splinter Review
Bug 19735: Update cpanfile (1.07 KB, patch)
2020-02-12 11:56 UTC, Jonathan Druart 		Details | Diff | Splinter Review
Bug 19735: Move Perl deps definitions into a cpanfile (29.28 KB, patch)
2020-02-12 13:33 UTC, Jonathan Druart 		Details | Diff | Splinter Review
Bug 19735: Remove C4::Installer::PerlDependancies (25.51 KB, patch)
2020-02-12 13:33 UTC, Jonathan Druart 		Details | Diff | Splinter Review
Bug 19735: (QA follow-up) Drop references to 'usage' (2.20 KB, patch)
2020-02-12 13:33 UTC, Jonathan Druart 		Details | Diff | Splinter Review
Bug 19735: Add support for max_ver (7.58 KB, patch)
2020-02-12 13:33 UTC, Jonathan Druart 		Details | Diff | Splinter Review
Bug 19735: Update cpanfile (1.14 KB, patch)
2020-02-12 13:33 UTC, Jonathan Druart 		Details | Diff | Splinter Review
Show Obsolete (17) View All Add an attachment (proposed patch, testcase, etc.)

Note You need to log in before you can comment on or make changes to this bug.
Description Julian Maurice 2017-12-02 08:56:37 UTC 
cpanfile is a format for describing CPAN dependencies for Perl applications.
It is more concise - thus easier to read and maintain - than C4::Installer::PerlDependencies, and allows to describe requirements more accurately (using version ranges or features for instance).
Additionally it can be read by tools such as cpanm or carton for an easy way to install dependencies on non-Debian-based systems.

For more information on cpanfile, see http://search.cpan.org/~miyagawa/Module-CPANfile-1.1002/lib/cpanfile.pod
Comment 1 Julian Maurice 2017-12-02 08:58:15 UTC 
Created attachment 69465 [details] [review]
Bug 19735: Script to generate cpanfile (DO NOT PUSH)

For testing purposes only
Comment 2 Julian Maurice 2017-12-02 08:58:18 UTC 
Created attachment 69466 [details] [review]
Bug 19735: Move Perl deps definitions into a cpanfile

cpanfile is a format for describing CPAN dependencies for Perl
applications.
It is more concise - thus easier to read and maintain - than
C4::Installer::PerlDependencies, and allows to describe requirements
more accurately (using version ranges or features for instance)
Additionally it can be read by tools such as cpanm or carton for an
easy way to install dependencies on non-Debian-based systems.

For more information on cpanfile, see
http://search.cpan.org/~miyagawa/Module-CPANfile-1.1002/lib/cpanfile.pod

This patch replace C4::Installer::PerlDependencies by an equivalent
cpanfile and update all scripts/modules that were using PerlDependencies
It also removes dead code from C4::Installer::PerlModules (some
subroutines were not used at all, except in unit tests)

Added dependencies:
 - Module::CPANfile
 - CPAN::Meta (dependency of Module::CPANfile, but we need a more recent
   version than the one Module::CPANfile requires)

Test plan:
  1. Go to About page, tab Perl modules and keep this browser tab open
  2. Apply patch
  3. Install Module::CPANfile and CPAN::Meta
    a. On Debian-based systems:
       # will install libcpan-meta-perl as a dependency
       sudo apt install libmodule-cpanfile-perl
    b. Others:
       # will install CPAN::Meta as a dependency
       sudo cpanm Module::CPANfile
  4. In a new browser tab, go to About page, tab Perl modules and compare
     the table with the one in the previous browser tab
     They should be identical, except for newly added dependencies
     (Module::CPANfile and CPAN::Meta)
  5. Do a 'standard' install
    a. perl Makefile.PL (select 'standard')
    b. make
    c. sudo make install
    d. Configure your database, web server, ... and go through the web
       install process
  6. Verify that the cpanfile got copied into PERL_MODULE_DIR (which
     should be /usr/share/koha/lib)
  7. Go to the about page of this fresh install and compare it with your
     dev install
  8. Verify that debian/list-deps still works
     This takes a lot of time and it may not be necessary to wait until
     the end. If you see some Debian package names that correspond to
     modules in cpanfile, it means it still works
     (you need apt-file for this script to work)
  9. Verify that koha_perl_deps.pl still works
 10. prove t/Installer_pm.t t/Installer_PerlModules.t
Comment 3 Roch D'Amour 2018-04-06 21:37:49 UTC 
I tried to apply the patch but got a merge conflict with current master and can't test. Sorry :/
Comment 4 Julian Maurice 2018-04-08 11:01:19 UTC 
Created attachment 73837 [details] [review]
Bug 19735: Script to generate cpanfile (DO NOT PUSH)

For testing purposes only
Comment 5 Julian Maurice 2018-04-08 11:01:23 UTC 
Created attachment 73838 [details] [review]
Bug 19735: Move Perl deps definitions into a cpanfile

cpanfile is a format for describing CPAN dependencies for Perl
applications.
It is more concise - thus easier to read and maintain - than
C4::Installer::PerlDependencies, and allows to describe requirements
more accurately (using version ranges or features for instance)
Additionally it can be read by tools such as cpanm or carton for an
easy way to install dependencies on non-Debian-based systems.

For more information on cpanfile, see
http://search.cpan.org/~miyagawa/Module-CPANfile-1.1002/lib/cpanfile.pod

This patch replace C4::Installer::PerlDependencies by an equivalent
cpanfile and update all scripts/modules that were using PerlDependencies
It also removes dead code from C4::Installer::PerlModules (some
subroutines were not used at all, except in unit tests)

Added dependencies:
 - Module::CPANfile
 - CPAN::Meta (dependency of Module::CPANfile, but we need a more recent
   version than the one Module::CPANfile requires)

Test plan:
  1. Go to About page, tab Perl modules and keep this browser tab open
  2. Apply patch
  3. Install Module::CPANfile and CPAN::Meta
    a. On Debian-based systems:
       # will install libcpan-meta-perl as a dependency
       sudo apt install libmodule-cpanfile-perl
    b. Others:
       # will install CPAN::Meta as a dependency
       sudo cpanm Module::CPANfile
  4. In a new browser tab, go to About page, tab Perl modules and compare
     the table with the one in the previous browser tab
     They should be identical, except for newly added dependencies
     (Module::CPANfile and CPAN::Meta)
  5. Do a 'standard' install
    a. perl Makefile.PL (select 'standard')
    b. make
    c. sudo make install
    d. Configure your database, web server, ... and go through the web
       install process
  6. Verify that the cpanfile got copied into PERL_MODULE_DIR (which
     should be /usr/share/koha/lib)
  7. Go to the about page of this fresh install and compare it with your
     dev install
  8. Verify that debian/list-deps still works
     This takes a lot of time and it may not be necessary to wait until
     the end. If you see some Debian package names that correspond to
     modules in cpanfile, it means it still works
     (you need apt-file for this script to work)
  9. Verify that koha_perl_deps.pl still works
 10. prove t/Installer_pm.t t/Installer_PerlModules.t
Comment 6 Julian Maurice 2018-04-08 11:01:51 UTC 
Patches rebased on master, cpanfile updated.
Comment 7 Alex Buckley 2018-10-30 00:46:17 UTC 
Can you please rebase your patches, the second patch is currently not applying cleanly on master, as the below output shows:

Apply? [(y)es, (n)o, (i)nteractive] y
Applying: Bug 19735: Script to generate cpanfile (DO NOT PUSH)
Applying: Bug 19735: Move Perl deps definitions into a cpanfile
Using index info to reconstruct a base tree...
M	C4/Installer.pm
M	C4/Installer/PerlDependencies.pm
M	C4/Installer/PerlModules.pm
M	Makefile.PL
M	about.pl
M	debian/list-deps
M	installer/install.pl
Falling back to patching base and 3-way merge...
Removing t/Installer_PerlDependencies.t
Auto-merging installer/install.pl
Auto-merging debian/list-deps
Auto-merging about.pl
Auto-merging Makefile.PL
Auto-merging C4/Installer/PerlModules.pm
CONFLICT (content): Merge conflict in C4/Installer/PerlModules.pm
CONFLICT (modify/delete): C4/Installer/PerlDependencies.pm deleted in Bug 19735: Move Perl deps definitions into a cpanfile and modified in HEAD. Version HEAD of C4/Installer/PerlDependencies.pm left in tree.
Auto-merging C4/Installer.pm
Failed to merge in the changes.
Patch failed at 0001 Bug 19735: Move Perl deps definitions into a cpanfile
The copy of the patch that failed is found in:
   /home/vagrant/kohaclone/.git/rebase-apply/patch
When you have resolved this problem run "git bz apply --continue".
If you would prefer to skip this patch, instead run "git bz apply --skip".
To restore the original branch and stop patching run "git bz apply --abort".
Patch left in /tmp/Bug-19735-Move-Perl-deps-definitions-into-a-cpanfi-S1GHo1.patch
Comment 8 Mark Tompsett 2018-10-30 04:59:03 UTC 
Not that lots of people run it anymore, but ./koha_perl_deps.pl should probably be tweaked too.
Comment 9 David Cook 2018-10-30 05:16:07 UTC 
Actually going to use this cpanfile now with plenv to do some troubleshooting...
Comment 10 David Cook 2018-10-30 05:18:59 UTC 
(In reply to David Cook from comment #9)
> Actually going to use this cpanfile now with plenv to do some
> troubleshooting...

Or rather I guess I'll install carton with plenv and go from there...
Comment 11 David Cook 2018-10-30 06:21:50 UTC 
I tried running this but I got a few failures:

! Installing the dependencies failed: Module 'DBD::mysql' is not installed, Module 'PDF::API2::Simple' is not installed, Module 'Text::Bidi' is not installed
! Bailing out the installation for /home/dcook/test/.
259 distributions installed
Installing modules failed
Comment 12 David Cook 2018-10-30 06:36:11 UTC 
(In reply to David Cook from comment #11)
> I tried running this but I got a few failures:
> 
> ! Installing the dependencies failed: Module 'DBD::mysql' is not installed,
> Module 'PDF::API2::Simple' is not installed, Module 'Text::Bidi' is not
> installed
> ! Bailing out the installation for /home/dcook/test/.
> 259 distributions installed
> Installing modules failed

Looking more at DBD::mysql... and it looks like it doesn't like mysql header files for MariaDB.  Hmmmmmmm.
Comment 13 David Cook 2018-10-30 06:38:42 UTC 
(In reply to David Cook from comment #12)
> (In reply to David Cook from comment #11)
> > I tried running this but I got a few failures:
> > 
> > ! Installing the dependencies failed: Module 'DBD::mysql' is not installed,
> > Module 'PDF::API2::Simple' is not installed, Module 'Text::Bidi' is not
> > installed
> > ! Bailing out the installation for /home/dcook/test/.
> > 259 distributions installed
> > Installing modules failed
> 
> Looking more at DBD::mysql... and it looks like it doesn't like mysql header
> files for MariaDB.  Hmmmmmmm.

Seems like a separate issue... https://github.com/perl5-dbi/DBD-mysql/issues/262
Comment 14 Julian Maurice 2018-10-30 08:36:03 UTC 
Created attachment 81612 [details] [review]
Bug 19735: Move Perl deps definitions into a cpanfile

cpanfile is a format for describing CPAN dependencies for Perl
applications.
It is more concise - thus easier to read and maintain - than
C4::Installer::PerlDependencies, and allows to describe requirements
more accurately (using version ranges or features for instance)
Additionally it can be read by tools such as cpanm or carton for an
easy way to install dependencies on non-Debian-based systems.

For more information on cpanfile, see
http://search.cpan.org/~miyagawa/Module-CPANfile-1.1002/lib/cpanfile.pod

This patch replace C4::Installer::PerlDependencies by an equivalent
cpanfile and update all scripts/modules that were using PerlDependencies
It also removes dead code from C4::Installer::PerlModules (some
subroutines were not used at all, except in unit tests)

Added dependencies:
 - Module::CPANfile
 - CPAN::Meta (dependency of Module::CPANfile, but we need a more recent
   version than the one Module::CPANfile requires)

Test plan:
  1. Go to About page, tab Perl modules and keep this browser tab open
  2. Apply patch
  3. Install Module::CPANfile and CPAN::Meta
    a. On Debian-based systems:
       # will install libcpan-meta-perl as a dependency
       sudo apt install libmodule-cpanfile-perl
    b. Others:
       # will install CPAN::Meta as a dependency
       sudo cpanm Module::CPANfile
  4. In a new browser tab, go to About page, tab Perl modules and compare
     the table with the one in the previous browser tab
     They should be identical, except for newly added dependencies
     (Module::CPANfile and CPAN::Meta)
  5. Do a 'standard' install
    a. perl Makefile.PL (select 'standard')
    b. make
    c. sudo make install
    d. Configure your database, web server, ... and go through the web
       install process
  6. Verify that the cpanfile got copied into PERL_MODULE_DIR (which
     should be /usr/share/koha/lib)
  7. Go to the about page of this fresh install and compare it with your
     dev install
  8. Verify that debian/list-deps still works
     This takes a lot of time and it may not be necessary to wait until
     the end. If you see some Debian package names that correspond to
     modules in cpanfile, it means it still works
     (you need apt-file for this script to work)
  9. Verify that koha_perl_deps.pl still works
 10. prove t/Installer_pm.t t/Installer_PerlModules.t
Comment 15 Julian Maurice 2018-10-30 08:38:48 UTC 
(In reply to M. Tompsett from comment #8)
> Not that lots of people run it anymore, but ./koha_perl_deps.pl should
> probably be tweaked too.

Testing koha_perl_deps.pl is part of the test plan, but no changes were required to make it work as it was already using C4::Installer::PerlModules
Comment 16 Mark Tompsett 2018-10-30 15:01:55 UTC 
Comment on attachment 73837 [details] [review]
Bug 19735: Script to generate cpanfile (DO NOT PUSH)

Review of attachment 73837 [details] [review]:
-----------------------------------------------------------------

::: misc/devel/generate_cpanfile.pl
@@ +14,5 @@
> +    $prereqs->{runtime}->{$type}->{$module} = $req->{min_ver};
> +}
> +
> +$prereqs->{runtime}->{requires}->{'Module::CPANfile'} = '1.1000';
> +$prereqs->{runtime}->{requires}->{'CPAN::Meta'} = '2.150006';

Before testing, I had 2.150005. I didn't follow the test plan (did a cpan2deb), so I'll need to recheck if the sudo apt install updates properly.
Comment 17 Mark Tompsett 2018-10-30 21:27:29 UTC 
Whatever for the 1-4 steps. I did the equivalent. It worked.

>   5. Do a 'standard' install
>     a. perl Makefile.PL (select 'standard')
>     b. make
>     c. sudo make install
>     d. Configure your database, web server, ... and go through the web
>        install process

Encountered a bump here, but got through it.
I was just accepting defaults mostly.


>   6. Verify that the cpanfile got copied into PERL_MODULE_DIR (which
>      should be /usr/share/koha/lib)

Yes.


>   7. Go to the about page of this fresh install and compare it with your
>      dev install

That's the problem. I'm stuck in a weird loop install in my kohadevbox. I'll probably destroy it, and up it just to restart. But perhaps there is something wrong in the installation steps?

Could you flush out the testing steps on a kohadevbox to painful detail?
Comment 18 David Cook 2018-10-30 23:19:02 UTC 
(In reply to M. Tompsett from comment #17)
> That's the problem. I'm stuck in a weird loop install in my kohadevbox. I'll
> probably destroy it, and up it just to restart. But perhaps there is
> something wrong in the installation steps?
> 

I sometimes get stuck in a weird install loop on kohadevbox too. Are you using Plack? I find restarting the Plack daemon usually fixes it.
Comment 19 Julian Maurice 2020-01-17 10:04:56 UTC 
Created attachment 97484 [details] [review]
Bug 19735: Script to generate cpanfile (DO NOT PUSH)

For testing purposes only
Comment 20 Julian Maurice 2020-01-17 10:05:00 UTC 
Created attachment 97485 [details] [review]
Bug 19735: Move Perl deps definitions into a cpanfile

cpanfile is a format for describing CPAN dependencies for Perl
applications.
It is more concise - thus easier to read and maintain - than
C4::Installer::PerlDependencies, and allows to describe requirements
more accurately (using version ranges or features for instance)
Additionally it can be read by tools such as cpanm or carton for an
easy way to install dependencies on non-Debian-based systems.

For more information on cpanfile, see
http://search.cpan.org/~miyagawa/Module-CPANfile-1.1002/lib/cpanfile.pod

This patch replace C4::Installer::PerlDependencies by an equivalent
cpanfile and update all scripts/modules that were using PerlDependencies
It also removes dead code from C4::Installer::PerlModules (some
subroutines were not used at all, except in unit tests)

Added dependencies:
 - Module::CPANfile
 - CPAN::Meta (dependency of Module::CPANfile, but we need a more recent
   version than the one Module::CPANfile requires)

Test plan:
  1. Go to About page, tab Perl modules and keep this browser tab open
  2. Apply patch
  3. Install Module::CPANfile and CPAN::Meta
    a. On Debian-based systems:
       # will install libcpan-meta-perl as a dependency
       sudo apt install libmodule-cpanfile-perl
    b. Others:
       # will install CPAN::Meta as a dependency
       sudo cpanm Module::CPANfile
  4. In a new browser tab, go to About page, tab Perl modules and compare
     the table with the one in the previous browser tab
     They should be identical, except for newly added dependencies
     (Module::CPANfile and CPAN::Meta)
  5. Do a 'standard' install
    a. perl Makefile.PL (select 'standard')
    b. make
    c. sudo make install
    d. Configure your database, web server, ... and go through the web
       install process
  6. Verify that the cpanfile got copied into PERL_MODULE_DIR (which
     should be /usr/share/koha/lib)
  7. Go to the about page of this fresh install and compare it with your
     dev install
  8. Verify that debian/list-deps still works
     This takes a lot of time and it may not be necessary to wait until
     the end. If you see some Debian package names that correspond to
     modules in cpanfile, it means it still works
     (you need apt-file for this script to work)
  9. Verify that koha_perl_deps.pl still works
 10. prove t/Installer_pm.t t/Installer_PerlModules.t
Comment 21 Julian Maurice 2020-01-17 10:05:15 UTC 
Patches rebased on master
Comment 22 Martin Renvoize (ashimema) 2020-01-21 17:03:29 UTC 
I like the idea of moving to cpanfile as it means moving away from koha specific code and toward well maintained wider community code..

This patchset all looks solid to me, but I'm not a packaging expert so I'm not sure how that side of things would be affected by this.. calling Mason and Mirko for comment.
Comment 23 David Cook 2020-01-22 01:41:01 UTC 
I'm really interested in this issue, but juggling a few things at the moment, so not going to do the full test.

However, I do want to say the following:

1. koha_perl_deps.pl is working great with this change
2. prove t/Installer_pm.t t/Installer_PerlModules.t passed its tests, although they spat out a bunch of Test::MockTime warnings on my non-Debian system running Perl 5.26.1. I imagine that's unrelated though.
Comment 24 Martin Renvoize (ashimema) 2020-02-07 14:28:03 UTC 
Created attachment 98568 [details] [review]
Bug 19735: Script to generate cpanfile (DO NOT PUSH)

For testing purposes only

Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Comment 25 Martin Renvoize (ashimema) 2020-02-07 14:28:06 UTC 
Created attachment 98569 [details] [review]
Bug 19735: Move Perl deps definitions into a cpanfile

cpanfile is a format for describing CPAN dependencies for Perl
applications.
It is more concise - thus easier to read and maintain - than
C4::Installer::PerlDependencies, and allows to describe requirements
more accurately (using version ranges or features for instance)
Additionally it can be read by tools such as cpanm or carton for an
easy way to install dependencies on non-Debian-based systems.

For more information on cpanfile, see
http://search.cpan.org/~miyagawa/Module-CPANfile-1.1002/lib/cpanfile.pod

This patch replace C4::Installer::PerlDependencies by an equivalent
cpanfile and update all scripts/modules that were using PerlDependencies
It also removes dead code from C4::Installer::PerlModules (some
subroutines were not used at all, except in unit tests)

Added dependencies:
 - Module::CPANfile
 - CPAN::Meta (dependency of Module::CPANfile, but we need a more recent
   version than the one Module::CPANfile requires)

Test plan:
  1. Go to About page, tab Perl modules and keep this browser tab open
  2. Apply patch
  3. Install Module::CPANfile and CPAN::Meta
    a. On Debian-based systems:
       # will install libcpan-meta-perl as a dependency
       sudo apt install libmodule-cpanfile-perl
    b. Others:
       # will install CPAN::Meta as a dependency
       sudo cpanm Module::CPANfile
  4. In a new browser tab, go to About page, tab Perl modules and compare
     the table with the one in the previous browser tab
     They should be identical, except for newly added dependencies
     (Module::CPANfile and CPAN::Meta)
  5. Do a 'standard' install
    a. perl Makefile.PL (select 'standard')
    b. make
    c. sudo make install
    d. Configure your database, web server, ... and go through the web
       install process
  6. Verify that the cpanfile got copied into PERL_MODULE_DIR (which
     should be /usr/share/koha/lib)
  7. Go to the about page of this fresh install and compare it with your
     dev install
  8. Verify that debian/list-deps still works
     This takes a lot of time and it may not be necessary to wait until
     the end. If you see some Debian package names that correspond to
     modules in cpanfile, it means it still works
     (you need apt-file for this script to work)
  9. Verify that koha_perl_deps.pl still works
 10. prove t/Installer_pm.t t/Installer_PerlModules.t

Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Comment 26 Martin Renvoize (ashimema) 2020-02-07 14:28:10 UTC 
Created attachment 98570 [details] [review]
Bug 19735: Remove C4::Installer::PerlDependancies

This patch does the actual removal of the PerlDependancies module.

Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Comment 27 Martin Renvoize (ashimema) 2020-02-07 14:29:56 UTC 
Signing off.. all works as expected.

I did a minor rebase whilst I was testing (new dependencies added to PerlDependancies.  As part of that I separated the 'remove PerlDependancies' step into it's own commit to make it easier to rebase and use the included test script for building an updated cpanfile at time of push.
Comment 28 Julian Maurice 2020-02-07 14:48:15 UTC 
(In reply to Martin Renvoize from comment #27)
> Signing off.. all works as expected.
> 
> I did a minor rebase whilst I was testing (new dependencies added to
> PerlDependancies.  As part of that I separated the 'remove PerlDependancies'
> step into it's own commit to make it easier to rebase and use the included
> test script for building an updated cpanfile at time of push.

I wanted to do that, but never did... Thanks :)
Comment 29 Martin Renvoize (ashimema) 2020-02-07 17:15:09 UTC 
Created attachment 98576 [details] [review]
Bug 19735: (QA follow-up) Drop references to 'usage'

The 'usage' key was dropped as part of this migration to cpanfile but
it was still refered to by the installed. This patch simply removes
references to it. After considering how we might add it back in I
decided that as it stands it wasn't a well maintained list and should be
re-implimented in the future in a more consistent form if we miss it.
Comment 30 Martin Renvoize (ashimema) 2020-02-07 17:34:12 UTC 
Created attachment 98577 [details] [review]
Bug 19735: Add support for max_ver
Comment 31 Martin Renvoize (ashimema) 2020-02-10 08:48:17 UTC 
Created attachment 98636 [details] [review]
Bug 19735: Script to generate cpanfile (DO NOT PUSH)

For testing purposes only

Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Comment 32 Martin Renvoize (ashimema) 2020-02-10 08:48:21 UTC 
Created attachment 98637 [details] [review]
Bug 19735: Move Perl deps definitions into a cpanfile

cpanfile is a format for describing CPAN dependencies for Perl
applications.
It is more concise - thus easier to read and maintain - than
C4::Installer::PerlDependencies, and allows to describe requirements
more accurately (using version ranges or features for instance)
Additionally it can be read by tools such as cpanm or carton for an
easy way to install dependencies on non-Debian-based systems.

For more information on cpanfile, see
http://search.cpan.org/~miyagawa/Module-CPANfile-1.1002/lib/cpanfile.pod

This patch replace C4::Installer::PerlDependencies by an equivalent
cpanfile and update all scripts/modules that were using PerlDependencies
It also removes dead code from C4::Installer::PerlModules (some
subroutines were not used at all, except in unit tests)

Added dependencies:
 - Module::CPANfile
 - CPAN::Meta (dependency of Module::CPANfile, but we need a more recent
   version than the one Module::CPANfile requires)

Test plan:
  1. Go to About page, tab Perl modules and keep this browser tab open
  2. Apply patch
  3. Install Module::CPANfile and CPAN::Meta
    a. On Debian-based systems:
       # will install libcpan-meta-perl as a dependency
       sudo apt install libmodule-cpanfile-perl
    b. Others:
       # will install CPAN::Meta as a dependency
       sudo cpanm Module::CPANfile
  4. In a new browser tab, go to About page, tab Perl modules and compare
     the table with the one in the previous browser tab
     They should be identical, except for newly added dependencies
     (Module::CPANfile and CPAN::Meta)
  5. Do a 'standard' install
    a. perl Makefile.PL (select 'standard')
    b. make
    c. sudo make install
    d. Configure your database, web server, ... and go through the web
       install process
  6. Verify that the cpanfile got copied into PERL_MODULE_DIR (which
     should be /usr/share/koha/lib)
  7. Go to the about page of this fresh install and compare it with your
     dev install
  8. Verify that debian/list-deps still works
     This takes a lot of time and it may not be necessary to wait until
     the end. If you see some Debian package names that correspond to
     modules in cpanfile, it means it still works
     (you need apt-file for this script to work)
  9. Verify that koha_perl_deps.pl still works
 10. prove t/Installer_pm.t t/Installer_PerlModules.t

Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Comment 33 Martin Renvoize (ashimema) 2020-02-10 08:48:25 UTC 
Created attachment 98638 [details] [review]
Bug 19735: Remove C4::Installer::PerlDependancies

This patch does the actual removal of the PerlDependancies module.

Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Comment 34 Martin Renvoize (ashimema) 2020-02-10 08:48:28 UTC 
Created attachment 98639 [details] [review]
Bug 19735: (QA follow-up) Drop references to 'usage'

The 'usage' key was dropped as part of this migration to cpanfile but
it was still refered to by the installed. This patch simply removes
references to it. After considering how we might add it back in I
decided that as it stands it wasn't a well maintained list and should be
re-implimented in the future in a more consistent form if we miss it.
Comment 35 Martin Renvoize (ashimema) 2020-02-10 08:48:31 UTC 
Created attachment 98640 [details] [review]
Bug 19735: Add support for max_ver

This patchset adds support for extracting 'max_ver' from the cpanfile so
we can use version ranges properly and report errors if we have modules
installed that do not fit within that version range.

Test plan:
1) Manually modify the module version of a required module in the cpanfile
   to have a max version greater than the version you have installed.
2) Run through the install proceedure and note the new warning that a
   module needs upgrade for the module in question.
3) The module should also be reported in the about page
Comment 36 Jonathan Druart 2020-02-12 11:56:01 UTC 
Created attachment 98743 [details] [review]
Bug 19735: Update cpanfile
Comment 37 Jonathan Druart 2020-02-12 11:57:52 UTC 
Does not work for me, I get:
Can't locate object method "phases" via package "CPAN::Meta::Prereqs" at /kohadevbox/koha/C4/Installer/PerlModules.pm line 55

on about.pl

% pmvers CPAN::Meta::Prereqs
2.150005

Debian 9.11

Am I missing something?

Also, do we really need the "Bug 19735: Add support for max_ver" patch? Maybe on its own bug report if needed?
Comment 38 Julian Maurice 2020-02-12 12:15:49 UTC 
(In reply to Jonathan Druart from comment #37)
> Does not work for me, I get:
> Can't locate object method "phases" via package "CPAN::Meta::Prereqs" at
> /kohadevbox/koha/C4/Installer/PerlModules.pm line 55
> 
> on about.pl
> 
> % pmvers CPAN::Meta::Prereqs
> 2.150005
> 
> Debian 9.11
> 
> Am I missing something?

You need CPAN::Meta 2.150006
Comment 39 Julian Maurice 2020-02-12 12:34:25 UTC 
(In reply to Jonathan Druart from comment #37)
> Also, do we really need the "Bug 19735: Add support for max_ver" patch?
> Maybe on its own bug report if needed?

On it own bug or not, it might become useful quickly (for example, see bug 24573)
Comment 40 Jonathan Druart 2020-02-12 12:58:28 UTC 
(In reply to Julian Maurice from comment #38)
> (In reply to Jonathan Druart from comment #37)
> > Does not work for me, I get:
> > Can't locate object method "phases" via package "CPAN::Meta::Prereqs" at
> > /kohadevbox/koha/C4/Installer/PerlModules.pm line 55
> > 
> > on about.pl
> > 
> > % pmvers CPAN::Meta::Prereqs
> > 2.150005
> > 
> > Debian 9.11
> > 
> > Am I missing something?
> 
> You need CPAN::Meta 2.150006

Weird, my docker container was not up-to-date, apt update then install fixed the problem.
I now have libcpan-meta-perl 2.150010-1

Note that the package is not in buster, Martin, Mason, how is that blocking?
Comment 41 Julian Maurice 2020-02-12 13:11:59 UTC 
(In reply to Jonathan Druart from comment #40)
> Note that the package is not in buster, Martin, Mason, how is that blocking?

CPAN::Meta is included in perl-modules-5.28 in buster
Comment 42 Jonathan Druart 2020-02-12 13:19:06 UTC 
(In reply to Julian Maurice from comment #41)
> (In reply to Jonathan Druart from comment #40)
> > Note that the package is not in buster, Martin, Mason, how is that blocking?
> 
> CPAN::Meta is included in perl-modules-5.28 in buster

Awesome!

Did you investigate those warnings?

t/Installer_PerlModules.t .. 1/6 [Wed Feb 12 13:16:29 2020] Installer_PerlModules.t: Subroutine CORE::GLOBAL::time redefined at /usr/share/perl5/Test/MockTime.pm line 20.
[Wed Feb 12 13:16:29 2020] Installer_PerlModules.t: Prototype mismatch: sub CORE::GLOBAL::time () vs none at /usr/share/perl5/Test/MockTime.pm line 20.
[Wed Feb 12 13:16:29 2020] Installer_PerlModules.t: Subroutine CORE::GLOBAL::localtime redefined at /usr/share/perl5/Test/MockTime.pm line 21.
[Wed Feb 12 13:16:29 2020] Installer_PerlModules.t: Prototype mismatch: sub CORE::GLOBAL::localtime (;$) vs none at /usr/share/perl5/Test/MockTime.pm line 21.
[Wed Feb 12 13:16:29 2020] Installer_PerlModules.t: Subroutine CORE::GLOBAL::gmtime redefined at /usr/share/perl5/Test/MockTime.pm line 22.
[Wed Feb 12 13:16:29 2020] Installer_PerlModules.t: Prototype mismatch: sub CORE::GLOBAL::gmtime (;$) vs none at /usr/share/perl5/Test/MockTime.pm line 22.
t/Installer_PerlModules.t .. ok
Comment 43 Julian Maurice 2020-02-12 13:24:13 UTC 
(In reply to Jonathan Druart from comment #42)
> Did you investigate those warnings?

No. I believe they are here on master too ? (not sure, I haven't checked in a while)
Comment 44 Jonathan Druart 2020-02-12 13:32:40 UTC 
(In reply to Julian Maurice from comment #43)
> (In reply to Jonathan Druart from comment #42)
> > Did you investigate those warnings?
> 
> No. I believe they are here on master too ? (not sure, I haven't checked in
> a while)

No, they are not.
Comment 45 Jonathan Druart 2020-02-12 13:33:18 UTC 
Created attachment 98744 [details] [review]
Bug 19735: Move Perl deps definitions into a cpanfile

cpanfile is a format for describing CPAN dependencies for Perl
applications.
It is more concise - thus easier to read and maintain - than
C4::Installer::PerlDependencies, and allows to describe requirements
more accurately (using version ranges or features for instance)
Additionally it can be read by tools such as cpanm or carton for an
easy way to install dependencies on non-Debian-based systems.

For more information on cpanfile, see
http://search.cpan.org/~miyagawa/Module-CPANfile-1.1002/lib/cpanfile.pod

This patch replace C4::Installer::PerlDependencies by an equivalent
cpanfile and update all scripts/modules that were using PerlDependencies
It also removes dead code from C4::Installer::PerlModules (some
subroutines were not used at all, except in unit tests)

Added dependencies:
 - Module::CPANfile
 - CPAN::Meta (dependency of Module::CPANfile, but we need a more recent
   version than the one Module::CPANfile requires)

Test plan:
  1. Go to About page, tab Perl modules and keep this browser tab open
  2. Apply patch
  3. Install Module::CPANfile and CPAN::Meta
    a. On Debian-based systems:
       # will install libcpan-meta-perl as a dependency
       sudo apt install libmodule-cpanfile-perl
    b. Others:
       # will install CPAN::Meta as a dependency
       sudo cpanm Module::CPANfile
  4. In a new browser tab, go to About page, tab Perl modules and compare
     the table with the one in the previous browser tab
     They should be identical, except for newly added dependencies
     (Module::CPANfile and CPAN::Meta)
  5. Do a 'standard' install
    a. perl Makefile.PL (select 'standard')
    b. make
    c. sudo make install
    d. Configure your database, web server, ... and go through the web
       install process
  6. Verify that the cpanfile got copied into PERL_MODULE_DIR (which
     should be /usr/share/koha/lib)
  7. Go to the about page of this fresh install and compare it with your
     dev install
  8. Verify that debian/list-deps still works
     This takes a lot of time and it may not be necessary to wait until
     the end. If you see some Debian package names that correspond to
     modules in cpanfile, it means it still works
     (you need apt-file for this script to work)
  9. Verify that koha_perl_deps.pl still works
 10. prove t/Installer_pm.t t/Installer_PerlModules.t

Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Comment 46 Jonathan Druart 2020-02-12 13:33:23 UTC 
Created attachment 98745 [details] [review]
Bug 19735: Remove C4::Installer::PerlDependancies

This patch does the actual removal of the PerlDependancies module.

Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Comment 47 Jonathan Druart 2020-02-12 13:33:27 UTC 
Created attachment 98746 [details] [review]
Bug 19735: (QA follow-up) Drop references to 'usage'

The 'usage' key was dropped as part of this migration to cpanfile but
it was still refered to by the installed. This patch simply removes
references to it. After considering how we might add it back in I
decided that as it stands it wasn't a well maintained list and should be
re-implimented in the future in a more consistent form if we miss it.

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Comment 48 Jonathan Druart 2020-02-12 13:33:31 UTC 
Created attachment 98747 [details] [review]
Bug 19735: Add support for max_ver

This patchset adds support for extracting 'max_ver' from the cpanfile so
we can use version ranges properly and report errors if we have modules
installed that do not fit within that version range.

Test plan:
1) Manually modify the module version of a required module in the cpanfile
   to have a max version greater than the version you have installed.
2) Run through the install proceedure and note the new warning that a
   module needs upgrade for the module in question.
3) The module should also be reported in the about page

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Comment 49 Jonathan Druart 2020-02-12 13:33:35 UTC 
Created attachment 98748 [details] [review]
Bug 19735: Update cpanfile

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Comment 50 Jonathan Druart 2020-02-12 14:33:30 UTC 
*** Bug 9979 has been marked as a duplicate of this bug. ***
Comment 51 Jonathan Druart 2020-02-12 14:35:11 UTC 
Either this one or bug 24642 will need to be adjusted.
Comment 52 Martin Renvoize (ashimema) 2020-02-12 16:31:32 UTC 
Nice work everyone!

Pushed to master for 20.05
Comment 53 Martin Renvoize (ashimema) 2020-02-12 16:39:02 UTC 
Note to RMaints.. this one is up to you whether you wish to backport it.. It removes C4::Installer::PerlDependancies and as such if you're backporting future bugs that require the addition of dependencies after this bug has been pushed, you will need to manually add them to PerlDependancies at backport time.
Comment 54 Jonathan Druart 2020-02-13 12:17:14 UTC 
Koha_Master_D9_MDB_Latest and Koha_Master_D9 are currently failing with:

koha_1       | #   Failed test 'No tests run for subtest "versions_info"'
koha_1       | #   at t/Installer_PerlModules.t line 35.
koha_1       | Can't locate object method "phases" via package "CPAN::Meta::Prereqs" at /kohadevbox/koha/C4/Installer/PerlModules.pm line 55.
koha_1       | # Looks like your test exited with 2 just after 6.
koha_1       | [10:05:40] t/Installer_PerlModules.t
Comment 55 Martin Renvoize (ashimema) 2020-02-16 12:01:29 UTC 
OK,

I've added the Meta dependencies as a cpan installed module for Jessie for master.. I don't believe we should go to the effort of backporting this dependency for Jessie given we will be deprecating support for that OS at the release of 20.05.

However, this does make me believe this enhancement should certainly NOT be backported.
Comment 56 Joy Nelson 2020-03-05 00:56:22 UTC 
Enhancement not pushed to 19.11.x