Bug 25591

Summary: Update list-deps for Debian 10 and Ubuntu 20.04
Product: Koha Reporter: Mason James <mtj>
Component: PackagingAssignee: Mason James <mtj>
Status: CLOSED FIXED QA Contact: Tomás Cohen Arazi <tomascohen>
Severity: blocker    
Priority: P1 - high CC: aleisha, jonathan.druart, lucas, martin.renvoize, mirko, tomascohen, victor
Version: 19.05   
Hardware: All   
OS: All   
See Also: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17084
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=26109
Change sponsored?: --- Patch complexity: ---
Documentation contact: Documentation submission:
Text to go in the release notes:
Version(s) released in:
20.11.00, 20.05.01, 19.11.07, 19.05.15
Bug Depends on: 25607    
Bug Blocks: 24167, 25640    
Attachments: add 5.26 and 5.30 to control
Bug 25591 - Update debian/control for debian 10
update list-deps
update list-deps
Bug 25591 - Update list-deps for Debian 10 and Ubuntu 20.04
Bug 25591 - Update list-deps for Debian 10 and Ubuntu 20.04
list-deps-debian10-master
list-deps-debian9-master
list-deps-debian9-master-with-25591
list-deps-debian10-master
list-deps-debian10-master-with-25591
list-deps-ubuntu20.04-master
list-deps-ubuntu20.04-master-with-25591
Bug 25591: Update list-deps for Debian 10 and Ubuntu 20.04
Bug 25591 - add libcpan-meta-perl to debina/control.in file
Bug 25591: Update list-deps for Debian 10 and Ubuntu 20.04

Description Mason James 2020-05-22 21:01:12 UTC
this patch updates the debian/control for koha 20.05 on debian 10
Comment 2 Mason James 2020-05-22 21:05:06 UTC
Created attachment 105282 [details] [review]
Bug 25591 - Update debian/control for debian 10

this patch updates the debian/control for koha 20.05 on debian 10
Comment 3 Mason James 2020-05-22 21:06:27 UTC
(In reply to Mason James from comment #2)
> Created attachment 105282 [details] [review] [review]
> Bug 25591 - Update debian/control for debian 10
> 
> this patch updates the debian/control for koha 20.05 on debian 10

..adds support perl-5.30 support for ubuntu focal too
Comment 4 Martin Renvoize 2020-05-22 21:32:58 UTC
Happy to include this and it makes total sense.. but, I'd really like to have a reminder if how this relates to bug 17084 and friends.
Comment 5 Martin Renvoize 2020-05-23 07:26:56 UTC
Nice work everyone!

Pushed to master for 20.05
Comment 8 Mason James 2020-05-24 03:17:36 UTC
Created attachment 105312 [details] [review]
Bug 25591 - Update list-deps for Debian 10 and Ubuntu 20.04

this patch removes version-specific packages from the debian/control file, which fixes a bug where koha-common can only be installed on the debian version that the package was built on
Comment 9 Victor Grousset/tuxayo 2020-05-25 00:47:01 UTC
What is the test plan? Just having the packages install successfully?
Comment 10 Martin Renvoize 2020-05-25 18:54:53 UTC
Created attachment 105326 [details] [review]
Bug 25591 - Update list-deps for Debian 10 and Ubuntu 20.04

This patch removes version-specific packages from the debian/control file,
which fixes a bug where koha-common can only be installed on the debian
version that the package was built on

Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Comment 11 Martin Renvoize 2020-05-25 18:56:59 UTC
This scans reasonably to me, but I must admit it's somewhat outside my area's of expertise. It would be great to have verification from Tomas too :)
Comment 12 Victor Grousset/tuxayo 2020-05-25 22:37:20 UTC
Created attachment 105334 [details]
list-deps-debian10-master
Comment 13 Victor Grousset/tuxayo 2020-05-25 22:38:34 UTC
Created attachment 105335 [details]
list-deps-debian9-master
Comment 14 Victor Grousset/tuxayo 2020-05-25 22:38:51 UTC
Created attachment 105336 [details]
list-deps-debian9-master-with-25591
Comment 15 Victor Grousset/tuxayo 2020-05-25 22:39:09 UTC
Created attachment 105337 [details]
list-deps-debian10-master
Comment 16 Victor Grousset/tuxayo 2020-05-25 22:39:20 UTC
Created attachment 105338 [details]
list-deps-debian10-master-with-25591
Comment 17 Victor Grousset/tuxayo 2020-05-25 22:39:32 UTC
Created attachment 105339 [details]
list-deps-ubuntu20.04-master
Comment 18 Victor Grousset/tuxayo 2020-05-25 22:39:51 UTC
Created attachment 105340 [details]
list-deps-ubuntu20.04-master-with-25591
Comment 19 Victor Grousset/tuxayo 2020-05-25 22:45:04 UTC
In case that helps reviewing, here attached are the results of my attempt to sign off. The list-deps was generated with and without the patch on Debian 9, Debian 10, Ubuntu 20.04.

There are sorted and trailing space was removed to clean the diffs.

As expected, packages perl-modules-5.XX and libperl5.XX aren't in the list anymore.

That's the only difference, no sign of collateral damage.
Comment 20 Tomás Cohen Arazi 2020-05-26 13:32:52 UTC
(In reply to Victor Grousset/tuxayo from comment #19)
> In case that helps reviewing, here attached are the results of my attempt to
> sign off. The list-deps was generated with and without the patch on Debian
> 9, Debian 10, Ubuntu 20.04.
> 
> There are sorted and trailing space was removed to clean the diffs.
> 
> As expected, packages perl-modules-5.XX and libperl5.XX aren't in the list
> anymore.
> 
> That's the only difference, no sign of collateral damage.

Thanks, Victor. I think the patch is incorrect, in the sense that what we need is to map perl dependencies to Debian package providing it. So in order to support more Debian versions, the generated output should have pipe-separated values for those libraries that map into perl-modules or perl-modules-<version>.

so we *should* expect a line containing:

perl-modules | perl-modules-5.22 | perl-modules-5.24 | perl-modules-5.28 | perl-modules-5.30

unless there's syntax for specifying regexps in module names, I belive we should stick with the current approach for this release, and clean things next.
Comment 21 Tomás Cohen Arazi 2020-05-26 13:53:01 UTC
I write this down so everyone knows how to check this (posterity):


`
docker run --volume=/home/tcohen/git/koha:/koha -ti debian:stretch bash
apt update
apt install apt-file wget gnupg
wget -q -O- https://debian.koha-community.org/koha/gpg.asc | apt-key add -
echo 'deb http://deb.debian.org/debian stretch-backports main' > /etc/apt/sources.list.d/stretch-backports.list
echo 'deb http://debian.koha-community.org/koha stable main' > /etc/apt/sources.list.d/koha.list
apt update
apt-install libmodern-perl-perl libcpan-meta-perl libmodule-cpanfile-perl
apt-file update
cd /koha
PERL5LIB=. perl debian/update-control
`

In order to support newer Debian/Ubuntu, the problem is perl-modules-<version> needs to cover their Perl version (5.28 and 5.30 respectively).
Comment 22 Tomás Cohen Arazi 2020-05-26 13:54:46 UTC
This needs to be run in the lowest supported Debian version. So stretch stuffs are to be pulled in this case. Then we patch for higher versions. This is what we are doing in list-deps. And sometimes the resulting control file might be patched for simplicity.
Comment 23 Mason James 2020-05-27 01:08:01 UTC
(In reply to Tomás Cohen Arazi from comment #20)
> 
> so we *should* expect a line containing:
> 
> perl-modules | perl-modules-5.22 | perl-modules-5.24 | perl-modules-5.28 |
> perl-modules-5.30
> 

hi Tomas, 
i did some experimenting and discovered that while explicitly listing all versions of perl-modules-5.XX in the control file, does work
...the control file can be simplified to just listing the 'perl-modules' virtual package


debian will correctly pull the correct perl-modules-5.XX package, when installing perl-modules

i've tested the simplified control file by building a package with it (on deb9), and successfully installing the package on deb9, deb10, deb11 and ubuntu 1804, 1910, 2004, 2010


you can test the package by running...

# docker run  -ti debian:testing bash
echo 'deb [trusted=yes] http://apt.kohaaloha.com/koha-nightly dev main' >  /etc/apt/sources.list.d/koha.list
apt update
apt install koha-common
dpkg -l | grep koha-common
 ii  koha-common  20.05.00.000~git+d9.20200523231519.38b762ea-1 all integrated (physical) library management system
Comment 24 Tomás Cohen Arazi 2020-05-27 03:05:24 UTC
(In reply to Mason James from comment #23)
> (In reply to Tomás Cohen Arazi from comment #20)
> > 
> > so we *should* expect a line containing:
> > 
> > perl-modules | perl-modules-5.22 | perl-modules-5.24 | perl-modules-5.28 |
> > perl-modules-5.30
> > 
> 
> hi Tomas, 
> i did some experimenting and discovered that while explicitly listing all
> versions of perl-modules-5.XX in the control file, does work
> ...the control file can be simplified to just listing the 'perl-modules'
> virtual package
> 
> 
> debian will correctly pull the correct perl-modules-5.XX package, when
> installing perl-modules
> 
> i've tested the simplified control file by building a package with it (on
> deb9), and successfully installing the package on deb9, deb10, deb11 and
> ubuntu 1804, 1910, 2004, 2010
> 
> 
> you can test the package by running...
> 
> # docker run  -ti debian:testing bash
> echo 'deb [trusted=yes] http://apt.kohaaloha.com/koha-nightly dev main' > 
> /etc/apt/sources.list.d/koha.list
> apt update
> apt install koha-common
> dpkg -l | grep koha-common
>  ii  koha-common  20.05.00.000~git+d9.20200523231519.38b762ea-1 all
> integrated (physical) library management system

Great job, Mason.
Comment 25 Tomás Cohen Arazi 2020-05-27 14:25:40 UTC
Created attachment 105397 [details] [review]
Bug 25591: Update list-deps for Debian 10 and Ubuntu 20.04

This patch removes version-specific packages from the debian/control file,
which fixes a bug where koha-common can only be installed on the debian
version that the package was built on

Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Comment 28 Mason James 2020-06-01 03:03:29 UTC
Created attachment 105450 [details] [review]
Bug 25591: Update list-deps for Debian 10 and Ubuntu 20.04

update list-deps for libcpan-meta-perl package
Comment 29 Jonathan Druart 2020-06-02 13:22:30 UTC
Pushed to master for 20.11, thanks to everybody involved!
Comment 30 Aleisha Amohia 2020-06-12 01:18:06 UTC
assuming this is for 20.05 only, not backporting to 19.11.x
Comment 31 Victor Grousset/tuxayo 2020-06-15 16:24:07 UTC
Should it be backported to 20.05? Because it seems to be only in master.
Comment 32 Mason James 2020-06-16 21:33:09 UTC
(In reply to Victor Grousset/tuxayo from comment #31)
> Should it be backported to 20.05? Because it seems to be only in master.

yes, please apply to 20.05.x branch
Comment 33 Lucas Gass 2020-06-17 20:43:29 UTC
backported to 20.05.x for 20.05.01
Comment 34 Aleisha Amohia 2020-06-17 21:50:20 UTC
backported to 19.11.x for 19.11.07, as recommended for Bug 22522
Comment 35 Victor Grousset/tuxayo 2020-06-18 17:13:49 UTC
IIUC Debian 10 support and Ubuntu 20.04 isn't for 19.05.x so not backporting this patch.
Comment 36 Mason James 2020-07-09 15:01:11 UTC
(In reply to Aleisha Amohia from comment #34)
> backported to 19.11.x for 19.11.07, as recommended for Bug 22522

this has been resolved now, closing
Comment 37 Mason James 2020-09-09 10:52:59 UTC
(In reply to Victor Grousset/tuxayo from comment #35)
> IIUC Debian 10 support and Ubuntu 20.04 isn't for 19.05.x so not backporting
> this patch.

hi Victor. can you add this patch to 19.05.x branch

it will fix 19.05 installation errors on...
 deb 8, 10, 11
 ubuntu 16.04, 18.04, 20.04, 20.10

the current 19.05 branch only installs on deb

ps: i have applied this patch to the last 3 19.05.x package releases
Comment 38 Mason James 2020-09-09 10:54:21 UTC
> the current 19.05 branch only installs on deb

oops, typo...
the current 19.05 branch only installs on deb9
Comment 39 Victor Grousset/tuxayo 2020-09-15 02:39:53 UTC
(In reply to Mason James from comment #37)
> it will fix 19.05 installation errors on...
>  deb 8, 10, 11
>  ubuntu 16.04, 18.04, 20.04, 20.10

😱😱😱 okay that's a big deal.

> ps: i have applied this patch to the last 3 19.05.x package releases

Thanks a lot!

Ok backporting.
Rats! Conflict which I can't resolve:

Next is the conflict part in the diff3 conflict style. Here is the structure:
<<<<<<< HEAD
here is the code in 19.05.x before the cherry-pick
||||||| parent of 248978bc62... Bug 25591: Update list-deps for Debian 10 and Ubuntu 20.04
here is the code in 19.11.x before the patch
=======
here is the code in 19.11.x after the patch
>>>>>>> 248978bc62... Bug 25591: Update list-deps for Debian 10 and Ubuntu 20.04


<<<<<<< HEAD
my $modules = C4::Installer::PerlModules->new();
my $prereqs = $modules->prereqs;
foreach my $phase ($prereqs->phases) {
    foreach my $type ($prereqs->types_in($phase)) {
        my $reqs = $prereqs->requirements_for($phase, $type);
||||||| parent of 248978bc62... Bug 25591: Update list-deps for Debian 10 and Ubuntu 20.04
    # Remove packages that are required/essential and always installed on
    # a Debian system. Debian packages should not have unversioned
    # dependencies on such packages.
    foreach my $line (@temp) {
        if ( $line ne "perl-base" ) {
            @lines = ( @lines, $line );
        }
    }
    if ( exists $overrides{$module} ) {
        print "$overrides{$module}\n";
    }
    elsif ( scalar(@lines) == 1 && $lines[0] ne "" ) {
        my $pkg = $lines[0];
        print "$pkg\n";
    }
    elsif ( scalar(@lines) > 1 ) {
        foreach my $pkg (@lines) {
            print " | " if ( $pkg ne $lines[0] );
            print "$pkg";
            print " | $pkg" . "-5.22" if ( $pkg eq "perl-modules" );
            print " | $pkg" . "-5.24" if ( $pkg eq "perl-modules" );
        }
        print "\n";
    }
    elsif ( scalar(@temp) != 0 ) {
=======
    # Remove packages that are required/essential and always installed on
    # a Debian system. Debian packages should not have unversioned
    # dependencies on such packages.

    # skip perl-base and problematic version specific libperl* and
    # perl-module* packages (they get installed as deps. anyway)
    foreach my $line (@temp) {
        if ( $line ne "perl-base" and $line !~ /^libperl5\./ and $line !~ /^perl-modules-5\./ ) {
            @lines = ( @lines, $line );
        }
    }
    if ( exists $overrides{$module} ) {
        print "$overrides{$module}\n";
    }
    elsif ( scalar(@lines) == 1 && $lines[0] ne "" ) {
        my $pkg = $lines[0];
        print "$pkg\n";
    }
    elsif ( scalar(@lines) > 1 ) {
        foreach my $pkg (@lines) {
            print " | " if ( $pkg ne $lines[0] );
            print "$pkg";
        }
        print "\n";
    }
    elsif ( scalar(@temp) != 0 ) {
>>>>>>> 248978bc62... Bug 25591: Update list-deps for Debian 10 and Ubuntu 20.04
Comment 40 Victor Grousset/tuxayo 2020-09-15 02:55:58 UTC
> Rats! Conflict which I can't resolve:

It was a mistake on my side.

Backported to 19.05.x branch for 19.05.15
Comment 41 Mason James 2020-10-06 07:34:16 UTC
*** Bug 26109 has been marked as a duplicate of this bug. ***