Bug 24845 - Web-based installer consistently fails while creating database tables.
Summary: Web-based installer consistently fails while creating database tables.
Status: CLOSED INVALID
Alias: None
Product: Koha
Classification: Unclassified
Component: Installation and upgrade (web-based installer) (show other bugs)
Version: 19.11
Hardware: All All
: P3 major (vote)
Assignee: Bugs List
QA Contact: Testopia
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2020-03-11 06:29 UTC by Dwight Tuinstra
Modified: 2020-11-30 21:45 UTC (History)
3 users (show)

See Also:
Change sponsored?: ---
Patch complexity: ---
Documentation contact:
Documentation submission:
Text to go in the release notes:
Version(s) released in:


Attachments
indexer-output.log, showing table-doesn't exist errors before web installer is run. (62.55 KB, text/x-log)
2020-03-11 06:29 UTC, Dwight Tuinstra
Details
intranet-error.log, showing table-doesn't exist errors generated during web-installer run. (153.89 KB, text/plain)
2020-03-11 06:31 UTC, Dwight Tuinstra
Details
Shell script to automate the manual, pre-web-installer steps for 19.11 on Debian 9 (5.60 KB, application/x-shellscript)
2020-03-11 21:19 UTC, Dwight Tuinstra
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Dwight Tuinstra 2020-03-11 06:29:42 UTC
Created attachment 100504 [details]
indexer-output.log, showing table-doesn't exist errors before web installer is run.

Installation by following the wiki document "Koha on Debian" fails. 

I am able to successfully get through the preliminary steps leading up to the web-based installer, and the first handful of screens in that installer. The problem occurs after clicking on the button to create the database
tables. After several minutes of no activity, the brower reports
"'Gateway Timeout' The gateway did not receive a timely response from the
upstream server or application." At this point, table creation in mysql/mariadb stops, after a varying number of tables (around 160 to 180) have been created.

There are two possible causes that I've been able to see so far:

POSSIBILITY 1:
There is an indexer problem that occurs even before the point where the web
installer is run. The file /var/log/koha/biblioteca/indexer-output.log begins to fill up with error messages -- see attached files. After this, the log continues to grow with the last two lines constantly added over and over. Curiously, though, this stops about the time that the web installer is run.

POSSIBILITY 2:
The creation of database tables takes so long that the browser or installer
times out, or race conditions arise. I watched table creation in another
window as the web installer was running. Table creation seemed to go slow
(roughly a table every second) even though there was plenty of free memory,
free disk space (with low disk activity), and the CPUs were over 95% idle.
Table creation stopped at 162 tables, at almost the exact same moment that
the browser gave the timeout message, so I'm not sure if that's all the
tables or if there are supposed to be more.

What's odd here is that the web installer writes a good number of messages
to /var/log/koha/biblioteca/intranet-error.log, each saying that one of these
database tables do not exist: language_descriptions,
language_subtag_registry, or systempreferences. However, all of these
tables *do* exist in the database when one examines it later, which
suggests a race condition.

This problem is perfectly repeatable -- I've gone so far as to script the manual steps from the wiki documentation to ensure all steps are done, and done as specified. 

This problem is preventing Koha from being installed and used.

OTHER INFO:
- Virtualization: VirtualBox 5.2.34_Ubuntu r133883, 2 i7 CPUs,
pre-allocated fixed disk (32Gb), 2GB RAM
- Debian version: 4.9.0-12-686-pae #1 SMP Debian 4.9.210-1 (2020-01-20) i686
- Koha version: 19.11
- Following: stable
- Database: mysql  Ver 15.1 Distrib 10.1.44-MariaDB, for debian-linux-gnu (i686) using readline 5.2
- During WebUI installer:
    - All Perl dependencies installed and OK
    - All DB settings look reasonable, connection established, and database exists and can be accessed
Comment 1 Dwight Tuinstra 2020-03-11 06:31:19 UTC
Created attachment 100505 [details]
intranet-error.log, showing table-doesn't exist errors generated during web-installer run.
Comment 2 Jonathan Druart 2020-03-11 09:25:34 UTC
You can ignore the indexed output log, it is run every minute even if the DB structure is not created yet.

Is MySQL on a remote server?

Did you try to increase Apache timeout?
Comment 3 Dwight Tuinstra 2020-03-11 15:19:14 UTC
MySQL (Mariadb, actually) is on the same server. The initial pages of the web installer say that the database connection and permissions are correct.

I have not tried adjusting the Apache timeout. How do I do that?
Comment 4 Jonathan Druart 2020-03-11 15:30:40 UTC
In the apache config file (/etc/apache2/apache2.conf), there is a "Timeout" variable.
Comment 5 Dwight Tuinstra 2020-03-11 20:58:21 UTC
I was finally able to get the table-creation part of the install to work by setting the apache2 Timeout value to 30 minutes. To try to speed up mariadb, I also set innodb_buffer_pool_size to 40M (it was not set initially). I don't know if that made any difference, I haven't had time to run a comparison. I also bumped up the virtual machine's memory to 4GB and dedicated 4 CPUs to it.

Actual creation of all 209 tables took over *20 minutes*, or roughly 6 seconds (an eternity in CPU cyles and disk seek time) per table. Once this was done, the "Selecting default settings" page took 20 to 30 minutes to finish.

Creating tables and filling in values should not take this long. I'm guessing there is a problem with the mariadb configuration itself, or some kind of interaction with the OS (or virtual machine manager) that is slowing down mariadb. Either way, the documentation or the installer should make the necessary adjustments or advise the user to do so. A person who runs into these problems is likely to just abandon their project and spend time looking for alternatives.

Database performance issues during install also suggest there will be similar problems during operation.

If the performance issues during install are inescapable, then the installer should at least give "percent-complete" progress feedback to the user. This would reassure them and also keep the web session alive.

I am attaching a script for automating the manual, pre-web-installer steps, in hopes this will make it easier for someone else to duplicate the problem.
Comment 6 Dwight Tuinstra 2020-03-11 21:19:23 UTC
Created attachment 100607 [details]
Shell script to automate the manual, pre-web-installer steps for 19.11 on Debian 9

This script automates most of the manual, pre-web-installer steps given in the "Koha on Debian" wiki documentation. See the comments near the top for the initial steps that still need to be done by hand. There are also some variables there that you may wish to change.

The script takes a fresh Debian 9 install and runs up through starting the web-based installer. It's meant as an assist for looking into problems, not as a script for general use by people installing Koha (it lacks error checking and other user-friendliness for such use).

The parts that ask the user to take a snapshot are for anyone running it in VM and who wish to take snapshots. If you're not running in a VM or don't care to take a snapshot, just hit "Enter" and the script will continue.

Step 02 installs VirtualBox Guest Additions. If you're not running Vbox, you might want to do the Step 01 commands by hand, then start the script at Step 03.
Comment 7 Jonathan Druart 2020-03-12 09:19:40 UTC
Dwight,

I confirm that the installation process takes less than few minutes, and the insert of the DB structure less than a minute.

Did you try to insert kohastructure.sql into a fresh DB manually?

Why are not you using a 64bit linux kernel? I guess your host has a 64bit architecture, so why not the VM?
Comment 8 Dwight Tuinstra 2020-03-13 03:36:20 UTC
Ooops!! My bad. Didn't notice the Debian ISO was 32-bit, and I've been away from Debian too long and didn't realize that i686 wasn't the 64-bit version.

So ... I redid everything with a 64-bit Debian guest VM. Ran into the same problems when the VM only had 1 CPU and 1GB of memory. Bumping that up to 4 CPUs,  2GB of memory, and enabling the guest SATA controller to "Use host I/O cache" did the trick -- all steps finished quickly. Not sure which change made the difference. Might be good to have a System Requirements section in the install docs.

Marking this as RESOLVED - INVALID.

Thank you for your help!