Bug 6434 - Many changes for Ukrainian and Russian sql-tables
Summary: Many changes for Ukrainian and Russian sql-tables
Status: CLOSED FIXED
Alias: None
Product: Koha
Classification: Unclassified
Component: I18N/L10N (show other bugs)
Version: 3.6
Hardware: All All
: P5 - low normal (vote)
Assignee: Frédéric Demians
QA Contact: Bugs List
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2011-05-31 10:44 UTC by Serhij Dubyk
Modified: 2013-12-05 19:59 UTC (History)
4 users (show)

See Also:
Change sponsored?: ---
Patch complexity: ---
Documentation contact:
Documentation submission:
Text to go in the release notes:
Version(s) released in:


Attachments
archived patch for upgrade Ukrainian and Russian sql (184.26 KB, patch)
2011-05-31 10:50 UTC, Serhij Dubyk
Details | Diff | Splinter Review
Bug 6434 - Ukrainian and Russian MARC sql-tables (75.37 KB, patch)
2012-03-30 15:33 UTC, Serhij Dubyk
Details | Diff | Splinter Review
Bug 6434 - Ukrainian and Russian MARC sql-tables (75.37 KB, application/x-bzip)
2012-03-30 15:34 UTC, Serhij Dubyk
Details
Bug 6434 - Ukrainian and Russian MARC sql-tables (760.59 KB, application/x-bzip)
2012-03-30 22:35 UTC, Serhij Dubyk
Details
Signed-off patch (761.51 KB, application/x-bzip)
2012-03-31 07:56 UTC, Frédéric Demians
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Serhij Dubyk 2011-05-31 10:44:42 UTC
Many changes for Ukrainian and Russian sql:
 * update (and translate) many sql-tables
 * reformatting unimarc-tables, split to local and general fields/subfields
 * global chande to use 995 subfields for items (952 now in obsolete)
 * initial support for marc21 (Russian and Ukrainian translation)
 * other updates, needed for 3.0.x > 3.2.x > 3.4.x
Comment 1 Serhij Dubyk 2011-05-31 10:50:18 UTC Comment hidden (obsolete)
Comment 2 Serhij Dubyk 2011-05-31 11:01:47 UTC
I want to upgrade most sql-files for Ukrainian and Russian localizations.
Process of submitting the patch is stopped at the next step:

git send-email -to koha-patches@lists.koha-community.org kohaclone/0001-Many-changes-for-Ukrainian-and-Russian-sql.patch
fatal: kohaclone/0001-Many-changes-for-Ukrainian-and-Russian-sql.patch: 1730: patch contains a line longer than 998 characters
warning: no patches were sent

I shortened the length of lines in some files but it still does not help.

This sql-tables was adapted for Koha 3.2.x and now for Koha 3.4.1.
System prefs synchonized with [koha.git]/installer/data/mysql/en/mandatory/sysprefs.sql few days ago.

Please commit this patch.
Thanks!
Comment 3 Frédéric Demians 2011-05-31 16:17:54 UTC
Your patch doesn't apply on HEAD. 

INSERT INTO authorised_values (category, authorised_value, lib) VALUES 
error: patch failed: installer/data/mysql/ru-RU/optional/patron_atributes.sql:1
error: installer/data/mysql/ru-RU/optional/patron_atributes.sql: patch does not apply
error: patch failed: installer/data/mysql/uk-UA/optional/patron_atributes.sql:1
error: installer/data/mysql/uk-UA/optional/patron_atributes.sql: patch does not apply
Did you hand edit your patch?
It does not apply to blobs recorded in its index.
Cannot fall back to three-way merge.
Comment 4 Serhij Dubyk 2011-06-01 07:38:35 UTC
(In reply to comment #3)
> Your patch doesn't apply on HEAD. 
> 
> INSERT INTO authorised_values (category, authorised_value, lib) VALUES 
> error: patch failed: installer/data/mysql/ru-RU/optional/patron_atributes.sql:1
> error: installer/data/mysql/ru-RU/optional/patron_atributes.sql: patch does not
> apply
> error: patch failed: installer/data/mysql/uk-UA/optional/patron_atributes.sql:1
> error: installer/data/mysql/uk-UA/optional/patron_atributes.sql: patch does not
> apply
> Did you hand edit your patch?
> It does not apply to blobs recorded in its index.
> Cannot fall back to three-way merge.

I not manually edited the patch. 
I do not understand why there is an error with these files - they are quite simple (look https://www.dropbox.com/s/f07d6nidu39mvdj/uk-UA/optional/patron_atributes.sql and https://www.dropbox.com/s/f07d6nidu39mvdj/ru-RU/optional/patron_atributes.sql).

Could something be wrong when I work with GIT..?
My steps:
1) git clone git://git.koha-community.org/koha.git kohaclone
2) cd kohaclone
git checkout -b dubyk_master origin/master
3) delete kohaclone/installer/data/mysql/uk-UA/ and kohaclone/installer/data/mysql/ru-RU/
4) Copy my above updated folders uk-UA and ru-RU to kohaclone/installer/data/mysql/
5) git commit -a
6) git format-patch -s --subject-prefix='PATCH] [SIGNED-OFF' origin/master
7) In result I have file:
kohaclone/0001-Many-changes-for-Ukrainian-and-Russian-sql.patch
which I tried to send 
(git send-email -to koha-patches@lists.koha-community.org ...)
and attached here.

Folders uk-UA/, ru-RU/ and related archives I posted also in Dropbox -
https://www.dropbox.com/s/f07d6nidu39mvdj#view:list
Comment 5 Serhij Dubyk 2011-06-21 07:07:40 UTC
Maybe this patch should be divided into many small?
With the help "git send-email" I can't do next changes in sql-tables:
1) Delete old tables
uk-UA / mandatory / system_preferences_full_optimal_for_install_only.sql
uk-UA / mandatory / system_preferences_only_translation_updating.sql
uk-UA / mandatory / system_preferences_optimal_values_insert_only.sql
ru-RU / mandatory / system_preferences_full_optimal_for_install_only.sql
ru-RU / mandatory / system_preferences_only_translation_updating.sql
ru-RU / mandatory / system_preferences_optimal_values_insert_only.sql 
and replace them by new ones
https://www.dropbox.com/s/f07d6nidu39mvdj/uk-UA/mandatory/system_preferences.sql
https://www.dropbox.com/s/f07d6nidu39mvdj/ru-RU/mandatory/system_preferences.sql
2) Replace old
uk-UA / mandatory / sample_notices.sql
ru-RU / mandatory / sample_notices.sql
with new
https://www.dropbox.com/s/f07d6nidu39mvdj/ru-RU/mandatory/sample_notices.sql
https://www.dropbox.com/s/f07d6nidu39mvdj/uk-UA/mandatory/sample_notices.sql
3) maybe some other sql-tables - I do not know how to see all files that are not suitable for "git send-email"

Other sql-tables I can send with many small patches - if this makes sense?
Comment 6 Serhij Dubyk 2012-03-30 09:03:54 UTC
I try to split this patch into smaller pieces and include them here ...
Comment 7 Frédéric Demians 2012-03-30 10:31:52 UTC
> I try to split this patch into smaller pieces and include them here ...

If you can't, just sent me the file by email. I will push it on a git repository as a branch.
Comment 8 Serhij Dubyk 2012-03-30 15:33:12 UTC Comment hidden (obsolete)
Comment 9 Serhij Dubyk 2012-03-30 15:34:35 UTC Comment hidden (obsolete)
Comment 10 Frédéric Demians 2012-03-30 17:57:50 UTC
Serhij, could say more about what your patch is supposed to do? Why is there two patches attached and not one? (it seems to be the same). Mark as obsolete one of them if necessary. The first patch is non manageable. If it is a bzip2 compressed patch, attach it as you have done for the second patch. I've been able bunzip2 the second patch which delete deprecated Ukrainian and Russian SQL tables but I can't see any new tables.
Comment 11 Serhij Dubyk 2012-03-30 22:30:47 UTC
Oh, I forgot run "git add -A".
Comment 12 Serhij Dubyk 2012-03-30 22:35:09 UTC Comment hidden (obsolete)
Comment 13 Frédéric Demians 2012-03-31 07:56:02 UTC
Created attachment 8734 [details]
Signed-off patch

It works, doesn't break anything, seems to be in Ukrainian/Russian (I can't
judge).

It's a huge patch (21M). That's why it's attached as a bz2 archive: so
the archive must be downloaded, decompressed, and then applied.

General remark: Adding this patch to Koha will increase substantially Koha git 
repository size. It shows that non-code files (.po files, SQL files)
could/should be managed outside Koha core repository. See this discussion:

http://wiki.koha-community.org/wiki/Git_Splitting_and_Shrinking

Here is my test plan:

[1]  Apply the patch on master
[2]  Install ru-RU and uk-UA languages:
     cd misc/translator
     ./translate ru-RU uk-UA
[3]  re-create Koha DB: 
     DROP DATABASE koha;
     CREATE DATABASE koha;
[4]  Load Koha pro interface: get web installer
[5]  Choose ru-RU language
[6]  Everything is in Russian. Click blindly on button until reaching the 
     page where to choose Unimarc/Marc21.
[7]  Choose Unimarc.
[8]  Tick on everything to load all UNIMARC SQL files.
[9]  All SQL are loaded without any error.
[10] Check Koha cataloguing.
[11] Go to [3] and choose Marc21 at step [7].
[12] Go to [3] and choose uk-UA at step [5].
[13] Go to [3] and choose uk-UA at step [5] and Marc21 at step [7].
Comment 14 Ian Walls 2012-03-31 20:21:52 UTC
Comment on attachment 8734 [details]
Signed-off patch

Removing patch marker, since this file is actually a .bz2, and cannot be applied like a patch without decompression.
Comment 15 Ian Walls 2012-03-31 20:49:47 UTC
The compressed, signed-off patch only affects SQL files (and their related txt descriptions) in ru-RU and uk-UA.  Given the size, and that I'm unfamiliar with Cyrillic, I haven't read the complete patch, but I did a scan for unintended side effects to the database (a limited one), and didn't find any adverse issues.

This is a huge patch, decompressed, and would greatly increase Koha's overall size.  Our choice, then, is whether to accept this size increase to the repo, forever, or to implement a splitting/shrinking plan and accept the consequences thereof.

I would recommend input from as many community members as possible here, both developers and those using non-English translations of Koha.  I'm loathe to lose any history from the repo, as it makes my job as a developer harder, but increasing the overall disk space requirement for Koha doesn't do users any favours.  I'm not sure I have good advice on how to proceed.

From a purely technical standpoint, this file looks safe, and can Pass QA.  As to exactly how it's contents are added to Koha, however, may require some additional discussion.

Marking as Passed QA, but with the recommendation that we discuss how to apply this before it's pushed.
Comment 16 Serhij Dubyk 2012-04-02 06:51:02 UTC
Work on these tables in partly sponsored by the libraries: LTS Library(http://koha.lts.lviv.ua:8008/,Unimarc), TNTU Scintific Technical Library(http://koha.tntu.edu.ua/,Unimarc), DCU Library (http://109.254.48.77:8008/,Unimarc), Dmitry Pozharsky University Library (http://lib.s-and-e.ru/,Marc21) and ServanTek Company (http://servantek.org.ua/).
I not so frequently updating these MARC-sql-table, so there are so many changes.
Old tables deleted in this patch (add ~ 2 Mb to patch). To facilitate further support all tables converted as follows: 1) separated standard MARC fields/subfields and specific for Koha 2) reformated all sql as follow (collected the appropriate fields/subfields and aligned):

INSERT INTO marc_tag_structure  (frameworkcode, tagfield, mandatory, repeatable, liblibrarian, libopac, authorised_value) VALUE
 ('', '011', '', 1, 'Міжнародний стандартний номер серіального видання (ISSN)', 'ISSN', '');
INSERT INTO  marc_subfield_structure (frameworkcode, authtypecode, tagfield, tagsubfield, mandatory, repeatable, liblibrarian,
 ('', '', '011', 'a', 0, 0, 'Номер (ISSN)', 'ISSN',                     0, NULL, 'biblioitems.issn', '', '', NULL, NULL, NULL,
 ('', '', '011', 'b', 0, 0, 'Уточнення', '',                            -1, NULL, '', '', '', NULL, NULL, NULL, NULL),
 ('', '', '011', 'd', 0, 0, 'Умови придбання і/або ціна', '',           -1, NULL, '', '', '', NULL, NULL, NULL, NULL),
 ('', '', '011', 'y', 0, 0, 'Анульований ISSN', '',                     -1, NULL, '', '', '', NULL, NULL, NULL, NULL),
 ('', '', '011', 'z', 0, 0, 'Помилковий ISSN', '',                      -1, NULL, '', '', '', NULL, NULL, NULL, NULL);
- for easier reading/editing by human.

In result, the size of the tables corresponding to the size of other languages from /data/mysql/ ​​(including the fact that that the Cyrillic characters are 2 bytes).

P.S. Also, I have scripts that perform such reformatting as above and the initial script, which compares the fields/subfields for all languages ​​from Koha-sql.  If someone is interested, I unload them somewhere publicly.
Comment 17 Paul Poulain 2012-04-06 14:27:48 UTC
this patch has been pushed to master on march 31th

For an unknown reason, it's not on a new/bug_6434 branch
(I suspect it's because it's a zipped patch so I could not use my automated tools with that use git bz and I made a mistake)
Comment 18 Jared Camins-Esakov 2012-05-09 16:58:52 UTC
This bug will be included in the Koha 3.6.5 release.