Description
Magnus Enger
2012-04-10 12:23:58 UTC
Sounds good - having large bulky .po-files makes it hard to prioritze during translation. Created attachment 26530 [details] [review] Bug 7939 - Separate po files for different MARC dialects [PRELIMINAR - not in final form but can be tested] This patch implements separate PO files for different MARC dialects. To do so it depends on correct filenames, i.e. it will build PO files using files with/without "unimarc|normarc|marc21" on their names. Prominent changes: A) tmpl_process3.pl three new options: exclude or include certain filenames and a new source dir that can have multiple values B) LangInstaller.pm added definitions to create or update nn-NO-{MARCFLAVOR}.po and a little modification on install procedure C) Standarization of filenames OPAC prog po file is now xx-YY-opac-prog.po STAFF po file is now xx-YY-staff-prog.po MARC dialects po files are xx-YY-marc-{MARCFLAVOUR}.po To test: 1) Update po files for your preferred language, ej. nn-NO cd misc/translator perl translate update nn-NO (do it twice if there are warnings) 2) Do some copying/renaming cp po/nn-NO-i-staff-t-prog-v-3006000.po po/nn-NO-marc-UNIMARC.po cp po/nn-NO-i-staff-t-prog-v-3006000.po po/nn-NO-marc-NORMARC.po cp po/nn-NO-i-staff-t-prog-v-3006000.po po/nn-NO-marc-MARC21.po mv po/nn-NO-i-staff-t-prog-v-3006000.po po/nn-NO-staff-prog.po mv po/nn-NO-i-opac-t-prog-v-3006000.po po/nn-NO-opac-prog.po (most MARC dialect strings are on staff) 3) Apply the patch 4) Update again to fix translation files, verbose perl translate update nn-NO -v 5) Install language, verbose, verify translations perl translate install nn-NO -v More tests to come. To be continued... (In reply to Magnus Enger from comment #0) > I think it would be a good idea to have separate po files for MARC21, > UNIMARC and NORMARC (value builders and XSLT). > > See also Bug 7934. Hi Magnus, do you want to try? Created attachment 26537 [details] [review] Bug 7939 - Separate po files for different MARC dialects [PRELIMINAR 2 - not in final form but can be tested] This patch implements separate PO files for different MARC dialects. It depends on correct filenames, i.e. it will build PO files using files with/without "unimarc|normarc|marc21" on their names. Prominent changes: A) tmpl_process3.pl three new options: exclude or include certain filenames and a new source dir that can have multiple values (I opted to add this last feature and not change current source dir in order not to brake current usage) Minor change on use of msgmerge, now there are no backup files (*~) and obsoleted translations are removed (to produce smaller files) [this can be on another bug, is not strictly related with the pourpose of this bug] B) LangInstaller.pm added definitions to create or update nn-NO-{MARCFLAVOR}.po and modification on install procedure to handle multiple target dirs C) Standarization of filenames OPAC prog po file is now xx-YY-opac-prog.po STAFF po file is now xx-YY-staff-prog.po MARC dialects po files are xx-YY-marc-{MARCFLAVOUR}.po [again, renaming not related, can be in it's own bug] To test: 1) Update po files for your preferred language, ej. nn-NO cd misc/translator perl translate update nn-NO (do it twice if there are warnings) 2) Do some copying/renaming cp po/nn-NO-i-staff-t-prog-v-3006000.po po/nn-NO-marc-UNIMARC.po cp po/nn-NO-i-staff-t-prog-v-3006000.po po/nn-NO-marc-NORMARC.po cp po/nn-NO-i-staff-t-prog-v-3006000.po po/nn-NO-marc-MARC21.po mv po/nn-NO-i-staff-t-prog-v-3006000.po po/nn-NO-staff-prog.po mv po/nn-NO-i-opac-t-prog-v-3006000.po po/nn-NO-opac-prog.po (most MARC dialect strings are on staff, so we use that as basis) 3) Apply the patch 4) Update again to fix translation files, verbose perl translate update nn-NO -v 5) Install language, verbose, verify translations perl translate install nn-NO -v 6) Create translation files rm po/nn-NO* perl translate create nn-NO we must have this list: po/nn-NO-marc-MARC21.po po/nn-NO-marc-NORMARC.po po/nn-NO-marc-UNIMARC.po po/nn-NO-opac-bootstrap.po po/nn-NO-opac-ccsr.po po/nn-NO-opac-prog.po po/nn-NO-pref.po po/nn-NO-staff-help.po po/nn-NO-staff-prog.po [staff-help could be more properly staff-prog-help, another rename] Additional tests: 7) Number of msgids 7.a) Before patch and after upgrade, extract and count msgids for i in $(ls po/nn-NO-*po); \ do msginit -i $i -o nn-old.po --no-translator --no-wrap --locale=nn_NO; \ egrep ^msgid nn-old.po >> old; \ done sort new | uniq | tee s-old | wc -l > n-old s-old: have all msgids n-old: number of msgids 7.b) After patch and after creation of new files Repeat procedure, diferent files (s-new, n-new) 7.c) Compare (diff s-old snew), they are the same (save for a strange UNIMARC char in my case) 8) Installed dirs/files 8.a) List of EN dirs/files cd koha-tmpl find | egrep "/en/" > en 8.b) List of nn-NO dirs/files. After patch and language install cd koha-tmpl find | egrep "/nn-NO/" | sed 's|/nn-NO/|/en/|' > nn 8.c) Compare (diff en nn), they are the same (save for a bogus help/nn-NO dir, without files. Working for a solution) On point 7.a), an error, must be sort old | uniq | tee s-old | wc -l > n-old Created attachment 26576 [details] [review] Bug 7939 - Separate po files for different MARC dialects This patch implements separate PO files for different MARC dialects. It depends on correct filenames, i.e. it will build PO files using files with/without "unimarc|normarc|marc21" on their names. Prominent changes: A) tmpl_process3.pl three new options: exclude or include certain filenames and a new source dir that can have multiple values (I opted to add this last feature and not change current source dir in order not to brake current usage) Minor change on use of msgmerge, added use of msgattrib. Now there are no backup files (*~) and obsoleted translations are removed (to produce smaller files) [this can be on another bug, is not strictly related with the pourpose of this bug] B) LangInstaller.pm Added definitions to create or update xx-YY-{MARCFLAVOR}.po, minor change to create and install procedure, and modification of install procedure to handle multiple target dirs. Updated documentation. C) Standarization of filenames OPAC prog po file is now xx-YY-opac-prog.po STAFF po file is now xx-YY-staff-prog.po MARC dialects po files are xx-YY-marc-{MARCFLAVOUR}.po [again, renaming not related, can be in it's own bug] To test: 1) Update po files for your preferred language, ej. nn-NO cd misc/translator perl translate update nn-NO (do it twice if there are warnings) 2) Do some copying/renaming cp po/nn-NO-i-staff-t-prog-v-3006000.po po/nn-NO-marc-UNIMARC.po cp po/nn-NO-i-staff-t-prog-v-3006000.po po/nn-NO-marc-NORMARC.po cp po/nn-NO-i-staff-t-prog-v-3006000.po po/nn-NO-marc-MARC21.po mv po/nn-NO-i-staff-t-prog-v-3006000.po po/nn-NO-staff-prog.po mv po/nn-NO-i-opac-t-prog-v-3006000.po po/nn-NO-opac-prog.po (most MARC dialect strings are on staff, so we use that as basis) 3) Apply the patch 4) Update again to fix translation files, verbose perl translate update nn-NO -v 5) Install language, verbose, verify translations perl translate install nn-NO -v 6) Create translation files rm po/nn-NO* perl translate create nn-NO we must have this list: po/nn-NO-marc-MARC21.po po/nn-NO-marc-NORMARC.po po/nn-NO-marc-UNIMARC.po po/nn-NO-opac-bootstrap.po po/nn-NO-opac-ccsr.po po/nn-NO-opac-prog.po po/nn-NO-pref.po po/nn-NO-staff-help.po po/nn-NO-staff-prog.po [staff-help could be more properly staff-prog-help, another rename] Additional tests: 7) Number of msgids 7.a) Before patch and after upgrade, extract and count msgids for i in $(ls po/nn-NO-*po); \ do msginit -i $i -o nn-old.po --no-translator --no-wrap --locale=nn_NO; \ egrep ^msgid nn-old.po >> old; \ done sort old | uniq | tee s-old | wc -l > n-old s-old: have all msgids n-old: number of msgids 7.b) After patch and after creation of new files Repeat procedure, diferent files (s-new, n-new) 7.c) Compare (diff s-old snew), they are the same (save for a strange UNIMARC char in my case, but it's present on corresponding PO file) 8) Installed dirs/files 8.a) List of EN dirs/files cd koha-tmpl find | egrep "/en/" > en 8.b) List of nn-NO dirs/files. After patch and language install cd koha-tmpl find | egrep "/nn-NO/" | sed 's|/nn-NO/|/en/|' > nn 8.c) Compare (diff en nn), they are the same Created attachment 26974 [details] [review] Bug 7939 - Separate po files for different MARC dialects This patch implements separate PO files for different MARC dialects. It depends on correct filenames, i.e. it will build PO files using files with/without "unimarc|normarc|marc21" on their names. Also depends on Bug 12061 Changes: A) LangInstaller.pm Added definitions to create or update xx-YY-{MARCFLAVOR}.po, minor change to create and install procedure, and modification of install procedure to handle multiple target dirs. Updated documentation. B) Standarization of filenames OPAC prog po file is now xx-YY-opac-prog.po STAFF po file is now xx-YY-staff-prog.po MARC dialects po files are xx-YY-marc-{MARCFLAVOUR}.po [renaming not related, can be in it's own bug] To test: 1) Update po files for your preferred language, ej. nn-NO cd misc/translator perl translate update nn-NO (do it twice if there are warnings) 2) Do some copying/renaming cp po/nn-NO-i-staff-t-prog-v-3006000.po po/nn-NO-marc-UNIMARC.po cp po/nn-NO-i-staff-t-prog-v-3006000.po po/nn-NO-marc-NORMARC.po cp po/nn-NO-i-staff-t-prog-v-3006000.po po/nn-NO-marc-MARC21.po mv po/nn-NO-i-staff-t-prog-v-3006000.po po/nn-NO-staff-prog.po mv po/nn-NO-i-opac-t-prog-v-3006000.po po/nn-NO-opac-prog.po (most MARC dialect strings are on staff, so we use that as basis) 3) Apply the patch 4) Update again to fix translation files, verbose perl translate update nn-NO -v 5) Install language, verbose, verify translations perl translate install nn-NO -v 6) Create translation files rm po/nn-NO* perl translate create nn-NO we must have this list: po/nn-NO-marc-MARC21.po po/nn-NO-marc-NORMARC.po po/nn-NO-marc-UNIMARC.po po/nn-NO-opac-bootstrap.po po/nn-NO-opac-ccsr.po po/nn-NO-opac-prog.po po/nn-NO-pref.po po/nn-NO-staff-help.po po/nn-NO-staff-prog.po [staff-help could be more properly staff-prog-help, another rename] Additional tests: 7) Number of msgids 7.a) Before patch and after upgrade, extract and count msgids for i in $(ls po/nn-NO-*po); \ do msginit -i $i -o nn-old.po --no-translator --no-wrap --locale=nn_NO; \ egrep ^msgid nn-old.po >> old; \ done sort old | uniq | tee s-old | wc -l > n-old s-old: have all msgids n-old: number of msgids 7.b) After patch and after creation of new files Repeat procedure, diferent files (s-new, n-new) 7.c) Compare (diff s-old snew), they are the same (save for a strange UNIMARC char in my case, but it's present on corresponding PO file) 8) Installed dirs/files 8.a) List of EN dirs/files cd koha-tmpl find | egrep "/en/" > en 8.b) List of nn-NO dirs/files. After patch and language install cd koha-tmpl find | egrep "/nn-NO/" | sed 's|/nn-NO/|/en/|' > nn 8.c) Compare (diff en nn), they are the same Created attachment 28472 [details] [review] Bug 7939 - Separate po files for different MARC dialects This patch implements separate PO files for different MARC dialects. It depends on correct filenames, i.e. it will build PO files using files with/without "unimarc|normarc|marc21" on their names. Also depends on Bug 12061 Changes: A) LangInstaller.pm Added definitions to create or update xx-YY-{MARCFLAVOR}.po, minor change to create and install procedure, and modification of install procedure to handle multiple target dirs. Updated documentation. B) Standarization of filenames OPAC prog po file is now xx-YY-opac-prog.po STAFF po file is now xx-YY-staff-prog.po MARC dialects po files are xx-YY-marc-{MARCFLAVOUR}.po [renaming not related, can be in it's own bug] To test: 1) Update po files for your preferred language, ej. nn-NO cd misc/translator perl translate update nn-NO (do it twice if there are warnings) 2) Do some copying/renaming cp po/nn-NO-i-staff-t-prog-v-3006000.po po/nn-NO-marc-UNIMARC.po cp po/nn-NO-i-staff-t-prog-v-3006000.po po/nn-NO-marc-NORMARC.po cp po/nn-NO-i-staff-t-prog-v-3006000.po po/nn-NO-marc-MARC21.po mv po/nn-NO-i-staff-t-prog-v-3006000.po po/nn-NO-staff-prog.po mv po/nn-NO-i-opac-t-prog-v-3006000.po po/nn-NO-opac-prog.po (most MARC dialect strings are on staff, so we use that as basis) 3) Apply the patch 4) Update again to fix translation files, verbose perl translate update nn-NO -v 5) Install language, verbose, verify translations perl translate install nn-NO -v 6) Create translation files rm po/nn-NO* perl translate create nn-NO we must have this list: po/nn-NO-marc-MARC21.po po/nn-NO-marc-NORMARC.po po/nn-NO-marc-UNIMARC.po po/nn-NO-opac-bootstrap.po po/nn-NO-opac-ccsr.po po/nn-NO-opac-prog.po po/nn-NO-pref.po po/nn-NO-staff-help.po po/nn-NO-staff-prog.po [staff-help could be more properly staff-prog-help, another rename] Additional tests: 7) Number of msgids 7.a) Before patch and after upgrade, extract and count msgids for i in $(ls po/nn-NO-*po); \ do msginit -i $i -o nn-old.po --no-translator --no-wrap --locale=nn_NO; \ egrep ^msgid nn-old.po >> old; \ done sort old | uniq | tee s-old | wc -l > n-old s-old: have all msgids n-old: number of msgids 7.b) After patch and after creation of new files Repeat procedure, diferent files (s-new, n-new) 7.c) Compare (diff s-old snew), they are the same (save for a strange UNIMARC char in my case, but it's present on corresponding PO file) 8) Installed dirs/files 8.a) List of EN dirs/files cd koha-tmpl find | egrep "/en/" > en 8.b) List of nn-NO dirs/files. After patch and language install cd koha-tmpl find | egrep "/nn-NO/" | sed 's|/nn-NO/|/en/|' > nn 8.c) Compare (diff en nn), they are the same Hi Bernardo, this needs a rework as prog has been removed from the OPAC now Created attachment 36241 [details] [review] Bug 7939 - Separate po files for different MARC dialects This patch implements separate PO files for different MARC dialects. It depends on correct filenames, i.e. it will build PO files using files with/without "unimarc|normarc|marc21" on their names. Changes: A) LangInstaller.pm Added definitions to create or update xx-YY-{MARCFLAVOR}.po, minor change to create and install procedure, and modification of install procedure to handle multiple target dirs. Updated documentation. B) Standarization of filenames STAFF po file is now xx-YY-staff-prog.po MARC dialects po files are xx-YY-marc-{MARCFLAVOUR}.po To test: 1) Update po files for your preferred language, ej. nn-NO cd misc/translator perl translate update nn-NO 2) Do some copying/renaming cp po/nn-NO-i-staff-t-prog-v-3006000.po po/nn-NO-marc-UNIMARC.po cp po/nn-NO-i-staff-t-prog-v-3006000.po po/nn-NO-marc-NORMARC.po cp po/nn-NO-i-staff-t-prog-v-3006000.po po/nn-NO-marc-MARC21.po mv po/nn-NO-i-staff-t-prog-v-3006000.po po/nn-NO-staff-prog.po (most MARC dialect strings are on staff, so we use that as basis) 3) Apply the patch 4) Update again to fix translation files, verbose perl translate update nn-NO -v 5) Install language, verbose, verify translations perl translate install nn-NO -v 6) Create translation files rm po/nn-NO* perl translate create nn-NO we must have this list: po/nn-NO-marc-MARC21.po po/nn-NO-marc-NORMARC.po po/nn-NO-marc-UNIMARC.po po/nn-NO-opac-bootstrap.po po/nn-NO-pref.po po/nn-NO-staff-help.po po/nn-NO-staff-prog.po Additional tests: 7) Number of msgids 7.a) Before patch and after upgrade, extract and count msgids for i in $(ls po/nn-NO-*po); \ do msginit -i $i -o nn-old.po --no-translator --no-wrap --locale=nn_NO; \ egrep ^msgid nn-old.po >> old; \ done sort old | uniq | tee s-old | wc -l > n-old s-old: have all msgids n-old: number of msgids 7.b) After patch and after creation of new files Repeat procedure, diferent files (s-new, n-new) 7.c) Compare (diff s-old snew), they are the same (save for a strange UNIMARC char in my case, but it's present on corresponding PO file) 8) Installed dirs/files 8.a) List of EN dirs/files cd koha-tmpl find | egrep "/en/" > en 8.b) List of nn-NO dirs/files. After patch and language install cd koha-tmpl find | egrep "/nn-NO/" | sed 's|/nn-NO/|/en/|' > nn 8.c) Compare (diff en nn), they are the same Created attachment 36378 [details] [review] Bug 7939 - Separate po files for different MARC dialects This patch implements separate PO files for different MARC dialects. It depends on correct filenames, i.e. it will build PO files using files with/without "unimarc|normarc|marc21" on their names. Changes: A) LangInstaller.pm Added definitions to create or update xx-YY-{MARCFLAVOR}.po, minor change to create and install procedure, and modification of install procedure to handle multiple target dirs. Updated documentation. B) Standarization of filenames STAFF po file is now xx-YY-staff-prog.po MARC dialects po files are xx-YY-marc-{MARCFLAVOUR}.po To test: 1) Update po files for your preferred language, ej. nn-NO cd misc/translator perl translate update nn-NO 2) Do some copying/renaming cp po/nn-NO-i-staff-t-prog-v-3006000.po po/nn-NO-marc-UNIMARC.po cp po/nn-NO-i-staff-t-prog-v-3006000.po po/nn-NO-marc-NORMARC.po cp po/nn-NO-i-staff-t-prog-v-3006000.po po/nn-NO-marc-MARC21.po mv po/nn-NO-i-staff-t-prog-v-3006000.po po/nn-NO-staff-prog.po (most MARC dialect strings are on staff, so we use that as basis) 3) Apply the patch 4) Update again to fix translation files, verbose perl translate update nn-NO -v 5) Install language, verbose, verify translations perl translate install nn-NO -v 6) Create translation files rm po/nn-NO* perl translate create nn-NO we must have this list: po/nn-NO-marc-MARC21.po po/nn-NO-marc-NORMARC.po po/nn-NO-marc-UNIMARC.po po/nn-NO-opac-bootstrap.po po/nn-NO-pref.po po/nn-NO-staff-help.po po/nn-NO-staff-prog.po Additional tests: 7) Number of msgids 7.a) Before patch and after upgrade, extract and count msgids for i in $(ls po/nn-NO-*po); \ do msginit -i $i -o nn-old.po --no-translator --no-wrap --locale=nn_NO; \ egrep ^msgid nn-old.po >> old; \ done sort old | uniq | tee s-old | wc -l > n-old s-old: have all msgids n-old: number of msgids 7.b) After patch and after creation of new files Repeat procedure, diferent files (s-new, n-new) 7.c) Compare (diff s-old snew), they are the same (save for a strange UNIMARC char in my case, but it's present on corresponding PO file) 8) Installed dirs/files 8.a) List of EN dirs/files cd koha-tmpl find | egrep "/en/" > en 8.b) List of nn-NO dirs/files. After patch and language install cd koha-tmpl find | egrep "/nn-NO/" | sed 's|/nn-NO/|/en/|' > nn 8.c) Compare (diff en nn), they are the same Signed-off-by: Magnus Enger <magnus@enger.priv.no> Followed the steps outlined by Bernardo, and everything works as expected. I think the most important points are that "perl translate create nn-NO" produces the right files, and translating anything in them, then doing "translate install" makes the translations show up in the interface. The numbers msgids in the nn-NO correspond well wit the number of msgids in other sets of .po files. I bet y'all will be happy when you don't have to see the stupid Norwegian strings when you translate! ;-) Bernardo, All looks good but I would like to know what is the plan when it will be pushed into master. Because update/install won't work until the po file is not recreated. Do you plan a big commit? I am wondering if the good idea wouldn't be to wait for the "git repo cleanup" you are doing (have done?) with Chris (see http://lists.koha-community.org/pipermail/koha-devel/2015-March/041293.html). It will avoid the creation and deletion of the same files in a short time. Note that this patch removes the version number in the filename Introduced by: commit 814795da60318c70059fc1dfbe7be4da37109db0 Author: Frédéric Demians <f.demians@tamil.fr> Date: Sun Dec 19 10:57:51 2010 +0100 Bug 5506 Improve translation script And updated only once: commit 69333c9f79bf2cb8ac4e58f46da714dca962a8ee Author: Chris Cormack <chrisc@catalyst.net.nz> Date: Sat Oct 22 05:33:40 2011 +1300 Moving po files to 3.6.0 filenames What was the idea behind that? I think in the past all the po files used to have the version number of the version they were applying to - but it was more trouble than useful to keep changing the file names. Currently the staff po file is the only file that still has a (wrong) version number. I am in favor of the change! :) Bernardo, can you please take a look? Tomas has set us a tight deadline! (In reply to Jonathan Druart from comment #12) > Bernardo, > All looks good but I would like to know what is the plan when it will be > pushed into master. > Because update/install won't work until the po file is not recreated. > Do you plan a big commit? Jonathan, if/when this bug is pushed i'll provide our RM with a patch to update/rename all po files. I've done something similar before, on every major release and to fix errors in translation files. But yes, until 'that' patch is pushed, translate will be problematic. But we can warn the RM to push this and the other together. > > I am wondering if the good idea wouldn't be to wait for the "git repo > cleanup" you are doing (have done?) with Chris (see > http://lists.koha-community.org/pipermail/koha-devel/2015-March/041293.html). > It will avoid the creation and deletion of the same files in a short time. We haven't agreed on a timeline yet, but in any case i'll be after 3.20 (In reply to Jonathan Druart from comment #13) > Note that this patch removes the version number in the filename > ... > What was the idea behind that? As Katrin said, it's a relic from the past, not updated in a long long time. It's a change with no practical justification, only a aesthetic one. I prefer not to have a wrong number. Created attachment 38423 [details] [review] Bug 7939 - Separate po files for different MARC dialects All looks good, great work Bernardo! Patches pushed to master. Thanks Bernardo! |