Bug 10963 - Simplified creation of MARC21 sample frameworks
Summary: Simplified creation of MARC21 sample frameworks
Status: CLOSED FIXED
Alias: None
Product: Koha
Classification: Unclassified
Component: Installation and upgrade (web-based installer) (show other bugs)
Version: Main
Hardware: All All
: P5 - low enhancement (vote)
Assignee: Bernardo Gonzalez Kriegel
QA Contact: Testopia
URL:
Keywords:
Depends on:
Blocks: 13897
  Show dependency treegraph
 
Reported: 2013-09-27 23:36 UTC by Bernardo Gonzalez Kriegel
Modified: 2018-02-16 16:45 UTC (History)
13 users (show)

See Also:
Change sponsored?: ---
Patch complexity: ---
Documentation contact:
Documentation submission:
Text to go in the release notes:
Version(s) released in:


Attachments
Bug 10963 - [1/9] - Simplified creation of MARC21 sample frameworks (17.86 KB, patch)
2013-09-28 03:18 UTC, Bernardo Gonzalez Kriegel
Details | Diff | Splinter Review
Bug 10963 - [2/9] - Simplified creation of MARC21 sample frameworks (465.08 KB, patch)
2013-09-28 03:18 UTC, Bernardo Gonzalez Kriegel
Details | Diff | Splinter Review
Bug 10963 - [3/9] - Simplified creation of MARC21 sample frameworks (458.62 KB, patch)
2013-09-28 03:18 UTC, Bernardo Gonzalez Kriegel
Details | Diff | Splinter Review
Bug 10963 - [4/9] - Simplified creation of MARC21 sample frameworks (458.89 KB, patch)
2013-09-28 03:18 UTC, Bernardo Gonzalez Kriegel
Details | Diff | Splinter Review
Bug 10963 - [5/9] - Simplified creation of MARC21 sample frameworks (458.81 KB, patch)
2013-09-28 03:18 UTC, Bernardo Gonzalez Kriegel
Details | Diff | Splinter Review
Bug 10963 - [6/9] - Simplified creation of MARC21 sample frameworks (458.62 KB, patch)
2013-09-28 03:18 UTC, Bernardo Gonzalez Kriegel
Details | Diff | Splinter Review
Bug 10963 - [7/9] - Simplified creation of MARC21 sample frameworks (458.73 KB, patch)
2013-09-28 03:18 UTC, Bernardo Gonzalez Kriegel
Details | Diff | Splinter Review
Bug 10963 - [8/9] - Simplified creation of MARC21 sample frameworks (458.69 KB, patch)
2013-09-28 03:19 UTC, Bernardo Gonzalez Kriegel
Details | Diff | Splinter Review
Bug 10963 - [9/9] - Simplified creation of MARC21 sample frameworks (462.34 KB, patch)
2013-09-28 03:19 UTC, Bernardo Gonzalez Kriegel
Details | Diff | Splinter Review
Bug 10963 - [1/9] - Simplified creation of MARC21 sample frameworks (17.96 KB, patch)
2013-10-07 19:06 UTC, Bernardo Gonzalez Kriegel
Details | Diff | Splinter Review
Framework for articles (272.17 KB, text/xml)
2013-10-21 22:20 UTC, Elaine Bradtke
Details
Fast add framework sample (74.47 KB, text/xml)
2013-10-21 22:22 UTC, Elaine Bradtke
Details
hidden-differences.txt (33.54 KB, text/plain)
2013-12-06 03:04 UTC, Bernardo Gonzalez Kriegel
Details
hidden-differences-no-new.txt (15.66 KB, text/plain)
2013-12-06 03:05 UTC, Bernardo Gonzalez Kriegel
Details
hidden-real.txt (2.56 KB, text/plain)
2013-12-06 03:05 UTC, Bernardo Gonzalez Kriegel
Details
Bug 10963 - [10] Simplified creation of MARC21 sample frameworks (57.34 KB, patch)
2013-12-06 23:34 UTC, Bernardo Gonzalez Kriegel
Details | Diff | Splinter Review
Bug 10963 - Simplified creation - FA framework (28.70 KB, patch)
2014-04-12 19:20 UTC, Bernardo Gonzalez Kriegel
Details | Diff | Splinter Review
Bug 10963 - Simplified creation - BKS framework (465.81 KB, patch)
2014-04-12 19:20 UTC, Bernardo Gonzalez Kriegel
Details | Diff | Splinter Review
Bug 10963 - Simplified creation - CF framework (459.16 KB, patch)
2014-04-12 19:20 UTC, Bernardo Gonzalez Kriegel
Details | Diff | Splinter Review
Bug 10963 - Simplified creation - SR framework (459.42 KB, patch)
2014-04-12 19:20 UTC, Bernardo Gonzalez Kriegel
Details | Diff | Splinter Review
Bug 10963 - Simplified creation - VR framework (459.27 KB, patch)
2014-04-12 19:20 UTC, Bernardo Gonzalez Kriegel
Details | Diff | Splinter Review
Bug 10963 - Simplified creation - AR framework (458.97 KB, patch)
2014-04-12 19:21 UTC, Bernardo Gonzalez Kriegel
Details | Diff | Splinter Review
Bug 10963 - Simplified creation - KT framework (458.93 KB, patch)
2014-04-12 19:21 UTC, Bernardo Gonzalez Kriegel
Details | Diff | Splinter Review
Bug 10963 - Simplified creation - IR framework (459.18 KB, patch)
2014-04-12 19:21 UTC, Bernardo Gonzalez Kriegel
Details | Diff | Splinter Review
Bug 10963 - Simplified creation - SER framework (462.66 KB, patch)
2014-04-12 19:21 UTC, Bernardo Gonzalez Kriegel
Details | Diff | Splinter Review
Bug 10963 - Revert old hidden values (46.11 KB, patch)
2014-04-12 19:21 UTC, Bernardo Gonzalez Kriegel
Details | Diff | Splinter Review
Bug 10963: Simplified creation - FA framework (28.71 KB, patch)
2015-03-04 11:58 UTC, Bernardo Gonzalez Kriegel
Details | Diff | Splinter Review
Bug 10963: Simplified creation - BKS framework (465.78 KB, patch)
2015-03-04 11:58 UTC, Bernardo Gonzalez Kriegel
Details | Diff | Splinter Review
Bug 10963: Simplified creation - CF framework (459.13 KB, patch)
2015-03-04 11:58 UTC, Bernardo Gonzalez Kriegel
Details | Diff | Splinter Review
Bug 10963: Simplified creation - SR framework (459.39 KB, patch)
2015-03-04 11:59 UTC, Bernardo Gonzalez Kriegel
Details | Diff | Splinter Review
Bug 10963: Simplified creation - VR framework (459.24 KB, patch)
2015-03-04 11:59 UTC, Bernardo Gonzalez Kriegel
Details | Diff | Splinter Review
Bug 10963: Simplified creation - AR framework (458.94 KB, patch)
2015-03-04 11:59 UTC, Bernardo Gonzalez Kriegel
Details | Diff | Splinter Review
Bug 10963: Simplified creation - KT framework (458.90 KB, patch)
2015-03-04 11:59 UTC, Bernardo Gonzalez Kriegel
Details | Diff | Splinter Review
Bug 10963: Simplified creation - IR framework (459.15 KB, patch)
2015-03-04 12:00 UTC, Bernardo Gonzalez Kriegel
Details | Diff | Splinter Review
Bug 10963: Simplified creation - SER framework (462.62 KB, patch)
2015-03-04 12:00 UTC, Bernardo Gonzalez Kriegel
Details | Diff | Splinter Review
Bug 10963: Revert old hidden values (46.11 KB, patch)
2015-03-04 12:00 UTC, Bernardo Gonzalez Kriegel
Details | Diff | Splinter Review
Bug 10963: Simplified creation - FA framework (28.77 KB, patch)
2015-03-04 13:59 UTC, Mirko Tietgen
Details | Diff | Splinter Review
Bug 10963: Simplified creation - BKS framework (465.84 KB, patch)
2015-03-04 14:00 UTC, Mirko Tietgen
Details | Diff | Splinter Review
Bug 10963: Simplified creation - CF framework (459.18 KB, patch)
2015-03-04 14:00 UTC, Mirko Tietgen
Details | Diff | Splinter Review
Bug 10963: Simplified creation - SR framework (459.44 KB, patch)
2015-03-04 14:00 UTC, Mirko Tietgen
Details | Diff | Splinter Review
Bug 10963: Simplified creation - VR framework (459.30 KB, patch)
2015-03-04 14:00 UTC, Mirko Tietgen
Details | Diff | Splinter Review
Bug 10963: Simplified creation - AR framework (459.00 KB, patch)
2015-03-04 14:00 UTC, Mirko Tietgen
Details | Diff | Splinter Review
Bug 10963: Simplified creation - KT framework (458.96 KB, patch)
2015-03-04 14:01 UTC, Mirko Tietgen
Details | Diff | Splinter Review
Bug 10963: Simplified creation - IR framework (459.21 KB, patch)
2015-03-04 14:01 UTC, Mirko Tietgen
Details | Diff | Splinter Review
Bug 10963: Simplified creation - SER framework (462.68 KB, patch)
2015-03-04 14:01 UTC, Mirko Tietgen
Details | Diff | Splinter Review
Bug 10963: Revert old hidden values (46.17 KB, patch)
2015-03-04 14:01 UTC, Mirko Tietgen
Details | Diff | Splinter Review
Bug 10963: Simplified creation - FA framework (28.73 KB, patch)
2015-04-25 13:18 UTC, Bernardo Gonzalez Kriegel
Details | Diff | Splinter Review
Diff between before and after the patches (4.13 MB, text/plain)
2015-06-19 11:51 UTC, Jonathan Druart
Details
Bug 10963: Simplified creation - BKS framework (465.90 KB, patch)
2015-06-19 15:07 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 10963: Simplified creation - CF framework (459.25 KB, patch)
2015-06-19 15:07 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 10963: Simplified creation - SR framework (459.51 KB, patch)
2015-06-19 15:08 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 10963: Simplified creation - VR framework (459.36 KB, patch)
2015-06-19 15:08 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 10963: Simplified creation - AR framework (459.06 KB, patch)
2015-06-19 15:08 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 10963: Simplified creation - KT framework (459.02 KB, patch)
2015-06-19 15:08 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 10963: Simplified creation - IR framework (459.27 KB, patch)
2015-06-19 15:08 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 10963: Simplified creation - SER framework (462.75 KB, patch)
2015-06-19 15:08 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 10963: Revert old hidden values (46.23 KB, patch)
2015-06-19 15:08 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 10963: Simplified creation - FA framework (28.80 KB, patch)
2015-06-19 15:08 UTC, Jonathan Druart
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description Bernardo Gonzalez Kriegel 2013-09-27 23:36:09 UTC
As mentioned on IRC, use of current sample frameworks is discouraged
because they are incomplete or out of sync with default framework.

I propose to build sample and fast add frameworks using default framework.
New files are smaller and we need to mantain a single framework.

Also there will be less strings to translate, when time comes for that.
Comment 1 Bernardo Gonzalez Kriegel 2013-09-28 03:18:07 UTC Comment hidden (obsolete)
Comment 2 Bernardo Gonzalez Kriegel 2013-09-28 03:18:15 UTC Comment hidden (obsolete)
Comment 3 Bernardo Gonzalez Kriegel 2013-09-28 03:18:23 UTC Comment hidden (obsolete)
Comment 4 Bernardo Gonzalez Kriegel 2013-09-28 03:18:31 UTC Comment hidden (obsolete)
Comment 5 Bernardo Gonzalez Kriegel 2013-09-28 03:18:40 UTC Comment hidden (obsolete)
Comment 6 Bernardo Gonzalez Kriegel 2013-09-28 03:18:49 UTC Comment hidden (obsolete)
Comment 7 Bernardo Gonzalez Kriegel 2013-09-28 03:18:57 UTC Comment hidden (obsolete)
Comment 8 Bernardo Gonzalez Kriegel 2013-09-28 03:19:05 UTC Comment hidden (obsolete)
Comment 9 Bernardo Gonzalez Kriegel 2013-09-28 03:19:14 UTC Comment hidden (obsolete)
Comment 10 Bernardo Gonzalez Kriegel 2013-09-28 03:20:12 UTC
Patch is too big to fit, so I splitted it in 9 patches

Here main description and test plan

Bug 10963 - Simplified creation of MARC21 sample frameworks

This patch series implements the creation of sample frameworks
using MARC21 default framework as base, just by copy it.

There are lots of obsoleted tags/subtags, but if default
is the recommended then as far as they remain hidden I
think there is no problem.

Only to new installs

To test:
1) Apply all patches
2) Remove all frameworks
3) Load default MARC21 framework
4) Load marc21_fastadd_framework.sql
5) Verify that Fast Add framework has same tags as before
6) Load marc21_simple_bib_frameworks.sql
7) Verify all sample frameworks are created
Comment 11 Bernardo Gonzalez Kriegel 2013-09-28 03:28:23 UTC
> Patch 4/9
> 
> This patch modifies simple bibs frameworks for VR

Wrong description but patch is good, modifies SR framework
Comment 12 I'm just a bot 2013-09-29 04:44:07 UTC
Patch applied cleanly, go forth and signoff
Comment 13 Bernardo Gonzalez Kriegel 2013-10-07 19:06:44 UTC Comment hidden (obsolete)
Comment 14 David Cook 2013-10-21 19:15:53 UTC
I really like the idea for this patch, as it will mean that we only need to manage the default framework for adding/modifying MARC fields and subfields when doing a fresh install.

However...it seems to me that it is completely copying the subfields as is which is different from what we're currently doing. If you run the following SQL:

"select group_concat(frameworkcode),hidden,tagfield,tagsubfield from marc_subfield_structure group by concat(tagfield,tagsubfield,hidden);"

You'll notice that the other sample frameworks hide different fields that the default framework displays.

I think we can still keep the methodology set out in these frameworks, but maybe we want to come up with some criteria (or copy the existing criteria) for hiding subfields/fields in these other sample frameworks.

Otherwise, libraries are just going to have to customize every framework and then it becomes pointless having "other" sample frameworks.
Comment 15 Bernardo Gonzalez Kriegel 2013-10-21 19:29:06 UTC
Thanks David, and I agree with you. 

This was just an idea, and I could rewrite it like auth frameworks in which we copy most fields/subfields except obsolete ones.

But as which fields to show or hide I have no opinion (not a librarian).

Maybe you could give me some hint, and I can rewrite the patch.
Comment 16 Elaine Bradtke 2013-10-21 22:08:22 UTC
I have a few custom frameworks that we use - modified from those available on the wiki. I will attach the two that I've edited. FA = Fast add - the original one was lacking some critical fields for our library.  ART = Articles - for periodical / journal articles the original of this was also lacking fields we needed.  I don't know how useful they will be, ours is an unusual library. I tried to cut out as many of the extra / obsolete fields as I could.
Comment 17 Elaine Bradtke 2013-10-21 22:17:52 UTC
I have a few custom frameworks that we use - modified from those available on the wiki. I will attach the two that I've edited. FA = Fast add - the original one was lacking some critical fields for our library.  ART = Articles - for periodical / journal articles the original of this was also lacking fields we needed.  I don't know how useful they will be, ours is an unusual library. I tried to cut out as many of the extra / obsolete fields as I could.
Comment 18 Elaine Bradtke 2013-10-21 22:20:22 UTC
Created attachment 22206 [details]
Framework for articles

Sample framework for journal articles used in the VWML
Comment 19 Elaine Bradtke 2013-10-21 22:22:37 UTC
Created attachment 22207 [details]
Fast add framework sample

Fast add framework in use at VWML
Comment 20 Katrin Fischer 2013-10-27 02:39:20 UTC
Hi Elaine,

the goal of this bug is a bit different, as the idea is to change the way we create the specified frameworks on database level. But we have a place for your frameworks on the wiki - http://wiki.koha-community.org/wiki/MARC_frameworks. If you want, you can upload them there or I can do it - just let me know.
Comment 21 Katrin Fischer 2013-12-05 22:15:23 UTC
Hm, looking at David's and Bernardo's conversation I think this needs a bit more work. I think we'd need to copy the standard framework, but keep the hidden values of the current simple frameworks.
Comment 22 Bernardo Gonzalez Kriegel 2013-12-06 03:04:06 UTC
Created attachment 23329 [details]
hidden-differences.txt
Comment 23 Bernardo Gonzalez Kriegel 2013-12-06 03:05:07 UTC
Created attachment 23330 [details]
hidden-differences-no-new.txt
Comment 24 Bernardo Gonzalez Kriegel 2013-12-06 03:05:45 UTC
Created attachment 23331 [details]
hidden-real.txt
Comment 25 Bernardo Gonzalez Kriegel 2013-12-06 03:11:16 UTC
Thank's Katrin, In Discussion is the rigth state!

I was working on this, now I will share what I found

1) Following David on comment #14 I did this query
(only default and simple fw loaded)

select tagfield, tagsubfield, hidden,
       group_concat(frameworkcode order by frameworkcode) as fws 
from marc_subfield_structure 
group by concat(tagfield,tagsubfield,hidden) 
having fws <> ',AR,BKS,CF,IR,KT,SER,SR,VR';

The result show where fw differs on hidden value between them.

2) The rearranged results, can be seen on attached file 
hidden-differences.txt (N for NULL value)

There are 635 lines on that file, most are from subfields present 
on default but not in simple fw

3) As we are looking for customizations, I'll discard all new subfields, they will be copied anyway.

Results on attached file hidden-differences-no-new.txt

Now they are "only" 296 lines.
But they are still a lot of updates to do on a copied fw, not practical

4) What I saw is that many subfields on simple fw have identical value on hidden field, but not on default fw.

PROPOSAL: 
when all simple fw agrees on a hidden value, 
we set default with that value

That will leave us with "real" customizations

5) With that proposal, results on attached file hidden-real.txt
There are only 39 lines now. 
And only hidden values are -6, -1 and 0
Just to remember:
 -6 => OPAC Intranet !Editor !Collapsed
 -1 => OPAC Intranet  Editor  Collapsed
  0 => OPAC Intranet  Editor !Collapsed

If we change -1 for 0 (if it's on editor, why collapsed?), we can take out a few more lines, but that's enough

6) Well, if there is a clear customization plan on hidden-real.txt, it's difficult for me to see it (except a few subfields)

For me is more like a pseudo-random distribution 
of 3 values on 39 subfields out of 3959 on 9 fw :-)

If we are going to provide simple fw with a meaning.
I expect directives like:

On serials and ..., hide 020 and ...
On books and ..., hide 022 and ...
etc.

In that way, if we have a default fw with some sort of visibility 
on all fields/subs, we can hide at field level on simple fw. 

Writing all update lines needed to implement hidden-real "as is" is doable,
and I already did it (without my proposal, around ~300 lines), so if I have no more comments on how to do this right, I will upload it and switch to need signoff.

Regards,
B
Comment 26 Bernardo Gonzalez Kriegel 2013-12-06 23:34:37 UTC Comment hidden (obsolete)
Comment 27 Bernardo Gonzalez Kriegel 2013-12-06 23:54:44 UTC
Ready to be signed!
It makes all simple and fastadd fw as copies of default and preserves previous hidden values, so no change except new fields.

But.. I think that we will have a big conflict with Bug 11330, first patch needs to be discarded except for changes on default, and Katrin's followup can stay.

Or.. this can be rewritten. It's one or the other.
Comment 28 Katrin Fischer 2014-03-23 17:37:05 UTC
Hi Bernardo, what is the next step here now that bug 11330 got pushed?
Comment 29 Bernardo Gonzalez Kriegel 2014-03-23 19:59:53 UTC
Hi Katrin, I will look into this again and try to make a new patch
Comment 30 Bernardo Gonzalez Kriegel 2014-04-12 19:20:28 UTC Comment hidden (obsolete)
Comment 31 Bernardo Gonzalez Kriegel 2014-04-12 19:20:36 UTC Comment hidden (obsolete)
Comment 32 Bernardo Gonzalez Kriegel 2014-04-12 19:20:44 UTC Comment hidden (obsolete)
Comment 33 Bernardo Gonzalez Kriegel 2014-04-12 19:20:52 UTC Comment hidden (obsolete)
Comment 34 Bernardo Gonzalez Kriegel 2014-04-12 19:20:59 UTC Comment hidden (obsolete)
Comment 35 Bernardo Gonzalez Kriegel 2014-04-12 19:21:07 UTC Comment hidden (obsolete)
Comment 36 Bernardo Gonzalez Kriegel 2014-04-12 19:21:15 UTC Comment hidden (obsolete)
Comment 37 Bernardo Gonzalez Kriegel 2014-04-12 19:21:24 UTC Comment hidden (obsolete)
Comment 38 Bernardo Gonzalez Kriegel 2014-04-12 19:21:31 UTC Comment hidden (obsolete)
Comment 39 Bernardo Gonzalez Kriegel 2014-04-12 19:21:39 UTC Comment hidden (obsolete)
Comment 40 Bernardo Gonzalez Kriegel 2014-04-12 19:25:12 UTC
New set of patches.
Some kind of test plan on Comment 26
I have checked that all old hidden values are present
Comment 41 Mirko Tietgen 2015-03-03 20:24:17 UTC
Patch does not apply. I already did half the test plan, I'd continue testing if you'd rebase it.

Apply? [(y)es, (n)o, (i)nteractive] i
Applying: Bug 10963 - Simplified creation - FA framework
Applying: Bug 10963 - Simplified creation - BKS framework
Using index info to reconstruct a base tree...
Falling back to patching base and 3-way merge...
Auto-merging installer/data/mysql/en/marcflavour/marc21/optional/marc21_simple_bib_frameworks.sql
CONFLICT (content): Merge conflict in installer/data/mysql/en/marcflavour/marc21/optional/marc21_simple_bib_frameworks.sql
Failed to merge in the changes.
Comment 42 Bernardo Gonzalez Kriegel 2015-03-04 11:58:36 UTC Comment hidden (obsolete)
Comment 43 Bernardo Gonzalez Kriegel 2015-03-04 11:58:46 UTC Comment hidden (obsolete)
Comment 44 Bernardo Gonzalez Kriegel 2015-03-04 11:58:57 UTC Comment hidden (obsolete)
Comment 45 Bernardo Gonzalez Kriegel 2015-03-04 11:59:13 UTC Comment hidden (obsolete)
Comment 46 Bernardo Gonzalez Kriegel 2015-03-04 11:59:28 UTC Comment hidden (obsolete)
Comment 47 Bernardo Gonzalez Kriegel 2015-03-04 11:59:44 UTC Comment hidden (obsolete)
Comment 48 Bernardo Gonzalez Kriegel 2015-03-04 11:59:54 UTC Comment hidden (obsolete)
Comment 49 Bernardo Gonzalez Kriegel 2015-03-04 12:00:06 UTC Comment hidden (obsolete)
Comment 50 Bernardo Gonzalez Kriegel 2015-03-04 12:00:19 UTC Comment hidden (obsolete)
Comment 51 Bernardo Gonzalez Kriegel 2015-03-04 12:00:29 UTC Comment hidden (obsolete)
Comment 52 Bernardo Gonzalez Kriegel 2015-03-04 12:02:18 UTC
(In reply to Mirko Tietgen from comment #41)
> Patch does not apply. I already did half the test plan, I'd continue testing
> if you'd rebase it.
> 

Patches rebased, Thanks for testing!!
Test on Comment #26 still valid.
Comment 53 Mirko Tietgen 2015-03-04 13:59:50 UTC Comment hidden (obsolete)
Comment 54 Mirko Tietgen 2015-03-04 14:00:09 UTC Comment hidden (obsolete)
Comment 55 Mirko Tietgen 2015-03-04 14:00:19 UTC Comment hidden (obsolete)
Comment 56 Mirko Tietgen 2015-03-04 14:00:29 UTC Comment hidden (obsolete)
Comment 57 Mirko Tietgen 2015-03-04 14:00:40 UTC Comment hidden (obsolete)
Comment 58 Mirko Tietgen 2015-03-04 14:00:50 UTC Comment hidden (obsolete)
Comment 59 Mirko Tietgen 2015-03-04 14:01:02 UTC Comment hidden (obsolete)
Comment 60 Mirko Tietgen 2015-03-04 14:01:12 UTC Comment hidden (obsolete)
Comment 61 Mirko Tietgen 2015-03-04 14:01:23 UTC Comment hidden (obsolete)
Comment 62 Mirko Tietgen 2015-03-04 14:01:29 UTC Comment hidden (obsolete)
Comment 63 Mirko Tietgen 2015-03-04 14:03:04 UTC
Followed the test plan in comment 26, diffs contain no lines.
Comment 64 Jonathan Druart 2015-04-17 11:19:28 UTC
Bernardo, 
This does not longer apply.
What is the plan here? I am under the impression we should not lag too much, it will be a pain to maintain this patch set if master moves!
Do you generate manually or using a script?
What is your confidence level in this patch set?
Comment 65 Bernardo Gonzalez Kriegel 2015-04-25 13:18:31 UTC Comment hidden (obsolete)
Comment 66 Bernardo Gonzalez Kriegel 2015-04-25 13:26:56 UTC
(In reply to Jonathan Druart from comment #64)
> Bernardo, 
> This does not longer apply.
> What is the plan here? I am under the impression we should not lag too much,
> it will be a pain to maintain this patch set if master moves!
> Do you generate manually or using a script?
> What is your confidence level in this patch set?

Hi, only needed to replace the first patch, a single line difference on FA fw.

The plan is simplify current sample fw sql files, no problem with that because they are not updated since long time.
What I check with a script are default frameworks (bib or auth)

With this patches we generate sample fw copying from default, so no need to update them, but only works at install time.

This need a followup to use the same procedure on each language,
will do that later.

Reverted to Signed-Off status
Comment 67 Jonathan Druart 2015-06-19 11:51:07 UTC
Created attachment 40364 [details]
Diff between before and after the patches

Bernardo,
What I did:
% git checkout master; git reset --hard origin/master
% mysql
mysql> DROP TABLE koha_test;
mysql> create database koha_test CHARACTER SET utf8 COLLATE utf8_unicode_ci;
% mysql koha_test < installer/data/mysql/kohastructure.sql
% mysql koha_test < installer/data/mysql/en/marcflavour/marc21/mandatory/marc21_framework_DEFAULT.sql
% mysql koha_test < installer/data/mysql/en/marcflavour/marc21/optional/marc21_fastadd_framework.sql 
% mysql koha_test < installer/data/mysql/en/marcflavour/marc21/optional/marc21_simple_bib_frameworks.sql

% mysqldump koha_test > dump_before.sql

% git checkout -b bug_10963; git bz apply 10963
# The following is the same as before
% mysql
mysql> DROP TABLE koha_test;
mysql> create database koha_test CHARACTER SET utf8 COLLATE utf8_unicode_ci;
% mysql koha_test < installer/data/mysql/kohastructure.sql
% mysql koha_test < installer/data/mysql/en/marcflavour/marc21/mandatory/marc21_framework_DEFAULT.sql
% mysql koha_test < installer/data/mysql/en/marcflavour/marc21/optional/marc21_fastadd_framework.sql 
% mysql koha_test < installer/data/mysql/en/marcflavour/marc21/optional/marc21_simple_bib_frameworks.sql

% mysqldump koha_test > dump_after.sql

% git add dump_before.sql
% git commit -m"before"
% mv dump_after.sql dump_before.sql
% git commit -a -m"after"

% git show HEAD --word-diff > /tmp/diff

This diff file is attached.

There is a lof of changes, is it expected?
Comment 68 Bernardo Gonzalez Kriegel 2015-06-19 13:13:30 UTC
(In reply to Jonathan Druart from comment #67)
> 
> There is a lof of changes, is it expected?

Yes.
The purpose of the patch is to have a the same number of fields/subfields on all frameworks. I'm only updating MARC21 default, and cait always asks what to do with the others.

Before patch we have

select a.frameworkcode, a.tag, b.subfield  from (select frameworkcode, count(*) as tag from marc_tag_structure group by frameworkcode) a left join (select frameworkcode, count(*) as subfield from marc_subfield_structure group by frameworkcode) b on a.frameworkcode = b.frameworkcode;
+---------------+-----+----------+
| frameworkcode | tag | subfield |
+---------------+-----+----------+
|               | 340 |     3978 |
| AR            | 313 |     3620 |
| BKS           | 313 |     3620 |
| CF            | 313 |     3620 |
| FA            |  17 |      118 |
| IR            | 313 |     3619 |
| KT            | 313 |     3620 |
| SER           | 313 |     3619 |
| SR            | 313 |     3620 |
| VR            | 313 |     3620 |
+---------------+-----+----------+
(There is a 'bug' on simple fw file, fixed here, 
on line 10399, fw code is incorrect 'SF' -> 'SR')

After patch
+---------------+-----+----------+
| frameworkcode | tag | subfield |
+---------------+-----+----------+
|               | 340 |     3978 |
| AR            | 340 |     3978 |
| BKS           | 340 |     3978 |
| CF            | 340 |     3978 |
| FA            |  18 |      173 |
| IR            | 340 |     3978 |
| KT            | 340 |     3978 |
| SER           | 340 |     3978 |
| SR            | 340 |     3978 |
| VR            | 340 |     3978 |
+---------------+-----+----------+

So yes, you will see differences.
All new (by me) fields/subfields have hidden = -6, they don't show on 'new' fw.
Comment 69 Jonathan Druart 2015-06-19 14:37:57 UTC
For instance:

('711','c','Location of meeting','Location of [-meeting',0,0,NULL,7,NULL,NULL,'',NULL,-1,'BKS','','',NULL,9999),('711','d','Date-]{+meeting',1,0,NULL,7,NULL,NULL,'',NULL,-1,'BKS','','',NULL,9999),('711','d','Date+} of meeting','Date of meeting',0,0,NULL,7,NULL,NULL,'',NULL,-1,'BKS','','',NULL,9999)

If my eyes are not too tired, it means that 711$c becomes repeatable.
Comment 70 Bernardo Gonzalez Kriegel 2015-06-19 14:55:37 UTC
(In reply to Jonathan Druart from comment #69)
> For instance:
> 
> ... 
> If my eyes are not too tired, it means that 711$c becomes repeatable.

Your eyes are right, but... it is repeatable :)
(http://www.loc.gov/marc/bibliographic/bd711.html)

If there is any problem, it's on default fw.
The others are just copying from it.
Couldn't find any discrepancy, but could be.
Comment 71 Jonathan Druart 2015-06-19 15:07:49 UTC
Created attachment 40395 [details] [review]
Bug 10963: Simplified creation - BKS framework

Patch 2/9

This patch rewrites BKS framework

Signed-off-by: Mirko Tietgen <mirko@abunchofthings.net>

Signed-off-by: Jonathan Druart <jonathan.druart@koha-community.org>
Comment 72 Jonathan Druart 2015-06-19 15:07:57 UTC
Created attachment 40396 [details] [review]
Bug 10963: Simplified creation - CF framework

Patch 3/9

This patch rewrites CF framework

Signed-off-by: Mirko Tietgen <mirko@abunchofthings.net>

Signed-off-by: Jonathan Druart <jonathan.druart@koha-community.org>
Comment 73 Jonathan Druart 2015-06-19 15:08:02 UTC
Created attachment 40397 [details] [review]
Bug 10963: Simplified creation - SR framework

Patch 4/9

This patch rewrites SR framework

Signed-off-by: Mirko Tietgen <mirko@abunchofthings.net>

Signed-off-by: Jonathan Druart <jonathan.druart@koha-community.org>
Comment 74 Jonathan Druart 2015-06-19 15:08:07 UTC
Created attachment 40398 [details] [review]
Bug 10963: Simplified creation - VR framework

Patch 5/9

This patch rewrites VR framework

Signed-off-by: Mirko Tietgen <mirko@abunchofthings.net>

Signed-off-by: Jonathan Druart <jonathan.druart@koha-community.org>
Comment 75 Jonathan Druart 2015-06-19 15:08:13 UTC
Created attachment 40399 [details] [review]
Bug 10963: Simplified creation - AR framework

Patch 6/9

This patch rewrites AR framework

Signed-off-by: Mirko Tietgen <mirko@abunchofthings.net>

Signed-off-by: Jonathan Druart <jonathan.druart@koha-community.org>
Comment 76 Jonathan Druart 2015-06-19 15:08:18 UTC
Created attachment 40400 [details] [review]
Bug 10963: Simplified creation - KT framework

Patch 7/9

This patch rewrites KT framework

Signed-off-by: Mirko Tietgen <mirko@abunchofthings.net>

Signed-off-by: Jonathan Druart <jonathan.druart@koha-community.org>
Comment 77 Jonathan Druart 2015-06-19 15:08:23 UTC
Created attachment 40401 [details] [review]
Bug 10963: Simplified creation - IR framework

Patch 8/9

This patch rewrites IR framework

Signed-off-by: Mirko Tietgen <mirko@abunchofthings.net>

Signed-off-by: Jonathan Druart <jonathan.druart@koha-community.org>
Comment 78 Jonathan Druart 2015-06-19 15:08:29 UTC
Created attachment 40402 [details] [review]
Bug 10963: Simplified creation - SER framework

Patch 9/9

This patch rewrites SER framework

Signed-off-by: Mirko Tietgen <mirko@abunchofthings.net>

Signed-off-by: Jonathan Druart <jonathan.druart@koha-community.org>
Comment 79 Jonathan Druart 2015-06-19 15:08:33 UTC
Created attachment 40403 [details] [review]
Bug 10963: Revert old hidden values

This patch reverts old hidden values to marc_subfield_structure.

Signed-off-by: Mirko Tietgen <mirko@abunchofthings.net>

Signed-off-by: Jonathan Druart <jonathan.druart@koha-community.org>
Comment 80 Jonathan Druart 2015-06-19 15:08:40 UTC
Created attachment 40404 [details] [review]
Bug 10963: Simplified creation - FA framework

Patch 1/9

This patch modifies Fast Add framework

Signed-off-by: Mirko Tietgen <mirko@abunchofthings.net>

Signed-off-by: Jonathan Druart <jonathan.druart@koha-community.org>
Comment 81 Tomás Cohen Arazi 2015-09-07 17:13:27 UTC
Patches pushed to master.

Good job Bernardo!