Patch is too big to fit, so I splitted it in 9 patches

Here main description and test plan

Bug 10963 - Simplified creation of MARC21 sample frameworks

This patch series implements the creation of sample frameworks
using MARC21 default framework as base, just by copy it.

There are lots of obsoleted tags/subtags, but if default
is the recommended then as far as they remain hidden I
think there is no problem.

Only to new installs

To test:
1) Apply all patches
2) Remove all frameworks
3) Load default MARC21 framework
4) Load marc21_fastadd_framework.sql
5) Verify that Fast Add framework has same tags as before
6) Load marc21_simple_bib_frameworks.sql
7) Verify all sample frameworks are created

> Patch 4/9
> 
> This patch modifies simple bibs frameworks for VR

Wrong description but patch is good, modifies SR framework

Patch applied cleanly, go forth and signoff

I really like the idea for this patch, as it will mean that we only need to manage the default framework for adding/modifying MARC fields and subfields when doing a fresh install.

However...it seems to me that it is completely copying the subfields as is which is different from what we're currently doing. If you run the following SQL:

"select group_concat(frameworkcode),hidden,tagfield,tagsubfield from marc_subfield_structure group by concat(tagfield,tagsubfield,hidden);"

You'll notice that the other sample frameworks hide different fields that the default framework displays.

I think we can still keep the methodology set out in these frameworks, but maybe we want to come up with some criteria (or copy the existing criteria) for hiding subfields/fields in these other sample frameworks.

Otherwise, libraries are just going to have to customize every framework and then it becomes pointless having "other" sample frameworks.

Thanks David, and I agree with you. 

This was just an idea, and I could rewrite it like auth frameworks in which we copy most fields/subfields except obsolete ones.

But as which fields to show or hide I have no opinion (not a librarian).

Maybe you could give me some hint, and I can rewrite the patch.

I have a few custom frameworks that we use - modified from those available on the wiki. I will attach the two that I've edited. FA = Fast add - the original one was lacking some critical fields for our library.  ART = Articles - for periodical / journal articles the original of this was also lacking fields we needed.  I don't know how useful they will be, ours is an unusual library. I tried to cut out as many of the extra / obsolete fields as I could.

I have a few custom frameworks that we use - modified from those available on the wiki. I will attach the two that I've edited. FA = Fast add - the original one was lacking some critical fields for our library.  ART = Articles - for periodical / journal articles the original of this was also lacking fields we needed.  I don't know how useful they will be, ours is an unusual library. I tried to cut out as many of the extra / obsolete fields as I could.

Created attachment 22206 [details]
Framework for articles

Sample framework for journal articles used in the VWML

Created attachment 22207 [details]
Fast add framework sample

Fast add framework in use at VWML

Hi Elaine,

the goal of this bug is a bit different, as the idea is to change the way we create the specified frameworks on database level. But we have a place for your frameworks on the wiki - http://wiki.koha-community.org/wiki/MARC_frameworks. If you want, you can upload them there or I can do it - just let me know.

Hm, looking at David's and Bernardo's conversation I think this needs a bit more work. I think we'd need to copy the standard framework, but keep the hidden values of the current simple frameworks.

Created attachment 23329 [details]
hidden-differences.txt

Created attachment 23330 [details]
hidden-differences-no-new.txt

Created attachment 23331 [details]
hidden-real.txt

Thank's Katrin, In Discussion is the rigth state!

I was working on this, now I will share what I found

1) Following David on comment #14 I did this query
(only default and simple fw loaded)

select tagfield, tagsubfield, hidden,
       group_concat(frameworkcode order by frameworkcode) as fws 
from marc_subfield_structure 
group by concat(tagfield,tagsubfield,hidden) 
having fws <> ',AR,BKS,CF,IR,KT,SER,SR,VR';

The result show where fw differs on hidden value between them.

2) The rearranged results, can be seen on attached file 
hidden-differences.txt (N for NULL value)

There are 635 lines on that file, most are from subfields present 
on default but not in simple fw

3) As we are looking for customizations, I'll discard all new subfields, they will be copied anyway.

Results on attached file hidden-differences-no-new.txt

Now they are "only" 296 lines.
But they are still a lot of updates to do on a copied fw, not practical

4) What I saw is that many subfields on simple fw have identical value on hidden field, but not on default fw.

PROPOSAL: 
when all simple fw agrees on a hidden value, 
we set default with that value

That will leave us with "real" customizations

5) With that proposal, results on attached file hidden-real.txt
There are only 39 lines now. 
And only hidden values are -6, -1 and 0
Just to remember:
 -6 => OPAC Intranet !Editor !Collapsed
 -1 => OPAC Intranet  Editor  Collapsed
  0 => OPAC Intranet  Editor !Collapsed

If we change -1 for 0 (if it's on editor, why collapsed?), we can take out a few more lines, but that's enough

6) Well, if there is a clear customization plan on hidden-real.txt, it's difficult for me to see it (except a few subfields)

For me is more like a pseudo-random distribution 
of 3 values on 39 subfields out of 3959 on 9 fw :-)

If we are going to provide simple fw with a meaning.
I expect directives like:

On serials and ..., hide 020 and ...
On books and ..., hide 022 and ...
etc.

In that way, if we have a default fw with some sort of visibility 
on all fields/subs, we can hide at field level on simple fw. 

Writing all update lines needed to implement hidden-real "as is" is doable,
and I already did it (without my proposal, around ~300 lines), so if I have no more comments on how to do this right, I will upload it and switch to need signoff.

Regards,
B

Ready to be signed!
It makes all simple and fastadd fw as copies of default and preserves previous hidden values, so no change except new fields.

But.. I think that we will have a big conflict with Bug 11330, first patch needs to be discarded except for changes on default, and Katrin's followup can stay.

Or.. this can be rewritten. It's one or the other.

Hi Bernardo, what is the next step here now that bug 11330 got pushed?

Hi Katrin, I will look into this again and try to make a new patch

New set of patches.
Some kind of test plan on Comment 26
I have checked that all old hidden values are present

Patch does not apply. I already did half the test plan, I'd continue testing if you'd rebase it.

Apply? [(y)es, (n)o, (i)nteractive] i
Applying: Bug 10963 - Simplified creation - FA framework
Applying: Bug 10963 - Simplified creation - BKS framework
Using index info to reconstruct a base tree...
Falling back to patching base and 3-way merge...
Auto-merging installer/data/mysql/en/marcflavour/marc21/optional/marc21_simple_bib_frameworks.sql
CONFLICT (content): Merge conflict in installer/data/mysql/en/marcflavour/marc21/optional/marc21_simple_bib_frameworks.sql
Failed to merge in the changes.

(In reply to Mirko Tietgen from comment #41)
> Patch does not apply. I already did half the test plan, I'd continue testing
> if you'd rebase it.
> 

Patches rebased, Thanks for testing!!
Test on Comment #26 still valid.

Followed the test plan in comment 26, diffs contain no lines.

Bernardo, 
This does not longer apply.
What is the plan here? I am under the impression we should not lag too much, it will be a pain to maintain this patch set if master moves!
Do you generate manually or using a script?
What is your confidence level in this patch set?

(In reply to Jonathan Druart from comment #64)
> Bernardo, 
> This does not longer apply.
> What is the plan here? I am under the impression we should not lag too much,
> it will be a pain to maintain this patch set if master moves!
> Do you generate manually or using a script?
> What is your confidence level in this patch set?

Hi, only needed to replace the first patch, a single line difference on FA fw.

The plan is simplify current sample fw sql files, no problem with that because they are not updated since long time.
What I check with a script are default frameworks (bib or auth)

With this patches we generate sample fw copying from default, so no need to update them, but only works at install time.

This need a followup to use the same procedure on each language,
will do that later.

Reverted to Signed-Off status

Created attachment 40364 [details]
Diff between before and after the patches

Bernardo,
What I did:
% git checkout master; git reset --hard origin/master
% mysql
mysql> DROP TABLE koha_test;
mysql> create database koha_test CHARACTER SET utf8 COLLATE utf8_unicode_ci;
% mysql koha_test < installer/data/mysql/kohastructure.sql
% mysql koha_test < installer/data/mysql/en/marcflavour/marc21/mandatory/marc21_framework_DEFAULT.sql
% mysql koha_test < installer/data/mysql/en/marcflavour/marc21/optional/marc21_fastadd_framework.sql 
% mysql koha_test < installer/data/mysql/en/marcflavour/marc21/optional/marc21_simple_bib_frameworks.sql

% mysqldump koha_test > dump_before.sql

% git checkout -b bug_10963; git bz apply 10963
# The following is the same as before
% mysql
mysql> DROP TABLE koha_test;
mysql> create database koha_test CHARACTER SET utf8 COLLATE utf8_unicode_ci;
% mysql koha_test < installer/data/mysql/kohastructure.sql
% mysql koha_test < installer/data/mysql/en/marcflavour/marc21/mandatory/marc21_framework_DEFAULT.sql
% mysql koha_test < installer/data/mysql/en/marcflavour/marc21/optional/marc21_fastadd_framework.sql 
% mysql koha_test < installer/data/mysql/en/marcflavour/marc21/optional/marc21_simple_bib_frameworks.sql

% mysqldump koha_test > dump_after.sql

% git add dump_before.sql
% git commit -m"before"
% mv dump_after.sql dump_before.sql
% git commit -a -m"after"

% git show HEAD --word-diff > /tmp/diff

This diff file is attached.

There is a lof of changes, is it expected?

(In reply to Jonathan Druart from comment #67)
> 
> There is a lof of changes, is it expected?

Yes.
The purpose of the patch is to have a the same number of fields/subfields on all frameworks. I'm only updating MARC21 default, and cait always asks what to do with the others.

Before patch we have

select a.frameworkcode, a.tag, b.subfield  from (select frameworkcode, count(*) as tag from marc_tag_structure group by frameworkcode) a left join (select frameworkcode, count(*) as subfield from marc_subfield_structure group by frameworkcode) b on a.frameworkcode = b.frameworkcode;
+---------------+-----+----------+
| frameworkcode | tag | subfield |
+---------------+-----+----------+
|               | 340 |     3978 |
| AR            | 313 |     3620 |
| BKS           | 313 |     3620 |
| CF            | 313 |     3620 |
| FA            |  17 |      118 |
| IR            | 313 |     3619 |
| KT            | 313 |     3620 |
| SER           | 313 |     3619 |
| SR            | 313 |     3620 |
| VR            | 313 |     3620 |
+---------------+-----+----------+
(There is a 'bug' on simple fw file, fixed here, 
on line 10399, fw code is incorrect 'SF' -> 'SR')

After patch
+---------------+-----+----------+
| frameworkcode | tag | subfield |
+---------------+-----+----------+
|               | 340 |     3978 |
| AR            | 340 |     3978 |
| BKS           | 340 |     3978 |
| CF            | 340 |     3978 |
| FA            |  18 |      173 |
| IR            | 340 |     3978 |
| KT            | 340 |     3978 |
| SER           | 340 |     3978 |
| SR            | 340 |     3978 |
| VR            | 340 |     3978 |
+---------------+-----+----------+

So yes, you will see differences.
All new (by me) fields/subfields have hidden = -6, they don't show on 'new' fw.

For instance:

('711','c','Location of meeting','Location of [-meeting',0,0,NULL,7,NULL,NULL,'',NULL,-1,'BKS','','',NULL,9999),('711','d','Date-]{+meeting',1,0,NULL,7,NULL,NULL,'',NULL,-1,'BKS','','',NULL,9999),('711','d','Date+} of meeting','Date of meeting',0,0,NULL,7,NULL,NULL,'',NULL,-1,'BKS','','',NULL,9999)

If my eyes are not too tired, it means that 711$c becomes repeatable.

(In reply to Jonathan Druart from comment #69)
> For instance:
> 
> ... 
> If my eyes are not too tired, it means that 711$c becomes repeatable.

Your eyes are right, but... it is repeatable :)
(http://www.loc.gov/marc/bibliographic/bd711.html)

If there is any problem, it's on default fw.
The others are just copying from it.
Couldn't find any discrepancy, but could be.

Created attachment 40395 [details] [review]
Bug 10963: Simplified creation - BKS framework

Patch 2/9

This patch rewrites BKS framework

Signed-off-by: Mirko Tietgen <mirko@abunchofthings.net>

Signed-off-by: Jonathan Druart <jonathan.druart@koha-community.org>

Created attachment 40396 [details] [review]
Bug 10963: Simplified creation - CF framework

Patch 3/9

This patch rewrites CF framework

Signed-off-by: Mirko Tietgen <mirko@abunchofthings.net>

Signed-off-by: Jonathan Druart <jonathan.druart@koha-community.org>

Created attachment 40397 [details] [review]
Bug 10963: Simplified creation - SR framework

Patch 4/9

This patch rewrites SR framework

Signed-off-by: Mirko Tietgen <mirko@abunchofthings.net>

Signed-off-by: Jonathan Druart <jonathan.druart@koha-community.org>

Created attachment 40398 [details] [review]
Bug 10963: Simplified creation - VR framework

Patch 5/9

This patch rewrites VR framework

Signed-off-by: Mirko Tietgen <mirko@abunchofthings.net>

Signed-off-by: Jonathan Druart <jonathan.druart@koha-community.org>

Created attachment 40399 [details] [review]
Bug 10963: Simplified creation - AR framework

Patch 6/9

This patch rewrites AR framework

Signed-off-by: Mirko Tietgen <mirko@abunchofthings.net>

Signed-off-by: Jonathan Druart <jonathan.druart@koha-community.org>

Created attachment 40400 [details] [review]
Bug 10963: Simplified creation - KT framework

Patch 7/9

This patch rewrites KT framework

Signed-off-by: Mirko Tietgen <mirko@abunchofthings.net>

Signed-off-by: Jonathan Druart <jonathan.druart@koha-community.org>

Created attachment 40401 [details] [review]
Bug 10963: Simplified creation - IR framework

Patch 8/9

This patch rewrites IR framework

Signed-off-by: Mirko Tietgen <mirko@abunchofthings.net>

Signed-off-by: Jonathan Druart <jonathan.druart@koha-community.org>

Created attachment 40402 [details] [review]
Bug 10963: Simplified creation - SER framework

Patch 9/9

This patch rewrites SER framework

Signed-off-by: Mirko Tietgen <mirko@abunchofthings.net>

Signed-off-by: Jonathan Druart <jonathan.druart@koha-community.org>

Created attachment 40403 [details] [review]
Bug 10963: Revert old hidden values

This patch reverts old hidden values to marc_subfield_structure.

Signed-off-by: Mirko Tietgen <mirko@abunchofthings.net>

Signed-off-by: Jonathan Druart <jonathan.druart@koha-community.org>

Created attachment 40404 [details] [review]
Bug 10963: Simplified creation - FA framework

Patch 1/9

This patch modifies Fast Add framework

Signed-off-by: Mirko Tietgen <mirko@abunchofthings.net>

Signed-off-by: Jonathan Druart <jonathan.druart@koha-community.org>

Patches pushed to master.

Good job Bernardo!