Bug 12477 - We need better ways to manage and update MARC Frameworks
Summary: We need better ways to manage and update MARC Frameworks
Status: RESOLVED WISHLIST
Alias: None
Product: Koha
Classification: Unclassified
Component: MARC Authority data support (show other bugs)
Version: Main
Hardware: All All
: P5 - low normal (vote)
Assignee: Galen Charlton
QA Contact: Testopia
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2014-06-25 03:10 UTC by David Cook
Modified: 2022-12-06 06:33 UTC (History)
8 users (show)

See Also:
Change sponsored?: ---
Patch complexity: ---
Documentation contact:
Documentation submission:
Text to go in the release notes:
Version(s) released in:


Attachments
Field 650 as XML (7.46 KB, text/plain)
2014-06-27 00:20 UTC, Bernardo Gonzalez Kriegel
Details
MARC21 bibliographic XML version (98.26 KB, application/gzip)
2014-07-17 23:06 UTC, Bernardo Gonzalez Kriegel
Details
Loader for XML files (55.61 KB, application/gzip)
2014-07-18 19:50 UTC, Bernardo Gonzalez Kriegel
Details
DBIC-based framework updater (3.23 KB, text/plain)
2014-07-23 05:01 UTC, David Cook
Details

Note You need to log in before you can comment on or make changes to this bug.
Description David Cook 2014-06-25 03:10:45 UTC
We need better ways to manage MARC frameworks for authority and bibliographic records (for new and existing instances).

After a review of updatedatabase.pl, I noted 1 change to auth_tag_structure and 12 changes to auth_subfield_structure. According to http://www.loc.gov/marc/status.html, there have been a lot more than 13 changes in the past 15 years.

I believe that bug 9826 and bug 10488 bring the authorities up to at least 2013 in terms of currency, which is great...for new installs.

However, if you're an existing install, your authority frameworks (and your bibliographic frameworks) are going to be horribly out of date.

As some people have pointed out, we can't necessarily automatically update frameworks, because people may have (ie probably have) customized their frameworks and an automatic update could lead to unexpected results.

That being said, what do we need to update?

1) Missing fields
2) Missing subfields
3) Changes to "repeat" value
4) Maybe changes to labels (for when fields are renamed)

The real "key", in my mind, to custom frameworks is visibility, the ability to add 'local' fields/subfields, and perhaps labels (for libraries who want less technical labels for cataloguing staff). 

If we automatically change "repeat" values transparently and push missing fields/subfields with a visibility of "hidden", then there are substantively no changes to the appearance or experience of frameworks in the editor or the detail/result/list pages. The one unexpected change might be if staff go to customize a framework and see a lot of standard fields that they deleted in the past. That...is a bit more of a puzzle.

However, that puzzle can probably be solved in a few ways:

1) Inform users that standard fields/subfields will be periodically added/updated in the frameworks; or
2) Create a "Refresh" type button next to each framework which puts the updating of frameworks into the hands of library staff

We could even include both these options and wrap the 1st one around a system preference that defaults to ON for new installs, and OFF for existing installs (so as to not change existing behaviour).

In order for this all to work, I think we would need to move away from maintaining default frameworks as SQL files, and instead store them in something like YAML which can be easily loaded into a script that does the work.

The majority of information stored in the frameworks is actually Koha-specific rather than MARC-specific. The only thing the frameworks contain relating to MARC are field codes, subfield codes, labels (i.e. liblibrarian and libopac), and repeat values.

We could actually just store this MARC-specific data in the YAML file, and then build the Default and other Custom frameworks based on a Perl script, which would probably be easier to maintain than the SQL that we maintain now.

Thoughts?
Comment 1 David Cook 2014-06-25 03:21:13 UTC
After reading bug 6007, I think perhaps it would be a good idea to have 'Display only used Tags/Subfields' on by default (provided that it's fixed/accurate).

That could solve the potential problem of unused fields/subfields showing up when library staff are editing their existing frameworks (where they might have deleted a lot of standard fields/subfields).
Comment 2 Bernardo Gonzalez Kriegel 2014-06-27 00:20:16 UTC
Created attachment 29316 [details]
Field 650 as XML

(In reply to David Cook from comment #0)
> We need better ways to manage MARC frameworks for authority and
> bibliographic records (for new and existing instances).
Agreed

> 
> However, that puzzle can probably be solved in a few ways:
> 
> 1) Inform users that standard fields/subfields will be periodically
> added/updated in the frameworks; or
> 2) Create a "Refresh" type button next to each framework which puts the
> updating of frameworks into the hands of library staff

If the field/subfield is missing, adding it as 'hidden' does not affect 
present fw. And you can always preserve liblibrarian/libopac as they are...
except when they change, as in last update (Bug 12435, bib 347$f).

As in other updates I proposed, I never try to update existing fw, for
the problems that you commented.
And there is a bigger problem: translations

> We could actually just store this MARC-specific data in the YAML file, and
> then build the Default and other Custom frameworks based on a Perl script,
> which would probably be easier to maintain than the SQL that we maintain now.

If it's agreed to store fw in other format, I propose we use XML (example attached). And the reason is translations.

We need a way to ease the translation of all configuration and sample data 
we have as SQL. I'm working intermittently on this, now I'm stuck with sample notices, but have worked out most sql files as XML and a script to build sql from them. And unless we propose another way, we need translated sql files at web install time. It's not practical that each language other than english have a lang dir commited to git.

I think that using YAML or XML is the same, in this case.

Talking again about frameworks, Bug 10963 is waiting for a review :)
Comment 3 David Cook 2014-07-16 07:53:01 UTC
Sorry for my delayed response! Competing priorities...

(In reply to Bernardo Gonzalez Kriegel from comment #2)
> If the field/subfield is missing, adding it as 'hidden' does not affect 
> present fw. And you can always preserve liblibrarian/libopac as they are...
> except when they change, as in last update (Bug 12435, bib 347$f).
> 
> As in other updates I proposed, I never try to update existing fw, for
> the problems that you commented.
>

It's a tough one. I'm thinking that there should be some mechanism for updating an existing framework. It might need to be triggered by library staff, so that they understand what is being updated, but there should be something there. Otherwise, if you're on Koha for a long time, you'll get stale frameworks. Since Koha used outdated frameworks for a long time, it's possible (re: probable) that many Koha users are using very out of date frameworks that need to be updated in a user-friendly way.

> And there is a bigger problem: translations
> 

I haven't done anything with translations, but I'm in favour of anything that makes translations easier.

> > We could actually just store this MARC-specific data in the YAML file, and
> > then build the Default and other Custom frameworks based on a Perl script,
> > which would probably be easier to maintain than the SQL that we maintain now.
> 
> If it's agreed to store fw in other format, I propose we use XML (example
> attached). And the reason is translations.
> 

I'm happy to go with XML. I suppose my only reservations are about the brittleness of XML and processing time involved in parsing a huge XML file. I suppose the brittleness shouldn't be a factor...we could validate the XML before committing it and parsing errors are very loud so there probably is no danger there. As for processing time...it's not like installing or updating frameworks would be a frequent operation so lengthy processing time might not matter (if it is even a thing).

> We need a way to ease the translation of all configuration and sample data 
> we have as SQL. I'm working intermittently on this, now I'm stuck with
> sample notices, but have worked out most sql files as XML and a script to
> build sql from them. And unless we propose another way, we need translated
> sql files at web install time. It's not practical that each language other
> than english have a lang dir commited to git.
> 

Now that I think about it... are "liblibrarian" and "libopac" the only fields that need to be translated? 

> Talking again about frameworks, Bug 10963 is waiting for a review :)

It looks good! I'll have to look at it soon.

--

More thoughts that I'm having...

marc_tag_structure:
1) MARC: tagfield, liblibrarian, libopac, repeatable
2) Koha: mandatory, authorised_value, frameworkcode

marc_subfield_structure:
1) MARC: tagfield, tagsubfield, liblibrarian, libopac, repeatable, maxlength
2) Koha: mandatory, kohafield, tab, authorised_value, authtypecode, value_builder, isurl, hidden, frameworkcode, seealso, link, defaultvalue

I'm wondering if it makes sense to maintain 2 separate sources of data.

The first XML file would be the data that comes from the actual MARC specification (denoted by the "MARC:" tag above.

The second XML file would be what Koha should do with the incoming data.

--

Oooh. Brain wave...we could use XSLT! We'd have a XML file containing the MARC specification data. Then we pass that through an XSLT which adds the relevant "Koha" data.

Think about it...in most cases those "Koha" fields are "null" or "". The most important fields are probably "tab" and "hidden". If we use defaults for most cases, but define exceptions in the XSLT on a field/subfield and framework basis...that would be much easier to maintain!

I don't know if that paragraph makes sense to anyone else...but I like this idea. It's easier to maintain exceptions rather than a whole comprehensive list, which can be dynamically generated.

Bernardo, if you could post an up-to-date XML file with that MARC data I mentioned above, I would be willing to take a look at creating a XSLT that adds the relevant Koha data and produces a final "Koha MARC framework" XML document in the end.

From there...we could use scripts to transform XML to SQL or to parse the XML and do the database updates via Perl. 

Anyway...just some thoughts for now. Have to run!
Comment 4 Bernardo Gonzalez Kriegel 2014-07-17 23:06:53 UTC
Created attachment 29810 [details]
MARC21 bibliographic XML version

Sample MARC21 as XML. There are a file for tags and another for subtags.
Also there are complete and simple ones. Complete have all attributes, simple ones with only marc specific values.
Comment 5 Bernardo Gonzalez Kriegel 2014-07-17 23:44:22 UTC
(In reply to David Cook from comment #3)
> Sorry for my delayed response! Competing priorities...
No problem :) This task will take some time

> I'm happy to go with XML. I suppose my only reservations are about the
> brittleness of XML and processing time involved in parsing a huge XML file.
> I suppose the brittleness shouldn't be a factor...we could validate the XML
> before committing it and parsing errors are very loud so there probably is
> no danger there. As for processing time...it's not like installing or
> updating frameworks would be a frequent operation so lengthy processing time
> might not matter (if it is even a thing).

I used XML::Simple to load and build sql files from the attached examples. It does not take to much time, and anyway is something to use at install time or when updating frameworks, not frequent tasks :)

> 
> Now that I think about it... are "liblibrarian" and "libopac" the only
> fields that need to be translated? 

Yes, and in EN and ES they are the same text. I wonder if there are libraries using different labels for opac/staff
In french UNIMARC frameworks I found that libopac is shortened or empty in most cases.

> marc_tag_structure:
> 1) MARC: tagfield, liblibrarian, libopac, repeatable
> 2) Koha: mandatory, authorised_value, frameworkcode

In this case authorised_value... is not used, or can't be used. FW editor allows to set this to an authorised value, supposedly to list allowed values for indicators, but this is not used on editor (there is another bug that mentions this) 

> marc_subfield_structure:
> 1) MARC: tagfield, tagsubfield, liblibrarian, libopac, repeatable, maxlength
> 2) Koha: mandatory, kohafield, tab, authorised_value, authtypecode,
> value_builder, isurl, hidden, frameworkcode, seealso, link, defaultvalue

In this case seealso is not used anymore (or that is implied by http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=6007#c3)

> 
> Oooh. Brain wave...we could use XSLT! We'd have a XML file containing the
> MARC specification data. Then we pass that through an XSLT which adds the
> relevant "Koha" data.
> 
> Think about it...in most cases those "Koha" fields are "null" or "". The
> most important fields are probably "tab" and "hidden". If we use defaults
> for most cases, but define exceptions in the XSLT on a field/subfield and
> framework basis...that would be much easier to maintain!
> 
> I don't know if that paragraph makes sense to anyone else...but I like this
> idea. It's easier to maintain exceptions rather than a whole comprehensive
> list, which can be dynamically generated.
> 
> Bernardo, if you could post an up-to-date XML file with that MARC data I
> mentioned above, I would be willing to take a look at creating a XSLT that
> adds the relevant Koha data and produces a final "Koha MARC framework" XML
> document in the end.
> 

On the attached file there are XMLs for tags and subtags, with/without Koha proper fields. They are at least well formed (xmllint --noout file.xml)

There's another thing to take into account: when MySQL 5.6 arrives to new debian/ubuntu versions, we'll have a problem. 5.6 client complains a lot to be used on CLI with password as argument, so Koha installer fails miserably because it uses 'mysql' command to load mandatory/optional values in database.
Another reason to find other way.
Comment 6 Bernardo Gonzalez Kriegel 2014-07-18 19:50:36 UTC
Created attachment 29869 [details]
Loader for XML files

This new attach has a variant of XML files, with one element to represent data types, and a simple script to load the values into Koha database.

I tried $dbh->do() for each value, or a prepare() and then execute(), but is too slow. It builds a single string with all values to insert, similar as the SQL files for frameworks.

In my tests the script takes a little more than three times as long as mysql client to load default framework (tags/subtags), but is a difference between ~0.7s and ~2.5s, perhaps acceptable...
Comment 7 David Cook 2014-07-21 07:49:00 UTC
(In reply to Bernardo Gonzalez Kriegel from comment #6)
> Created attachment 29869 [details]
> Loader for XML files
> 
> This new attach has a variant of XML files, with one element to represent
> data types, and a simple script to load the values into Koha database.
> 
> I tried $dbh->do() for each value, or a prepare() and then execute(), but is
> too slow. It builds a single string with all values to insert, similar as
> the SQL files for frameworks.
> 
> In my tests the script takes a little more than three times as long as mysql
> client to load default framework (tags/subtags), but is a difference between
> ~0.7s and ~2.5s, perhaps acceptable...

As you said before, the speed shouldn't matter too much if it's only being run during initial install and then during rare subsequent updates.

I've been playing today with your XML files, and have been working on a script for updating existing frameworks using Koha::Database/DBIx::Class. 

The "find_or_new" method is quite useful in that case, as it'll insert new rows and it'll give you a chance to do further actions for rows that already exist (using the primary key as a reference point). 

At the moment, I have it set up to add missing fields/subfields, and update the "repeatable" column for existing fields if the XML values are different. 

I'm going to add an option for updating "libopac" and "liblibrarian" as well. I figure some libraries use custom text so they might not necessarily want to override it according to the MARC specification. 

Anyway...

--

I can put together a demo DBIC loader for the default framework XML, but I'm wondering now how we want to handle installing the other sample frameworks. I took a look at bug 10963 and it looks really good. I like the idea you're going with.

I'm thinking maybe something like this: http://search.cpan.org/~ribasushi/DBIx-Class-0.08270/lib/DBIx/Class/ResultSet.pm#Multiple_queries

 my $framework = $schema->resultset('MarcTagStructure')->search({
    frameworkcode => '',
  });

then...

$framework->create({ frameworkcode => 'SR' });
$framework->create({ frameworkcode => 'BKS' });

Then probably some updates to change the hidden values. I'm not sure where we would want to store the data about which subfields to hide in which framework though. I suppose we could hard-code it into the script, but it might be nicer to keep that information in a config file.

Not 100% sure what the best way to update the hidden column will be...but I think that might be the way to do it.
Comment 8 David Cook 2014-07-22 01:22:42 UTC
Just having a thought about "hidden" values...

These are the "hidden" values:

-9 => Future use
-8 => Flag
-7 => OPAC !Intranet !Editor Collapsed
-6 => OPAC Intranet !Editor !Collapsed
-5 => OPAC Intranet !Editor Collapsed
-4 => OPAC !Intranet !Editor !Collapsed
-3 => OPAC !Intranet Editor Collapsed
-2 => OPAC !Intranet Editor !Collapsed
-1 => OPAC Intranet Editor Collapsed
0 => OPAC Intranet Editor !Collapsed
1 => !OPAC Intranet Editor Collapsed
2 => !OPAC !Intranet Editor !Collapsed
3 => !OPAC !Intranet Editor Collapsed
4 => !OPAC Intranet Editor !Collapsed
5 => !OPAC !Intranet !Editor Collapsed
6 => !OPAC Intranet !Editor !Collapsed
7 => !OPAC Intranet !Editor Collapsed
8 => !OPAC !Intranet !Editor !Collapsed
9 => Future use • URL : if checked, the subfield is an URL, and can be clicked

These are their frequencies in "marc21_subtag.xml".

"hidden="-9" (0 hits)
"hidden="-8" (0 hits)
"hidden="-7" (0 hits)
"hidden="-6" (3037 hits)
"hidden="-5" (59 hits)
"hidden="-4" (0 hits)
"hidden="-3" (0 hits)
"hidden="-2" (0 hits)
"hidden="-1" (240 hits)
"hidden="9" (0 hits)
"hidden="8" (0 hits)
"hidden="7" (3 hits)
"hidden="6" (4 hits)
"hidden="5" (379 hits)
"hidden="4" (1 hit)
"hidden="3" (0 hits)
"hidden="2" (0 hits)
"hidden="1" (11 hits)
"hidden="0" (225 hits)

Some of these don't make all that much sense.

For instance, -5 is functionally equivalent to -6. They're both in the OPAC and Intranet, and neither is in the Editor. The Collapsed quality really only applies to the Editor...so we may as well make the subfields with -5 be -6 by default.

6 and 7 are also functionally equivalent...although they're a bit bizarre. I don't know why you'd ever need something to show in the Intranet but not in the OPAC or the Editor.

If a field doesn't appear in the Editor, and it gets re-saved...any fields not appearing in the Editor will be destroyed.

I figure...if it doesn't appear in the Editor, it probably _doesn't_ need to appear anywhere else.

Conversely, it _does_ make sense that fields that appear in the Editor might not show up in the OPAC or the Intranet (as they might be system values or just not relevant for most views).

Following that logic, I find -6 to be a bit silly. I'd prefer 5 or 8, which are functionally equivalent.

Of course, such distinctions get a bit...pedantic, so I rather just go with 1 "hidden" value for a function. Perhaps "-6" is the best one to use, since it's the most common for fields that we want to hide.

Anyway, these are just some thoughts... when it comes to updating frameworks, I'm going to use -6 for the sake of consistency.
Comment 9 Jacek Ablewicz 2014-07-22 11:18:49 UTC
(In reply to David Cook from comment #8)

> If a field doesn't appear in the Editor, and it gets re-saved...any fields
> not appearing in the Editor will be destroyed.

Does it really work that way for biblos MARC frameworks?? If so, that would be a _major_ disaster IMO.

I was under the impression that if the certain subfields are set up as "hidden in editor", they still get saved (not getting destroyed on re-save). Also, at least in the manner of speaking, they do appear in editor ;), their contents are just not visible and not editable (AFAIRC authority editor uses "display: none" for that purpose, I'm not sure about biblio editor).

> I figure...if it doesn't appear in the Editor, it probably _doesn't_ need to
> appear anywhere else.

There are some scenarios when we e.g. may want some fields/subfields to be: preserved in the MARC record (control numbers, some linking data for records imported from union catalogues etc.), visible in staff client, but not directly editable by the librarians.

Possibility to define (perhaps even greatly) simplified frameworks, intended for some specific editing tasks (= with a lot of existing record fields/subfields hidden on edit) might often come in handy as well.
Comment 10 Bernardo Gonzalez Kriegel 2014-07-22 19:42:35 UTC
(In reply to Jacek Ablewicz from comment #9)
> (In reply to David Cook from comment #8)
> 
> > If a field doesn't appear in the Editor, and it gets re-saved...any fields
> > not appearing in the Editor will be destroyed.
> 
> Does it really work that way for biblos MARC frameworks?? If so, that would
> be a _major_ disaster IMO.
> 

I created a new fw, set hidden = -6 for all subtags. A new record using this fw will only show those tags/subtags that are mandatory.
Editing an existing record will show any tag/subtag that have a value, even hidden ones, and on saving no information is loose.

I think that default and sample (or simple) frameworks needs only 3 hidden values: visible (0), editable but not visible (2), and hidden (5) [visibility on opac/staff]. (5) could be used for obsoleted/deprecated tags/subtags (e.g. 440), but again you can't hide something in the editor with a value

(-6) is the value I was told to use when adding new tags/subtags, they do not display on editor (for new) and produce no functional change, but imported record can show values [opac/staff].

Whichever we use, my thinking is that we need to use a few hidden values as possible, and default FW needs to show as many tags as possible

Then sample FW could be built by copying default, and _hiding_ (editor) all but a selected subset of tags. Which subset... I don't care, but it need a certain rationale behind. Not an universal agreement though.

But if they have all tags/subs then no information will be loose.
That's what we need to ensure
Comment 11 David Cook 2014-07-23 04:15:54 UTC
(In reply to Bernardo Gonzalez Kriegel from comment #10)
> I created a new fw, set hidden = -6 for all subtags. A new record using this
> fw will only show those tags/subtags that are mandatory.
> Editing an existing record will show any tag/subtag that have a value, even
> hidden ones, and on saving no information is loose.

Ah, my mistake! That's good! I'm glad that I was wrong! 

It appears that this is also the case when using Z39.50. I think the problem I may have encountered in the past was losing data when importing a record that contained fields/subfields that didn't exist in that framework.

*checks with an example*

Aha! Yes! That's what it is! It makes sense...if the field/subfield doesn't exist in the framework, it has no way of including it in the record. 

I suppose this might be both a pro and a con...

When copy-cataloguing, you might only want to import a certain subset of fields, so you'd want to ignore fields not defined in that framework.

On the other hand, if you're cataloguing RDA records and you're not familiar with the extra fields and subfields needed for RDA, you'll lose data that you might want to keep when importing. That said, I think the onus needs to be on librarians to properly configure their frameworks for that.

I don't know if this distinction is necessarily apparent in Koha. I imagine it's possible that a librarian could create a new framework with a subset of fields, and lose data when importing records because they forgot a field/subfield. 

But...you could argue the onus is still on the person configuring the framework. If they didn't include the field/subfield, they have no one but themselves to blame for lost data.

> I think that default and sample (or simple) frameworks needs only 3 hidden
> values: visible (0), editable but not visible (2), and hidden (5)
> [visibility on opac/staff]. (5) could be used for obsoleted/deprecated
> tags/subtags (e.g. 440), but again you can't hide something in the editor
> with a value
> 
> (-6) is the value I was told to use when adding new tags/subtags, they do
> not display on editor (for new) and produce no functional change, but
> imported record can show values [opac/staff].

That sounds reasonable to me. 


> Whichever we use, my thinking is that we need to use a few hidden values as
> possible, and default FW needs to show as many tags as possible

Agreed.

> Then sample FW could be built by copying default, and _hiding_ (editor) all
> but a selected subset of tags. Which subset... I don't care, but it need a
> certain rationale behind. Not an universal agreement though.

That also makes sense. I don't know which subset either. While there are cataloguing standards, I find practice is often highly localized, so they'll need to be re-configured by end-users anyway. It might be useful to get input from cataloguers that work with a range of materials for this one. I mostly catalogued books, so I would say Leader,005,008,100(a),245(a,b,c),260(a,b,c) (maybe also 264(a,b,c) as it's used in RDA),300(a),336(a,2),337(a,2),338(a,2),490(a,v,x),500(a),505(a,r,t),520(a,b),700(a,t,d,f).

I don't have a lot of experience with serials but it should probably include 773(a,t,d,g,w).

Other frameworks would probably benefit from cataloguers with more specialized knowledge and experience.

> But if they have all tags/subs then no information will be loose.
> That's what we need to ensure

Agreed. In that case, what do you think about the Fast Add Framework? It only includes a small collection of fields/subfields. Wouldn't it make more sense to include all the fields, and only show the subset that it currently shows?

While I'm somewhat skeptical of the utility of many of the sample frameworks, I think the Fast Add Framework is quite useful. Admittedly, I don't know if it needs to have all the fields, as I doubt one would ever really be importing records into it. I suppose it would be good to have all the fields in case you accidentally switched to the Fast Add Framework and saved a record. That would lose data...
Comment 12 David Cook 2014-07-23 04:36:31 UTC
As for using DBIx::Class, it's literally this simple:

<code>

use Koha::Database;
use XML::Simple;

my $schema = Koha::Database->new()->schema();
my $SimpleXML = new XML::Simple;

my $field_file = "../marc21_tag.xml";
my $canonical_field_structure = '';

if ( -e $field_file){
    my $canonical_field_structure = $SimpleXML->XMLin($field_file, KeyAttr => []);
    foreach my $structure (@{$canonical_field_structure->{marc_tag_structure}}){
        $schema->resultset("MarcTagStructure")->find_or_create($structure);
    }
}

</code>

Of course, this is just an example. I'd prefer to use XML::LibXML rather than XML::Simple (especially as XML::Simple itself states that new code should avoid using XML::Simple), and set up the logical structure a bit differently, but yeah...that's it.
Comment 13 David Cook 2014-07-23 05:01:26 UTC
Created attachment 29961 [details]
DBIC-based framework updater

Here is some more complex code that I've devised for updating frameworks.

You give it an arrayref of hashrefs, a DBIC resultset, and an arrayref of framework codes.

The script will check to see if a row matching the hashref (using the primary key specified in the resultset) exists for each framework. 

If it exists, it runs an optional "update" callback. 

If it doesn't exist, it runs an optional "insert" callback (or just inserts it into the database if no callback has been specified).

Prior to the database search, the "canon_callback" can be used to modify the hashref for each frameworkcode. In my use case, if I'm loading marc_subfield_structure, I want to change the hidden value to -6 as I'm updating existing frameworks. Of course, if we were loading marc_tag_structure or were installing a framework for the first time, we wouldn't want to do that.

I also use the insert/update callbacks to build messages that I sent to the template UI so that the user knows what has been updated.

It's not very fast, but DBIC itself says "DBIx::Class is not built for speed, it's built for convenience and ease of use". It certainly is convenient and easy to use :).

I have debated the utility of this class a bit. I figure it could be handy when we're installing the sample frameworks, as we could use the "canon_callback" to change the "hidden" value on a per framework and subfield basis. We wouldn't even need the insert/update callbacks.



[Note: The DBIx::Class docs do mention that using find_or_create (and thus find_or_new or update_or_create) is subject to a race condition and that it should be run within a transaction. 

Mind you, lots of what we do in Koha could be subject to a race condition, so I don't know how important it is to get this right at the moment.

In any case, there seems to be a few ways of using transactions with DBIC (http://search.cpan.org/dist/DBIx-Class/lib/DBIx/Class/Manual/Cookbook.pod#TRANSACTIONS). 

It's not really a big deal when initially installing frameworks, as there shouldn't be anyone to race against, but worth mentioning for updates and in general.]
Comment 14 David Cook 2014-08-01 01:03:08 UTC
I'm thinking more about how to keep frameworks current...

In theory, all MARC Bibliographic Frameworks should include all MARC Bibliographic fields and subfields. 

I can only think of two scenarios where this theory would be false:

1) You've bent the MARC Bibliographic Framework in an attempt to support a different sort of metadata format. 

[Note: I don't think this is an overly valid reason not to auto-update frameworks.]

2) You want a framework that will discard certain fields when importing records. For instance, you might never ever want to keep the 035 for incoming records (although it's a nice bit of data to keep). So you delete the 035 from the framework.

[Note: This scenario is more interesting. By deleting certain fields, you could potentially increase the efficiency of your cataloguing workflow. That is, you would reduce the amount of time spent deleting unwanted data when importing records.

Yet... I can't help but feel that the "xkcd: Workflow" comic (http://xkcd.com/1172/) is relevant here.

I suppose a way to mitigate this might be to add an "auto-update" flag to the frameworks. You could turn it off for frameworks whose structure you don't want to change.

Yet...that doesn't make sense either...because there will come a time where you need to add fields/subfields (such as the RDA fields/subfields).

I suppose that leads us back to doing manual updates... but that can be painful for a number of reasons. One of those being that the full list of missing fields/subfields will show up each time, so you need to hand-pick which fields/subfields you want to add.

Plus, if you're doing manual updates, you need a prompt to do the updates. I suppose this could be done with a staff client news item...but then when to do framework update checks...]

---

I've been thinking more about visibility. Regardless of what metadata format we use, we'll run into this problem again and again.

Instead of using one column ("hidden"), it might make sense to use three columns ("internal","external","editor"). 

"internal" and "external" would control visibility in the staff client and the OPAC (maybe OAI as well...hence my use of "external" as a label). 

"editor" would have more options: "used","used - collapsed", "hidden", "unused". In this case, "used" shows up uncollapsed, "used - collapsed" shows up as a collapsed field, "hidden" doesn't show up unless you're importing records with a value in that subfield, "unused" means that there can never be a value in this field.

Alternatively... we keep the current "hidden" column and add a "used" column which defaults to "on". If someone wants a framework that will ignore certain fields when importing, they can turn "used" to "off". 

In support of this mechanism, it might make sense to remove the ability for people to delete fields/subfields from frameworks (unless they're local fields/subfields that they've added). Of course, that's easy to circumvent by just importing a framework that doesn't have certain fields/subfields (until the next auto-update comes around which re-populates those missing fields/subfields).

---

I don't know... I've looked at this a bit in DSpace as well, and I don't really like the way they do it either. In DSpace, there is a metadata registry, then you define input forms (which would be analogous to our frameworks in a way). It gets incredibly unwieldy after the first form or two, especially if you have a lot of fields per form.

In the past, I've mostly done original cataloguing, so I'm not 100% sure what other ILSes/LMSes do for templates with copy cataloguing. I recall cataloguers using Horizon would have to delete unwanted data when importing records. I think the function of the templates were more so to provide a minimal set of fields/subfields that you should fill out. (You then had the option to add more fields/subfields manually as necessary.)

---
Comment 15 David Cook 2014-08-01 01:13:45 UTC
tl;dr

The top priority should be making sure that no unwanted data loss occurs.

One solution might be to make sure that incoming fields/subfields are saved, even if they don't appear in a framework.

(I suppose this solution has flaws in terms of determining labels and repeatability... I suppose you could say "unknown field/subfield" and default to being repeatable in the editor.)

(For cataloguers who count on the workflow of fields/subfields not being saved if they're not in the framework, we could have a "strict" flag at the framework level which could control whether or not "unknown subfields/fields" are saved.)

This solution would actually negate some of the need to update frameworks. Updates to labels and whether it's "repeatable" would be the only thing needed really, as the frameworks would be acting more as templates than as strict frameworks. 

Thoughts?
Comment 16 David Cook 2014-08-05 05:30:04 UTC
Sadly...it appears that fixing the cataloguing editor, so that it doesn't discard fields/subfields not present in the framework, would require re-writing at least the "build_tabs" sub in addbiblio.pl (ie the cataloguing editor).

Since that there is a core bit of Koha functionality, I'm not planning to touch it.

Half-tempted to turn "build_tabs" into a syspref controlled wrapper that can switch between the current "build_tabs" code and a new sub.

On one hand, that would keep the traditional interface untouched, so no regressions would happen.

On the other hand, it would mean maintaining two slightly different versions of the same editor, and then trying to merge those together over time. Unfortunately, we already have that problem with the QueryParser :/. When a syspref or fallbacks are in place, there's no incentive to improve the new development...
Comment 17 emjhet 2018-05-30 21:52:00 UTC
Re: David Cook: "I'm thinking that there should be some mechanism for updating an existing framework."

Koha requires that the "Default" Framework must remain, unedited and available, in Koha. This is the area to do all MARC21 updates, hopefully with each update of Koha. 

As David Cook noted, a library may set up a unique Framework that does not have all of the MARC fields and subfields when such a library is specifically attempting to keep things simplified and efficient. 

I am the only cataloger among seven people who add bibliographic records to our system. I ran reports to see which MARC fields were used, checked the ones that were used only a handful of times (to see if they were used correctly), and changed all of our records back to the Koha Default Framework. I deleted all the other frameworks (as none of their fixed fields were coded correctly) and painstakingly created a "new" default framework, deleting fields and subfields that are obsolete or not relevant to the work that we do. 

The most important thing to me, regarding MARC updates, is that I do not have to start from scratch using the Koha Default Framework. As long as the information is added to the Koha Default Framework, I can manually look it up and manually add it (correctly) to any of my unique Frameworks.
Comment 18 Héctor Eduardo Castro Avalos 2018-10-26 05:30:55 UTC
Hi everyone

This is a problematic subject, keep-up to date the frameworks. We fix this updating SQL manually. Right now I encourage all my co-worker to start cataloging in Rancor, rather than frameworks. You'll see that this is a better option with the unique inconvenience that you cannot obtain stadistics with frameworks anymore.

Of course Rancor got some problems (bugs) yet, but if you learn how to avoid them, you'll see a good option for cataloging process. And after all it's much faster cataloging there than the old frameworks.

We used frameworks with students and inexperienced librarians.

Cataloger thought.
Comment 19 Katrin Fischer 2018-10-28 19:03:59 UTC
(In reply to Héctor Eduardo Castro Avalos from comment #18)
> Hi everyone
> 
> This is a problematic subject, keep-up to date the frameworks. We fix this
> updating SQL manually. Right now I encourage all my co-worker to start
> cataloging in Rancor, rather than frameworks. You'll see that this is a
> better option with the unique inconvenience that you cannot obtain
> stadistics with frameworks anymore.
> 
> Of course Rancor got some problems (bugs) yet, but if you learn how to avoid
> them, you'll see a good option for cataloging process. And after all it's
> much faster cataloging there than the old frameworks.
> 
> We used frameworks with students and inexperienced librarians.
> 
> Cataloger thought.

Hector, just a note: Even when using Rancor, always make sure that you define fields/subfields used in the default framework. It will cause you data loss if the record is opened in the old editor otherwise.
Comment 20 Katrin Fischer 2019-12-12 07:30:00 UTC
Tbh, I have not read all comments to the end, but this appears to be a discussion on how to:

- better store our frameworks
- update the existing frameworks in an installation to changes in the standard.

While both are great goals, I don't think this warrants a 'major'. Might even be categorized as enh? Lowering severity to normal for now.
Comment 21 David Cook 2022-12-06 06:33:30 UTC
I don't think this is going to be anyone's priority so going to close this one especially given its age