This bug is mentioned in:
Pull request: Bug 5528 analytical records http://lists.koha-community.org/pipermail/koha-patches/2011-January/013521.html
Pull request: Bug 5528 analytical records http://lists.koha-community.org/pipermail/koha-patches/2011-January/013524.html
Pull request: Bug 5528 analytical records http://lists.koha-community.org/pipermail/koha-patches/2011-January/013527.html

This bug is mentioned in:
Pull request: Bug 5528 - analytical records http://lists.koha-community.org/pipermail/koha-patches/2011-March/014346.html

This is rebased on master, UNIMARC support to be added.

git://kohadev.osslabs.biz/analy_3.6.git

*** Bug 6831 has been marked as a duplicate of this bug. ***

I have tested the latest code in the OSS Labs repository, and I can confirm that it works, and I cannot find any regressions. My one objection previously was the lack of framework installation in updatedatabase.pl, and that has been added in this iteration. I am prepared to sign off on this branch. Unfortunately, my git-fu is not strong enough to do the actual signing off step (and I'm rarely at my development computer, so I can't wait until Chris is around to walk me through the procedure and be confident that I won't miss the cut off for 3.6), so please accept this comment as my official sign-off.

Hi,
I have insert the last modifications to support UNIMARC.
For me, with this patch, the UNIMARC support is present

I have pulled down the analy-3.6 branch from the git://kohadev.osslabs.biz/analy_3.6.git branch, and squashed it into a commit based on today's copy of master.  Here is the list of issues I've discovered so far:

1. merge conflicts on updatedatabase.pl and kohaversion.pl.  The patches set versioning themselves, instead of waiting on DB rev assignment by RM.  Easy fix
2. Appears to be based on 3.4.1 instead of master... a 3.04.001.000 DB rev exists and needs removing.
3. Minor merge issue on addbiblio.tt with bugfix 6576; easy fix
4. Includes a full set of German templates in /koha-tt; will need to bulk remove those from the commit, since translation is handled by other mechanisms.
5. Legacy cataloguing/linkitem.tmpl file needs removing (there is a .tt version)

All the above are minor and easy to clean.  But I've got some bigger concerns:

This development manually pulls in MARC21 and UNIMARC fields/subfields throughout it's code; there is no utilisation of the Frameworks to determine field semantics.  One side effect:  we wind up ignoring NORMARC completely!  Another is that we're limited to a single relationship type (Host Item Entry) and it's forcibly tied to the existing MARC representation.

Not using the Frameworks seems to stem from not having a database table with which to connect the frameworks with.  Creating such a table is a little more complex, since multiple MARC fields could map into the same table structure, just with a different 'type' value.  Perhaps specifying it as "table:type.field", and modifying the code to strip the ":type" part out (one line in C4::Biblio::TransformKohaToMarcOneField might do it).

Ian, 

Could we not use the Keyword->MARC maybe, this would not leave out NORMARC, (or any future MARC). It would mean the link would be the same for all frameworks tho.

But may be enough to pass QA, with a further enhancement to get it using frameworks to follow?

Hi Chris and Ian,

The last develop that I have done to insert Unimarc is not present in git://kohadev.osslabs.biz/analy_3.6.git.
Savitra doesn't add this patch in the branch

You find the patch in attach 5494 [Last modifications for UNIMARC]
Few lines but essential for Unimarc.

shouldn't this be set "failed QA" for now ?

Keyword to MARC might work... but it's only really respected in Normal view, not XSLT view.

The most direct analogue for this kind of linkage in a Framework is Thesaurus, which links biblios to authorities of a type.  Analytics are just links to biblios of a type, so the structure would be very similar.

I've spoken with Magnus, and he informs me that NORMARC uses the same standard here as MARC21, so we can easily group those two together.  That, as well as integrating Zeno's patch, will be required before we can commit this.  I'm marking this patch as Failed QA for right now, to reflect this.

I think it would be safe to include the hard-coded field/subfield values for right now, provided we can verify that there are no regressions, and folks understand that to use this feature, they HAVE to use the right MARC.  Increasing the flexibility of the feature can come later.

I had no time to look at this, although I had planned to do it for a longer time. Now the time runs out and I am not sure I will be able to. 
As there is already one implementation for analytics in place - using 773 and 001 fields for linking and completely independant from items - it would be important to us that both features can work as alternatives.

Katrin,

This feature works *only* with items, so it will not conflict with existing analytics code.

Regards,
Jared

Created attachment 5798 [details] [review]
Squashed patch incorporating QA changes

Attached is a patch which incorporates changes 1-5 and the other changes required by Ian for QA.
1. Chris resolved the merge conflicts when he squashed the branch for me
2. Chris fixed this at the same time.
3. This one too.
4. Thanks to Chris for this as well.
5. I removed it.

In every place where there is MARC flavor-specific code for this patch, NORMARC is being treated the same as MARC21. While I was at it, I fixed a few copy-and-paste issues where the only difference between MARC21/NORMARC and UNIMARC was the hardcoded tag number. Although the analytics tags are still hardcoded, the item tags are not, which is a small step in the direction of framework-based configuration.

Regarding the MARC-keyword mappings, this would be a new use for them, nothing to do with display. Adding analytics as a thesaurus-like option makes sense to me, however.

Commit message follows:


Bug 5528: Analytic records support

Display links to parent biblios, show linked items in holdings, allow holds on
linked items. This uses MARC to maintain relationships.

Sponsored by the Mississippi Department of Archives and History and RapidRadio
Solution. Originally developed by Savitra Sirohi and Amit Gupta at OSSLabs, with
UNIMARC support added by Zeno Tajoli. Commits squashed and merge conflicts
resolved by Chris Cormack from Catalyst. Respect for NORMARC and some small
framework portability fixes made by Jared Camins-Esakov of C & P Bibliography
Services.

IMPORTANT NOTE: A bug in the 773 coding for MARC21 was corrected from the
original OSS Labs code. The 773s generated by the pre-release code did not have
the first indicator set to '0', which means that they were not supposed to
display. Going forward, the first indicator will be set correctly, but existing
records created with this code will no longer appear (they appeared before only
due to another bug). To correct this, you could globally (or, to make sure you
only modify records created with the Analytics tool, for records with 773$0)
change the first indicator of the 773 from blank to '0'.

== Background ==
An analytic record for an item is a more detailed, monographic biblio for an
item attached to a serial record .  This is often used for special issues of a
journal that are released as books on their own (assigned an ISBN, as well as an
ISSN/volume/issue).  It is important for researchers to be able to search for
these items both as issues of the serial, and as monographs.  It is equally
important for the library to not have duplicate item records for the item in
question to have to keep synchronized.

== Establishing relationships ==
Analytical records are connected to items belonging to parent or host
bibliographic records. This can be accomplished by:
* From an analytical bibliographic record linking to an host item by providing
  the item barcode as input
* From a host item by using option "analyze", this creates a new empty
  bibliographic record with field 773 (MARC21) populated
* Running a new CLI script that establishes a relationship between the
  analytical record and the host item identified by the barcode in the
  analytical record's 773$o (MARC21)

== Connecting Records ==
The relationships are maintained in the MARC records, we have not used database
tables at all.

== MARC Representation ==
In MARC21/NORMARC we have used:
* 773$9 to store the Koha item number of the host item
* 773$0 to store the Koha biblio number of the host bibliographic record

The above fields are used to display the relationships in various screens in the
OPAC and the staff interface. Additionally, when populating field 773 with host
item's details, we have used following MARC 21 mapping:
* 'a' <= 100/110/111 $a (author main)
* 'b' <= 250$a (edition)
* 'd' <= 260$a, 260$b, 260$c (place, publisher, year)
* 'o' <= barcode
* 't' <= 245$a (title)
* 'w' <= (003)001 --> if no 001 is available, we can populate biblionumber
* 'x' <= 022$a (issn)
* 'z' <= 020$a (isbn)

In UNIMARC, this code uses:
* 461$9 to store the Koha item number of the host item
* 461$0 to store the Koha biblio number of the host bibliographic record

When populating field 461 in UNIMARC, the following mapping is used:
* 't' <= 200$a (title)

== Treatment of Holds ==
A key requirement was to allow holds to be placed on host items from the
analytical record. We have accomplished this by allowing holds on specific
copies only. Biblio level holds are not allowed. This ensures that holds are
placed on specific items that are relevant to the analytical record.

== Deleting host items with linked analytical records ==
As we have not used database tables to maintain relationships, we had to use
search to find out if any linked analytical records are present. If 1 or more
analytical are present, we do not allow deletion of items. This is similar to
what we see when we try to delete authority records.

== Importing analytical records ==
Analytical records can be imported using bulkmarcimport or the GUI tools. The
new CLI script can be executed after the import to establish relationships with
host items. The script will establish relationships using the host item's
barcode, the barcode must be present in 773$o of the analytical record.

== What if there are two or more copies of the host item? ==
The current design will require that there be two host (773) fields, one for
each copy.

== What if there is no barcode available for the host item? ==
It is still possible to establish a relationship, by populating 773$9 with the
host's item number. However the CLI script uses barcode in 773$o to establish
relationships so it won't work where barcodes are unavailable. Also from an
analytical record, it is possible to establish a relationship to a host item by
providing the barcode as input, this option will not be available as well.

Commits that added the following features were squashed by Chris Cormack (this
is not a list of every commit):
* Display links to host records from biblio detail screens
* Support for UNIMARC, respecting the system preference 'marcflavor'
* Support holds from the OPAC
* Ability to link to items belong to host records from a analytical record
* Display items belonging to host records in the moredetail page
* Ability to edit items belonging to host records, also ability to delink from
  them
* Move get host items code into a C4 routine, also calling the new routine in
  related perl scripts
* Move host field population to a C4 routine, all changes in pl files to call
  new routine
* Allow only specific copy holds for analytical records plus changes to use new
  C4 routines
* Support for holds on items linked via host records
* Storing bibnumber and itemnumber in subfields 0 and 9, plus other mapping
  changes
* New command line script that establishes relationships between analytical
  records and host items and bibs. The script looks for host field (MARC21 773)
  in records, and based on barcode in subfield 'o' populates host bibnumber in
  subfield '0' and host itemnumber in subfield '9'. The script can be run after
  an import of analytical records, it can also be run in the crontab to maintain
  the relationships
* Ability to create analytical records from items, to view linked analytics, and
  prevent deletion of items that have linked analytics
* New template for catalogue/detail.pl (NOTE: not a new template file, just a
  new way of displaying analytics), template displays linked analytics and
  allows creation of analytical records
* New zebra index for item number in host fields. This index will be used to
  display links to analytical records from host records
* Display title of host record instead of the phrase host record
* Using detail.tmpl for analytics tab instead of a new template file
* Improved qualification info prepration in Prephostmarcfield
* Check for linked analytics before deleting item
* Display link to host record and more meaningful anchor text for edit item link
* Analytical record: Unimarc index in record.abs and help in
  create_analytical_rel.pl
* Adding a sys pref that controls display of options to create analytical
  relationships
* Add host entry in XSLT stylesheet in staff item detail
* Added host record support to OPAC detail XSLT
* Adding 773$0 and 773$9 to all frameworks
* Adding 773 subfields 0 and 9 to default marc framework via updatedatabase.pl
* Display create analytics and used in links in catalog detail
* Fixed problem where analytical records not showing in OPAC search results
  because GetMarcBiblio now needs a flag to add item records
* Fixed problem where analytics count was set to 1 for all records, not just
  those with analytics
* Fixed catalogue detail page not to show analytics counts if count is 0

Conflicts:
	installer/data/mysql/updatedatabase.pl
	koha-tmpl/intranet-tmpl/prog/en/modules/cataloguing/addbiblio.tt
	kohaversion.pl

Signed-off-by: Jared Camins-Esakov <jcamins@cpbibliography.com>

Thank you, Jared, for incorporating the necessary changes and attaching as a patch.  The detailed commit message is MUCH appreciated, as it provides clear documentation on what this feature does and how.

Reviewing the code, I see that things are rewritten very modularly, making minimal changes to existing functionality (and in those cases usually wrapping the changes in conditional checks).  I think we're pretty well safe from regressions.

I'd like to see the database table connections, and the 773 links be bibliolevel instead of item level, but those are just my preferences, and do not actually affect the functionality of this development.

Marking as Passed QA.  Chris, would it be possible for you to add in the Author credits in Git so that everyone who has worked on this squashed patch can get the proper credit?

Ian and Jared,

A patch to the manual explaining this (if you think it's helpful) would be awesome :)

Nicole

Pushed please test thoroughly

A bit late but, 'hostitemsflag' is passed several times to the templates, but never used.
Is there something missing or this flag is useless?