Adding a see also to 21708. It helps to insert the 003/024/035 fields for PIDs at the right position in the MARC record.

Interesting read:
https://journal.code4lib.org/articles/14978

Created attachment 106460 [details] [review]
Bug 24544: Database revision for two new preferences

We add PID_Domain and PID_Field here.

Test plan:
Run install or upgrade.

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>

Created attachment 106461 [details] [review]
Bug 24544: Introduce Koha::PID::Controller with unit test

This new module will facilitate inserting/updating PIDs in authority,
biblio and item records.

Test plan:
Run t/db_dependent/Koha/PID/Controller.t

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>

Created attachment 106462 [details] [review]
Bug 24544: Add script to insert persistent identifiers

Test plan: See patch with example plugins.
This script can only be tested fully when you enable a plugin to
generate new identifiers. The next patch provides samples to do
that.

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>

Created attachment 106463 [details] [review]
DO NOT PUSH: Example plugins

This patch contains a few example plugins for testing and
illustrating the use of the various plugin hooks.

Prerequisites:
[1] Fill PID_Domain with http://[your_domain]
    Note: This domain is used for pid recognition.
[2] Fill PID_Field with say 'auth:024,biblio:024,item'.
    In other words: enable PIDs for all categories and use 024.
[3] Where we refer to (un)install plugin X below, we mean:
    perl -MKoha::Plugin::Test::PID::X -e"Koha::Plugin::Test::PID::X->install;"
    perl -MKoha::Plugin::Test::PID::X -e"Koha::Plugin::Test::PID::X->uninstall;"
    Replace X by the plugin name.

Test plan:
[1] Koha::Plugin::Test::PID::External with opac/svc/pid_test
    This part shows the use of pid_create_hook.
[a] Install the External plugin.
[b] Execute shell command:
    export PID_SERVICE=http://[your_server:opac_port]/cgi-bin/koha/svc/pid_test
[c] Edit your first authority record, add a 024 with $a. Make sure that
    it fully includes your PID_Domain.
[d] Run insert_persistent_id.pl -a -c -v -records 2
    Should report back: records 2, updates 1.
[e] Verify that your first authority was not modified again and that
    your second authority record contains a new 024.

[2] Koha::Plugin::Test::PID::Orgcode
    This part shows all three hooks, one hook from External and
    two other hooks from Orgcode. For instance note the move from
    $a to $z in the insert hook here.
[a] Keep the External plugin, also install the Orgcode plugin.
[b] Replace PID_Field by biblio:035$a
[c] Fill MARCOrgCode with some string.
[d] Edit first biblio record. Add a 035 with $a containing orgcode
    but with another domain than PID_Domain.
    Edit the second biblio record. Add a 035 with PID_Domain in $a
    but without a prefixing orgcode.
[e] Run insert_persistent_id.pl -a -c -v -updates 2
    Should report back: records 2, updates 2
[f] Check that the first two biblio records contain a 035 with your
    orgcode in $a before PID_Domain and a new identifier.
    The first biblio should contain a 035 with $a and $z where $z
    contains the edit from step [d].
    The second biblio record should also contain the 035 from step [d].
[g] Replace PID_Field by: item
[h] Run insert_persistent_id.pl -a -c -v -updates 4
    Should insert pid's in 4 item records.

[3] Koha::Plugin::Test::PID::Cleanup
    This part shows a 'creative' use of get and insert in order to
    actually remove identifiers from a record, just illustrating the
    power of plugins.
[a] Uninstall plugins External and Orgcode.
[b] Install the Cleanup plugin.
[c] Run insert_persistent_id.pl -a -c -v -updates 3
    Verify that first three items were cleaned up. Fourth untouched.
[d] Uninstall Cleanup.

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>

A potential follow-up will add a call for inserting PIDs when storing a new item, biblio or authority. Did not want to add them here yet.

Created attachment 106464 [details] [review]
Bug 24544: Add new cronjob to cron files

Added the additional parameter -hours 2 to prevent a first time
run at night to do a full update (perhaps unexpected).
I would recommend to run a full update after configuring the
preferences.

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>

While I see the advantage of using PIDs (I use them in DSpace and other GLAM systems), after browsing these patches, I don't see the advantages of adding these patches.

What are you trying to achieve here?

Personally, I think that it would be better to add a plugin to the authority/biblio/item records and allow you to mint PIDs while cataloguing. That's how most systems I've used have worked.

That said, sometimes you want to batch mint PIDs, but I think that would be up to the entity doing the batch work. I don't see why we'd need anything special for that in Koha.

Or are you trying to create a local Koha PID minting service? If that's the case, I think storing the data in MARC fields is not a wise direction. It would be better to have a Koha PID lookup table mapping the PID to the record, so that PID resolution could be handled that way.

And if we do want to store the PID at the record level, I think it would be best to put that into the relational database (we could export it into the MARC at indexing/exporting time).

(In reply to David Cook from comment #11)
Thanks for your feedback.

> Personally, I think that it would be better to add a plugin to the
> authority/biblio/item records and allow you to mint PIDs while cataloguing.
> That's how most systems I've used have worked.

See also comment9. I chose here to not add stuff directly to core Koha routines. But work my way thru the data with a cron job. Surely this would be a next step. Problem with a cataloguing plugin is that you just know the record number only after saving  it. Although a PID generator might theoretically not need it, many implementations, including my own, do use it.

> That said, sometimes you want to batch mint PIDs, but I think that would be
> up to the entity doing the batch work. I don't see why we'd need anything
> special for that in Koha.

Not sure how much you saw from the patches, but this patch set provides an interface via plugins to an external PID service.

> Or are you trying to create a local Koha PID minting service? If that's the
> case, I think storing the data in MARC fields is not a wise direction. It
> would be better to have a Koha PID lookup table mapping the PID to the
> record, so that PID resolution could be handled that way.

See before. No Koha should not mint its own PIDs. Formally the PID generator may have its own PID lookup table. We do not really care here. (My local generator does not, since it is based on a Koha identifier. Its result can be found with a Standard-identifier index in Koha or even another future ILS. In that way actually turning my ES or Zebra index into a PID lookup table..)
But even with a full resolver having its own table, I would still argue to save a copy of the PID in the MARC record too for optimization, while respecting the lookup table as authoritative.
As a side note: Could you give me another example of vital data on biblio level that we do not store in MARC? Not meaning optimization or calculated aggregates etc.

(In reply to David Cook from comment #12)
> And if we do want to store the PID at the record level, I think it would be
> best to put that into the relational database (we could export it into the
> MARC at indexing/exporting time).

See ^^

Adding a local comment in our discussion basket ;)

What is the right moment for a biblio record to receive a persistent identifier btw? Thinking about local procedures around acquisition etc.

The choice in this patch set is still influenced by the partial/meta resolving (see former comments). A replace by Z3950/SRU might result in two PIDs in the authoritative table and one local copy (the newest) in Koha.

(In reply to Marcel de Rooy from comment #13)
> See also comment9. I chose here to not add stuff directly to core Koha
> routines. But work my way thru the data with a cron job. Surely this would
> be a next step. Problem with a cataloguing plugin is that you just know the
> record number only after saving  it. Although a PID generator might
> theoretically not need it, many implementations, including my own, do use it.
> 

I could see the utility of that cronjob for you, but maybe not for all Koha users?

That's a good point about the PID generator. 

> Not sure how much you saw from the patches, but this patch set provides an
> interface via plugins to an external PID service.
> 

I skimmed through that code, but found it a bit difficult to read.

> No Koha should not mint its own PIDs. 

Ok excellent.

> Formally the PID generator
> may have its own PID lookup table. We do not really care here. (My local
> generator does not, since it is based on a Koha identifier. Its result can
> be found with a Standard-identifier index in Koha or even another future
> ILS. In that way actually turning my ES or Zebra index into a PID lookup
> table..)

I'm not sure that I understand this part. 

So you use the Koha identifier to mint a PID with a local non-Koha generator, then you store that in the Koha record and index it. 

Your organisation resolver then forward to a local non-Koha resolver which then queries Zebra/ES to get the record that matches... I assume not a full URL but a partial path?

> But even with a full resolver having its own table, I would still argue to
> save a copy of the PID in the MARC record too for optimization, while
> respecting the lookup table as authoritative.
> As a side note: Could you give me another example of vital data on biblio
> level that we do not store in MARC? Not meaning optimization or calculated
> aggregates etc.

I don't know what you mean by "vital data" in this case, but some standouts are biblio.frameworkcode, biblio.datecreated (debateable), biblio_metadata.format, biblio_metadata.schema. 

To be honest, I think that I see where you're coming from. In the past, I wanted to store OAI-PMH identifiers in the 024 field for imported MARC bibliographic records and then look them up with a Zebra index search, but then I realized was problematic for me. While those MARC fields exist, the data in those fields aren't descriptive metadata about the record. They're metadata about the metadata record. In my case, I ended up putting the OAI-PMH identifier in the relational database, as it was much more robust than putting it in the MARC record and indexing it into Zebra (especially since there can be a lag for index updates).

(In reply to Marcel de Rooy from comment #15)
> What is the right moment for a biblio record to receive a persistent
> identifier btw? Thinking about local procedures around acquisition etc.
> 
> The choice in this patch set is still influenced by the partial/meta
> resolving (see former comments). A replace by Z3950/SRU might result in two
> PIDs in the authoritative table and one local copy (the newest) in Koha.

That's an interesting question. In my mind, a biblionumber represents a conceptual metadata entity. While the actual metadata can change, that entity is tied to the point in time that the biblionumber is created. While we use the MySQL/MariaDB autonumber for that at the moment, in theory we could use a UUID or some other generated number (like we do for the item barcode rather than the item number).

For me, the MARCXML isn't the bibliographic record, but rather a view of it. In the future, that bibliographic record might have a RDF/XML view as well. They're both describing the same bibliographic entity, and that entity is the thing that should have the PID. After all, HTML, MARCXML, RDF/XML, they're just different content types available via content negotiation, right?

The replacement by Z3950/SRU is interesting. If we had versioned records, it might make sense to have a PID per version I suppose...

It could be interesting to see how DSpace manages this.

(In reply to David Cook from comment #17)
> That's an interesting question. In my mind, a biblionumber represents a
> conceptual metadata entity. While the actual metadata can change, that
> entity is tied to the point in time that the biblionumber is created. While
> we use the MySQL/MariaDB autonumber for that at the moment, in theory we
> could use a UUID or some other generated number (like we do for the item
> barcode rather than the item number).

Yes, I think that creation time works best.

> For me, the MARCXML isn't the bibliographic record, but rather a view of it.
> In the future, that bibliographic record might have a RDF/XML view as well.
> They're both describing the same bibliographic entity, and that entity is
> the thing that should have the PID. After all, HTML, MARCXML, RDF/XML,
> they're just different content types available via content negotiation,
> right?

Just as the biblionumber the persistent identifier belongs/points to a conceptual thing, not talking about where we store it. This thing can indeed be described in MARCXML, RDF etc. What would be wrong about showing the PID in such views just like we often include biblionumber? Finally, storing it in a MARC field (just as many save biblionumber in 001) or not (only exporting or displaying it) is an implementation thing/technical optimization?

>I don't know what you mean by "vital data" in this case, but some standouts are biblio.frameworkcode, biblio.datecreated (debateable), biblio_metadata.format, biblio_metadata.schema. 

I was trying to follow the discussion here a bit, if I understand correctly, the difference between the PID and the fields you list would be that the PID is supposed to be exported via Z39.50/SRU etc. - so if not included in the MARC file, it would make things much more complicated?

(In reply to Katrin Fischer from comment #19)
> I was trying to follow the discussion here a bit, if I understand correctly,
> the difference between the PID and the fields you list would be that the PID
> is supposed to be exported via Z39.50/SRU etc. - so if not included in the
> MARC file, it would make things much more complicated?

Thx for asking.

For me frameworkcode or metadata format are quite different from the persistent identifier. The frameworkcode is bound to the MARC view/representation. Which may not stay the only view. The reason we made metadata format. That field is repeatable and belongs to the corresponding view.
The PID on the other hand is a unique identifier that does not belong to a particular view but imo c/should be shown in each view. MARC21 and UNIMARC also added fields for such identifiers (auth 024, unimarc 003, etc.)
Although we could save the field outside MARC and just copy it in at export time or display time, I dont mind adding it in MARC as a technical optimization. See comment18.
Strict formally, the PID has been generated outside Koha and can be dereferenced outside Koha.

From David Cook on dev mailing list:

Sorry that I haven’t had a chance to reply on Bugzilla. I’ve been super busy.

For what it’s worth, I’m fine with it being stored in the MARC, so long as it’s also stored in the relational database. I thought it was an either/or question rather than a both/and question.

Curious to hear what people have to say!

From myself on July 20 on the dev mailing list:
===
What do you think? If your authority, biblio or item has a PID, should we save it in MARC too? Noting that MARC is just a representation or view of your data. We already have a biblio_metadata table that allows for more. What if we already had multiple views? If you can, you are welcome to contribute to that discussion on the bug report.
===

Note that this did not trigger any response or discussion sofar.

(In reply to Marcel de Rooy from comment #22)
> From myself on July 20 on the dev mailing list:
> ===
> What do you think? If your authority, biblio or item has a PID, should we
> save it in MARC too? Noting that MARC is just a representation or view of
> your data. We already have a biblio_metadata table that allows for more.
> What if we already had multiple views? If you can, you are welcome to
> contribute to that discussion on the bug report.
> ===
> 
> Note that this did not trigger any response or discussion sofar.

It was heartwarming to see the amount of interest and feedback on this development ;) I decided to move things to local plugins. And will close this report now.

(In reply to Marcel de Rooy from comment #23)
> It was heartwarming to see the amount of interest and feedback on this
> development ;) I decided to move things to local plugins. And will close
> this report now.

I empathize. I have had a few developments that have been dear to me but haven't received much interest/feedback from others, unfortunately. 

I hope that local plugins are able to provide an adequate solution for you.