Bug 22509

Summary: Add a script to generate MARC fields containing date formatted strings
Product: Koha Reporter: Jonathan Druart <jonathan.druart>
Component: Command-line UtilitiesAssignee: Jonathan Druart <jonathan.druart>
Status: CLOSED FIXED QA Contact: Testopia <testopia>
Severity: enhancement    
Priority: P5 - low CC: fridolin.somers, hagud, katrin.fischer, m.de.rooy, martin.renvoize, robin
Version: master   
Hardware: All   
OS: All   
Change sponsored?: --- Patch complexity: Small patch
Documentation contact: Documentation submission:
Text to go in the release notes:
Version(s) released in:
19.11.00
Bug Depends on:    
Bug Blocks: 23924    
Attachments: Bug 22509: Add new script to add MARC fields with date formatted stringsto bib records
Bug 22509: Add the --unless-exists option to the script
Bug 22509: Add new script to add MARC fields with date formatted stringsto bib records
Bug 22509: Add the --unless-exists option to the script
Bug 22509: Add new script to add MARC fields with date formatted stringsto bib records
Bug 22509: Add the --unless-exists option to the script
Bug 22509: Fix POD error
Bug 22509: Fix POD error
Bug 22509: Add more POD and fix --where
Bug 22509: Clarify with an example the usage of --unless-exists
Bug 22509: Clarify with an example the usage of --unless-exists

Description Jonathan Druart 2019-03-13 19:32:39 UTC
This will add a simple script to add MARC fields to bibliographic records with date information.
Comment 1 Jonathan Druart 2019-03-13 19:37:54 UTC
Created attachment 86586 [details] [review]
Bug 22509: Add new script to add MARC fields with date formatted stringsto bib records

Say you want to add, into a new MARC fields, the following string:
  2/2019/Mar-03/13
Depending on today's date.

Test plan:
Use the script after with the different options (see --help option for the list)

Sponsored-by: Orex Digital
Comment 2 Hugo Agud 2019-04-01 12:48:28 UTC
Ths script allow a better integration with vufind, and allow to create hierarchy facets by date 1st level year, 2 level mont, third level day

NOw the script add the same date to all records but
1. Each record should have its onw date, perhaps import from a marc tag? 
2. the script should only apply to records without 905, not recreated
Comment 3 Jonathan Druart 2019-04-05 02:03:17 UTC
Created attachment 87435 [details] [review]
Bug 22509: Add the --unless-exists option to the script

It will avoid to create the fields several times if it is not needed.
Comment 4 Jonathan Druart 2019-04-05 02:03:40 UTC
(In reply to Hugo Agud from comment #2)
> Ths script allow a better integration with vufind, and allow to create
> hierarchy facets by date 1st level year, 2 level mont, third level day
> 
> NOw the script add the same date to all records but
> 1. Each record should have its onw date, perhaps import from a marc tag? 

How would be formatted the date?

> 2. the script should only apply to records without 905, not recreated

Done in the last patch, see the new options --unless-exists.
Comment 5 Hugo Agud 2019-04-09 08:35:17 UTC
Created attachment 87588 [details] [review]
Bug 22509: Add new script to add MARC fields with date formatted stringsto bib records

Say you want to add, into a new MARC fields, the following string:
  2/2019/Mar-03/13
Depending on today's date.

Test plan:
Use the script after with the different options (see --help option for the list)

Sponsored-by: Orex Digital

Signed-off-by: Hugo Agud <hagud@orex.es>
Comment 6 Hugo Agud 2019-04-09 08:35:20 UTC
Created attachment 87589 [details] [review]
Bug 22509: Add the --unless-exists option to the script

It will avoid to create the fields several times if it is not needed.

Signed-off-by: Hugo Agud <hagud@orex.es>
Comment 7 Hugo Agud 2019-04-23 16:25:06 UTC
Created attachment 88532 [details] [review]
Bug 22509: Add new script to add MARC fields with date formatted stringsto bib records

Say you want to add, into a new MARC fields, the following string:
  2/2019/Mar-03/13
Depending on today's date.

Test plan:
Use the script after with the different options (see --help option for the list)

Sponsored-by: Orex Digital

Signed-off-by: Hugo Agud <hagud@orex.es>
Comment 8 Hugo Agud 2019-04-23 16:25:09 UTC
Created attachment 88533 [details] [review]
Bug 22509: Add the --unless-exists option to the script

It will avoid to create the fields several times if it is not needed.

Signed-off-by: Hugo Agud <hagud@orex.es>
Comment 9 Katrin Fischer 2019-04-29 13:49:39 UTC
Can you explain the use case for this one? It seems a rather specific script.
Comment 10 Hugo Agud 2019-04-29 13:53:19 UTC
This script it is designed for a better integration of koha database into discoveries, the info of this script is used by discoveries for creating hierarchical Facets.

This script created a data like 2019-04-29 into a formatted tag that allows to create hierarchical Facets, in a way that end users may seach for records by year, month or day at the end.
Comment 11 Katrin Fischer 2019-04-29 13:56:36 UTC
Which date would you index like that? The record creation date? I have never seen anything like that in discoveries. It still appears very specific.
Comment 12 Hugo Agud 2019-04-29 14:09:02 UTC
usually the creation date yes, and yes again it is specific but for press search is really useful
Comment 13 Marcel de Rooy 2019-05-10 09:29:45 UTC
   FAIL   pod
                *** ERROR: Apparent command =item not preceded by blank line  in file misc/add_date_fields_to_marc_records.pl
Comment 14 Jonathan Druart 2019-05-10 12:59:49 UTC
Created attachment 89546 [details] [review]
Bug 22509: Fix POD error

"Apparent command =item not preceded by blank line"
Comment 15 Katrin Fischer 2019-05-11 16:45:36 UTC
I think what Hugo is referring to with the hierarchical facets is this:

https://vufind.org/wiki/configuration:facets

So it seems that VuFind can build hierarchical facets on all values that are concatenated with / and an ending /. So the 'relationship' to VuFind is there, but it's not very strong :)

This script is more general as it allows to add arbitrary formatted date information to any MARC subfield. But it's also a bit limited, as it will use today's date.

I have some notes:

1) The misc directory is already a bit chaotic. I would suggest to move this into a new directory in misc where we could keep such scripts in the future. Maybe something like: marc_tools, data_manipulation, marc_manipulation ?

2) If I pass nothing in, the script is run in dry mode. I think it would be better to display the help, since the needed parameters to actually do something are missing.

3) When I run it with -h it advertises:
perl add_date_fields_to_marc_records.pl --help 
But the help displayed doesn't explain the options like it's supposed to looking at the script itself.

Can you please take a look at fixing the help?
Comment 16 Jonathan Druart 2019-05-11 17:07:47 UTC
Created attachment 89616 [details] [review]
Bug 22509: Fix POD error

1. "Apparent command =item not preceded by blank line"
2. Display the options for help
3. Do not run if no --field passed
Comment 17 Jonathan Druart 2019-05-11 17:09:55 UTC
(In reply to Katrin Fischer from comment #15)

Thanks Katrin for taking a look at this one.

POD (2 and 3) fixed.

About the path I would prefer to let the RM pick the correct location. IMO it will be weird to have only this script in a new directory.
Comment 18 Katrin Fischer 2019-05-11 17:15:47 UTC
(In reply to Jonathan Druart from comment #17)
> (In reply to Katrin Fischer from comment #15)
> 
> Thanks Katrin for taking a look at this one.
> 
> POD (2 and 3) fixed.
> 
> About the path I would prefer to let the RM pick the correct location. IMO
> it will be weird to have only this script in a new directory.

I think starting with one would be ok, we have to start somewhere. And moving them later is more painful for people, as they have to fix their paths.
Comment 19 Katrin Fischer 2019-05-11 19:50:53 UTC
(In reply to Katrin Fischer from comment #18)
> (In reply to Jonathan Druart from comment #17)
> > (In reply to Katrin Fischer from comment #15)
> > 
> > Thanks Katrin for taking a look at this one.
> > 
> > POD (2 and 3) fixed.
> > 
> > About the path I would prefer to let the RM pick the correct location. IMO
> > it will be weird to have only this script in a new directory.
> 
> I think starting with one would be ok, we have to start somewhere. And
> moving them later is more painful for people, as they have to fix their
> paths.

Leaving this to the RM indeed :)

1) Can you add a usage example for the WHERE condition? I tried:

./add_date_fields_to_marc_records.pl --where 'biblionumber <10' --field='245$a=1/test/%Y/%b-%m'

It should tell which table is queried. It also feels a little odd that it seems like you have to enter --where 'where...'? It seems the where argument on rebuild_zebra.pl works without.

2) The use of unless-exists was not clear to me from the description, I think an example below the option description would be helpful too.

Apart from documentation this seems to work fine. If you update that, I am willing to PQA :)
Comment 20 Jonathan Druart 2019-05-11 23:58:50 UTC
Created attachment 89626 [details] [review]
Bug 22509: Add more POD and fix --where

We should not need the "WHERE" keyword to have the same behavior as
existing script (rebuild_zebra.pl)
Comment 21 Jonathan Druart 2019-05-11 23:58:53 UTC
Created attachment 89627 [details] [review]
Bug 22509: Clarify with an example the usage of --unless-exists
Comment 22 Jonathan Druart 2019-05-11 23:59:39 UTC
(In reply to Katrin Fischer from comment #19)

Hope it makes more sense now.
Comment 23 Katrin Fischer 2019-05-13 19:05:18 UTC
Created attachment 89680 [details] [review]
Bug 22509: Clarify with an example the usage of --unless-exists

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Comment 24 Katrin Fischer 2019-05-13 19:07:51 UTC
RM, please see comment 18 for some discussion about where to store this script!
Comment 25 Martin Renvoize 2019-06-24 14:47:56 UTC
Nice work!

Pushed to master for 19.11.00
Comment 26 Fridolin Somers 2019-07-24 07:18:58 UTC
Enhancement not pushed to 19.05.x