Bug 11046 - Better handling of uncertain years for publicationyear/copyrightdate
Summary: Better handling of uncertain years for publicationyear/copyrightdate
Status: CLOSED FIXED
Alias: None
Product: Koha
Classification: Unclassified
Component: Cataloging (show other bugs)
Version: master
Hardware: All All
: P5 - low enhancement (vote)
Assignee: Marcel de Rooy
QA Contact: Josef Moravec
URL:
Keywords:
Depends on: 19096
Blocks: 24674 26516
  Show dependency treegraph
 
Reported: 2013-10-13 21:06 UTC by Mathieu Saby
Modified: 2021-09-20 14:08 UTC (History)
6 users (show)

See Also:
Change sponsored?: ---
Patch complexity: Small patch
Documentation contact:
Documentation submission:
Text to go in the release notes:
Version(s) released in:


Attachments
Bug 11046: Better handling of uncertain years for publicationyear (3.15 KB, patch)
2017-08-28 08:23 UTC, Marcel de Rooy
Details | Diff | Splinter Review
Bug 11046: Add the form YYY-? for uncertain years (2.19 KB, patch)
2017-08-28 11:16 UTC, Marcel de Rooy
Details | Diff | Splinter Review
Bug 11046: Better handling of uncertain years for publicationyear (3.24 KB, patch)
2017-10-21 07:22 UTC, Alex Buckley
Details | Diff | Splinter Review
Bug 11046: Add the form YYY-? for uncertain years (2.27 KB, patch)
2017-10-21 07:24 UTC, Alex Buckley
Details | Diff | Splinter Review
Bug 11046: Better handling of uncertain years for publicationyear (3.30 KB, patch)
2018-01-02 21:04 UTC, Josef Moravec
Details | Diff | Splinter Review
Bug 11046: Add the form YYY-? for uncertain years (2.33 KB, patch)
2018-01-02 21:04 UTC, Josef Moravec
Details | Diff | Splinter Review
Bug 11046: Improve the readability of the regex using named capture (1.38 KB, patch)
2018-01-09 20:14 UTC, Jonathan Druart
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description Mathieu Saby 2013-10-13 21:06:16 UTC
In C4:Biblio::TransformMarcToKoha, publicationyear and copyrightdate are filled from MARC dates.
MARC dates could be uncertain. like "200X", "2012?", "ca. 2012".
If there is a "X" or a blank, the date is not copied in database.
Maybe this could be fixed. But in that case, publicationyear and copyrightdate will sometimes contain non numerical data. Maybe that's an issue?

Sometime MARC dates are a range, like "2010-2013". In that case, Koha is currently using only the first one. Why not both?

Last remark : In UNIMARC, I don't know the difference between publicationyear and copyrightdate. Is it obvious for MARC21 people ?

M. Saby
Comment 1 Marcel de Rooy 2017-07-17 13:29:15 UTC
(In reply to mathieu saby from comment #0)
> In C4:Biblio::TransformMarcToKoha, publicationyear and copyrightdate are
> filled from MARC dates.
> MARC dates could be uncertain. like "200X", "2012?", "ca. 2012".
> If there is a "X" or a blank, the date is not copied in database.
> Maybe this could be fixed. But in that case, publicationyear and
> copyrightdate will sometimes contain non numerical data. Maybe that's an
> issue?
> 
> Sometime MARC dates are a range, like "2010-2013". In that case, Koha is
> currently using only the first one. Why not both?

I would feel for saving 18.. or 18XX etc. to 1800. Or 197X to 1970. Etc.
We cannot save two dates in one field obviously. So we have to choose and pick the first one.

> Last remark : In UNIMARC, I don't know the difference between
> publicationyear and copyrightdate. Is it obvious for MARC21 people ?
In Koha the UNIMARC side uses publicationyear and MARC21 uses copyrightdate.
Comment 2 Marcel de Rooy 2017-08-24 13:48:16 UTC
(In reply to mathieu saby from comment #0)
> Sometime MARC dates are a range, like "2010-2013". In that case, Koha is
> currently using only the first one. Why not both?

We cannot pick both. But could someone confirm that picking the first year is the preferred approach ? Or would the last year be better ??
We currently pick the first one.
Comment 3 Marcel de Rooy 2017-08-28 08:23:18 UTC
Created attachment 66549 [details] [review]
Bug 11046: Better handling of uncertain years for publicationyear

This patch makes it possible that uncertain year like 18.. or 197x are
converted to 1800 or 1970 in Koha field copyrightdate (MARC21) or
publicationyear (UNIMARC). (The corresponding MARC record will not be
changed obviously.)

This change will allow for better results when sorting search results or
list contents on copyrightdate. Currently, things like 18.. will sort
together with zero.

Note: The regex now allows four possible uncertain year markers: x or X,
question mark or dot.

Test plan:
[1] Run t/db_dependent/Biblio/TransformMarcToKoha.t
[2] Edit a biblio record. Save 18.. into 260c. Check biblio.copyrightdate.

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Comment 4 Marcel de Rooy 2017-08-28 08:24:31 UTC
Watch the dependencies !
Comment 5 Marcel de Rooy 2017-08-28 11:16:05 UTC
Created attachment 66552 [details] [review]
Bug 11046: Add the form YYY-? for uncertain years

This form occurred in Dutch ISBD rules.
The question mark should follow the hyphen(s).

Test plan:
Run t/db_dependent/Biblio/TransformMarcToKoha.t

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Comment 6 Alex Buckley 2017-10-21 07:22:27 UTC
Created attachment 68349 [details] [review]
Bug 11046: Better handling of uncertain years for publicationyear

This patch makes it possible that uncertain year like 18.. or 197x are
converted to 1800 or 1970 in Koha field copyrightdate (MARC21) or
publicationyear (UNIMARC). (The corresponding MARC record will not be
changed obviously.)

This change will allow for better results when sorting search results or
list contents on copyrightdate. Currently, things like 18.. will sort
together with zero.

Note: The regex now allows four possible uncertain year markers: x or X,
question mark or dot.

Test plan:
[1] Run t/db_dependent/Biblio/TransformMarcToKoha.t
[2] Edit a biblio record. Save 18.. into 260c. Check biblio.copyrightdate.

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>

Followed test plan, patch worked as described, it also passed QA test
tool

Signed-off-by: Alex Buckley <alexbuckley@catalyst.net.nz>
Comment 7 Alex Buckley 2017-10-21 07:24:47 UTC
Created attachment 68350 [details] [review]
Bug 11046: Add the form YYY-? for uncertain years

This form occurred in Dutch ISBD rules.
The question mark should follow the hyphen(s).

Test plan:
Run t/db_dependent/Biblio/TransformMarcToKoha.t

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>

TransformMarcToKoha tests passed. Also this patch passed QA test tool

Signed-off-by: Alex Buckley <alexbuckley@catalyst.net.nz>
Comment 8 Marcel de Rooy 2017-10-23 06:46:44 UTC
(In reply to Alex Buckley from comment #7)
> Signed-off-by: Alex Buckley <alexbuckley@catalyst.net.nz>

Thanks Alex
Comment 9 Josef Moravec 2018-01-02 21:04:01 UTC
Created attachment 70235 [details] [review]
Bug 11046: Better handling of uncertain years for publicationyear

This patch makes it possible that uncertain year like 18.. or 197x are
converted to 1800 or 1970 in Koha field copyrightdate (MARC21) or
publicationyear (UNIMARC). (The corresponding MARC record will not be
changed obviously.)

This change will allow for better results when sorting search results or
list contents on copyrightdate. Currently, things like 18.. will sort
together with zero.

Note: The regex now allows four possible uncertain year markers: x or X,
question mark or dot.

Test plan:
[1] Run t/db_dependent/Biblio/TransformMarcToKoha.t
[2] Edit a biblio record. Save 18.. into 260c. Check biblio.copyrightdate.

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>

Followed test plan, patch worked as described, it also passed QA test
tool

Signed-off-by: Alex Buckley <alexbuckley@catalyst.net.nz>

Signed-off-by: Josef Moravec <josef.moravec@gmail.com>
Comment 10 Josef Moravec 2018-01-02 21:04:06 UTC
Created attachment 70236 [details] [review]
Bug 11046: Add the form YYY-? for uncertain years

This form occurred in Dutch ISBD rules.
The question mark should follow the hyphen(s).

Test plan:
Run t/db_dependent/Biblio/TransformMarcToKoha.t

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>

TransformMarcToKoha tests passed. Also this patch passed QA test tool

Signed-off-by: Alex Buckley <alexbuckley@catalyst.net.nz>

Signed-off-by: Josef Moravec <josef.moravec@gmail.com>
Comment 11 Marcel de Rooy 2018-01-05 06:19:44 UTC
(In reply to Josef Moravec from comment #10)
> Signed-off-by: Josef Moravec <josef.moravec@gmail.com>

Thanks Josef.
Comment 12 Jonathan Druart 2018-01-09 20:14:17 UTC
Created attachment 70394 [details] [review]
Bug 11046: Improve the readability of the regex using named capture

The tests tell me I am good
Comment 13 Jonathan Druart 2018-01-09 20:15:38 UTC
Can we have a patch for the manual please? :)
Comment 14 Jonathan Druart 2018-01-09 20:27:31 UTC
Pushed to master for 18.05, thanks to everybody involved!
Comment 15 Marcel de Rooy 2018-01-10 12:15:36 UTC
(In reply to Jonathan Druart from comment #13)
> Can we have a patch for the manual please? :)

Does the Manual keyword trigger someone of the Documentation team to do so?
Comment 16 Katrin Fischer 2018-01-12 07:18:36 UTC
The manual team is still small, maybe write something up and send it to them? ;)
Comment 17 Jonathan Druart 2018-01-12 13:17:31 UTC
(In reply to Marcel de Rooy from comment #15)
> (In reply to Jonathan Druart from comment #13)
> > Can we have a patch for the manual please? :)
> 
> Does the Manual keyword trigger someone of the Documentation team to do so?

I think the author or someone involved in the development/testing of the patch should write something to explain to the manual team what the feature/change is.
And then the manual team could adapt it correctly to the manual and make a patch.
Comment 18 Nick Clemens 2018-01-16 12:25:50 UTC
Enhancement, skipping for 17.11.x.
Awesome work everybody!
Comment 19 Marcel de Rooy 2018-02-22 08:40:35 UTC
For documentation or release:

The patches of this report make that unknown publication/copyright years like 19.. or 19xx or 19XX or 19?? or 19--? (and similar forms) are interpreted as 1900 instead of 0 when saving copyrightdate (MARC21) or publicationyear (UNIMARC).
These fields are so-called kohafields (fields extracted from the MARC record and saved additionally in the database for optimization).
Since it is now possible to connect multiple MARC fields to one kohafield, you can enter something like 1xxx or 201x into MARC21 fields like 260$c or 264$c. When you save the record, the estimated publication year (1000 or 2010) will be saved into the corresponding kohafield. This field is used for sorting lists [virtual shelves] by copyrightdate among others.
Comment 20 Marcel de Rooy 2021-09-20 14:08:43 UTC
Just a late observation about the last changes:

-    } elsif( $retval =~ m/(\d)[.Xx?]{3}|(\d\d)[.Xx?]{2}|(\d{3})[.Xx?]/ ) {
-        my $digits = $1 || $2 || $3;
-        $retval = $digits * ( 10 ** ( 4 - length($digits) ));
-    } elsif( $retval =~ m/(\d)[-]{3}\?|(\d\d)[-]{2}\?|(\d{3})[-]\?/ ) {
-        # the form 198-? occurred in Dutch ISBD rules
-        my $digits = $1 || $2 || $3;
+    } elsif( $retval =~ m/
+             (?<year>\d)[-]?[.Xx?]{3}
+            |(?<year>\d{2})[.Xx?]{2}
+            |(?<year>\d{3})[.Xx?]
+            |(?<year>\d)[-]{3}\?
+            |(?<year>\d\d)[-]{2}\?
+            |(?<year>\d{3})[-]\?
+    /xms ) { # the form 198-? occurred in Dutch ISBD rules

This part of the regex made me wonder:
             (?<year>\d)[-]?[.Xx?]{3}
What does the minus do here after the first digit? Do we allow 1-234 or so?
Looking at the above, this must have been a typo (copy/paste) after all..

Continued on bug 24674