In C4:Biblio::TransformMarcToKoha, publicationyear and copyrightdate are filled from MARC dates. MARC dates could be uncertain. like "200X", "2012?", "ca. 2012". If there is a "X" or a blank, the date is not copied in database. Maybe this could be fixed. But in that case, publicationyear and copyrightdate will sometimes contain non numerical data. Maybe that's an issue? Sometime MARC dates are a range, like "2010-2013". In that case, Koha is currently using only the first one. Why not both? Last remark : In UNIMARC, I don't know the difference between publicationyear and copyrightdate. Is it obvious for MARC21 people ? M. Saby
(In reply to mathieu saby from comment #0) > In C4:Biblio::TransformMarcToKoha, publicationyear and copyrightdate are > filled from MARC dates. > MARC dates could be uncertain. like "200X", "2012?", "ca. 2012". > If there is a "X" or a blank, the date is not copied in database. > Maybe this could be fixed. But in that case, publicationyear and > copyrightdate will sometimes contain non numerical data. Maybe that's an > issue? > > Sometime MARC dates are a range, like "2010-2013". In that case, Koha is > currently using only the first one. Why not both? I would feel for saving 18.. or 18XX etc. to 1800. Or 197X to 1970. Etc. We cannot save two dates in one field obviously. So we have to choose and pick the first one. > Last remark : In UNIMARC, I don't know the difference between > publicationyear and copyrightdate. Is it obvious for MARC21 people ? In Koha the UNIMARC side uses publicationyear and MARC21 uses copyrightdate.
(In reply to mathieu saby from comment #0) > Sometime MARC dates are a range, like "2010-2013". In that case, Koha is > currently using only the first one. Why not both? We cannot pick both. But could someone confirm that picking the first year is the preferred approach ? Or would the last year be better ?? We currently pick the first one.
Created attachment 66549 [details] [review] Bug 11046: Better handling of uncertain years for publicationyear This patch makes it possible that uncertain year like 18.. or 197x are converted to 1800 or 1970 in Koha field copyrightdate (MARC21) or publicationyear (UNIMARC). (The corresponding MARC record will not be changed obviously.) This change will allow for better results when sorting search results or list contents on copyrightdate. Currently, things like 18.. will sort together with zero. Note: The regex now allows four possible uncertain year markers: x or X, question mark or dot. Test plan: [1] Run t/db_dependent/Biblio/TransformMarcToKoha.t [2] Edit a biblio record. Save 18.. into 260c. Check biblio.copyrightdate. Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Watch the dependencies !
Created attachment 66552 [details] [review] Bug 11046: Add the form YYY-? for uncertain years This form occurred in Dutch ISBD rules. The question mark should follow the hyphen(s). Test plan: Run t/db_dependent/Biblio/TransformMarcToKoha.t Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Created attachment 68349 [details] [review] Bug 11046: Better handling of uncertain years for publicationyear This patch makes it possible that uncertain year like 18.. or 197x are converted to 1800 or 1970 in Koha field copyrightdate (MARC21) or publicationyear (UNIMARC). (The corresponding MARC record will not be changed obviously.) This change will allow for better results when sorting search results or list contents on copyrightdate. Currently, things like 18.. will sort together with zero. Note: The regex now allows four possible uncertain year markers: x or X, question mark or dot. Test plan: [1] Run t/db_dependent/Biblio/TransformMarcToKoha.t [2] Edit a biblio record. Save 18.. into 260c. Check biblio.copyrightdate. Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl> Followed test plan, patch worked as described, it also passed QA test tool Signed-off-by: Alex Buckley <alexbuckley@catalyst.net.nz>
Created attachment 68350 [details] [review] Bug 11046: Add the form YYY-? for uncertain years This form occurred in Dutch ISBD rules. The question mark should follow the hyphen(s). Test plan: Run t/db_dependent/Biblio/TransformMarcToKoha.t Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl> TransformMarcToKoha tests passed. Also this patch passed QA test tool Signed-off-by: Alex Buckley <alexbuckley@catalyst.net.nz>
(In reply to Alex Buckley from comment #7) > Signed-off-by: Alex Buckley <alexbuckley@catalyst.net.nz> Thanks Alex
Created attachment 70235 [details] [review] Bug 11046: Better handling of uncertain years for publicationyear This patch makes it possible that uncertain year like 18.. or 197x are converted to 1800 or 1970 in Koha field copyrightdate (MARC21) or publicationyear (UNIMARC). (The corresponding MARC record will not be changed obviously.) This change will allow for better results when sorting search results or list contents on copyrightdate. Currently, things like 18.. will sort together with zero. Note: The regex now allows four possible uncertain year markers: x or X, question mark or dot. Test plan: [1] Run t/db_dependent/Biblio/TransformMarcToKoha.t [2] Edit a biblio record. Save 18.. into 260c. Check biblio.copyrightdate. Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl> Followed test plan, patch worked as described, it also passed QA test tool Signed-off-by: Alex Buckley <alexbuckley@catalyst.net.nz> Signed-off-by: Josef Moravec <josef.moravec@gmail.com>
Created attachment 70236 [details] [review] Bug 11046: Add the form YYY-? for uncertain years This form occurred in Dutch ISBD rules. The question mark should follow the hyphen(s). Test plan: Run t/db_dependent/Biblio/TransformMarcToKoha.t Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl> TransformMarcToKoha tests passed. Also this patch passed QA test tool Signed-off-by: Alex Buckley <alexbuckley@catalyst.net.nz> Signed-off-by: Josef Moravec <josef.moravec@gmail.com>
(In reply to Josef Moravec from comment #10) > Signed-off-by: Josef Moravec <josef.moravec@gmail.com> Thanks Josef.
Created attachment 70394 [details] [review] Bug 11046: Improve the readability of the regex using named capture The tests tell me I am good
Can we have a patch for the manual please? :)
Pushed to master for 18.05, thanks to everybody involved!
(In reply to Jonathan Druart from comment #13) > Can we have a patch for the manual please? :) Does the Manual keyword trigger someone of the Documentation team to do so?
The manual team is still small, maybe write something up and send it to them? ;)
(In reply to Marcel de Rooy from comment #15) > (In reply to Jonathan Druart from comment #13) > > Can we have a patch for the manual please? :) > > Does the Manual keyword trigger someone of the Documentation team to do so? I think the author or someone involved in the development/testing of the patch should write something to explain to the manual team what the feature/change is. And then the manual team could adapt it correctly to the manual and make a patch.
Enhancement, skipping for 17.11.x. Awesome work everybody!
For documentation or release: The patches of this report make that unknown publication/copyright years like 19.. or 19xx or 19XX or 19?? or 19--? (and similar forms) are interpreted as 1900 instead of 0 when saving copyrightdate (MARC21) or publicationyear (UNIMARC). These fields are so-called kohafields (fields extracted from the MARC record and saved additionally in the database for optimization). Since it is now possible to connect multiple MARC fields to one kohafield, you can enter something like 1xxx or 201x into MARC21 fields like 260$c or 264$c. When you save the record, the estimated publication year (1000 or 2010) will be saved into the corresponding kohafield. This field is used for sorting lists [virtual shelves] by copyrightdate among others.
Just a late observation about the last changes: - } elsif( $retval =~ m/(\d)[.Xx?]{3}|(\d\d)[.Xx?]{2}|(\d{3})[.Xx?]/ ) { - my $digits = $1 || $2 || $3; - $retval = $digits * ( 10 ** ( 4 - length($digits) )); - } elsif( $retval =~ m/(\d)[-]{3}\?|(\d\d)[-]{2}\?|(\d{3})[-]\?/ ) { - # the form 198-? occurred in Dutch ISBD rules - my $digits = $1 || $2 || $3; + } elsif( $retval =~ m/ + (?<year>\d)[-]?[.Xx?]{3} + |(?<year>\d{2})[.Xx?]{2} + |(?<year>\d{3})[.Xx?] + |(?<year>\d)[-]{3}\? + |(?<year>\d\d)[-]{2}\? + |(?<year>\d{3})[-]\? + /xms ) { # the form 198-? occurred in Dutch ISBD rules This part of the regex made me wonder: (?<year>\d)[-]?[.Xx?]{3} What does the minus do here after the first digit? Do we allow 1-234 or so? Looking at the above, this must have been a typo (copy/paste) after all.. Continued on bug 24674