Bug 12424 - ddc sorting of call numbers truncates long Cutter parts
Summary: ddc sorting of call numbers truncates long Cutter parts
Status: CLOSED FIXED
Alias: None
Product: Koha
Classification: Unclassified
Component: Cataloging (show other bugs)
Version: master
Hardware: All All
: P5 - low normal (vote)
Assignee: simith.doliveira
QA Contact: Testopia
URL:
Keywords:
Depends on:
Blocks: 8837
  Show dependency treegraph
 
Reported: 2014-06-14 04:59 UTC by giuseppe.angilella
Modified: 2015-06-04 23:33 UTC (History)
8 users (show)

See Also:
Change sponsored?: ---
Patch complexity: Small patch
Documentation contact:
Documentation submission:
Text to go in the release notes:
Version(s) released in:


Attachments
Bug 12424 - ddc sorting of call numbers truncates long Cutter parts (7.05 KB, patch)
2014-08-06 12:42 UTC, simith.doliveira
Details | Diff | Splinter Review
Patch updated. Rebase done. (7.05 KB, patch)
2014-08-12 13:19 UTC, simith.doliveira
Details | Diff | Splinter Review
Bug 12424 - ddc sorting of call numbers truncates long Cutter parts (7.12 KB, patch)
2014-08-12 15:54 UTC, Biblibre Sandboxes
Details | Diff | Splinter Review
Bug 12424 - ddc sorting of call numbers truncates long Cutter parts (8.59 KB, patch)
2014-08-21 18:48 UTC, simith.doliveira
Details | Diff | Splinter Review
Bug 12424 - ddc sorting of call numbers truncates long Cutter parts (8.59 KB, patch)
2014-10-15 16:09 UTC, Chris Cormack
Details | Diff | Splinter Review
[PASSED QA] Bug 12424 - ddc sorting of call numbers truncates long Cutter parts (8.86 KB, patch)
2014-10-16 13:14 UTC, Katrin Fischer
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description giuseppe.angilella 2014-06-14 04:59:47 UTC
Occasionally, I need quite long Cutter parts in the itemcallnumber of specific items, e.g.

530 F435 1996 v2p1

for part 1 (p1) of volume 2 (v2) of the 1996 edition of the Feynman's (Cutter F435) Lectures in Physics (Dewey 530).

All substrings are essential information, as another edition (2007) exists (and is present in my library's collection). Moreover, 1996 and 2007 cannot be abridged to 96 and 07, respectively, as this would obviously result in a wrong sorting.

Now, since "530" has no decimal part in this case, Dewey's class sorting routine (C4/ClassSortRoutine/Dewey.pm) recognizes "1996" as the "second digit group", and therefore converts it into a "15-digit long group, padded on right with zeroes" (see documentation e.g. at https://github.com/colinsc/koha/blob/master/C4/ClassSortRoutine/Dewey.pm ). This results in having this item sorted at the very end of all items having an 530.* Dewey class (yet before items with 531).

I could overcome this, by attaching the year part to the proper Cutter part by a character different from a whitespace or a decimal point, e.g.

530 F435_1996 v2p1

In this way, 1996 is not an isolated "second digit group" any longer, and "F435_1996" is not modified by the class sorting routine.

However, the resulting cn_sort field for this item looks like:

530_000000000000000_F435_1996_

namely, the volume and part substring have been truncated by MySQL, since items.cn_sort is only a varchar(30) subfield.

The situation would be even worse for longer Cutter strings (e.g., including trailing letter(s) from the record's title), or with item call numbers including copy numbers.

This behaviour could be enhanced by increasing the width of the items.cn_sort subfield in the Koha database by a sufficient number of additional characters, to avoid truncation of long Cutter parts.

(I am not sure how one could avoid misunderstanding the year part for the Dewey decimal part without the inclusion of a non-splitting character, if not by modifying Dewey.pm, which is definitely not what I am suggesting here.)

Many thanks!
Comment 1 giuseppe.angilella 2014-06-21 14:57:40 UTC
I've actually tried to modify the database, with

alter table items modify cn_sort varchar(40) default null;

Although this does enlarge the column's size (without seemingly affecting its present content), new cn_sort's still have exactly 30 characters at most, which seems to imply that Dewey.pm should also be modified.

If that's right, this makes it a Cataloging issue, rather than a Database issue, and I am going to change the Component above of this bug or feature accordingly.
Comment 2 giuseppe.angilella 2014-06-22 06:26:42 UTC
After some trial and error, I've found out that it was not Dewey.pm which needs be modified, but /usr/share/koha/lib/C4/ClassSortRoutine.pm , where the dimension of the "key" variable is set to 30: it must be increased to 40, to comply with my previous change in the database:

    return substr($key, 0, 40);

Summarizing, my sorting issue was solved by:

1. attaching the year part to the Cutter part by an underscore; this prevents Dewey.pm from treating the year part as the "second numeric part", and padding it with zeroes therefore;

1a. If the same author (author with the same Cutter) has other items, with record-specific Cutters, such as F435wh, these must be re-Cuttered as F435_wh;

2. items.cn_sort must be increased to 40 characters (modify Koha's database as described above);

3. the length of $key must also be increased to 40 (modify ClassSortRoutine.pm as described above).
Comment 3 Eric Bégin 2014-08-01 14:48:49 UTC
Do you foresee problem if we were increasing the length of cn_sort to 255 chars ?
Comment 4 giuseppe.angilella 2014-08-02 03:56:15 UTC
I wouldn't foresee any problem. Thanks for looking into that!
Comment 5 Chris Cormack 2014-08-02 07:52:21 UTC
I think this accidentally got switch to needs signoff, without having a patched attached to sign off on.
Comment 6 simith.doliveira 2014-08-06 12:42:32 UTC Comment hidden (obsolete)
Comment 7 giuseppe.angilella 2014-08-08 14:47:35 UTC
I tried to test this patch with the sandbox system, but it does not work. Here is the result I've received from biblibre:

The sandbox you've requested is not ready.
Some problems occurred applying patches from bug 12424:
<h1>Something went wrong !</h1>Applying: Bug 12424 - ddc sorting of call
numbers truncates long Cutter parts
Using index info to reconstruct a base tree...
Falling back to patching base and 3-way merge...
Auto-merging installer/data/mysql/updatedatabase.pl
CONFLICT (content): Merge conflict in installer/data/mysql/updatedatabase.pl
Failed to merge in the changes.
Patch failed at 0001 Bug 12424 - ddc sorting of call numbers truncates long
Cutter parts
When you have resolved this problem run git bz apply --continue.
If you would prefer to skip this patch, instead run git bz apply --skip.
To restore the original branch and stop patching run git bz apply --abort.
Bug 12424 - ddc sorting of call numbers truncates long Cutter parts

30553 - Bug 12424 - ddc sorting of call numbers truncates long Cutter parts

Apply? [(y)es, (n)o, (i)nteractive] Patch left in
/tmp/Bug-12424---ddc-sorting-of-call-numbers-truncates--1zbYCL.patch .
Comment 8 simith.doliveira 2014-08-12 13:19:53 UTC Comment hidden (obsolete)
Comment 9 Biblibre Sandboxes 2014-08-12 15:54:31 UTC
Patch tested with a sandbox, by Giuseppe Angilella <giuseppe.angilella@ct.infn.it>
Comment 10 Biblibre Sandboxes 2014-08-12 15:54:52 UTC Comment hidden (obsolete)
Comment 11 giuseppe.angilella 2014-08-12 16:04:20 UTC
It seems to work: thank you very much for your efforts!

Note: It is still necessary to include an underscore between the Cutter part and the year (or any further numerical string, other than the Dewey number proper, not prepended by an alphabetic character), i.e. "F435_1996" rather than "F435 1996".
Comment 12 Katrin Fischer 2014-08-17 21:06:42 UTC
This patch is looking good, no problems spotted in code review. 

But there is one thing that I am not sure about: There is another cn_sort field in biblioitems and deletedbiblioitems that I feel should also be updated to avoid problems and make them all match. Could you provide a follow-up?

Leaving this as Signed off for now.
Comment 13 simith.doliveira 2014-08-21 18:48:46 UTC Comment hidden (obsolete)
Comment 14 Katrin Fischer 2014-08-23 17:40:02 UTC
Hi Simith, 
this would have been better as a separate follow-up patch (follow-up usually meaning a new and separate patch that applies on the alreay existing patches). This way you could have kept the sign-off from Giuseppe.
Comment 15 Katrin Fischer 2014-08-23 17:46:01 UTC
Ah, I had missed you switched back to 'needs sign-off' - so it's ok. Just trying to help with procedures - if QA asks for something, most of the time you can keep the 'signed off' status with a separate patch if the change is not too big.
Comment 16 Chris Cormack 2014-10-15 16:09:15 UTC Comment hidden (obsolete)
Comment 17 Katrin Fischer 2014-10-16 13:14:29 UTC
Created attachment 32407 [details] [review]
[PASSED QA] Bug 12424 - ddc sorting of call numbers truncates long Cutter parts

This patch increases the cn_sort field length to 255 chars.

Modified:

C4/ClassSortRoutine.pm                    - change the hardcoded cn_sort length to 255
installer/data/mysql/kohastructure.sql    - alter tables items and deleteditems,
                                            biblioitems and deletedbiblioitems
installer/data/mysql/updatedatabase.pl    - alter tables items and deleteditems,
                                            biblioitems and deletedbiblioitems

Testing:

Before applying the patch:
0) Select an item
1) Edit the item selected
2) Change "Full call number" option to 530 F435_1996 v2p1
3) Save changes
4) Valide 530_000000000000000_F435_1996_ (table items - cn_sort column).

After applying the patch:
5) Edit again the item selected in 0
3) Save changes
4) Valide 530_000000000000000_F435_1996_V2P1 (table items - cn_sort column).

Signed-off-by: Chris Cormack <chris@bigballofwax.co.nz>
Signed-off-by: Katrin Fischer <Katrin.Fischer.83@web.de>
Passes tests and QA script.
Works as described.
Comment 18 Katrin Fischer 2014-10-16 13:14:53 UTC
Thx for your patience!
Comment 19 Tomás Cohen Arazi 2014-10-18 13:58:08 UTC
Patch pushed to master.

Thanks Simith!