Bug 12375 - Store serials enumeration data in separate fields
Summary: Store serials enumeration data in separate fields
Status: CLOSED FIXED
Alias: None
Product: Koha
Classification: Unclassified
Component: Serials (show other bugs)
Version: master
Hardware: All All
: P5 - low enhancement (vote)
Assignee: Kyle M Hall
QA Contact: Testopia
URL:
Keywords:
: 8956 (view as bug list)
Depends on:
Blocks: 13420 16678 16692 17292
  Show dependency treegraph
 
Reported: 2014-06-05 16:30 UTC by Kyle M Hall
Modified: 2018-06-04 20:10 UTC (History)
11 users (show)

See Also:
Change sponsored?: ---
Patch complexity: ---
Documentation contact:
Documentation submission:
Text to go in the release notes:
Version(s) released in:


Attachments
Bug 12375 [1] - Update database (2.77 KB, patch)
2014-06-05 16:33 UTC, Kyle M Hall
Details | Diff | Splinter Review
Bug 12375 [2] - Update Serial schema file (3.27 KB, patch)
2014-06-05 16:33 UTC, Kyle M Hall
Details | Diff | Splinter Review
Bug 12375 [3] - Store serials enumeration data in separate fields (4.34 KB, patch)
2014-06-05 16:33 UTC, Kyle M Hall
Details | Diff | Splinter Review
Bug 12375 [4] - Replace use of DBI with Koha::Database in C4::Serials::NewIssue (2.43 KB, patch)
2014-06-05 16:33 UTC, Kyle M Hall
Details | Diff | Splinter Review
Bug 12375 [4] - Replace use of DBI with Koha::Database in C4::Serials::NewIssue (3.12 KB, patch)
2014-06-05 16:36 UTC, Kyle M Hall
Details | Diff | Splinter Review
Bug 12375 [1] - Update database (2.63 KB, patch)
2014-06-05 16:51 UTC, Kyle M Hall
Details | Diff | Splinter Review
Bug 12375 [2] - Update Serial schema file (3.27 KB, patch)
2014-06-05 16:51 UTC, Kyle M Hall
Details | Diff | Splinter Review
Bug 12375 [3] - Store serials enumeration data in separate fields (4.34 KB, patch)
2014-06-05 16:51 UTC, Kyle M Hall
Details | Diff | Splinter Review
Bug 12375 [4] - Replace use of DBI with Koha::Database in C4::Serials::NewIssue (3.12 KB, patch)
2014-06-05 16:51 UTC, Kyle M Hall
Details | Diff | Splinter Review
Bug 12375 [1] - Update database (2.68 KB, patch)
2014-06-13 15:21 UTC, Biblibre Sandboxes
Details | Diff | Splinter Review
Bug 12375 [2] - Update Serial schema file (3.32 KB, patch)
2014-06-13 15:21 UTC, Biblibre Sandboxes
Details | Diff | Splinter Review
Bug 12375 [3] - Store serials enumeration data in separate fields (4.39 KB, patch)
2014-06-13 15:21 UTC, Biblibre Sandboxes
Details | Diff | Splinter Review
Bug 12375 [4] - Replace use of DBI with Koha::Database in C4::Serials::NewIssue (3.17 KB, patch)
2014-06-13 15:21 UTC, Biblibre Sandboxes
Details | Diff | Splinter Review
28846 - Bug 12375 [5] [QA Followup] - Don't assume X Y and Z are used in order (3.05 KB, patch)
2014-07-21 14:06 UTC, Kyle M Hall
Details | Diff | Splinter Review
Bug 12375 [5] [QA Followup] - Don't assume X Y and Z are used in order (3.04 KB, patch)
2014-07-21 14:07 UTC, Kyle M Hall
Details | Diff | Splinter Review
Bug 12375 [1] - Update database (2.68 KB, patch)
2014-07-25 13:31 UTC, Kyle M Hall
Details | Diff | Splinter Review
Bug 12375 [2] - Update Serial schema file (3.32 KB, patch)
2014-07-25 13:32 UTC, Kyle M Hall
Details | Diff | Splinter Review
Bug 12375 [3] - Store serials enumeration data in separate fields (4.39 KB, patch)
2014-07-25 13:32 UTC, Kyle M Hall
Details | Diff | Splinter Review
Bug 12375 [4] - Replace use of DBI with Koha::Database in C4::Serials::NewIssue (3.17 KB, patch)
2014-07-25 13:32 UTC, Kyle M Hall
Details | Diff | Splinter Review
Bug 12375 [5] [QA Followup] - Don't assume X Y and Z are used in order (3.04 KB, patch)
2014-07-25 13:32 UTC, Kyle M Hall
Details | Diff | Splinter Review
Bug 12375 [6] [QA Followup] - Supress warnings for undefined indexes (1.19 KB, patch)
2014-07-25 13:32 UTC, Kyle M Hall
Details | Diff | Splinter Review
Bug 12375: QA followup - remove trailing whitespace (1.01 KB, patch)
2014-07-25 13:52 UTC, Julian Maurice
Details | Diff | Splinter Review
Bug 12375 [1] - Update database (2.75 KB, patch)
2015-01-13 13:49 UTC, Kyle M Hall
Details | Diff | Splinter Review
Bug 12375 [2] - Update Serial schema file (2.00 KB, patch)
2015-01-13 13:49 UTC, Kyle M Hall
Details | Diff | Splinter Review
Bug 12375 [3] - Store serials enumeration data in separate fields (4.39 KB, patch)
2015-01-13 13:49 UTC, Kyle M Hall
Details | Diff | Splinter Review
Bug 12375 [4] - Replace use of DBI with Koha::Database in C4::Serials::NewIssue (3.17 KB, patch)
2015-01-13 13:49 UTC, Kyle M Hall
Details | Diff | Splinter Review
Bug 12375 [5] [QA Followup] - Don't assume X Y and Z are used in order (3.04 KB, patch)
2015-01-13 13:49 UTC, Kyle M Hall
Details | Diff | Splinter Review
Bug 12375 [6] [QA Followup] - Supress warnings for undefined indexes (1.19 KB, patch)
2015-01-13 13:49 UTC, Kyle M Hall
Details | Diff | Splinter Review
Bug 12375: QA followup - remove trailing whitespace (1.02 KB, patch)
2015-01-13 13:49 UTC, Kyle M Hall
Details | Diff | Splinter Review
Bug 12375 [1] - Update database (3.03 KB, patch)
2015-06-29 11:58 UTC, Kyle M Hall
Details | Diff | Splinter Review
Bug 12375 [2] - Update Serial schema file (2.00 KB, patch)
2015-06-29 11:58 UTC, Kyle M Hall
Details | Diff | Splinter Review
Bug 12375 [3] - Store serials enumeration data in separate fields (4.42 KB, patch)
2015-06-29 11:58 UTC, Kyle M Hall
Details | Diff | Splinter Review
Bug 12375 [4] - Replace use of DBI with Koha::Database in C4::Serials::NewIssue (3.24 KB, patch)
2015-06-29 11:58 UTC, Kyle M Hall
Details | Diff | Splinter Review
Bug 12375 [5] [QA Followup] - Don't assume X Y and Z are used in order (3.04 KB, patch)
2015-06-29 11:58 UTC, Kyle M Hall
Details | Diff | Splinter Review
Bug 12375 [6] [QA Followup] - Supress warnings for undefined indexes (1.19 KB, patch)
2015-06-29 11:58 UTC, Kyle M Hall
Details | Diff | Splinter Review
Bug 12375: QA followup - remove trailing whitespace (1.02 KB, patch)
2015-06-29 11:58 UTC, Kyle M Hall
Details | Diff | Splinter Review
Bug 12375 [1] - Update database (2.97 KB, patch)
2015-12-31 12:31 UTC, Kyle M Hall
Details | Diff | Splinter Review
Bug 12375 [2] - Update Serial schema file (1.99 KB, patch)
2015-12-31 12:31 UTC, Kyle M Hall
Details | Diff | Splinter Review
Bug 12375 [3] - Store serials enumeration data in separate fields (4.52 KB, patch)
2015-12-31 12:31 UTC, Kyle M Hall
Details | Diff | Splinter Review
Bug 12375 [4] - Replace use of DBI with Koha::Database in C4::Serials::NewIssue (3.23 KB, patch)
2015-12-31 12:31 UTC, Kyle M Hall
Details | Diff | Splinter Review
Bug 12375 [5] [QA Followup] - Don't assume X Y and Z are used in order (3.04 KB, patch)
2015-12-31 12:31 UTC, Kyle M Hall
Details | Diff | Splinter Review
Bug 12375 [6] [QA Followup] - Supress warnings for undefined indexes (1.19 KB, patch)
2015-12-31 12:31 UTC, Kyle M Hall
Details | Diff | Splinter Review
Bug 12375 [7] - Update to use Koha::Object classes (11.93 KB, patch)
2015-12-31 12:31 UTC, Kyle M Hall
Details | Diff | Splinter Review
Bug 12375 - Add new serial fields to kohastructure (1.37 KB, patch)
2016-01-06 12:18 UTC, Kyle M Hall
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description Kyle M Hall 2014-06-05 16:30:29 UTC
Currently, all serials enumeration data is stored in conglomerated fashion in serial.serialseq. This makes it extremely difficult to do any reporting based on the individual sequence values due to this fact. In addition to the formatted version of the sequence, we should be storing the individual values as well.
Comment 1 Kyle M Hall 2014-06-05 16:32:41 UTC
*** Bug 8956 has been marked as a duplicate of this bug. ***
Comment 2 Kyle M Hall 2014-06-05 16:33:37 UTC Comment hidden (obsolete)
Comment 3 Kyle M Hall 2014-06-05 16:33:39 UTC Comment hidden (obsolete)
Comment 4 Kyle M Hall 2014-06-05 16:33:41 UTC Comment hidden (obsolete)
Comment 5 Kyle M Hall 2014-06-05 16:33:43 UTC Comment hidden (obsolete)
Comment 6 Kyle M Hall 2014-06-05 16:36:53 UTC Comment hidden (obsolete)
Comment 7 Kyle M Hall 2014-06-05 16:51:21 UTC Comment hidden (obsolete)
Comment 8 Kyle M Hall 2014-06-05 16:51:37 UTC Comment hidden (obsolete)
Comment 9 Kyle M Hall 2014-06-05 16:51:40 UTC Comment hidden (obsolete)
Comment 10 Kyle M Hall 2014-06-05 16:51:42 UTC Comment hidden (obsolete)
Comment 11 Biblibre Sandboxes 2014-06-13 15:20:47 UTC
Patch tested with a sandbox, by Paul Landers <paul.landers@ttuhsc.edu>
Comment 12 Biblibre Sandboxes 2014-06-13 15:21:27 UTC Comment hidden (obsolete)
Comment 13 Biblibre Sandboxes 2014-06-13 15:21:30 UTC Comment hidden (obsolete)
Comment 14 Biblibre Sandboxes 2014-06-13 15:21:32 UTC Comment hidden (obsolete)
Comment 15 Biblibre Sandboxes 2014-06-13 15:21:35 UTC Comment hidden (obsolete)
Comment 16 Julian Maurice 2014-06-16 14:10:48 UTC
If a numbering pattern uses {X}, {X}{Y} or {X}{Y}{Z}, in this order, then the updatedatabase works.
But if I have a numbering pattern that uses only {Z} for example, then the Z part will be stored in serialseq_x (and future serials will have the Z part in serialseq_z column)
If my numbering pattern uses only {X} and {Z}, the Z part will be in serialseq_y.
I think this have to be fixed, so failing QA here.

Also the original serialseq column is not removed. Is it intended ? I think we should not keep values in database that can be easily generated on the fly.
Comment 17 Kyle M Hall 2014-07-21 14:06:59 UTC Comment hidden (obsolete)
Comment 18 Kyle M Hall 2014-07-21 14:07:36 UTC Comment hidden (obsolete)
Comment 19 Kyle M Hall 2014-07-21 14:13:08 UTC
(In reply to Julian Maurice from comment #16)
> If a numbering pattern uses {X}, {X}{Y} or {X}{Y}{Z}, in this order, then
> the updatedatabase works.
> But if I have a numbering pattern that uses only {Z} for example, then the Z
> part will be stored in serialseq_x (and future serials will have the Z part
> in serialseq_z column)
> If my numbering pattern uses only {X} and {Z}, the Z part will be in
> serialseq_y.
> I think this have to be fixed, so failing QA here.

Fixed! The attached qa followup should allow the database update to handle these situations.

> Also the original serialseq column is not removed. Is it intended ? I think
> we should not keep values in database that can be easily generated on the
> fly.

We need to keep the serialseq column, at least for the time being. Right now the new columns are for reporting. In the future we can also use them to improve the sorting of serials. In the long run we may want to deprecate serialseq, but each library would need to go through all their serials to verify the sequence values are correct before upgrading ( that is, the values may have been migrated from a previous ILS and need to be updated ).
Comment 20 Julian Maurice 2014-07-25 11:52:18 UTC
Ok, it works well now, but it generates a lot of warnings

Use of uninitialized value $indexes[1] in hash element at installer/data/mysql/updatedatabase.pl line 8605.
Use of uninitialized value $indexes[2] in hash element at installer/data/mysql/updatedatabase.pl line 8605.
...

line 8605 for me is...

(
    undef,
    $enumeration_data{ $indexes[0] },  # <= ... here
    $enumeration_data{ $indexes[1] },
    $enumeration_data{ $indexes[2] }
) = split( /__SPLIT__/, $serialseq );

Could you fix that, please ?
Comment 21 Kyle M Hall 2014-07-25 13:31:54 UTC Comment hidden (obsolete)
Comment 22 Kyle M Hall 2014-07-25 13:32:08 UTC Comment hidden (obsolete)
Comment 23 Kyle M Hall 2014-07-25 13:32:10 UTC Comment hidden (obsolete)
Comment 24 Kyle M Hall 2014-07-25 13:32:13 UTC Comment hidden (obsolete)
Comment 25 Kyle M Hall 2014-07-25 13:32:16 UTC Comment hidden (obsolete)
Comment 26 Kyle M Hall 2014-07-25 13:32:18 UTC Comment hidden (obsolete)
Comment 27 Kyle M Hall 2014-07-25 13:33:48 UTC
This new followup should suppress those warnings. Please give it a try!

(In reply to Julian Maurice from comment #20)
> Ok, it works well now, but it generates a lot of warnings
> 
> Use of uninitialized value $indexes[1] in hash element at
> installer/data/mysql/updatedatabase.pl line 8605.
> Use of uninitialized value $indexes[2] in hash element at
> installer/data/mysql/updatedatabase.pl line 8605.
> ...
> 
> line 8605 for me is...
> 
> (
>     undef,
>     $enumeration_data{ $indexes[0] },  # <= ... here
>     $enumeration_data{ $indexes[1] },
>     $enumeration_data{ $indexes[2] }
> ) = split( /__SPLIT__/, $serialseq );
> 
> Could you fix that, please ?
Comment 28 Julian Maurice 2014-07-25 13:52:10 UTC
Great! Thanks.
koha-qa is only complaining about a trailing whitespace in updatedatabase.pl which I'll fix in a follow-up patch.

Passed QA.
Comment 29 Julian Maurice 2014-07-25 13:52:23 UTC Comment hidden (obsolete)
Comment 30 Kyle M Hall 2015-01-13 13:49:34 UTC Comment hidden (obsolete)
Comment 31 Kyle M Hall 2015-01-13 13:49:42 UTC Comment hidden (obsolete)
Comment 32 Kyle M Hall 2015-01-13 13:49:45 UTC Comment hidden (obsolete)
Comment 33 Kyle M Hall 2015-01-13 13:49:48 UTC Comment hidden (obsolete)
Comment 34 Kyle M Hall 2015-01-13 13:49:51 UTC Comment hidden (obsolete)
Comment 35 Kyle M Hall 2015-01-13 13:49:54 UTC Comment hidden (obsolete)
Comment 36 Kyle M Hall 2015-01-13 13:49:57 UTC Comment hidden (obsolete)
Comment 37 Tomás Cohen Arazi 2015-06-25 13:27:16 UTC
Kyle, could you please rebase this one? The updatedatabase patch was easy, but C4/Serials.pm got a bit more complicated.
Comment 38 Kyle M Hall 2015-06-29 11:58:04 UTC Comment hidden (obsolete)
Comment 39 Kyle M Hall 2015-06-29 11:58:11 UTC Comment hidden (obsolete)
Comment 40 Kyle M Hall 2015-06-29 11:58:13 UTC Comment hidden (obsolete)
Comment 41 Kyle M Hall 2015-06-29 11:58:31 UTC Comment hidden (obsolete)
Comment 42 Kyle M Hall 2015-06-29 11:58:34 UTC Comment hidden (obsolete)
Comment 43 Kyle M Hall 2015-06-29 11:58:36 UTC Comment hidden (obsolete)
Comment 44 Kyle M Hall 2015-06-29 11:58:38 UTC Comment hidden (obsolete)
Comment 45 Kyle M Hall 2015-12-31 12:31:29 UTC
Created attachment 46122 [details] [review]
Bug 12375 [1] - Update database

Signed-off-by: Paul Landers <paul.landers@ttuhsc.edu>
Comment 46 Kyle M Hall 2015-12-31 12:31:38 UTC
Created attachment 46123 [details] [review]
Bug 12375 [2] - Update Serial schema file

Signed-off-by: Paul Landers <paul.landers@ttuhsc.edu>
Comment 47 Kyle M Hall 2015-12-31 12:31:41 UTC
Created attachment 46124 [details] [review]
Bug 12375 [3] - Store serials enumeration data in separate fields

Signed-off-by: Paul Landers <paul.landers@ttuhsc.edu>
Comment 48 Kyle M Hall 2015-12-31 12:31:43 UTC
Created attachment 46125 [details] [review]
Bug 12375 [4] - Replace use of DBI with Koha::Database in C4::Serials::NewIssue

Currently, all serials enumeration data is stored in conglomerated
fashion in serial.serialseq. This makes it extremely difficult to do any
reporting based on the individual sequence values due to this fact. In
addition to the formatted version of the sequence, we should be storing
the individual values as well.

Test Plan:
1) Apply this patch
2) Run updatedatabase.pl
3) Create a new subscription, and add serials to it. The new x y and
   z fields should be populated with the same data the replaces {X} {Y} and
   {Z} in the enumeration field.
4) Inspect some existing serials. The update script will have made a
   best effort to recover the existing enumeration data to store in
   separate fields.

Signed-off-by: Paul Landers <paul.landers@ttuhsc.edu>
Comment 49 Kyle M Hall 2015-12-31 12:31:46 UTC
Created attachment 46126 [details] [review]
Bug 12375 [5] [QA Followup] - Don't assume X Y and Z are used in order
Comment 50 Kyle M Hall 2015-12-31 12:31:49 UTC
Created attachment 46127 [details] [review]
Bug 12375 [6] [QA Followup] - Supress warnings for undefined indexes
Comment 51 Kyle M Hall 2015-12-31 12:31:52 UTC
Created attachment 46128 [details] [review]
Bug 12375 [7] - Update to use Koha::Object classes

This code was written before the introduction of Koha::Object(s)

Considering the triviality of updating the code, we should go ahead
and switch this code to use Koha::Object(s) now.
Comment 52 Kyle M Hall 2015-12-31 12:36:50 UTC
Pushed to master!
Comment 53 Kyle M Hall 2016-01-06 12:18:00 UTC
Created attachment 46319 [details] [review]
Bug 12375 - Add new serial fields to kohastructure
Comment 54 Nicole C. Engard 2016-01-27 19:18:56 UTC
*** Bug 13420 has been marked as a duplicate of this bug. ***
Comment 55 Jonathan Druart 2016-05-05 14:32:39 UTC
Jenkins raises a warning:

No method update! at /srv/jenkins/workspace/Koha_Master_D8/C4/Serials.pm line 1602.

It seems that ->update should be replaced with ->store
Comment 56 Jonathan Druart 2016-06-08 14:09:24 UTC
(In reply to Jonathan Druart from comment #55)
> Jenkins raises a warning:
> 
> No method update! at /srv/jenkins/workspace/Koha_Master_D8/C4/Serials.pm
> line 1602.
> 
> It seems that ->update should be replaced with ->store

Will be fixed by bug 16692