Bug 11368 - Add script to import Lexile scores
Summary: Add script to import Lexile scores
Status: CLOSED FIXED
Alias: None
Product: Koha
Classification: Unclassified
Component: Command-line Utilities (show other bugs)
Version: master
Hardware: All All
: P5 - low enhancement (vote)
Assignee: Kyle M Hall
QA Contact: Testopia
URL:
Whiteboard:
Keywords:
Depends on: 10500
Blocks:
  Show dependency treegraph
 
Reported: 2013-12-10 15:26 UTC by Kyle M Hall
Modified: 2016-06-21 21:39 UTC (History)
9 users (show)

See Also:
Change sponsored?: ---
Patch complexity: ---
Bot Control: ---
When did the bot last check this:
Who signed the patch off:
Text to go in the release notes:


Attachments
LexileTitlesTruncated.txt (534 bytes, text/plain)
2013-12-10 15:29 UTC, Kyle M Hall
Details
Bug 11368 - Add script to import Lexile scores (6.23 KB, patch)
2013-12-10 15:35 UTC, Kyle M Hall
Details | Diff | Splinter Review
Bug 11368 - Add script to import Lexile scores (7.68 KB, patch)
2014-05-08 12:38 UTC, Kyle M Hall
Details | Diff | Splinter Review
Bug 11368 - Add script to import Lexile scores (7.72 KB, patch)
2015-03-04 08:28 UTC, Mirko Tietgen
Details | Diff | Splinter Review
Bug 11368 - Add script to import Lexile scores (7.73 KB, patch)
2015-05-14 12:42 UTC, Kyle M Hall
Details | Diff | Splinter Review
Bug 11368 [QA Followup] (3.41 KB, patch)
2015-05-14 12:42 UTC, Kyle M Hall
Details | Diff | Splinter Review
Bug 11368 [QA Followup 2] (1.69 KB, patch)
2015-09-16 14:34 UTC, Kyle M Hall
Details | Diff | Splinter Review
Bug 11368 - Add script to import Lexile scores (7.71 KB, patch)
2015-09-17 10:10 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 11368 [QA Followup] (3.40 KB, patch)
2015-09-17 10:10 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 11368 [QA Followup 2] (1.75 KB, patch)
2015-09-17 10:10 UTC, Jonathan Druart
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description Kyle M Hall 2013-12-10 15:26:52 UTC
Koha needs a script to automate the importing of Lexile score data for titles that have available scores but are not currently in the title's record.
Comment 1 Kyle M Hall 2013-12-10 15:29:03 UTC
Created attachment 23379 [details]
LexileTitlesTruncated.txt
Comment 2 Kyle M Hall 2013-12-10 15:35:44 UTC Comment hidden (obsolete)
Comment 3 Kyle M Hall 2014-05-08 12:38:58 UTC Comment hidden (obsolete)
Comment 4 Galen Charlton 2014-05-08 15:36:45 UTC
Out of curiosity, who provides Lexile scores in that format?
Comment 5 Mirko Tietgen 2015-03-04 08:28:08 UTC Comment hidden (obsolete)
Comment 6 Katrin Fischer 2015-04-06 11:39:53 UTC
Hi Kyle,

0) The QA script fails:

 FAIL	misc/migration_tools/import_lexile.pl
   FAIL	  critic
		I/O layer ":utf8" used at line 39, column 1. Use ":encoding(UTF-8)" to get strict validation.
   FAIL	  forbidden patterns
		forbidden pattern: Koha is now under the GPLv3 license (line 9)
   OK	  pod
   OK	  valid

1) Is copyright 2013 correct?

2) I am not sure what one of the options means from the description, could you clarify?
+    --source-value          Defines the value to put stored in the "Source" subfield ( default: "Lexile" )

3) The documentation for --file says: "acquired from Lexile.com", but the actual file format is not documented. I think maybe we need to decide here if this is a vendor specific or a general purpose script. If it's vendor specific, we might want to reflect that more clearly in the file name and description. If it's general purpose, we should include notes about the file format to make this easier to reuse.

4) We didn't really finish our discussions about the standard options for CLI scripts (http://wiki.koha-community.org/wiki/Coding_Guidelines#Command-line_argument_conventions), but maybe this should have a -c for confirming the action as it's going to change data?
Comment 7 Kyle M Hall 2015-05-14 12:38:39 UTC
> 0) The QA script fails:
> 
>  FAIL	misc/migration_tools/import_lexile.pl
>    FAIL	  critic
> 		I/O layer ":utf8" used at line 39, column 1. Use ":encoding(UTF-8)" to get
> strict validation.
>    FAIL	  forbidden patterns
> 		forbidden pattern: Koha is now under the GPLv3 license (line 9)
>    OK	  pod
>    OK	  valid

Will fix!

> 1) Is copyright 2013 correct?

Will fix!
> 
> 2) I am not sure what one of the options means from the description, could
> you clarify?
> +    --source-value          Defines the value to put stored in the "Source"
> subfield ( default: "Lexile" )

Please take a look at http://www.loc.gov/marc/bibliographic/bd521.html

Basically, 521 fields can store all types of target audience data. 521$a stores the 'rating' while 521$b stores the type of the rating. In this case, the rating is a Lexile score.
 
> 3) The documentation for --file says: "acquired from Lexile.com", but the
> actual file format is not documented. I think maybe we need to decide here
> if this is a vendor specific or a general purpose script. If it's vendor
> specific, we might want to reflect that more clearly in the file name and
> description. If it's general purpose, we should include notes about the file
> format to make this easier to reuse.

I'll add the file format to the help. This is meant specifically for importing Lexile.com files ( as they own and control Lexile rankings ) but this script is general purpose enough to import any CSV file that matches the schema.

> 
> 4) We didn't really finish our discussions about the standard options for
> CLI scripts
> (http://wiki.koha-community.org/wiki/Coding_Guidelines#Command-
> line_argument_conventions), but maybe this should have a -c for confirming
> the action as it's going to change data?

I'll add the -c and -t options.
Comment 8 Kyle M Hall 2015-05-14 12:42:22 UTC Comment hidden (obsolete)
Comment 9 Kyle M Hall 2015-05-14 12:42:29 UTC Comment hidden (obsolete)
Comment 10 Mirko Tietgen 2015-05-14 18:46:03 UTC
(In reply to Katrin Fischer from comment #6)

> If it's vendor
> specific, we might want to reflect that more clearly in the file name and
> description. If it's general purpose, we should include notes about the file
> format to make this easier to reuse.

I thought about broadening the scope of the script when I signed off on the patch, checking with the table for Antolin (German reading platform with quizzes for school kids, run by a publisher).

I don't think it's feasible to guess all possible fields and options that may occur in such lists, and given the non-standardized nature of these things, you never know if a format will change at some point.

It is easily possible to tweak the script if you know what you need, so I'd say keeping it vendor specific and having more vendor specific scripts of this nature if needed makes sense. This patch is a good starting point for similar tasks.
Comment 11 Jonathan Druart 2015-05-26 15:40:14 UTC
Comment on attachment 39182 [details] [review]
Bug 11368 - Add script to import Lexile scores

Review of attachment 39182 [details] [review]:
-----------------------------------------------------------------

I get this warning:
DBIx::Class::ResultSet::search_rs(): search( %condition ) is deprecated, use search( \%condition ) instead at misc/migration_tools/import_lexile.pl line 147

::: misc/migration_tools/import_lexile.pl
@@ +65,5 @@
> +    'source=s'               => $subfield_source,
> +    'source-value=s'         => $subfield_source_value,
> +);
> +
> +my $usage = << 'ENDUSAGE';

Should not we use Pod::Usage?

@@ +82,5 @@
> +
> +ENDUSAGE
> +
> +unless ($file) {
> +    say $usage;

Prefer to call pod2usage.

@@ +88,5 @@
> +}
> +
> +my $schema = Koha::Database->new()->schema();
> +
> +my $csv = Text::CSV->new( { binary => 1, sep_char => "\t" } )

Shouldn't we use the pref "separator" and add an option to specify another one?

@@ +121,5 @@
> +            push( @isbns, $row->{$_} );
> +            eval { push( @isbns, GetVariationsOfISBN( $row->{$_} ) ) };
> +        }
> +    }
> +    @isbns = grep( $_, @isbns );

Not sure to undestand what this does :)

@@ +142,5 @@
> +
> +        if ($verbose) {
> +            say "Found matching record! Biblionumber: $biblionumber";
> +
> +            if ( $verbose > 2 ) {

You should mention this in the POD.
Comment 12 Kyle M Hall 2015-09-16 14:34:08 UTC Comment hidden (obsolete)
Comment 13 Kyle M Hall 2015-09-16 14:36:53 UTC
(In reply to Jonathan Druart from comment #11)
> Comment on attachment 39182 [details] [review] [review]
> Bug 11368 - Add script to import Lexile scores
> 
> Review of attachment 39182 [details] [review] [review]:
> -----------------------------------------------------------------
> 
> I get this warning:
> DBIx::Class::ResultSet::search_rs(): search( %condition ) is deprecated, use
> search( \%condition ) instead at misc/migration_tools/import_lexile.pl line
> 147

Fixed!

> 
> ::: misc/migration_tools/import_lexile.pl
> @@ +65,5 @@
> > +    'source=s'               => $subfield_source,
> > +    'source-value=s'         => $subfield_source_value,
> > +);
> > +
> > +my $usage = << 'ENDUSAGE';
> 
> Should not we use Pod::Usage?
> 
> @@ +82,5 @@
> > +
> > +ENDUSAGE
> > +
> > +unless ($file) {
> > +    say $usage;
> 
> Prefer to call pod2usage.

I don't think the developer guidelines require pod2usage. That being said, I'd welcome a followup!

> @@ +88,5 @@
> > +}
> > +
> > +my $schema = Koha::Database->new()->schema();
> > +
> > +my $csv = Text::CSV->new( { binary => 1, sep_char => "\t" } )
> 
> Shouldn't we use the pref "separator" and add an option to specify another
> one?

The file format is fixed with tabs as the separator, so we should definitely not use the syspref here.

> @@ +121,5 @@
> > +            push( @isbns, $row->{$_} );
> > +            eval { push( @isbns, GetVariationsOfISBN( $row->{$_} ) ) };
> > +        }
> > +    }
> > +    @isbns = grep( $_, @isbns );
> 
> Not sure to undestand what this does :)

This line filters out any 'empty' isbns.

> @@ +142,5 @@
> > +
> > +        if ($verbose) {
> > +            say "Found matching record! Biblionumber: $biblionumber";
> > +
> > +            if ( $verbose > 2 ) {
> 
> You should mention this in the POD.

Done!
Comment 14 Jonathan Druart 2015-09-17 10:10:21 UTC
Created attachment 42641 [details] [review]
Bug 11368 - Add script to import Lexile scores

Koha needs a script to automate the importing of Lexile score data for
titles that have available scores but are not currently in the title's
record.

This script will take a CSV file of Lexile scores, and locate any
matching records in the Koha database ( by ISBN ). If the record already
has a score, it will be updated. If not, the Lexile score field will be
created.

Test Plan:
1) Apply this patch
2) Catalog a record for each of the following ISBNs:
   0789170191
   9780673779410
3) Download the file LexileTitlesTruncated.txt attached
   to this bug report
4) Run the script from the command line:
   ./misc/migraction_tools/import_lexile.pl -v --file /path/to/LexileTitlesTruncated.txt
5) View those records in Koha
6) Note those records now have valid Lexile scores
7) Edit the Lexile score ( 521$a ) and change the value to something else
8) Repeat step 4
9) Note the original Lexile score has been restored

Signed-off-by: Mirko Tietgen <mirko@abunchofthings.net>
Comment 15 Jonathan Druart 2015-09-17 10:10:24 UTC
Created attachment 42642 [details] [review]
Bug 11368 [QA Followup]

* Fix QA failures
* Fix copyright
* Add file format documentation
* Add -c --confirm option
* Add -t --test option
* Add -h --help option
Comment 16 Jonathan Druart 2015-09-17 10:10:28 UTC
Created attachment 42643 [details] [review]
Bug 11368 [QA Followup 2]

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Comment 17 Tomás Cohen Arazi 2015-09-18 15:48:07 UTC
Patches pushed to master.

Thanks Kyle!