The totalissues column in the biblioitems table appears to not be populated. I have tested on my system and had another Koha user test on theirs. This column should be populated or it should be removed if it has been replaced with the items.issues column. Nicole
Nicole, The funny thing is, I'm discovering, that the *index* of this is what's used to derive popularity sorting the in OPAC. So it's used--but nothing in Koha ever sets it. I can set it in migration scripts easily (filling 942$0 with the sum of the 952$l values), but nothing will ever update that value in Koha.
Yikes! That's not good at all. Upping the priority a bit here.
Part 1. Basic description. Although Koha has had the option to sort search results by “popularity” for years, the feature was never fully implemented. A working popularity sort would allow patrons (and staff) to view their search results sorted according to which items are most popular. In order to implement this feature 942$0 (total issues) will need to be automatically updated when items are checked out, and the indexing configuration adjusted slightly to take advantage of popularity data. Part 2. Additional features. Several additional changes are required to take advantage of the improved popularity support: * A system preference is required to allow users to disable the automatic popularity index update * The DOM indexing configuration should be set up so that popularity can automatically be computed by summing the number of times each item was issued (this will be done on a separate bug report, since the DOM indexing configuration is dependent on bug 7818, and the rest of the work on this bug is not) * A script to populate field 942$0 with historical data is needed, since Koha has not been keeping 942$0 up to date
Created attachment 9765 [details] [review] Bug 6557: Syspref to control total issue counting Because updating the total issues count associated with a bibliographic record on issue could cause a significant load on the server, this commit adds the syspref UpdateTotalIssuesOnCirc (which defaults to OFF to match existing behavior). The syspref has the following description: Do/Do not update a bibliographic record's total issues count whenever an item is issued (WARNING! This increases server load significantly; if performance is a concern, use the update_totalissues.pl cron job to update the total issues count).
Created attachment 9766 [details] [review] Bug 6557: automatically increment totalissues Adds the ability to automatically increment biblioitems.totalissues whenever an item is issued. To test: 1) Choose a record with at least one item that can circulate 2) Check the value of 942$0 (you may need to look at the plain MARC view on the OPAC). Most likely there won't be any 942$0 at all 3) Enable UpdateTotalIssuesOnCirc 4) Check out the item you selected 5) Check the value of 942$0 (you may need to look at the plain MARC view on the OPAC). That value should now be one greater than before 6) Discharge the item 7) Disable UpdateTotalIssuesOnCirc 8) Check out the item you selected again 9) Check the value of 942$0 (you may need to look at the plain MARC view on the OPAC). That value should not have changed
Created attachment 9767 [details] [review] Bug 6557: add script to update totalissues from stats NAME update_totalissues.pl SYNOPSIS update_totalissues.pl --use-stats update_totalissues.pl --use-items update_totalissues.pl --commit=1000 update_totalissues.pl --since='2012-01-01' update_totalissues.pl --interval=30d DESCRIPTION This batch job populates bibliographic records' total issues count based on historical issue statistics. --help Prints this help -v|--verbose Provide verbose log information (list every bib modified). --use-stats Use the data in the statistics table for populating total issues. --use-items Use items.issues data for populating total issues. Note that issues data from the items table does not respect the --since or --interval options, by definition. Also note that if both --use-stats and --use-items are specified, the count of biblios processed will be misleading. -s|--since=DATE Only process issues recorded in the statistics table since DATE. -i|--interval=S Only process issues recorded in the statistics table in the last N units of time. The interval should consist of a number with a one-letter unit suffix. The valid suffixes are h (hours), d (days), w (weeks), m (months), and y (years). The default unit is days. --incremental Add the number of issues found in the statistics table to the existing total issues count. Intended so that this script can be used as a cron job to update popularity information during low-usage periods. --commit=N Commit the results to the database after every N records are processed. --test Only test the popularity population script. === TESTING PLAN === NOTE: in order to test this script, you will need to have some sort of circulation data already existing in your Koha installation. 1) Disable UpdateTotalIssuesOnCirc 2) Run: misc/cronjobs/update_total_issues.pl --use-items -t -v 3) If you have total checkout data in your item records (i.e. anything in 952$l), you should see messages like "Processing bib 43 (1 issues)" 4) Choose one of the lines that shows more than 0 issues, and view the record with that biblionumber in the staff client, choosing the "Items" tab (moredetail.pl). Add up the "Total checkouts" listed for each item, and confirm it matches what the script reported 5) Run: misc/cronjobs/update_total_issues.pl --use-stats -t -v 6) If you have any circulation statistics in your database (i.e. any 'issue' entries in your statistics table), you should see messages like "Processing bib 43 (1 issues)"; 7) Choose one of the lines and view the record with that biblionumber in the staff client, choosing the "Items" tab (moredetail.pl). If you count the number of checkouts listed in each item's checkout history, the total should match what the script reported. 8) Check out an item 9) Run: misc/cronjobs/update_total_issues.pl --use-stats --since=1h -t -v 10) You should see one line reporting a single circ for the bib record associated with the item you just checked out (there may be more if you checked out any books in the hour prior to running these tests 11) If the results in steps 4, 7, and 10 match the predictions, the script worked
Changing to normal because after some thought I've decided this could conceivably be safe for backporting.
I found a problem in step 6 of the testing protocol - I have issues in my DB but every item shows 0 issues when the script is run. Trivial problem - the documentation refers to update_total_issues.pl, the name of the script in the filesystem is update_totalissues.pl.
Created attachment 9957 [details] [review] Bug 6557: Syspref to control total issue counting Because updating the total issues count associated with a bibliographic record on issue could cause a significant load on the server, this commit adds the syspref UpdateTotalIssuesOnCirc (which defaults to OFF to match existing behavior). The syspref has the following description: Do/Do not update a bibliographic record's total issues count whenever an item is issued (WARNING! This increases server load significantly; if performance is a concern, use the update_totalissues.pl cron job to update the total issues count). This patch to Koha was sponsored by the Arcadia Public Library and the Arcadia Public Library Foundation in honor of Jackie Faust-Moreno, late director of the Arcadia Public Library.
Created attachment 9958 [details] [review] Bug 6557: automatically increment totalissues Adds the ability to automatically increment biblioitems.totalissues whenever an item is issued. To test: 1) Choose a record with at least one item that can circulate 2) Check the value of 942$0 (you may need to look at the plain MARC view on the OPAC). Most likely there won't be any 942$0 at all 3) Enable UpdateTotalIssuesOnCirc 4) Check out the item you selected 5) Check the value of 942$0 (you may need to look at the plain MARC view on the OPAC). That value should now be one greater than before 6) Discharge the item 7) Disable UpdateTotalIssuesOnCirc 8) Check out the item you selected again 9) Check the value of 942$0 (you may need to look at the plain MARC view on the OPAC). That value should not have changed This patch to Koha was sponsored by the Arcadia Public Library and the Arcadia Public Library Foundation in honor of Jackie Faust-Moreno, late director of the Arcadia Public Library.
Created attachment 9959 [details] [review] Bug 6557: add script to update totalissues from stats NAME update_totalissues.pl SYNOPSIS update_totalissues.pl --use-stats update_totalissues.pl --use-items update_totalissues.pl --commit=1000 update_totalissues.pl --since='2012-01-01' update_totalissues.pl --interval=30d DESCRIPTION This batch job populates bibliographic records' total issues count based on historical issue statistics. --help Prints this help -v|--verbose Provide verbose log information (list every bib modified). --use-stats Use the data in the statistics table for populating total issues. --use-items Use items.issues data for populating total issues. Note that issues data from the items table does not respect the --since or --interval options, by definition. Also note that if both --use-stats and --use-items are specified, the count of biblios processed will be misleading. -s|--since=DATE Only process issues recorded in the statistics table since DATE. -i|--interval=S Only process issues recorded in the statistics table in the last N units of time. The interval should consist of a number with a one-letter unit suffix. The valid suffixes are h (hours), d (days), w (weeks), m (months), and y (years). The default unit is days. --incremental Add the number of issues found in the statistics table to the existing total issues count. Intended so that this script can be used as a cron job to update popularity information during low-usage periods. --commit=N Commit the results to the database after every N records are processed. --test Only test the popularity population script. === TESTING PLAN === NOTE: in order to test this script, you will need to have some sort of circulation data already existing in your Koha installation. 1) Disable UpdateTotalIssuesOnCirc 2) Run: misc/cronjobs/update_totalissues.pl --use-items -t -v 3) If you have total checkout data in your item records (i.e. anything in 952$l), you should see messages like "Processing bib 43 (1 issues)" 4) Choose one of the lines that shows more than 0 issues, and view the record with that biblionumber in the staff client, choosing the "Items" tab (moredetail.pl). Add up the "Total checkouts" listed for each item, and confirm it matches what the script reported 5) Run: misc/cronjobs/update_totalissues.pl --use-stats -t -v 6) If you have any circulation statistics in your database (i.e. any 'issue' entries in your statistics table), you should see messages like "Processing bib 43 (1 issues)"; 7) Choose one of the lines and view the record with that biblionumber in the staff client, choosing the "Items" tab (moredetail.pl). If you count the number of checkouts listed in each item's checkout history, the total should match what the script reported. 8) Check out an item 9) Run: misc/cronjobs/update_totalissues.pl --use-stats --since=1h -t -v 10) You should see one line reporting a single circ for the bib record associated with the item you just checked out (there may be more if you checked out any books in the hour prior to running these tests 11) If the results in steps 4, 7, and 10 match the predictions, the script worked This patch to Koha was sponsored by the Arcadia Public Library and the Arcadia Public Library Foundation in honor of Jackie Faust-Moreno, late director of the Arcadia Public Library.
Created attachment 9961 [details] [review] Bug 6557: add script to update totalissues from stats NAME update_totalissues.pl SYNOPSIS update_totalissues.pl --use-stats update_totalissues.pl --use-items update_totalissues.pl --commit=1000 update_totalissues.pl --since='2012-01-01' update_totalissues.pl --interval=30d DESCRIPTION This batch job populates bibliographic records' total issues count based on historical issue statistics. --help Prints this help -v|--verbose Provide verbose log information (list every bib modified). --use-stats Use the data in the statistics table for populating total issues. --use-items Use items.issues data for populating total issues. Note that issues data from the items table does not respect the --since or --interval options, by definition. Also note that if both --use-stats and --use-items are specified, the count of biblios processed will be misleading. -s|--since=DATE Only process issues recorded in the statistics table since DATE. -i|--interval=S Only process issues recorded in the statistics table in the last N units of time. The interval should consist of a number with a one-letter unit suffix. The valid suffixes are h (hours), d (days), w (weeks), m (months), and y (years). The default unit is days. --incremental Add the number of issues found in the statistics table to the existing total issues count. Intended so that this script can be used as a cron job to update popularity information during low-usage periods. --commit=N Commit the results to the database after every N records are processed. --test Only test the popularity population script. === TESTING PLAN === NOTE: in order to test this script, you will need to have some sort of circulation data already existing in your Koha installation. 1) Disable UpdateTotalIssuesOnCirc 2) Run: misc/cronjobs/update_totalissues.pl --use-items -t -v 3) If you have total checkout data in your item records (i.e. anything in 952$l), you should see messages like "Processing bib 43 (1 issues)" 4) Choose one of the lines that shows more than 0 issues, and view the record with that biblionumber in the staff client, choosing the "Items" tab (moredetail.pl). Add up the "Total checkouts" listed for each item, and confirm it matches what the script reported 5) Run: misc/cronjobs/update_totalissues.pl --use-stats -t -v 6) If you have any circulation statistics in your database (i.e. any 'issue' entries in your statistics table), you should see messages like "Processing bib 43 (1 issues)"; 7) Choose one of the lines and view the record with that biblionumber in the staff client, choosing the "Items" tab (moredetail.pl). If you count the number of checkouts listed in each item's checkout history, the total should match what the script reported. 8) Check out an item 9) Run: misc/cronjobs/update_totalissues.pl --use-stats --incremental --since=1h -t -v 10) You should see one line reporting a single circ for the bib record associated with the item you just checked out (there may be more if you checked out any books in the hour prior to running these tests 11) If the results in steps 4, 7, and 10 match the predictions, the script worked This patch to Koha was sponsored by the Arcadia Public Library and the Arcadia Public Library Foundation in honor of Jackie Faust-Moreno, late director of the Arcadia Public Library.
Created attachment 9962 [details] [review] Bug 6557: add script to update totalissues from stats NAME update_totalissues.pl SYNOPSIS update_totalissues.pl --use-stats update_totalissues.pl --use-items update_totalissues.pl --commit=1000 update_totalissues.pl --since='2012-01-01' update_totalissues.pl --interval=30d DESCRIPTION This batch job populates bibliographic records' total issues count based on historical issue statistics. --help Prints this help -v|--verbose Provide verbose log information (list every bib modified). --use-stats Use the data in the statistics table for populating total issues. --use-items Use items.issues data for populating total issues. Note that issues data from the items table does not respect the --since or --interval options, by definition. Also note that if both --use-stats and --use-items are specified, the count of biblios processed will be misleading. -s|--since=DATE Only process issues recorded in the statistics table since DATE. -i|--interval=S Only process issues recorded in the statistics table in the last N units of time. The interval should consist of a number with a one-letter unit suffix. The valid suffixes are h (hours), d (days), w (weeks), m (months), and y (years). The default unit is days. --incremental Add the number of issues found in the statistics table to the existing total issues count. Intended so that this script can be used as a cron job to update popularity information during low-usage periods. --commit=N Commit the results to the database after every N records are processed. --test Only test the popularity population script. === TESTING PLAN === NOTE: in order to test this script, you will need to have some sort of circulation data already existing in your Koha installation. 1) Disable UpdateTotalIssuesOnCirc 2) Run: misc/cronjobs/update_totalissues.pl --use-items -t -v 3) If you have total checkout data in your item records (i.e. anything in 952$l), you should see messages like "Processing bib 43 (1 issues)" 4) Choose one of the lines that shows more than 0 issues, and view the record with that biblionumber in the staff client, choosing the "Items" tab (moredetail.pl). Add up the "Total checkouts" listed for each item, and confirm it matches what the script reported 5) Run: misc/cronjobs/update_totalissues.pl --use-stats -t -v 6) If you have any circulation statistics in your database (i.e. any 'issue' entries in your statistics table), you should see messages like "Processing bib 43 (1 issues)"; 7) Choose one of the lines and view the record with that biblionumber in the staff client, choosing the "Items" tab (moredetail.pl). If you count the number of checkouts listed in each item's checkout history, the total should match what the script reported. 8) Check out an item 9) Run: misc/cronjobs/update_totalissues.pl --use-stats --incremental --since=1h -t -v 10) You should see one line reporting a single circ for the bib record associated with the item you just checked out (there may be more if you checked out any books in the hour prior to running these tests 11) If the results in steps 4, 7, and 10 match the predictions, the script worked This patch to Koha was sponsored by the Arcadia Public Library and the Arcadia Public Library Foundation in honor of Jackie Faust-Moreno, late director of the Arcadia Public Library.
Created attachment 9963 [details] [review] Bug 6557: add script to update totalissues from stats NAME update_totalissues.pl SYNOPSIS update_totalissues.pl --use-stats update_totalissues.pl --use-items update_totalissues.pl --commit=1000 update_totalissues.pl --since='2012-01-01' update_totalissues.pl --interval=30d DESCRIPTION This batch job populates bibliographic records' total issues count based on historical issue statistics. --help Prints this help -v|--verbose Provide verbose log information (list every bib modified). --use-stats Use the data in the statistics table for populating total issues. --use-items Use items.issues data for populating total issues. Note that issues data from the items table does not respect the --since or --interval options, by definition. Also note that if both --use-stats and --use-items are specified, the count of biblios processed will be misleading. -s|--since=DATE Only process issues recorded in the statistics table since DATE. -i|--interval=S Only process issues recorded in the statistics table in the last N units of time. The interval should consist of a number with a one-letter unit suffix. The valid suffixes are h (hours), d (days), w (weeks), m (months), and y (years). The default unit is days. --incremental Add the number of issues found in the statistics table to the existing total issues count. Intended so that this script can be used as a cron job to update popularity information during low-usage periods. --commit=N Commit the results to the database after every N records are processed. --test Only test the popularity population script. === TESTING PLAN === NOTE: in order to test this script, you will need to have some sort of circulation data already existing in your Koha installation. 1) Disable UpdateTotalIssuesOnCirc 2) Run: misc/cronjobs/update_totalissues.pl --use-items -t -v 3) If you have total checkout data in your item records (i.e. anything in 952$l), you should see messages like "Processing bib 43 (1 issues)" 4) Choose one of the lines that shows more than 0 issues, and view the record with that biblionumber in the staff client, choosing the "Items" tab (moredetail.pl). Add up the "Total checkouts" listed for each item, and confirm it matches what the script reported 5) Run: misc/cronjobs/update_totalissues.pl --use-stats -t -v 6) If you have any circulation statistics in your database (i.e. any 'issue' entries in your statistics table), you should see messages like "Processing bib 43 (1 issues)"; 7) Choose one of the lines and view the record with that biblionumber in the staff client, choosing the "Items" tab (moredetail.pl). If you count the number of checkouts listed in each item's checkout history, the total should match what the script reported. 8) Check out an item 9) Run: misc/cronjobs/update_totalissues.pl --use-stats --incremental --interval=1h -t -v 10) You should see one line reporting a single circ for the bib record associated with the item you just checked out (there may be more if you checked out any books in the hour prior to running these tests 11) If the results in steps 4, 7, and 10 match the predictions, the script worked NOTE: The --interval argument takes an interval identifier like "1h" and the --since argument takes an actual date like "2012-06-06 12:00:00". Also note that if the time on your database server does not match the time on your Koha server, you will need to adjust the intervals. This patch to Koha was sponsored by the Arcadia Public Library and the Arcadia Public Library Foundation in honor of Jackie Faust-Moreno, late director of the Arcadia Public Library.
Created attachment 9964 [details] [review] Bug 6557: add script to update totalissues from stats NAME update_totalissues.pl SYNOPSIS update_totalissues.pl --use-stats update_totalissues.pl --use-items update_totalissues.pl --commit=1000 update_totalissues.pl --since='2012-01-01' update_totalissues.pl --interval=30d DESCRIPTION This batch job populates bibliographic records' total issues count based on historical issue statistics. --help Prints this help -v|--verbose Provide verbose log information (list every bib modified). --use-stats Use the data in the statistics table for populating total issues. --use-items Use items.issues data for populating total issues. Note that issues data from the items table does not respect the --since or --interval options, by definition. Also note that if both --use-stats and --use-items are specified, the count of biblios processed will be misleading. -s|--since=DATE Only process issues recorded in the statistics table since DATE. -i|--interval=S Only process issues recorded in the statistics table in the last N units of time. The interval should consist of a number with a one-letter unit suffix. The valid suffixes are h (hours), d (days), w (weeks), m (months), and y (years). The default unit is days. --incremental Add the number of issues found in the statistics table to the existing total issues count. Intended so that this script can be used as a cron job to update popularity information during low-usage periods. --commit=N Commit the results to the database after every N records are processed. --test Only test the popularity population script. === TESTING PLAN === NOTE: in order to test this script, you will need to have some sort of circulation data already existing in your Koha installation. 1) Disable UpdateTotalIssuesOnCirc 2) Run: misc/cronjobs/update_totalissues.pl --use-items -t -v 3) If you have total checkout data in your item records (i.e. anything in 952$l), you should see messages like "Processing bib 43 (1 issues)" 4) Choose one of the lines that shows more than 0 issues, and view the record with that biblionumber in the staff client, choosing the "Items" tab (moredetail.pl). Add up the "Total checkouts" listed for each item, and confirm it matches what the script reported 5) Run: misc/cronjobs/update_totalissues.pl --use-stats -t -v 6) If you have any circulation statistics in your database (i.e. any 'issue' entries in your statistics table), you should see messages like "Processing bib 43 (1 issues)"; 7) Choose one of the lines and view the record with that biblionumber in the staff client, choosing the "Items" tab (moredetail.pl). If you count the number of checkouts listed in each item's checkout history, the total should match what the script reported. 8) Check out an item 9) Run: misc/cronjobs/update_totalissues.pl --use-stats --incremental --interval=1h -t -v 10) You should see one line reporting a single circ for the bib record associated with the item you just checked out (there may be more if you checked out any books in the hour prior to running these tests 11) If the results in steps 4, 7, and 10 match the predictions, the script worked NOTE: The --interval argument takes an interval identifier like "1h" and the --since argument takes an actual date like "2012-06-06 12:00:00". Also note that if the time on your database server does not match the time on your Koha server, you will need to adjust the intervals. This patch to Koha was sponsored by the Arcadia Public Library and the Arcadia Public Library Foundation in honor of Jackie Faust-Moreno, late director of the Arcadia Public Library.
Created attachment 9965 [details] [review] Bug 6557: add script to update totalissues from stats NAME update_totalissues.pl SYNOPSIS update_totalissues.pl --use-stats update_totalissues.pl --use-items update_totalissues.pl --commit=1000 update_totalissues.pl --since='2012-01-01' update_totalissues.pl --interval=30d DESCRIPTION This batch job populates bibliographic records' total issues count based on historical issue statistics. --help Prints this help -v|--verbose Provide verbose log information (list every bib modified). --use-stats Use the data in the statistics table for populating total issues. --use-items Use items.issues data for populating total issues. Note that issues data from the items table does not respect the --since or --interval options, by definition. Also note that if both --use-stats and --use-items are specified, the count of biblios processed will be misleading. -s|--since=DATE Only process issues recorded in the statistics table since DATE. -i|--interval=S Only process issues recorded in the statistics table in the last N units of time. The interval should consist of a number with a one-letter unit suffix. The valid suffixes are h (hours), d (days), w (weeks), m (months), and y (years). The default unit is days. --incremental Add the number of issues found in the statistics table to the existing total issues count. Intended so that this script can be used as a cron job to update popularity information during low-usage periods. If neither --since or --interval are specified, incremental mode will default to processing the last twenty-four hours. --commit=N Commit the results to the database after every N records are processed. --test Only test the popularity population script. WARNING If the time on your database server does not match the time on your Koha server you will need to take that into account, and probably use the --since argument instead of the --interval argument for incremental updating. === TESTING PLAN === NOTE: in order to test this script, you will need to have some sort of circulation data already existing in your Koha installation. 1) Disable UpdateTotalIssuesOnCirc 2) Run: misc/cronjobs/update_totalissues.pl --use-items -t -v 3) If you have total checkout data in your item records (i.e. anything in 952$l), you should see messages like "Processing bib 43 (1 issues)" 4) Choose one of the lines that shows more than 0 issues, and view the record with that biblionumber in the staff client, choosing the "Items" tab (moredetail.pl). Add up the "Total checkouts" listed for each item, and confirm it matches what the script reported 5) Run: misc/cronjobs/update_totalissues.pl --use-stats -t -v 6) If you have any circulation statistics in your database (i.e. any 'issue' entries in your statistics table), you should see messages like "Processing bib 43 (1 issues)"; 7) Choose one of the lines and view the record with that biblionumber in the staff client, choosing the "Items" tab (moredetail.pl). If you count the number of checkouts listed in each item's checkout history, the total should match what the script reported. 8) Check out an item 9) Run: misc/cronjobs/update_totalissues.pl --use-stats --incremental --interval=1h -t -v 10) You should see one line reporting a single circ for the bib record associated with the item you just checked out (there may be more if you checked out any books in the hour prior to running these tests 11) If the results in steps 4, 7, and 10 match the predictions, the script worked This patch to Koha was sponsored by the Arcadia Public Library and the Arcadia Public Library Foundation in honor of Jackie Faust-Moreno, late director of the Arcadia Public Library.
Created attachment 9966 [details] [review] Bug 6557: Syspref to control total issue counting Because updating the total issues count associated with a bibliographic record on issue could cause a significant load on the server, this commit adds the syspref UpdateTotalIssuesOnCirc (which defaults to OFF to match existing behavior). The syspref has the following description: Do/Do not update a bibliographic record's total issues count whenever an item is issued (WARNING! This increases server load significantly; if performance is a concern, use the update_totalissues.pl cron job to update the total issues count). Bug 6557: automatically increment totalissues Adds the ability to automatically increment biblioitems.totalissues whenever an item is issued. To test: 1) Choose a record with at least one item that can circulate 2) Check the value of 942$0 (you may need to look at the plain MARC view on the OPAC). Most likely there won't be any 942$0 at all 3) Enable UpdateTotalIssuesOnCirc 4) Check out the item you selected 5) Check the value of 942$0 (you may need to look at the plain MARC view on the OPAC). That value should now be one greater than before 6) Discharge the item 7) Disable UpdateTotalIssuesOnCirc 8) Check out the item you selected again 9) Check the value of 942$0 (you may need to look at the plain MARC view on the OPAC). That value should not have changed Bug 6557: add script to update totalissues from stats NAME update_totalissues.pl SYNOPSIS update_totalissues.pl --use-stats update_totalissues.pl --use-items update_totalissues.pl --commit=1000 update_totalissues.pl --since='2012-01-01' update_totalissues.pl --interval=30d DESCRIPTION This batch job populates bibliographic records' total issues count based on historical issue statistics. --help Prints this help -v|--verbose Provide verbose log information (list every bib modified). --use-stats Use the data in the statistics table for populating total issues. --use-items Use items.issues data for populating total issues. Note that issues data from the items table does not respect the --since or --interval options, by definition. Also note that if both --use-stats and --use-items are specified, the count of biblios processed will be misleading. -s|--since=DATE Only process issues recorded in the statistics table since DATE. -i|--interval=S Only process issues recorded in the statistics table in the last N units of time. The interval should consist of a number with a one-letter unit suffix. The valid suffixes are h (hours), d (days), w (weeks), m (months), and y (years). The default unit is days. --incremental Add the number of issues found in the statistics table to the existing total issues count. Intended so that this script can be used as a cron job to update popularity information during low-usage periods. If neither --since or --interval are specified, incremental mode will default to processing the last twenty-four hours. --commit=N Commit the results to the database after every N records are processed. --test Only test the popularity population script. WARNING If the time on your database server does not match the time on your Koha server you will need to take that into account, and probably use the --since argument instead of the --interval argument for incremental updating. === TESTING PLAN === NOTE: in order to test this script, you will need to have some sort of circulation data already existing in your Koha installation. 1) Disable UpdateTotalIssuesOnCirc 2) Run: misc/cronjobs/update_totalissues.pl --use-items -t -v 3) If you have total checkout data in your item records (i.e. anything in 952$l), you should see messages like "Processing bib 43 (1 issues)" 4) Choose one of the lines that shows more than 0 issues, and view the record with that biblionumber in the staff client, choosing the "Items" tab (moredetail.pl). Add up the "Total checkouts" listed for each item, and confirm it matches what the script reported 5) Run: misc/cronjobs/update_totalissues.pl --use-stats -t -v 6) If you have any circulation statistics in your database (i.e. any 'issue' entries in your statistics table), you should see messages like "Processing bib 43 (1 issues)"; 7) Choose one of the lines and view the record with that biblionumber in the staff client, choosing the "Items" tab (moredetail.pl). If you count the number of checkouts listed in each item's checkout history, the total should match what the script reported. 8) Check out an item 9) Run: misc/cronjobs/update_totalissues.pl --use-stats --incremental --interval=1h -t -v 10) You should see one line reporting a single circ for the bib record associated with the item you just checked out (there may be more if you checked out any books in the hour prior to running these tests 11) If the results in steps 4, 7, and 10 match the predictions, the script worked This patch to Koha was sponsored by the Arcadia Public Library and the Arcadia Public Library Foundation in honor of Jackie Faust-Moreno, late director of the Arcadia Public Library. Signed-off-by: Liz Rea <wizzyrea@gmail.com> Tested this with my test data - numbers are correct and updated appropriately. More importantly - if I do a popularity search, the most popular items *come up first*. Amazing.
UpdateTotalIssues() has an avoidable inefficiency: it calls both GetBiblioData, and GetMarcBiblio. GetBiblioData returns all columns from biblio, biblioitems and itemtypes. We only need biblio.frameworkcode, biblioitems.totalissues and the $record (which can be obtained from biblioitems.marcxml). Additionally, the call from C4/Circulation to UpdateTotalIssues() occurs in AddIssue, which already has a copy of the biblio pulled in from a call to GetBiblioFromItemnumber (biblioitems.marc and biblioitems.marcxml). So we already have all the data we need in AddIssues to do the update; it would just be a matter of creating the MARC::Record object and doing a ModBiblio to update the right field. Modifying the cronjob to also return the MARC record and frameworkcode would let both means interface with the subroutine, so the library can still decide whether to take the slightened performance hit (or build from stats the first time).
Talking this out in IRC, while it may be more efficient to modify the subroutine to take in parts that we already have in AddIssue, it would not make for clean, readable code. And since computing popularity really doesn't need to be an up-to-the-minute thing, and a script is provided to do it on cron, I just can't justify holding up QA for this. Code is very clean, well documented and non-invasive. Provides a mechanism for both real-time and scheduled updates, and the initial population can be handled using the cron script as well. Assigning to Marcel for QA
Created attachment 9986 [details] [review] Bug 6557: Record bib popularity in totalissues
QA Comment Nice feature. Well documented incl. test plan. Code looks good. Just a thought on UpdateTotalIssues: BTW Note the comment from Ian earlier. Also: something simple: checking if biblionumber makes sense? Record exists? (Strange things happen..) Could be handled in the same followup suggested for Ians remarks. Will your use of commit trigger commit-ineffective-warnings in most cases (autocommit will be on by default) ? You do not have begin_work / rollback stuff. Since you offer a commit parameter, I wonder if you promise too much? Normally, the update is written directly to the database. If you actually would use a transaction with begin_work, you should check if it is possible etc. too. No blocker for me. process_stats could have used a few comments on the various queries used. See now what it does, but should not need that deduction ;) Passed QA
Some questions about this patch: * why do you use a cronjob instead of UPDATing when there is any circulation. The items.issues field is updated this way, for consistency biblioitems.totalissue should be too * I can't see a case where a library would want NOT to record totalissue. Thus, I can't see a case where the syspref would be set to 0. So, is the syspref needed ? The fact that you declared this bug as "normal" and not as "enhancement" make me feel it's more a bugguy behaviour thant something the library would be able to change.
Paul, (In reply to comment #22) > Some questions about this patch: > * why do you use a cronjob instead of UPDATing when there is any > circulation. The items.issues field is updated this way, for consistency > biblioitems.totalissue should be too Updating the biblio at circ is very expensive. It is much better not to edit the bib record at all at circulation. By making the update-at-circ optional, we won't be giving any libraries with underpowered servers an unpleasant shock. > * I can't see a case where a library would want NOT to record totalissue. > Thus, I can't see a case where the syspref would be set to 0. So, is the > syspref needed ? The fact that you declared this bug as "normal" and not as > "enhancement" make me feel it's more a bugguy behaviour thant something the > library would be able to change.
I can't decide myself to push this patch. Let me share my comments: * it add a syspref to decide wether we should update this field through a cron or in real time. The FAUSP (Fighters Against Useless SysPrefs) commitee has some concern: if it's long to do, who will set this syspref to 1 ? Small libraries have a low-grade hardware, large libraries circulate a lot. My conclusion is that noone will set the syspref to 1. So the syspref could/should be removed. * OTOH, the UpdateTotalIssues sub uses ModBiblio. It appears that it's because the popularity sorting is based on 942$0 field. that's why it's so long & I can't see another method to populate this field. * In UNIMARC default setup, the totalissue field is attached to 942$0 like in MARC21. Quite surprising according to me, but not wrong afaik * The install documentation should be updated to reflect the misc/cronjobs/update_totalissues.pl (can be done later. Having crontab.example updated is a good 1st step) Please comment my point 1 (2, 3 and 4 are just here as a reminder, even if 4 will require a follow-up or another bugzilla entry)
Paul, (In reply to comment #24) > I can't decide myself to push this patch. Let me share my comments: > * it add a syspref to decide wether we should update this field through a > cron or in real time. The FAUSP (Fighters Against Useless SysPrefs) commitee > has some concern: if it's long to do, who will set this syspref to 1 ? Small > libraries have a low-grade hardware, large libraries circulate a lot. My > conclusion is that noone will set the syspref to 1. So the syspref > could/should be removed. I think it is worth having the syspref and the option for realtime update for two reasons: 1. in the future performance may improve, and this is definitely a feature that people might want. By including the syspref now, we forestall the requests that it be added. 2. For very small libraries, setting up a cron job can be a significant hardship, and if you're circulating one item per minute on average, the performance hit from realtime updating doesn't really matter. 3. (I'm like the Spanish inquisition, with fear, surprise, and ruthless efficiency... can I start over?) Although I consider the performance hit to be significant, the fact is we're not looking at an operation any more inefficient than checking out any item was in 3.2. > * OTOH, the UpdateTotalIssues sub uses ModBiblio. It appears that it's > because the popularity sorting is based on 942$0 field. that's why it's so > long & I can't see another method to populate this field. > * In UNIMARC default setup, the totalissue field is attached to 942$0 like > in MARC21. Quite surprising according to me, but not wrong afaik I noticed that. And was baffled by it. But I figure, if it ain't broke... > * The install documentation should be updated to reflect the > misc/cronjobs/update_totalissues.pl (can be done later. Having > crontab.example updated is a good 1st step)
(In reply to comment #24) > The FAUSP (Fighters Against Useless SysPrefs) commitee I recently saw this patch removing a pref :-) http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=8263 (In reply to comment #25) > Although I consider the performance hit to be significant Could you make this more concrete? Did you test the difference in performance? Instead of just mentioning some milliseconds on a specific server, could we come to a reasonably accurate percentage or so? If the performance increase would be below say 10%, we may not need two ways. But we probably do if we reach 50%?
(In reply to comment #26) > (In reply to comment #24) > > The FAUSP (Fighters Against Useless SysPrefs) commitee > I recently saw this patch removing a pref :-) > http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=8263 > > (In reply to comment #25) > > Although I consider the performance hit to be significant > Could you make this more concrete? Did you test the difference in > performance? Instead of just mentioning some milliseconds on a specific > server, could we come to a reasonably accurate percentage or so? > If the performance increase would be below say 10%, we may not need two > ways. But we probably do if we reach 50%? As I recall, the performance hit was around 30%. Reducing the performance hit at the cost of clean code separation is -- as I discussed with Ian -- not a good idea, because it would make maintenance a nightmare, and move us one step closer to C4::Search levels of mess. I would say that the syspref would be a candidate for future removal, once DOM has entirely replaced GRS-1 (can we hope this will be true for 3.14? Well, maybe 3.16), but I am not comfortable with adding a feature that legacy libraries will be unable to take advantage of (instant popularity updates).
30% would justify the pref IMO. Speaking about committees, is there already a committee to fight against Yet Another Cron Job ? ;) Some day we need a layer on top of prefs and cronjobs. One intelligent cronjob that does everything based on settings and yes a few more prefs ..
(In reply to comment #28) > 30% would justify the pref IMO. > Speaking about committees, is there already a committee to fight against Yet > Another Cron Job ? ;) > Some day we need a layer on top of prefs and cronjobs. One intelligent > cronjob that does everything based on settings and yes a few more prefs .. That would be very nice. I'd like to kill both the instant updating and the cron job (making it into a script that only needs to be run on special occasions or if you want something weird for popularity) for 3.16, but as I said, I'm just not comfortable with doing it now because of the ramifications this would have for libraries with legacy data.
Created attachment 10561 [details] [review] Bug 6557 follow-up UNIMARC 924$0 in default french frameworks
Created attachment 10562 [details] [review] Bug 6557 follow-up UNIMARC 942$0 in default french frameworks
Patch pushed but NOTE : * i've provided a follow-up for french UNIMARC frameworks, that had no 942$0 at all, making the script badly fail: $ misc/cronjobs/update_totalissues.pl --use-items Tag "0" is not a valid tag. at /home/paul/koha.dev/koha-community//C4/Biblio.pm line 3870 A follow-up could be written to add a more clear error message: if biblioitems.totalissues isn't linked to a MARC field, issue a message before dying. * I'll also send a mail to koha-devel & koha-translate to warn ppl in case there is another language framework that has this problem * I'll advertise this patch in my RM newsletter * chris_c = in my opinion, it's a bugfix, but a very long standing one, and, if you want to backport it to 3.8, you'll have to write very clearly how to deal with it in the release note ! (I think it [c|sh]ould wait for 3.10)
There have been no further reports of problems so I am marking this bug resolved.