Bug 21066

Summary: Replace opac_news.timestamp by published_on and add updated_on as timestamp
Product: Koha Reporter: Marcel de Rooy <m.de.rooy>
Component: ToolsAssignee: Marcel de Rooy <m.de.rooy>
Status: CLOSED FIXED QA Contact: Testopia <testopia>
Severity: enhancement    
Priority: P5 - low CC: jonathan.druart, lucas, tomascohen
Version: Main   
Hardware: All   
OS: All   
Change sponsored?: --- Patch complexity: Medium patch
Documentation contact: Documentation submission:
Text to go in the release notes:
Version(s) released in:
20.11.00
Circulation function:
Attachments: Bug 21066: Database revision
Bug 21066: Code changes needed for replacing opac_news.timestamp
Bug 21066: DBIx changes for opac_news
Bug 21066: Database revision
DO_NOT_PUSH: DBIx schema changes
Bug 21066: Code changes needed for replacing opac_news.timestamp
Bug 21066: Resolve SQL errors in strict mode
Bug 21066: Update NewsChannels.t
Bug 21066: Update English sample notices
Bug 21066: Update translated news and notices
Bug 21066: Update translated news and notices
Bug 21066: Database revision
DO_NOT_PUSH: DBIx schema changes
Bug 21066: Code changes needed for replacing opac_news.timestamp
Bug 21066: Resolve SQL errors in strict mode
Bug 21066: Update NewsChannels.t
Bug 21066: Update English sample notices
Bug 21066: Update translated news and notices
Bug 21066: Database revision
DO_NOT_PUSH: DBIx schema changes
Bug 21066: Code changes needed for replacing opac_news.timestamp
Bug 21066: Resolve SQL errors in strict mode
Bug 21066: Update NewsChannels.t
Bug 21066: Update English sample notices
Bug 21066: Update translated news and notices
Bug 21066: (QA follow-up) Fix publication date display in OPAC
Bug 21066: (QA follow-up) Check for field existence in dbrev
Bug 21066: DBRev 20.06.00.033
Bug 21066: Rename publicationdate with published_on

Description Marcel de Rooy 2018-07-12 13:40:18 UTC
I would propose to use the opac_news.timestamp field readonly and fill/update it automatically via SQL.

Or do we still need the original publish date and add a modified/updated_on timestamp?

What do you think?
Comment 1 Marcel de Rooy 2018-07-17 14:24:38 UTC
Created attachment 77037 [details] [review]
Bug 21066: Database revision

*Convert* timestamp to publisheddate, add updated_on as 'real' timestamp.
Comment 2 Marcel de Rooy 2018-07-17 14:24:43 UTC
Created attachment 77038 [details] [review]
Bug 21066: Code changes needed for replacing opac_news.timestamp

Note: This is not finished yet.
TODO installer/data/mysql/en/mandatory/sample_notices.sql:<p class="newsfooter" style="font-size: 8pt; font-style:italic; margin-bottom: 1px; margin-top: 1px">Posted on <<opac_news.timestamp>></p>
TODO installer/data/mysql/en/optional/sample_news.sql:INSERT INTO `opac_news` (`title`, `new`, `lang`, `timestamp`, `expirationdate`, `number`) ETC
TODO t/db_dependent/NewsChannels.t
Comment 3 Marcel de Rooy 2018-07-17 14:24:47 UTC
Created attachment 77039 [details] [review]
Bug 21066: DBIx changes for opac_news
Comment 4 Marcel de Rooy 2018-07-17 14:27:30 UTC
Well, this is just a preview of my proposal. Replace timestamp by publisheddate (a real date) and add updated_on (a real timestamp).
These are currently 16.11 patches.
I hope to work later on moving them upwards, depending on community interest and increased momentum on other proposed developments..
Comment 5 Katrin Fischer 2018-10-28 18:52:43 UTC
Hi Marcel,

I think maybe have:
updated_on
created_on

And use the publication date from the entry in the notices? With the TT syntax being more widely available the formatting should not be as problematic as it used to be.

I am not so happy about publisheddate because the date the entry is published (visible) might be quite different from the date the entry was created.

Hope I understood correctly!
Comment 6 Katrin Fischer 2018-10-28 18:56:24 UTC
Argh... should have looked more closely at the table. I completely understand your problem now.

I think making the timestamp a publishingdate (better fit with expirationdate and might be in the future) and having another real timestamp field makes totally sense.
Comment 7 Marcel de Rooy 2020-08-04 14:27:53 UTC
Created attachment 107769 [details] [review]
Bug 21066: Database revision

Change timestamp to publicationdate, add updated_on as 'real' timestamp.

Test plan:
Run install or upgrade. Check table opac_news.

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Comment 8 Marcel de Rooy 2020-08-04 14:27:57 UTC
Created attachment 107770 [details] [review]
DO_NOT_PUSH: DBIx schema changes
Comment 9 Marcel de Rooy 2020-08-04 14:28:02 UTC
Created attachment 107771 [details] [review]
Bug 21066: Code changes needed for replacing opac_news.timestamp

Test plan:
Perform some CRUD operations on the news page in staff.
Verify that news on opac and staff respect publicationdate.

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Comment 10 Marcel de Rooy 2020-08-04 14:28:06 UTC
Created attachment 107772 [details] [review]
Bug 21066: Resolve SQL errors in strict mode

DBD::mysql::st execute failed: Incorrect integer value: '' for column `koha_master`.`opac_news`.`number` at row 1 [for Statement "INSERT INTO opac_news ( content,number,lang,title,branchcode,borrowernumber,expirationdate,publicationdate ) VALUES ( ?,?,?,?,?,?,?,? )" with ParamValues: 0="test", 1="", 2="", 3="test", 4=undef, 5="51", 6='2020-08-11', 7='2020-08-04'] at /usr/share/koha/C4/NewsChannels.pm line 68.
DBD::mysql::st execute failed: Incorrect integer value: 'a' for column `koha_master`.`opac_news`.`number` at row 1 [for Statement "UPDATE opac_news SET lang = ?,publicationdate = ?,expirationdate = ?,number = ?,content = ?,branchcode = ?,title = ? WHERE idnew = ?;" with ParamValues: 0="", 1='2020-08-03', 2=undef, 3="a", 4="test", 5=undef, 6="test", 7="9"] at /usr/share/koha/C4/NewsChannels.pm line 107.

Test plan:
Do not apply this patch.
Enable strict mode in koha-conf.
Add new record in tools/news leaving 'Appear in position' blank.
The record is not added; you have a warn in your plack-error log.
Apply patch and try again.

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Comment 11 Marcel de Rooy 2020-08-04 14:28:10 UTC
Created attachment 107773 [details] [review]
Bug 21066: Update NewsChannels.t

Removing the weird '' => 1 from the hash too.
NOTE: The following line in the module seems to be useless:
    $data->{$data->{'lang'}} = 1 if defined $data->{lang};

Test plan:
Run t/db_dependent/NewsChannels.t
Run t/db_dependent/Koha/News.t

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Comment 12 Marcel de Rooy 2020-08-04 14:28:15 UTC
Created attachment 107774 [details] [review]
Bug 21066: Update English sample notices

Applies to sample notices and sample news.
Replace timestamp by publicationdate.

Also adjust notice in TemplateToolkit.t.

Test plan:
Run dbrev and verify if ISSUESLIP refers to opac_news.publicationdate.
Run t/db_dependent/Letters/TemplateToolkit.t

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Comment 13 Marcel de Rooy 2020-08-04 14:28:19 UTC Comment hidden (obsolete)
Comment 14 Marcel de Rooy 2020-08-04 14:29:33 UTC
(In reply to Katrin Fischer from comment #6)
> Argh... should have looked more closely at the table. I completely
> understand your problem now.
> 
> I think making the timestamp a publishingdate (better fit with
> expirationdate and might be in the future) and having another real timestamp
> field makes totally sense.

Okay, second try. With publicationdate. Perhaps even a bit better than publishingdate ;)
Comment 15 Marcel de Rooy 2020-08-04 14:31:20 UTC
Created attachment 107776 [details] [review]
Bug 21066: Update translated news and notices

Replace timestamp by publicationdate in translations.
Applies to sample notices and sample news.

Test plan:
Run a new install or pick a sql file and run it on cmdline.

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Comment 16 Owen Leonard 2020-08-26 12:08:22 UTC
Created attachment 109148 [details] [review]
Bug 21066: Database revision

Change timestamp to publicationdate, add updated_on as 'real' timestamp.

Test plan:
Run install or upgrade. Check table opac_news.

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>

Signed-off-by: Owen Leonard <oleonard@myacpl.org>
Comment 17 Owen Leonard 2020-08-26 12:08:25 UTC
Created attachment 109149 [details] [review]
DO_NOT_PUSH: DBIx schema changes

https://bugs.koha-community.org/show_bug.cgi?id=21066

Signed-off-by: Owen Leonard <oleonard@myacpl.org>
Comment 18 Owen Leonard 2020-08-26 12:08:29 UTC
Created attachment 109150 [details] [review]
Bug 21066: Code changes needed for replacing opac_news.timestamp

Test plan:
Perform some CRUD operations on the news page in staff.
Verify that news on opac and staff respect publicationdate.

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>

Signed-off-by: Owen Leonard <oleonard@myacpl.org>
Comment 19 Owen Leonard 2020-08-26 12:08:32 UTC
Created attachment 109151 [details] [review]
Bug 21066: Resolve SQL errors in strict mode

DBD::mysql::st execute failed: Incorrect integer value: '' for column `koha_master`.`opac_news`.`number` at row 1 [for Statement "INSERT INTO opac_news ( content,number,lang,title,branchcode,borrowernumber,expirationdate,publicationdate ) VALUES ( ?,?,?,?,?,?,?,? )" with ParamValues: 0="test", 1="", 2="", 3="test", 4=undef, 5="51", 6='2020-08-11', 7='2020-08-04'] at /usr/share/koha/C4/NewsChannels.pm line 68.
DBD::mysql::st execute failed: Incorrect integer value: 'a' for column `koha_master`.`opac_news`.`number` at row 1 [for Statement "UPDATE opac_news SET lang = ?,publicationdate = ?,expirationdate = ?,number = ?,content = ?,branchcode = ?,title = ? WHERE idnew = ?;" with ParamValues: 0="", 1='2020-08-03', 2=undef, 3="a", 4="test", 5=undef, 6="test", 7="9"] at /usr/share/koha/C4/NewsChannels.pm line 107.

Test plan:
Do not apply this patch.
Enable strict mode in koha-conf.
Add new record in tools/news leaving 'Appear in position' blank.
The record is not added; you have a warn in your plack-error log.
Apply patch and try again.

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>

Signed-off-by: Owen Leonard <oleonard@myacpl.org>
Comment 20 Owen Leonard 2020-08-26 12:08:36 UTC
Created attachment 109152 [details] [review]
Bug 21066: Update NewsChannels.t

Removing the weird '' => 1 from the hash too.
NOTE: The following line in the module seems to be useless:
    $data->{$data->{'lang'}} = 1 if defined $data->{lang};

Test plan:
Run t/db_dependent/NewsChannels.t
Run t/db_dependent/Koha/News.t

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>

Signed-off-by: Owen Leonard <oleonard@myacpl.org>
Comment 21 Owen Leonard 2020-08-26 12:08:39 UTC
Created attachment 109153 [details] [review]
Bug 21066: Update English sample notices

Applies to sample notices and sample news.
Replace timestamp by publicationdate.

Also adjust notice in TemplateToolkit.t.

Test plan:
Run dbrev and verify if ISSUESLIP refers to opac_news.publicationdate.
Run t/db_dependent/Letters/TemplateToolkit.t

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>

Signed-off-by: Owen Leonard <oleonard@myacpl.org>
Comment 22 Owen Leonard 2020-08-26 12:08:43 UTC
Created attachment 109154 [details] [review]
Bug 21066: Update translated news and notices

Replace timestamp by publicationdate in translations.
Applies to sample notices and sample news.

Test plan:
Run a new install or pick a sql file and run it on cmdline.

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>

Signed-off-by: Owen Leonard <oleonard@myacpl.org>
Comment 23 Katrin Fischer 2020-08-26 19:21:34 UTC
Created attachment 109192 [details] [review]
Bug 21066: Database revision

Change timestamp to publicationdate, add updated_on as 'real' timestamp.

Test plan:
Run install or upgrade. Check table opac_news.

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>

Signed-off-by: Owen Leonard <oleonard@myacpl.org>

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Comment 24 Katrin Fischer 2020-08-26 19:21:38 UTC
Created attachment 109193 [details] [review]
DO_NOT_PUSH: DBIx schema changes

https://bugs.koha-community.org/show_bug.cgi?id=21066

Signed-off-by: Owen Leonard <oleonard@myacpl.org>

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Comment 25 Katrin Fischer 2020-08-26 19:21:42 UTC
Created attachment 109194 [details] [review]
Bug 21066: Code changes needed for replacing opac_news.timestamp

Test plan:
Perform some CRUD operations on the news page in staff.
Verify that news on opac and staff respect publicationdate.

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>

Signed-off-by: Owen Leonard <oleonard@myacpl.org>

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Comment 26 Katrin Fischer 2020-08-26 19:21:46 UTC
Created attachment 109195 [details] [review]
Bug 21066: Resolve SQL errors in strict mode

DBD::mysql::st execute failed: Incorrect integer value: '' for column `koha_master`.`opac_news`.`number` at row 1 [for Statement "INSERT INTO opac_news ( content,number,lang,title,branchcode,borrowernumber,expirationdate,publicationdate ) VALUES ( ?,?,?,?,?,?,?,? )" with ParamValues: 0="test", 1="", 2="", 3="test", 4=undef, 5="51", 6='2020-08-11', 7='2020-08-04'] at /usr/share/koha/C4/NewsChannels.pm line 68.
DBD::mysql::st execute failed: Incorrect integer value: 'a' for column `koha_master`.`opac_news`.`number` at row 1 [for Statement "UPDATE opac_news SET lang = ?,publicationdate = ?,expirationdate = ?,number = ?,content = ?,branchcode = ?,title = ? WHERE idnew = ?;" with ParamValues: 0="", 1='2020-08-03', 2=undef, 3="a", 4="test", 5=undef, 6="test", 7="9"] at /usr/share/koha/C4/NewsChannels.pm line 107.

Test plan:
Do not apply this patch.
Enable strict mode in koha-conf.
Add new record in tools/news leaving 'Appear in position' blank.
The record is not added; you have a warn in your plack-error log.
Apply patch and try again.

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>

Signed-off-by: Owen Leonard <oleonard@myacpl.org>

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Comment 27 Katrin Fischer 2020-08-26 19:21:49 UTC
Created attachment 109196 [details] [review]
Bug 21066: Update NewsChannels.t

Removing the weird '' => 1 from the hash too.
NOTE: The following line in the module seems to be useless:
    $data->{$data->{'lang'}} = 1 if defined $data->{lang};

Test plan:
Run t/db_dependent/NewsChannels.t
Run t/db_dependent/Koha/News.t

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>

Signed-off-by: Owen Leonard <oleonard@myacpl.org>

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Comment 28 Katrin Fischer 2020-08-26 19:21:53 UTC
Created attachment 109197 [details] [review]
Bug 21066: Update English sample notices

Applies to sample notices and sample news.
Replace timestamp by publicationdate.

Also adjust notice in TemplateToolkit.t.

Test plan:
Run dbrev and verify if ISSUESLIP refers to opac_news.publicationdate.
Run t/db_dependent/Letters/TemplateToolkit.t

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>

Signed-off-by: Owen Leonard <oleonard@myacpl.org>

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Comment 29 Katrin Fischer 2020-08-26 19:21:57 UTC
Created attachment 109198 [details] [review]
Bug 21066: Update translated news and notices

Replace timestamp by publicationdate in translations.
Applies to sample notices and sample news.

Test plan:
Run a new install or pick a sql file and run it on cmdline.

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>

Signed-off-by: Owen Leonard <oleonard@myacpl.org>

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Comment 30 Katrin Fischer 2020-08-26 19:22:01 UTC
Created attachment 109199 [details] [review]
Bug 21066: (QA follow-up) Fix publication date display in OPAC

The date would no longer show, changed the variable name and
removed the hours, as the field in the database is a date
and not a datetime.

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Comment 31 Marcel de Rooy 2020-08-27 05:59:50 UTC
(In reply to Katrin Fischer from comment #30)
> Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>

Thanks
Comment 32 Katrin Fischer 2020-08-30 21:30:29 UTC
Hm, was checking this for a similar bug I am working on and there are 2 issues with the first database update:

1) It should check for the new column existing (idempotent)
2) It should specify where the new column should be added with AFTER
Comment 33 Marcel de Rooy 2020-08-31 11:53:22 UTC
Created attachment 109354 [details] [review]
Bug 21066: (QA follow-up) Check for field existence in dbrev

Test plan:
Run the upgrade twice.

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Comment 34 Marcel de Rooy 2020-08-31 11:54:10 UTC
(In reply to Katrin Fischer from comment #32)
> Hm, was checking this for a similar bug I am working on and there are 2
> issues with the first database update:
> 
> 1) It should check for the new column existing (idempotent)
> 2) It should specify where the new column should be added with AFTER

Thx for noting. Fixed
Comment 35 Jonathan Druart 2020-09-02 12:10:04 UTC
Comment on attachment 109192 [details] [review]
Bug 21066: Database revision

Review of attachment 109192 [details] [review]:
-----------------------------------------------------------------

::: installer/data/mysql/kohastructure.sql
@@ +1723,4 @@
>    `title` varchar(250) NOT NULL default '', -- title of the news article
>    `content` MEDIUMTEXT NOT NULL, -- the body of your news article
>    `lang` varchar(50) NOT NULL default '', -- location for the article (koha is the staff client, slip is the circulation receipt and language codes are for the opac)
> +  `publicationdate` date DEFAULT NULL, -- publication date

Should not it be published_on?
Comment 36 Tomás Cohen Arazi (tcohen) 2020-09-02 12:16:26 UTC
(In reply to Jonathan Druart from comment #35)
> Comment on attachment 109192 [details] [review] [review]
> Bug 21066: Database revision
> 
> Review of attachment 109192 [details] [review] [review]:
> -----------------------------------------------------------------
> 
> ::: installer/data/mysql/kohastructure.sql
> @@ +1723,4 @@
> >    `title` varchar(250) NOT NULL default '', -- title of the news article
> >    `content` MEDIUMTEXT NOT NULL, -- the body of your news article
> >    `lang` varchar(50) NOT NULL default '', -- location for the article (koha is the staff client, slip is the circulation receipt and language codes are for the opac)
> > +  `publicationdate` date DEFAULT NULL, -- publication date
> 
> Should not it be published_on?

I agree.
Comment 37 Marcel de Rooy 2020-09-02 12:20:53 UTC
(In reply to Jonathan Druart from comment #35)
> Should not it be published_on?

These patches date from 2018. Have gone through QA. I think this tiny change is quite arbitrary and should not block this patch set now.
Comment 38 Jonathan Druart 2020-09-02 12:36:55 UTC
Created attachment 109514 [details] [review]
Bug 21066: DBRev 20.06.00.033
Comment 39 Jonathan Druart 2020-09-02 12:37:00 UTC
Created attachment 109515 [details] [review]
Bug 21066: Rename publicationdate with published_on
Comment 40 Jonathan Druart 2020-09-02 12:37:24 UTC
(In reply to Marcel de Rooy from comment #37)
> (In reply to Jonathan Druart from comment #35)
> > Should not it be published_on?
> 
> These patches date from 2018. Have gone through QA. I think this tiny change
> is quite arbitrary and should not block this patch set now.

I don't want to block it. Do you agree with the renaming?
Comment 41 Marcel de Rooy 2020-09-02 12:40:27 UTC
(In reply to Jonathan Druart from comment #40)
> (In reply to Marcel de Rooy from comment #37)
> > (In reply to Jonathan Druart from comment #35)
> > > Should not it be published_on?
> > 
> > These patches date from 2018. Have gone through QA. I think this tiny change
> > is quite arbitrary and should not block this patch set now.
> 
> I don't want to block it. Do you agree with the renaming?

Yes, no problem glancing thru the patch diagonally..
Comment 42 Jonathan Druart 2020-09-02 13:35:51 UTC
Pushed to master for 20.11, thanks to everybody involved!
Comment 43 Marcel de Rooy 2020-09-02 13:37:57 UTC
(In reply to Jonathan Druart from comment #42)
> Pushed to master for 20.11, thanks to everybody involved!

Cool
Comment 44 Lucas Gass (lukeg) 2020-09-04 21:19:00 UTC
enhancement will not be backported to 20.05.x
Comment 45 Owen Leonard 2020-09-10 18:42:22 UTC
*** Bug 23430 has been marked as a duplicate of this bug. ***