Bug 9819

Summary: stopwords related code should be removed
Product: Koha Reporter: Tomás Cohen Arazi <tomascohen>
Component: SearchingAssignee: Tomás Cohen Arazi <tomascohen>
Status: CLOSED FIXED QA Contact:
Severity: minor    
Priority: P5 - low CC: jonathan.druart, katrin.fischer, kyle, m.de.rooy, mathsabypro, nengard, tomascohen
Version: Main   
Hardware: All   
OS: All   
Change sponsored?: Sponsored Patch complexity: Medium patch
Documentation contact: Documentation submission:
Text to go in the release notes:
Version(s) released in:
Bug Depends on: 7440    
Bug Blocks:    
Attachments: Bug 9819 - 'stopwords'-related code removed
Bug 9819 - Remove 'stopwords' table from kohastructure.sql
Bug 9819 - Make updatedatabase.pl remove the 'stopwords' table
Bug 9819 - 'stopwords'-related code removed
Bug 9819 - Remove 'stopwords' table from kohastructure.sql
Bug 9819 - Make updatedatabase.pl remove the 'stopwords' table
Bug 9819 - QA Followup: Stopwords vestiges in Pg and es-ES removed
Bug 9819 - 'stopwords'-related code removed
Bug 9819 - Remove 'stopwords' table from kohastructure.sql
Bug 9819 - Make updatedatabase.pl remove the 'stopwords' table
Bug 9819 - QA Followup: Stopwords vestiges in es-ES removed
Bug 9819: Make the tests pass
Bug 9819 - 'stopwords'-related code removed
Bug 9819 - Remove 'stopwords' table from kohastructure.sql
Bug 9819 - Make updatedatabase.pl remove the 'stopwords' table
Bug 9819 - QA Followup: Stopwords vestiges in es-ES removed
Bug 9819: Make the tests pass
Bug 9819 - 'stopwords'-related code removed
Bug 9819 - Remove 'stopwords' table from kohastructure.sql
Bug 9819 - Make updatedatabase.pl remove the 'stopwords' table
Bug 9819 - QA Followup: Stopwords vestiges in es-ES removed
Bug 9819: Make the tests pass
Bug 9819 [QA Followup] - Removed missed stopwords refs, remove Schema file

Description Tomás Cohen Arazi 2013-03-14 18:35:41 UTC

    
Comment 1 Katrin Fischer 2013-04-10 09:19:01 UTC
When stopwords related code is removed this should include the database table stopwords.
Comment 2 Tomás Cohen Arazi 2013-04-10 14:34:56 UTC Comment hidden (obsolete)
Comment 3 Tomás Cohen Arazi 2013-04-10 14:35:02 UTC Comment hidden (obsolete)
Comment 4 Tomás Cohen Arazi 2013-04-10 14:35:12 UTC Comment hidden (obsolete)
Comment 5 Marcel de Rooy 2013-04-10 14:40:06 UTC
Hi Tomas,
Could you also explain on this bug report why the stopwords should be removed?
Comment 6 Katrin Fischer 2013-04-10 14:49:25 UTC
The use of stopwords was limited to NoZebra - so this bug is a consequence of removing all NoZebra related code.
Comment 7 Kyle M Hall 2013-04-19 18:01:14 UTC Comment hidden (obsolete)
Comment 8 Kyle M Hall 2013-04-19 18:01:22 UTC Comment hidden (obsolete)
Comment 9 Kyle M Hall 2013-04-19 18:01:28 UTC Comment hidden (obsolete)
Comment 10 Jonathan Druart 2013-04-22 10:39:48 UTC
QA comment:

Great job Tomás. All looks good.
But there are 2 smalls things
1/ the structure file for PostgreSQL contains the "create table" statement for the stopwords table. I know this file is not updated, but while you're at it you could delete it.
2/ The stopwords sql file for spanish has been introduced into master before you provided your first patch. It should be deleted.


Marked as Failed QA for 2.
Comment 11 Tomás Cohen Arazi 2013-04-22 12:11:36 UTC Comment hidden (obsolete)
Comment 12 Jonathan Druart 2013-04-22 13:14:49 UTC
Tomás, thank you for your followup!

I found another issue: some unit tests in t/db_dependent/Search.t fail:

#   Failed test 'getRecords on _ALLRECORDS PQF returned all records'
#   at t/db_dependent/Search.t line 387.
#          got: undef
#     expected: '178'

#   Failed test 'getRecords PQF author search for Lessig returned proper number of matches'
#   at t/db_dependent/Search.t line 393.
#          got: undef
#     expected: '4'

#   Failed test 'getRecords CCL author search for Lessig returned proper number of matches'
#   at t/db_dependent/Search.t line 399.
#          got: undef
#     expected: '4'

#   Failed test 'getRecords CQL author search for Lessig returned proper number of matches'
#   at t/db_dependent/Search.t line 405.
#          got: undef
#     expected: '4'
# Looks like you failed 4 tests of 75.

They passed with success before yours patches.
Comment 13 Tomás Cohen Arazi 2013-04-22 13:30:41 UTC
(In reply to comment #12)
> They passed with success before yours patches.

What data set can I use to test it in my box?
Comment 14 Jonathan Druart 2013-04-22 13:56:37 UTC
(In reply to comment #13)
> (In reply to comment #12)
> > They passed with success before yours patches.
> 
> What data set can I use to test it in my box?

No idea!
On master, the prove t/db_dependent/Search.t command does not return 0?
Comment 15 Katrin Fischer 2014-11-09 23:22:32 UTC
It looks like we got stuck on this... but would be a nice clean-up? :)
Some of the code seems to have been removed since - patches don't apply.
Comment 16 Katrin Fischer 2014-11-10 00:25:02 UTC
*** Bug 11110 has been marked as a duplicate of this bug. ***
Comment 17 Nicole C. Engard 2015-03-06 08:33:33 UTC
I'd also like to see this make it's way in to Koha
Comment 18 Jonathan Druart 2015-12-15 10:08:38 UTC Comment hidden (obsolete)
Comment 19 Jonathan Druart 2015-12-15 10:08:42 UTC Comment hidden (obsolete)
Comment 20 Jonathan Druart 2015-12-15 10:08:45 UTC Comment hidden (obsolete)
Comment 21 Jonathan Druart 2015-12-15 10:08:48 UTC Comment hidden (obsolete)
Comment 22 Jonathan Druart 2015-12-15 10:08:51 UTC Comment hidden (obsolete)
Comment 23 Jonathan Druart 2015-12-15 10:09:26 UTC
Patches rebased against master, please test.
Comment 24 Chris Cormack 2015-12-17 23:59:34 UTC Comment hidden (obsolete)
Comment 25 Chris Cormack 2015-12-17 23:59:46 UTC Comment hidden (obsolete)
Comment 26 Chris Cormack 2015-12-18 00:00:00 UTC Comment hidden (obsolete)
Comment 27 Chris Cormack 2015-12-18 00:00:13 UTC Comment hidden (obsolete)
Comment 28 Chris Cormack 2015-12-18 00:00:25 UTC Comment hidden (obsolete)
Comment 29 Kyle M Hall 2015-12-18 18:00:48 UTC
Created attachment 45854 [details] [review]
Bug 9819 - 'stopwords'-related code removed

This patch removes code related to stopwords usage. The following methods are removed:

C4::Search->remove_stopwords
C4::Context->stopwords
C4::Context->_new_stopwords

And the buildQuery API was changed (removed the \@removed_stopwords return value).
A follow-up is provided for database changes, to make rebasing easier.

To test:
- Apply this patch
- Do some searches in both intranet and opac interfaces
- Nothing should break

Sponsored-by: Universidad Nacional de Córdoba
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Chris Cormack <chrisc@catalyst.net.nz>
Comment 30 Kyle M Hall 2015-12-18 18:00:58 UTC
Created attachment 45855 [details] [review]
Bug 9819 - Remove 'stopwords' table from kohastructure.sql

Just that.

To test:
- Apply the patch
- Do a fresh install
- The database structure should be created flawlessly
- There should not be a 'stopwords' table created

Sponsored-by: Universidad Nacional de Córdoba
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Chris Cormack <chrisc@catalyst.net.nz>
Comment 31 Kyle M Hall 2015-12-18 18:01:01 UTC
Created attachment 45856 [details] [review]
Bug 9819 - Make updatedatabase.pl remove the 'stopwords' table

To test:
- Apply the patch
- Will be prompted to run the updater
- The 'stopwords' database should be removed
  ( test on a mysql prompt with
    > SHOW TABLES LIKE 'stopwords';
   and should get no results)

Sponsored-by: Universidad Nacional de Córdoba
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Chris Cormack <chrisc@catalyst.net.nz>
Comment 32 Kyle M Hall 2015-12-18 18:01:04 UTC
Created attachment 45857 [details] [review]
Bug 9819 - QA Followup: Stopwords vestiges in es-ES removed

Gotcha.
To+

Signed-off-by: Chris Cormack <chrisc@catalyst.net.nz>

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Comment 33 Kyle M Hall 2015-12-18 18:01:07 UTC
Created attachment 45858 [details] [review]
Bug 9819: Make the tests pass

Signed-off-by: Chris Cormack <chrisc@catalyst.net.nz>

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Comment 34 Kyle M Hall 2015-12-30 15:55:32 UTC
Created attachment 46092 [details] [review]
Bug 9819 [QA Followup] - Removed missed stopwords refs, remove Schema file
Comment 35 Kyle M Hall 2015-12-30 16:14:43 UTC
Pushed to master, thanks Tomas and Jonathan!