It'd be handy to have one place to look for string normalization routines.
Created attachment 55571 [details] [review] Bug 17302: Add Koha::Util::Normalize for normalization functions This patch introduces Koha::Util::Normalize, which includes the following normalization routines that need no explanation: - remove_spaces - upper_case - lower_case and it also includes: - default: which basically does what C4::Matcher::_normalize does. All routines functionality are fully tested with the included in the included tests. To test: - Apply the patch - Run: $ prove t/Koha/Util/Normalize.t => SUCCESS: All tests pass - Sign off :-D
Exporter ?
I still think that the normalization done by C4::Matcher::_normalize is a bad idea to use as a default. We shouldn't be pre-normalizing data before it hits Zebra, because Zebra is going to try to normalize it again for searching using the same method it used to normalize data for indexing.
(In reply to David Cook from comment #3) > I still think that the normalization done by C4::Matcher::_normalize is a > bad idea to use as a default. Yeah, but first things first, if we want this on master. Look at bug 17304, which is part of this effort to make normalization configurable.
Created attachment 55591 [details] [review] Bug 17302: Add Koha::Util::Normalize for normalization functions This patch introduces Koha::Util::Normalize, which includes the following normalization routines that need no explanation: - remove_spaces - upper_case - lower_case and it also includes: - legacy_default: which basically does what C4::Matcher::_normalize does. All routines functionality are fully tested with the included in the included tests. To test: - Apply the patch - Run: $ prove t/Koha/Util/Normalize.t => SUCCESS: All tests pass - Sign off :-D Edit: Added Exporter to explicitly export the routines.
Created attachment 55600 [details] [review] Bug 17302: Add Koha::Util::Normalize for normalization functions This patch introduces Koha::Util::Normalize, which includes the following normalization routines that need no explanation: - remove_spaces - upper_case - lower_case and it also includes: - legacy_default: which basically does what C4::Matcher::_normalize does. All routines functionality are fully tested with the included in the included tests. To test: - Apply the patch - Run: $ prove t/Koha/Util/Normalize.t => SUCCESS: All tests pass - Sign off :-D Edit: Added Exporter to explicitly export the routines. Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Created attachment 55601 [details] [review] Bug 17302: [Follow-up] Make Normalize.pm undef-resistent Trivial changes for passing undef to the norm routines. Added a few dumb tests too. Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Created attachment 55649 [details] [review] [PASSED QA] Bug 17302: Add Koha::Util::Normalize for normalization functions This patch introduces Koha::Util::Normalize, which includes the following normalization routines that need no explanation: - remove_spaces - upper_case - lower_case and it also includes: - legacy_default: which basically does what C4::Matcher::_normalize does. All routines functionality are fully tested with the included in the included tests. To test: - Apply the patch - Run: $ prove t/Koha/Util/Normalize.t => SUCCESS: All tests pass - Sign off :-D Edit: Added Exporter to explicitly export the routines. Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl> Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
Created attachment 55650 [details] [review] [PASSED QA] Bug 17302: [Follow-up] Make Normalize.pm undef-resistent Trivial changes for passing undef to the norm routines. Added a few dumb tests too. Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl> Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
(In reply to Tomás Cohen Arazi from comment #4) > Yeah, but first things first, if we want this on master. Look at bug 17304, > which is part of this effort to make normalization configurable. I'm not sure how bug #17304 relates to making normalization configurable. Isn't that just adding unit tests? I'm not convinced that there should be any normalization - configurable or hard-coded - either. If we want to normalize data for matching, we should probably be looking at Zebra's normalization rules. On that note, I'd be curious to see how ElasticSearch handles normalization. I looked a bit at how Solr did it a while ago and I recall it leaving a lot to be desired...
Pushed to master for 16.11, thanks Marcel, Tomas!