Bug 20551

Summary: Add option for including deleted records in export_records.pl
Product: Koha Reporter: David Gustafsson <glasklas>
Component: Command-line UtilitiesAssignee: David Gustafsson <glasklas>
Status: Signed Off --- QA Contact: Testopia <testopia>
Severity: enhancement    
Priority: P5 - low CC: arthur.suzuki, baptiste.bayche, bibliwho, dcook, eric.begin, frank.hansen, fridolin.somers, jonathan.druart, m.de.rooy, pedro.amorim, robin, theod
Version: Main   
Hardware: All   
OS: All   
GIT URL: Change sponsored?: ---
Patch complexity: Medium patch Documentation contact:
Documentation submission: Text to go in the release notes:
Version(s) released in:
Circulation function:
Bug Depends on:    
Bug Blocks: 23009    
Attachments: Bug 20551: Add option for including deleted records in export_records.pl
Bug 20551: Add option for including deleted records in export_records.pl
Bug 20551: Add option for including deleted records in export_records.pl
Bug 20551: Add option for including deleted records in export_records.pl
Bug 20551: Add option for including deleted records in export_records.pl
Bug 20551: Add option for including deleted records in export_records.pl
Bug 20551: Add option for including deleted records in export_records.pl
Bug 20551: Fix option name in pod
Bug 20551: Rename "marcflavour" field to "schema"
Bug 20551: Add option for including deleted records in export_records.pl
Bug 20551: Fix option name in pod
Bug 20551: Rename "marcflavour" field to "schema"
Bug 20551: Improve documentation and make compatible with more options
Bug 20551: Add option for including deleted records in export_records.pl
Bug 20551: Fix option name in pod
Bug 20551: Rename "marcflavour" field to "schema"
Bug 20551: Improve documentation and make compatible with more options
Bug 20551: fix tests
Bug 20551: Add missing subroutine import
Bug 20551: Fix tests
Bug 20551: Add option for including deleted records in export_records.pl
Bug 20551: Fix option name in pod
Bug 20551: Rename "marcflavour" field to "schema"
Bug 20551: Improve documentation and make compatible with more options
Bug 20551: Add missing subroutine import
Bug 20551: Fix tests
Bug 20551: Add option for including deleted records in export_records.pl
Bug 20551: Fix option name in pod
Bug 20551: Rename "marcflavour" field to "schema"
Bug 20551: Improve documentation and make compatible with more options
Bug 20551: Add missing subroutine import
Bug 20551: Fix tests
Bug 20551: Fetch deleted biblios through _get_record_export
Bug 20551: Refactor unmotivated partially applied validator function
Bug 20551: Add option for including deleted records in export_records.pl
Bug 20551: Fix option name in pod
Bug 20551: Rename "marcflavour" field to "schema"
Bug 20551: Improve documentation and make compatible with more options
Bug 20551: Add missing subroutine import
Bug 20551: Fix tests
Bug 20551: Fetch deleted biblios through _get_record_export
Bug 20551: Refactor unmotivated partially applied validator function
Bug 20551: (QA follow-up) perltidy and test fixes
Bug 20551: Move some code that might be reused into functions
Bug 20551: Add option for including deleted records in export_records.pl
Bug 20551: Fix option name in pod
Bug 20551: Rename "marcflavour" field to "schema"
Bug 20551: Improve documentation and make compatible with more options
Bug 20551: Add missing subroutine import
Bug 20551: Fix tests
Bug 20551: Fetch deleted biblios through _get_record_export
Bug 20551: Refactor unmotivated partially applied validator function
Bug 20551: (QA follow-up) perltidy and test fixes
Bug 20551: Move some code that might be reused into functions
Bug 20551: (QA follow-up) don't read record content if there is an error during decoding
Bug 20551: Get biblionumber safly in validator function
Bug 20551: Get biblionumber safely in validator function
Bug 20551: Use iterator for retrieving records to save memory
Bug 20551: Use iterator for retrieving records to save memory
Bug 20551: Use iterator for retrieving records to save memory
Bug 20551: Use iterator for retrieving records to save memory
Bug 20551: Use iterator for retrieving records to save memory
Bug 20551: Cache resultset instead of passing to sub
Bug 20551: Warn and return if deleted biblio not found
Bug 20551: Use timetstamp of deletedbiblio_metadata instead of deletedbibio
Bug 20551: Return 0 instead of undef from _get_record_for_export not to break iterator loop
Bug 20551: Return 0 instead of undef from _get_deleted_record_for_export
Bug 20551: Add option for including deleted records in export_records.pl
Bug 20551: Fix option name in pod
Bug 20551: Rename "marcflavour" field to "schema"
Bug 20551: Improve documentation and make compatible with more options
Bug 20551: Add missing subroutine import
Bug 20551: Fix tests
Bug 20551: Fetch deleted biblios through _get_record_export
Bug 20551: Refactor unmotivated partially applied validator function
Bug 20551: (QA follow-up) perltidy and test fixes
Bug 20551: Move some code that might be reused into functions
Bug 20551: (QA follow-up) don't read record content if there is an error during decoding
Bug 20551: Get biblionumber safely in validator function
Bug 20551: Use iterator for retrieving records to save memory
Bug 20551: Cache resultset instead of passing to sub
Bug 20551: Warn and return if deleted biblio not found
Bug 20551: Use timetstamp of deletedbiblio_metadata instead of deletedbibio
Bug 20551: Return 0 instead of undef from _get_record_for_export not to break iterator loop
Bug 20551: Return 0 instead of undef from _get_deleted_record_for_export

Description David Gustafsson 2018-04-10 09:17:23 UTC
Add an option "--include_deleted" to also include possibly deleted biblios in export. This option can only be used together with --date, and --record-type=biblio. The rational behind this is that use case does not makes much sense (if you are doing a full export, why would you include deleted biblios) and thus some extra complexity can be avoided.
 
Superficially related to https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17693, not really quite different as this includes the whole biblio-record (with record status set to "d" in leader) in the regular export, not just biblio ids. This is a requirement for synchronizing with the discovery-system we are currently using for example and imho is a more standard and MARC-compliant way of supporting export of record deletions.
Comment 1 David Gustafsson 2018-04-10 09:33:17 UTC Comment hidden (obsolete)
Comment 2 David Gustafsson 2018-04-10 12:19:56 UTC Comment hidden (obsolete)
Comment 3 David Gustafsson 2018-04-10 12:20:38 UTC
As some might need it, also added an '--deleted_only' option, to export only the deleted biblios.
Comment 4 David Gustafsson 2018-05-07 12:15:41 UTC Comment hidden (obsolete)
Comment 5 David Gustafsson 2018-05-07 12:15:56 UTC
Rebased against Koha master.
Comment 6 David Gustafsson 2018-12-04 15:12:48 UTC
Created attachment 82843 [details] [review]
Bug 20551: Add option for including deleted records in export_records.pl

Add option "--include_deleted" to include deleted
biblios in export_records.pl as marc records with
record status (in leader) set to "d" and
"--deleted_only" to export only deleted biblios.

How to test:
1) Run tests in t/db_dependent/Exporter/Record.t
2) All tests should pass
3) Delete a biblio record in Koha
4) Run the export script as:
   `export_records.pl --date=<date-just-before-deletion>
   --include_deleted`
5) Open the generated koha.mrc, and verify that contains
   the recently deleted record, and that the record header
   has record status "d".
6) Perform the same check with:
   `export_records.pl --date=<date-just-before-deletion>
      --include_deleted --format=xml`
7) Run:
   `export_records.pl --date=<date-just-before-deletion>
      --deleted_only --format=xml`
    and verify that only the deleted biblio was included
    in the export.

Sponsored-by: Gothenburg University Library
Comment 7 David Gustafsson 2018-12-04 15:14:51 UTC
Rebase against master (untested, will test later and verify that still works).
Comment 8 Owen Leonard 2018-12-05 15:44:06 UTC
The functionality of this works as described in the test plan. However, the QA tools raise this warning:

 FAIL   Koha/Exporter/Record.pm
   FAIL   critic
                # Subroutines::ProhibitExplicitReturnUndef: Got 1 violation(s).

And t/db_dependent/Exporter/Record.t returns this error:

t/db_dependent/Exporter/Record.t .. 1/6     # No tests run!

#   Failed test 'No tests run for subtest "export xml"'
#   at t/db_dependent/Exporter/Record.t line 187.
    # No tests run!

#   Failed test 'No tests run for subtest "export iso2709"'
#   at t/db_dependent/Exporter/Record.t line 219.
# Looks like you failed 2 tests of 6.
t/db_dependent/Exporter/Record.t .. Dubious, test returned 2 (wstat 512, 0x200)
Failed 2/6 subtests

Test Summary Report
-------------------
t/db_dependent/Exporter/Record.t (Wstat: 512 Tests: 6 Failed: 2)
  Failed tests:  2-3
  Non-zero exit status: 2
Files=1, Tests=6,  3 wallclock secs ( 0.02 usr  0.01 sys +  1.41 cusr  0.69 csys =  2.13 CPU)
Result: FAIL
Comment 9 David Gustafsson 2018-12-06 15:37:23 UTC
Created attachment 82914 [details] [review]
Bug 20551: Add option for including deleted records in export_records.pl

Add option "--include_deleted" to include deleted
biblios in export_records.pl as marc records with
record status (in leader) set to "d" and
"--deleted_only" to export only deleted biblios.

How to test:
1) Run tests in t/db_dependent/Exporter/Record.t
2) All tests should pass
3) Delete a biblio record in Koha
4) Run the export script as:
   `export_records.pl --date=<date-just-before-deletion>
   --include_deleted`
5) Open the generated koha.mrc, and verify that contains
   the recently deleted record, and that the record header
   has record status "d".
6) Perform the same check with:
   `export_records.pl --date=<date-just-before-deletion>
      --include_deleted --format=xml`
7) Run:
   `export_records.pl --date=<date-just-before-deletion>
      --deleted_only --format=xml`
    and verify that only the deleted biblio was included
    in the export.

Sponsored-by: Gothenburg University Library
Comment 10 David Gustafsson 2018-12-06 15:38:58 UTC
Found the culprit (next in map that accidentally got there in rebase, where should just return empty list), now tests should pass.
Comment 11 David Gustafsson 2018-12-06 15:39:48 UTC
Forgot to fix the perl critic issue, will do.
Comment 12 David Gustafsson 2018-12-06 15:54:38 UTC
Created attachment 82918 [details] [review]
Bug 20551: Add option for including deleted records in export_records.pl

Add option "--include_deleted" to include deleted
biblios in export_records.pl as marc records with
record status (in leader) set to "d" and
"--deleted_only" to export only deleted biblios.

How to test:
1) Run tests in t/db_dependent/Exporter/Record.t
2) All tests should pass
3) Delete a biblio record in Koha
4) Run the export script as:
   `export_records.pl --date=<date-just-before-deletion>
   --include_deleted`
5) Open the generated koha.mrc, and verify that contains
   the recently deleted record, and that the record header
   has record status "d".
6) Perform the same check with:
   `export_records.pl --date=<date-just-before-deletion>
      --include_deleted --format=xml`
7) Run:
   `export_records.pl --date=<date-just-before-deletion>
      --deleted_only --format=xml`
    and verify that only the deleted biblio was included
    in the export.

Sponsored-by: Gothenburg University Library
Comment 13 Frank Hansen 2019-05-13 09:45:01 UTC
Patch didn't apply for me. I got:

CONFLICT (content): Merge conflict in misc/export_records.pl
Failed to merge in the changes.
Patch failed at 0001 Bug 20551: Add option for including deleted records in export_records.pl
Comment 14 Katrin Fischer 2019-05-13 11:27:05 UTC
Same for me, marking "Patch doesn't apply". Frnak, you can set that status too if you run into this the next time!
Comment 15 David Gustafsson 2019-05-29 12:24:30 UTC
Created attachment 90184 [details] [review]
Bug 20551: Add option for including deleted records in export_records.pl

Add option "--include_deleted" to include deleted
biblios in export_records.pl as marc records with
record status (in leader) set to "d" and
"--deleted_only" to export only deleted biblios.

How to test:
1) Run tests in t/db_dependent/Exporter/Record.t
2) All tests should pass
3) Delete a biblio record in Koha
4) Run the export script as:
   `export_records.pl --date=<date-just-before-deletion>
   --include_deleted`
5) Open the generated koha.mrc, and verify that contains
   the recently deleted record, and that the record header
   has record status "d".
6) Perform the same check with:
   `export_records.pl --date=<date-just-before-deletion>
      --include_deleted --format=xml`
7) Run:
   `export_records.pl --date=<date-just-before-deletion>
      --deleted_only --format=xml`
    and verify that only the deleted biblio was included
    in the export.

Sponsored-by: Gothenburg University Library
Comment 16 David Gustafsson 2019-05-29 12:24:47 UTC
Rebased against master.
Comment 17 David Gustafsson 2019-05-29 14:39:30 UTC
Created attachment 90190 [details] [review]
Bug 20551: Fix option name in pod
Comment 18 David Gustafsson 2019-05-31 10:42:29 UTC
Created attachment 90220 [details] [review]
Bug 20551: Rename "marcflavour" field to "schema"
Comment 19 Eric Bégin 2019-10-03 14:11:49 UTC
Test during hackfest 2019

error: sha1 information is lacking or useless (Koha/Exporter/Record.pm).
error: could not build fake ancestor
Patch failed at 0001 Bug 20551: Rename "marcflavour" field to "schema"
Comment 20 David Gustafsson 2019-10-03 14:24:44 UTC
Created attachment 93619 [details] [review]
Bug 20551: Add option for including deleted records in export_records.pl

Add option "--include_deleted" to include deleted
biblios in export_records.pl as marc records with
record status (in leader) set to "d" and
"--deleted_only" to export only deleted biblios.

How to test:
1) Run tests in t/db_dependent/Exporter/Record.t
2) All tests should pass
3) Delete a biblio record in Koha
4) Run the export script as:
   `export_records.pl --date=<date-just-before-deletion>
   --include_deleted`
5) Open the generated koha.mrc, and verify that contains
   the recently deleted record, and that the record header
   has record status "d".
6) Perform the same check with:
   `export_records.pl --date=<date-just-before-deletion>
      --include_deleted --format=xml`
7) Run:
   `export_records.pl --date=<date-just-before-deletion>
      --deleted_only --format=xml`
    and verify that only the deleted biblio was included
    in the export.

Sponsored-by: Gothenburg University Library
Comment 21 David Gustafsson 2019-10-03 14:24:49 UTC
Created attachment 93620 [details] [review]
Bug 20551: Fix option name in pod
Comment 22 David Gustafsson 2019-10-03 14:24:52 UTC
Created attachment 93621 [details] [review]
Bug 20551: Rename "marcflavour" field to "schema"
Comment 23 David Gustafsson 2019-10-03 14:25:25 UTC
Should now be rebased against master.
Comment 24 Frank Hansen 2020-04-14 11:32:03 UTC
(In reply to David Gustafsson from comment #23)
> Should now be rebased against master.

The patch itself applies and works fine for me. However, the test doesn't pass:

t/db_dependent/Exporter/Record.t .. 1/6
    #   Failed test 'Deleted record has the correct leader value'
    #   at t/db_dependent/Exporter/Record.t line 186.
    #          got: '00134dam a22000617a 4500'
    #     expected: '00136dam a22000617a 4500'
    # Looks like you failed 1 test of 4.

#   Failed test 'export xml'
#   at t/db_dependent/Exporter/Record.t line 187.

    #   Failed test 'Deleted record has the correct leader value'
    #   at t/db_dependent/Exporter/Record.t line 218.
    #          got: '00134dam a22000617a 4500'
    #     expected: '00136dam a22000617a 4500'
    # Looks like you failed 1 test of 4.

#   Failed test 'export iso2709'
#   at t/db_dependent/Exporter/Record.t line 219.
t/db_dependent/Exporter/Record.t .. 5/6 # Looks like you failed 2 tests of 6.
t/db_dependent/Exporter/Record.t .. Dubious, test returned 2 (wstat 512, 0x200)
Failed 2/6 subtests

Test Summary Report
-------------------
t/db_dependent/Exporter/Record.t (Wstat: 512 Tests: 6 Failed: 2)
  Failed tests:  2-3
  Non-zero exit status: 2
Files=1, Tests=6,  4 wallclock secs ( 0.02 usr  0.00 sys +  2.94 cusr  0.28 csys =  3.24 CPU)
Result: FAIL
Comment 25 Frank Hansen 2020-04-15 08:49:21 UTC
The test failed because I ran the test on an empty database.
I will now sign off the patch.
Comment 26 Jonathan Druart 2020-04-15 15:22:12 UTC
1. Doesn't it make sense to have this for authority records as well?

2. Why is --date mandatory?

3. POD does not tell that --deleted_only and --include_deleted is only available for bibs

4. --id_list_file does not work with --deleted_only

5. Items are not exported, I guess that's expected but maybe we should have it noted somewhere?
Comment 27 Jonathan Druart 2020-04-16 10:30:50 UTC
*** Bug 17693 has been marked as a duplicate of this bug. ***
Comment 28 Frank Hansen 2020-04-17 09:38:05 UTC
(In reply to Jonathan Druart from comment #26)
> 1. Doesn't it make sense to have this for authority records as well?

The authority records are stored in auth_header table. But deleted records is not kept anywhere else. As I can see, this would require to keep deleted authority records in an another table like deletedauth_header.
Comment 29 Katrin Fischer 2020-05-23 11:43:54 UTC
Hi Frank, you are right, we do not keep the deleted authorities, so that takes care of 1. I think some of the other questions could still be answered/improved:


(In reply to Jonathan Druart from comment #26)
> 1. Doesn't it make sense to have this for authority records as well?

Because we don't keep them ;)

> 2. Why is --date mandatory?

Not sure about this one, can you explain?

> 3. POD does not tell that --deleted_only and --include_deleted is only
> available for bibs

Easy fix to documentation I think and it could help people who are not aware of database structure.

> 4. --id_list_file does not work with --deleted_only

Cannot tell about this one, but sounds worth looking into.

> 5. Items are not exported, I guess that's expected but maybe we should have
> it noted somewhere?

Maybe also a note in the docs?
Comment 30 David Gustafsson 2020-05-27 14:37:21 UTC
(In reply to Jonathan Druart from comment #26)
> 1. Doesn't it make sense to have this for authority records as well?
> 
> 2. Why is --date mandatory?
> 
> 3. POD does not tell that --deleted_only and --include_deleted is only
> available for bibs
> 
> 4. --id_list_file does not work with --deleted_only
> 
> 5. Items are not exported, I guess that's expected but maybe we should have
> it noted somewhere?

2) I think I just assumed it was required since included in the SQL-query, also before the patch. Did not realize that 
"SELECT * FROM biblio_metadata WHERE timestamp > ''" is equivalent with "SELECT * FROM biblio_metadata" if every timestamp entry has a value. So with that in mind an empty timestamp is ok, though the code is a real eyesore :)

3) I think it's already pretty clear since will produce an error message is --included_deleted or --deleted_only is used in conjunction with record_type other than 'bibs'. But I clarified this in the usage documentation for said options. 

4) Did not work since have to add special case for @deleted_record_ids, fixed this.

5). There actually is a "deleteditems" table so I guess it would be possible to include these, even though it's hard to imagine why you would want to do this. Since the default is to include items perhaps it would be best to leave this as is and never include items for deleted biblios, or you will get more data than you probably want by default. It would also increase code complexity, probably with little benefit. If someone requests this it can be fixed later perhaps. Will point it out in the documentation that items are not included.
Comment 31 David Gustafsson 2020-05-27 14:50:16 UTC
Created attachment 105403 [details] [review]
Bug 20551: Improve documentation and make compatible with more options
Comment 32 Frank Hansen 2021-02-05 15:04:39 UTC
Created attachment 116408 [details] [review]
Bug 20551: Add option for including deleted records in export_records.pl

Add option "--include_deleted" to include deleted
biblios in export_records.pl as marc records with
record status (in leader) set to "d" and
"--deleted_only" to export only deleted biblios.

How to test:
1) Run tests in t/db_dependent/Exporter/Record.t
2) All tests should pass
3) Delete a biblio record in Koha
4) Run the export script as:
   `export_records.pl --date=<date-just-before-deletion>
   --include_deleted`
5) Open the generated koha.mrc, and verify that contains
   the recently deleted record, and that the record header
   has record status "d".
6) Perform the same check with:
   `export_records.pl --date=<date-just-before-deletion>
      --include_deleted --format=xml`
7) Run:
   `export_records.pl --date=<date-just-before-deletion>
      --deleted_only --format=xml`
    and verify that only the deleted biblio was included
    in the export.

Sponsored-by: Gothenburg University Library

Great job!

Signed-off-by: Frank Hansen <frank.hansen@ub.lu.se>
Comment 33 Frank Hansen 2021-02-05 15:04:43 UTC
Created attachment 116409 [details] [review]
Bug 20551: Fix option name in pod

Signed-off-by: Frank Hansen <frank.hansen@ub.lu.se>
Comment 34 Frank Hansen 2021-02-05 15:04:46 UTC
Created attachment 116410 [details] [review]
Bug 20551: Rename "marcflavour" field to "schema"

Signed-off-by: Frank Hansen <frank.hansen@ub.lu.se>
Comment 35 Frank Hansen 2021-02-05 15:04:50 UTC
Created attachment 116411 [details] [review]
Bug 20551: Improve documentation and make compatible with more options

Signed-off-by: Frank Hansen <frank.hansen@ub.lu.se>
Comment 36 Katrin Fischer 2021-05-02 14:11:52 UTC
> 
> 5). There actually is a "deleteditems" table so I guess it would be possible
> to include these, even though it's hard to imagine why you would want to do
> this. Since the default is to include items perhaps it would be best to
> leave this as is and never include items for deleted biblios, or you will
> get more data than you probably want by default. It would also increase code
> complexity, probably with little benefit. If someone requests this it can be
> fixed later perhaps. Will point it out in the documentation that items are
> not included.

I think introducing the export of deleted records and deleted items separately would be a good idea. Let's start with this well defined feature and think about the deleted items some more.

While records have a flag to say "deleted", we don't have the same for items. So we could start exporting deleted items that belonged to deleted records, but what about other deleted items on existing records? How to "mark" them as deleted?
Comment 37 Katrin Fischer 2021-05-02 14:12:55 UTC
I am sorry, but tests are failing :(

kohadev-koha@kohadevbox:/home/vagrant/kohaclone$ prove t/db_dependent/Exporter/Record.t
t/db_dependent/Exporter/Record.t .. 1/6 Use of uninitialized value in concatenation (.) or string at /usr/share/perl5/MARC/File/XML.pm line 397, <$fh> chunk 7.

    #   Failed test 'Deleted record has the correct leader value'
    #   at t/db_dependent/Exporter/Record.t line 179.
    #          got: '00134dam a22000617a 4500'
    #     expected: '00136dam a22000617a 4500'
    # Looks like you failed 1 test of 4.

#   Failed test 'export xml'
#   at t/db_dependent/Exporter/Record.t line 180.

    #   Failed test 'Deleted record has the correct leader value'
    #   at t/db_dependent/Exporter/Record.t line 211.
    #          got: '00134dam a22000617a 4500'
    #     expected: '00136dam a22000617a 4500'
    # Looks like you failed 1 test of 4.

#   Failed test 'export iso2709'
#   at t/db_dependent/Exporter/Record.t line 212.
# Looks like you failed 2 tests of 6.
t/db_dependent/Exporter/Record.t .. Dubious, test returned 2 (wstat 512, 0x200)
Failed 2/6 subtests
Comment 38 David Gustafsson 2021-07-23 11:58:27 UTC
Created attachment 123099 [details] [review]
Bug 20551: fix tests
Comment 39 David Gustafsson 2021-07-23 12:00:49 UTC
Created attachment 123101 [details] [review]
Bug 20551: Add missing subroutine import
Comment 40 David Gustafsson 2021-07-23 12:00:54 UTC
Created attachment 123102 [details] [review]
Bug 20551: Fix tests
Comment 41 David Gustafsson 2021-07-23 12:16:12 UTC
(In reply to Katrin Fischer from comment #36)
> > 
> > 5). There actually is a "deleteditems" table so I guess it would be possible
> > to include these, even though it's hard to imagine why you would want to do
> > this. Since the default is to include items perhaps it would be best to
> > leave this as is and never include items for deleted biblios, or you will
> > get more data than you probably want by default. It would also increase code
> > complexity, probably with little benefit. If someone requests this it can be
> > fixed later perhaps. Will point it out in the documentation that items are
> > not included.
> 
> I think introducing the export of deleted records and deleted items
> separately would be a good idea. Let's start with this well defined feature
> and think about the deleted items some more.
> 
> While records have a flag to say "deleted", we don't have the same for
> items. So we could start exporting deleted items that belonged to deleted
> records, but what about other deleted items on existing records? How to
> "mark" them as deleted?

Now the tests are fixed. Regarding export of deleted items from a first glance I can see how you would expect this just as matter of symmetry with the regular export, but both conceptually and for lots of other reasons it makes no sense at all.

1. Exporting deleted records is probably never done to save the record information somewhere else, but to use data in records to match a record somewhere else to remove. I cannot imagine a case where this data would be found on an item-level.

2. It would require us to refactor C4::Biblio::EmbedItemsInMarcBiblio just for this specific case (which is an edge case that in practice will probably never be utilized), further increasing the burden of code maintenance and risk of introducing new bugs.

If I'm incorrect and this feature is requested by many users it's always possible to implement later, but I would be very surprised if this turns out to be the case.
Comment 42 David Gustafsson 2021-09-23 16:38:51 UTC
Created attachment 125211 [details] [review]
Bug 20551: Add option for including deleted records in export_records.pl

Add option "--include_deleted" to include deleted
biblios in export_records.pl as marc records with
record status (in leader) set to "d" and
"--deleted_only" to export only deleted biblios.

How to test:
1) Run tests in t/db_dependent/Exporter/Record.t
2) All tests should pass
3) Delete a biblio record in Koha
4) Run the export script as:
   `export_records.pl --date=<date-just-before-deletion>
   --include_deleted`
5) Open the generated koha.mrc, and verify that contains
   the recently deleted record, and that the record header
   has record status "d".
6) Perform the same check with:
   `export_records.pl --date=<date-just-before-deletion>
      --include_deleted --format=xml`
7) Run:
   `export_records.pl --date=<date-just-before-deletion>
      --deleted_only --format=xml`
    and verify that only the deleted biblio was included
    in the export.

Sponsored-by: Gothenburg University Library

Signed-off-by: Frank Hansen <frank.hansen@ub.lu.se>
Comment 43 David Gustafsson 2021-09-23 16:38:55 UTC
Created attachment 125212 [details] [review]
Bug 20551: Fix option name in pod

Signed-off-by: Frank Hansen <frank.hansen@ub.lu.se>
Comment 44 David Gustafsson 2021-09-23 16:39:00 UTC
Created attachment 125213 [details] [review]
Bug 20551: Rename "marcflavour" field to "schema"

Signed-off-by: Frank Hansen <frank.hansen@ub.lu.se>
Comment 45 David Gustafsson 2021-09-23 16:39:05 UTC
Created attachment 125214 [details] [review]
Bug 20551: Improve documentation and make compatible with more options

Signed-off-by: Frank Hansen <frank.hansen@ub.lu.se>
Comment 46 David Gustafsson 2021-09-23 16:39:13 UTC
Created attachment 125215 [details] [review]
Bug 20551: Add missing subroutine import
Comment 47 David Gustafsson 2021-09-23 16:39:19 UTC
Created attachment 125216 [details] [review]
Bug 20551: Fix tests
Comment 48 David Gustafsson 2021-09-23 16:45:08 UTC
Rebase against master.
Comment 49 Frank Hansen 2021-12-23 17:14:44 UTC
Created attachment 128895 [details] [review]
Bug 20551: Add option for including deleted records in export_records.pl

Add option "--include_deleted" to include deleted
biblios in export_records.pl as marc records with
record status (in leader) set to "d" and
"--deleted_only" to export only deleted biblios.

How to test:
1) Run tests in t/db_dependent/Exporter/Record.t
2) All tests should pass
3) Delete a biblio record in Koha
4) Run the export script as:
   `export_records.pl --date=<date-just-before-deletion>
   --include_deleted`
5) Open the generated koha.mrc, and verify that contains
   the recently deleted record, and that the record header
   has record status "d".
6) Perform the same check with:
   `export_records.pl --date=<date-just-before-deletion>
      --include_deleted --format=xml`
7) Run:
   `export_records.pl --date=<date-just-before-deletion>
      --deleted_only --format=xml`
    and verify that only the deleted biblio was included
    in the export.

Sponsored-by: Gothenburg University Library

Signed-off-by: Frank Hansen <frank.hansen@ub.lu.se>
Comment 50 Frank Hansen 2021-12-23 17:14:49 UTC
Created attachment 128896 [details] [review]
Bug 20551: Fix option name in pod

Signed-off-by: Frank Hansen <frank.hansen@ub.lu.se>
Comment 51 Frank Hansen 2021-12-23 17:14:54 UTC
Created attachment 128897 [details] [review]
Bug 20551: Rename "marcflavour" field to "schema"

Signed-off-by: Frank Hansen <frank.hansen@ub.lu.se>
Comment 52 Frank Hansen 2021-12-23 17:15:00 UTC
Created attachment 128898 [details] [review]
Bug 20551: Improve documentation and make compatible with more options

Signed-off-by: Frank Hansen <frank.hansen@ub.lu.se>
Comment 53 Frank Hansen 2021-12-23 17:15:06 UTC
Created attachment 128899 [details] [review]
Bug 20551: Add missing subroutine import
Comment 54 Frank Hansen 2021-12-23 17:15:11 UTC
Created attachment 128900 [details] [review]
Bug 20551: Fix tests

Signed-off-by: Frank Hansen <frank.hansen@ub.lu.se>

Works fine for me.
Signing off
Comment 55 Cab Vinton 2021-12-23 19:24:03 UTC
Will this enhancement be available from the staff interface? i.e., Tools > Export data (.../cgi-bin/koha/tools/export.pl)
Comment 56 Katrin Fischer 2022-01-02 15:27:22 UTC
(In reply to Cab Vinton from comment #55)
> Will this enhancement be available from the staff interface? i.e., Tools >
> Export data (.../cgi-bin/koha/tools/export.pl)

From looking at the patch set this will only appear in the command line script. But as this has been waiting a while now, I suggest we file a separate report for making this available on GUI side. What do you think?
Comment 57 Marcel de Rooy 2022-03-18 09:59:17 UTC
+    my $marc_flavour = C4::Context->preference('marcflavour');
+    my $biblio_metadata = $resultset->find({
+        'biblionumber' => $biblionumber,
+        'format' => 'marcxml',
+        'marcflavour' => $marc_flavour
+    });
+    my $marc_xml = $biblio_metadata->metadata;
+    $marc_xml = StripNonXmlChars($marc_xml);
+
+    my $record = eval {
+        MARC::Record::new_from_xml($marc_xml, 'UTF-8', $marc_flavour)
+    };
+    if (!$record) {

You test $record, but you should have tested $biblio_metadata too ?
Otherwise you already crash on ->metadata..
Comment 58 Marcel de Rooy 2022-03-18 10:03:14 UTC
+    return unless (@{$record_ids} || @{$deleted_record_ids} && $format ne 'csv');

Parentheses here would be more helpful inside. The outer ones are not needed.
But in order to read easier, you could do A || ( B && C )
Comment 59 Marcel de Rooy 2022-03-18 10:14:37 UTC
sub export

        binmode $fh, ':encoding(UTF-8)' unless $format eq 'csv';
    } else {
        binmode STDOUT, ':encoding(UTF-8)' unless $format eq 'csv';

What happens if we choose csv? Are we not doing UTF-8 ?
Later we do:

print marc2csv( $record_ids, $csv_profile_id, $itemnumbers );
Comment 60 Marcel de Rooy 2022-03-18 10:16:08 UTC
(In reply to Marcel de Rooy from comment #58)
> +    return unless (@{$record_ids} || @{$deleted_record_ids} && $format ne
> 'csv');
> 
> Parentheses here would be more helpful inside. The outer ones are not needed.
> But in order to read easier, you could do A || ( B && C )

And functionally: why not deleted records in csv ?
Comment 61 Marcel de Rooy 2022-03-18 10:18:42 UTC
+                my ($record_type) = @_;

+            my $validator = $encoding_validator->('Record');

+                $validator = $encoding_validator->('Deleted record');

Deleted flag ?
Comment 62 Marcel de Rooy 2022-03-18 10:24:09 UTC
-            my $record = _get_record_for_export( { %$params, record_id => $record_id } );

+            for my $record (@records, @deleted_records) {
+                print MARC::File::XML::record($record);

Noting that get_record_for_export does a lot more and we seem to loose that now?
Comment 63 Marcel de Rooy 2022-03-18 10:30:46 UTC
Ok, not all my questions have the same importance and I realize that this dev is already too long in the pipeline, but I like to have feedback now.

Changing status to reflect that
Comment 64 Cab Vinton 2022-03-18 10:33:06 UTC
(In reply to Katrin Fischer from comment #56)
> From looking at the patch set this will only appear in the command line
> script. But as this has been waiting a while now, I suggest we file a
> separate report for making this available on GUI side. What do you think?

I missed this earlier, but Yes, a separate report for the GUI side makes sense. I would wait until this one is fully signed off, but others are welcome to proceed if they wish.

Not sure how prevalent this is, but I believe quite a few union catalogs still ask contributing members to regularly send updates w/ lists of records added & deleted. So this feature would come in handy for all the folks who don't have access to the command line.
Comment 65 David Gustafsson 2022-04-01 13:30:54 UTC
(In reply to Marcel de Rooy from comment #57)
> +    my $marc_flavour = C4::Context->preference('marcflavour');
> +    my $biblio_metadata = $resultset->find({
> +        'biblionumber' => $biblionumber,
> +        'format' => 'marcxml',
> +        'marcflavour' => $marc_flavour
> +    });
> +    my $marc_xml = $biblio_metadata->metadata;
> +    $marc_xml = StripNonXmlChars($marc_xml);
> +
> +    my $record = eval {
> +        MARC::Record::new_from_xml($marc_xml, 'UTF-8', $marc_flavour)
> +    };
> +    if (!$record) {
> 
> You test $record, but you should have tested $biblio_metadata too ?
> Otherwise you already crash on ->metadata..

Hi, thanks for the feedback. There is no definite right or wrong, but personally try to avoid defensive programming where things are very unlikely to fail. Ids should be valid since have been previously fetched from database, in case of the XML-decoding it's highly unlikely to go wrong, but I can imagine cases where it could.
Comment 66 David Gustafsson 2022-04-01 13:36:38 UTC
(In reply to Marcel de Rooy from comment #57)
> +    my $marc_flavour = C4::Context->preference('marcflavour');
> +    my $biblio_metadata = $resultset->find({
> +        'biblionumber' => $biblionumber,
> +        'format' => 'marcxml',
> +        'marcflavour' => $marc_flavour
> +    });
> +    my $marc_xml = $biblio_metadata->metadata;
> +    $marc_xml = StripNonXmlChars($marc_xml);
> +
> +    my $record = eval {
> +        MARC::Record::new_from_xml($marc_xml, 'UTF-8', $marc_flavour)
> +    };
> +    if (!$record) {
> 
> You test $record, but you should have tested $biblio_metadata too ?
> Otherwise you already crash on ->metadata..

Hi, thanks for the feedback. There is no definite right or wrong, but personally try to avoid defensive programming where things are very unlikely to fail. Ids should be valid since have been previously fetched from database, in case of the XML-decoding it's highly unlikely to go wrong, but I can imagine cases where it could.(In reply to Marcel de Rooy from comment #58)
> +    return unless (@{$record_ids} || @{$deleted_record_ids} && $format ne
> 'csv');
> 
> Parentheses here would be more helpful inside. The outer ones are not needed.
> But in order to read easier, you could do A || ( B && C )

Yes,they could be removed. Personally I think it's more confusing, or at least a bit unsightly, to use parenthesis for anything else than changing operator precedence. At least with "||" and "&&" where there really should be no confusion.
Comment 67 David Gustafsson 2022-04-01 13:38:14 UTC
(In reply to Marcel de Rooy from comment #59)
> sub export
> 
>         binmode $fh, ':encoding(UTF-8)' unless $format eq 'csv';
>     } else {
>         binmode STDOUT, ':encoding(UTF-8)' unless $format eq 'csv';
> 
> What happens if we choose csv? Are we not doing UTF-8 ?
> Later we do:
> 
> print marc2csv( $record_ids, $csv_profile_id, $itemnumbers );

I had a look at this and don't think this block is part of the patch.
Comment 68 David Gustafsson 2022-04-01 13:50:21 UTC
(In reply to Marcel de Rooy from comment #59)
> sub export
> 
>         binmode $fh, ':encoding(UTF-8)' unless $format eq 'csv';
>     } else {
>         binmode STDOUT, ':encoding(UTF-8)' unless $format eq 'csv';
> 
> What happens if we choose csv? Are we not doing UTF-8 ?
> Later we do:
> 
> print marc2csv( $record_ids, $csv_profile_id, $itemnumbers );

marc2csv uses GetMarcBiblio to load biblios that does not support loading deleted biblios which resides in another table. marc2csv would have to be carefully reviewed and refactored to ensure it can handle deleted biblios, this would increase the scope of this patch and I just did not think it was worth the effort. A separate issues could perhaps be created in case there is an interest implementing support for this.
Comment 69 David Gustafsson 2022-04-01 13:52:36 UTC
Sorry, the last comment was meant to be a response to:

> And functionally: why not deleted records in csv ?
Comment 70 David Gustafsson 2022-04-01 14:06:36 UTC
(In reply to Marcel de Rooy from comment #61)
> +                my ($record_type) = @_;
> 
> +            my $validator = $encoding_validator->('Record');
> 
> +                $validator = $encoding_validator->('Deleted record');
> 
> Deleted flag ?

I don't think I understand. The string is correct, but looking at the code I wonder why creating a closure instead of just providing the argument further down, I should probably change this.
Comment 71 David Gustafsson 2022-04-01 14:23:13 UTC
(In reply to Marcel de Rooy from comment #62)
> -            my $record = _get_record_for_export( { %$params, record_id =>
> $record_id } );
> 
> +            for my $record (@records, @deleted_records) {
> +                print MARC::File::XML::record($record);
> 
> Noting that get_record_for_export does a lot more and we seem to loose that
> now?

Yes, right now I can't think of any reason why deleted records should not be a new record type in _get_record_for_export and processed in the same way as the others (authorities and biblios). I will look into this.
Comment 72 David Gustafsson 2022-04-01 14:26:18 UTC
(_get_record_for_export is still called to fetch the other records though, so no functionality has been removed there, but reading what it does don't se how most of the same operations should not work also on deleted biblios)
Comment 73 David Gustafsson 2022-04-04 13:12:49 UTC
Created attachment 132936 [details] [review]
Bug 20551: Fetch deleted biblios through _get_record_export
Comment 74 David Gustafsson 2022-04-04 13:25:39 UTC
Created attachment 132937 [details] [review]
Bug 20551: Refactor unmotivated partially applied validator function
Comment 75 David Gustafsson 2022-04-04 13:32:35 UTC
Thanks again for the feedback, I think at least the most important issues should now be fixed.
Comment 76 David Gustafsson 2024-03-01 15:27:31 UTC
Must have forgotten to change status, setting to Needs Signoff.
Comment 77 Arthur Suzuki 2024-11-18 10:30:06 UTC
(In reply to David Gustafsson from comment #76)
> Must have forgotten to change status, setting to Needs Signoff.

Hi David,
The patches do not apply anymore but I'd be happy to signoff if you rebase it on current main :)
Best, Arthur
Comment 78 Arthur Suzuki 2025-01-16 10:41:50 UTC
Created attachment 176623 [details] [review]
Bug 20551: Add option for including deleted records in export_records.pl

Add option "--include_deleted" to include deleted
biblios in export_records.pl as marc records with
record status (in leader) set to "d" and
"--deleted_only" to export only deleted biblios.

How to test:
1) Run tests in t/db_dependent/Exporter/Record.t
2) All tests should pass
3) Delete a biblio record in Koha
4) Run the export script as:
   `export_records.pl --date=<date-just-before-deletion>
   --include_deleted`
5) Open the generated koha.mrc, and verify that contains
   the recently deleted record, and that the record header
   has record status "d".
6) Perform the same check with:
   `export_records.pl --date=<date-just-before-deletion>
      --include_deleted --format=xml`
7) Run:
   `export_records.pl --date=<date-just-before-deletion>
      --deleted_only --format=xml`
    and verify that only the deleted biblio was included
    in the export.

Sponsored-by: Gothenburg University Library

Signed-off-by: Frank Hansen <frank.hansen@ub.lu.se>
Comment 79 Arthur Suzuki 2025-01-16 10:42:06 UTC
Created attachment 176624 [details] [review]
Bug 20551: Fix option name in pod

Signed-off-by: Frank Hansen <frank.hansen@ub.lu.se>
Comment 80 Arthur Suzuki 2025-01-16 10:42:15 UTC
Created attachment 176625 [details] [review]
Bug 20551: Rename "marcflavour" field to "schema"

Signed-off-by: Frank Hansen <frank.hansen@ub.lu.se>
Comment 81 Arthur Suzuki 2025-01-16 10:42:21 UTC
Created attachment 176626 [details] [review]
Bug 20551: Improve documentation and make compatible with more options

Signed-off-by: Frank Hansen <frank.hansen@ub.lu.se>
Comment 82 Arthur Suzuki 2025-01-16 10:42:25 UTC
Created attachment 176627 [details] [review]
Bug 20551: Add missing subroutine import
Comment 83 Arthur Suzuki 2025-01-16 10:42:31 UTC
Created attachment 176628 [details] [review]
Bug 20551: Fix tests

Signed-off-by: Frank Hansen <frank.hansen@ub.lu.se>

Works fine for me.
Signing off
Comment 84 Arthur Suzuki 2025-01-16 10:42:36 UTC
Created attachment 176629 [details] [review]
Bug 20551: Fetch deleted biblios through _get_record_export
Comment 85 Arthur Suzuki 2025-01-16 10:42:40 UTC
Created attachment 176630 [details] [review]
Bug 20551: Refactor unmotivated partially applied validator function
Comment 86 Arthur Suzuki 2025-01-16 10:42:45 UTC
Created attachment 176631 [details] [review]
Bug 20551: (QA follow-up) perltidy and test fixes
Comment 87 Arthur Suzuki 2025-01-16 10:44:09 UTC
Just made a rebase a small patch to make the QA scripts happy.
Arthur
Comment 88 David Gustafsson 2025-02-05 17:28:39 UTC
Created attachment 177536 [details] [review]
Bug 20551: Move some code that might be reused into functions
Comment 89 Fridolin Somers 2025-02-18 14:36:28 UTC
Ohhhh there is an issue when biblio.biblionumber is on control field like 001 :

my ($id_tag, $id_subfield) = GetMarcFromKohaField('biblio.biblionumber', '');
my $record_id = $record->subfield($id_tag, $id_subfield);

Fails with :
Control fields (generally, just tags below 010) do not have subfields, use data() at /usr/share/perl5/MARC/Record.pm line 240.
Comment 90 Fridolin Somers 2025-02-18 14:40:08 UTC
Maybe when record can not be encoded we should not try to read its content
Comment 91 Arthur Suzuki 2025-02-19 14:59:41 UTC
Created attachment 178343 [details] [review]
Bug 20551: Add option for including deleted records in export_records.pl

Add option "--include_deleted" to include deleted
biblios in export_records.pl as marc records with
record status (in leader) set to "d" and
"--deleted_only" to export only deleted biblios.

How to test:
1) Run tests in t/db_dependent/Exporter/Record.t
2) All tests should pass
3) Delete a biblio record in Koha
4) Run the export script as:
   `export_records.pl --date=<date-just-before-deletion>
   --include_deleted`
5) Open the generated koha.mrc, and verify that contains
   the recently deleted record, and that the record header
   has record status "d".
6) Perform the same check with:
   `export_records.pl --date=<date-just-before-deletion>
      --include_deleted --format=xml`
7) Run:
   `export_records.pl --date=<date-just-before-deletion>
      --deleted_only --format=xml`
    and verify that only the deleted biblio was included
    in the export.

Sponsored-by: Gothenburg University Library

Signed-off-by: Frank Hansen <frank.hansen@ub.lu.se>
Comment 92 Arthur Suzuki 2025-02-19 14:59:45 UTC
Created attachment 178344 [details] [review]
Bug 20551: Fix option name in pod

Signed-off-by: Frank Hansen <frank.hansen@ub.lu.se>
Comment 93 Arthur Suzuki 2025-02-19 14:59:50 UTC
Created attachment 178345 [details] [review]
Bug 20551: Rename "marcflavour" field to "schema"

Signed-off-by: Frank Hansen <frank.hansen@ub.lu.se>
Comment 94 Arthur Suzuki 2025-02-19 14:59:54 UTC
Created attachment 178346 [details] [review]
Bug 20551: Improve documentation and make compatible with more options

Signed-off-by: Frank Hansen <frank.hansen@ub.lu.se>
Comment 95 Arthur Suzuki 2025-02-19 14:59:58 UTC
Created attachment 178347 [details] [review]
Bug 20551: Add missing subroutine import
Comment 96 Arthur Suzuki 2025-02-19 15:00:02 UTC
Created attachment 178348 [details] [review]
Bug 20551: Fix tests

Signed-off-by: Frank Hansen <frank.hansen@ub.lu.se>

Works fine for me.
Signing off
Comment 97 Arthur Suzuki 2025-02-19 15:00:07 UTC
Created attachment 178349 [details] [review]
Bug 20551: Fetch deleted biblios through _get_record_export
Comment 98 Arthur Suzuki 2025-02-19 15:00:11 UTC
Created attachment 178350 [details] [review]
Bug 20551: Refactor unmotivated partially applied validator function
Comment 99 Arthur Suzuki 2025-02-19 15:00:15 UTC
Created attachment 178351 [details] [review]
Bug 20551: (QA follow-up) perltidy and test fixes
Comment 100 Arthur Suzuki 2025-02-19 15:00:19 UTC
Created attachment 178352 [details] [review]
Bug 20551: Move some code that might be reused into functions
Comment 101 Arthur Suzuki 2025-02-19 15:00:23 UTC
Created attachment 178353 [details] [review]
Bug 20551: (QA follow-up) don't read record content if there is an error during decoding
Comment 102 Arthur Suzuki 2025-02-19 15:04:49 UTC
just made a rebase using autorebase script + small patch in reply to fridolin's comment
Comment 103 David Gustafsson 2025-02-19 19:29:22 UTC
Created attachment 178378 [details] [review]
Bug 20551: Get biblionumber safly in validator function
Comment 104 David Gustafsson 2025-02-19 19:30:41 UTC
Created attachment 178379 [details] [review]
Bug 20551: Get biblionumber safely in validator function
Comment 105 David Gustafsson 2025-02-19 19:32:40 UTC
(In reply to Arthur Suzuki from comment #102)
> just made a rebase using autorebase script + small patch in reply to
> fridolin's comment

Thanks for the rebase. As it's useful to still get the biblionumber and not that hard to fix I restored including it in the error message, but now done so will not crash if not stored in a subfield.
Comment 106 Fridolin Somers 2025-02-24 09:13:33 UTC
I think for performance, this call should be outside validator :

my ( $id_tag, $id_code ) = GetMarcFromKohaField( 'biblio.biblionumber', '' );

Because validator is called in the loop.

We have some use of export more that 1 million records :)
Comment 107 Fridolin Somers 2025-02-24 09:17:01 UTC
Arg I've been using this feature on an a big export and DB server crashed.
I see the output file is not filled little by little.

I think it comes from this change : 

+        @records = map {
+            my $record = _get_record_for_export( { %{$params}, record_id => $_ } );
+            $record ? $record : ();
+        } @{$record_ids};

For a huge export the @records object will take too much RAM.

Without this path the records are build one by one.

Please make some tests to confirm.
Comment 108 David Gustafsson 2025-02-24 13:50:54 UTC
Created attachment 178575 [details] [review]
Bug 20551: Use iterator for retrieving records to save memory
Comment 109 David Gustafsson 2025-02-24 18:30:58 UTC
Created attachment 178617 [details] [review]
Bug 20551: Use iterator for retrieving records to save memory
Comment 110 David Gustafsson 2025-02-24 18:33:18 UTC
(In reply to Fridolin Somers from comment #107)
> Arg I've been using this feature on an a big export and DB server crashed.
> I see the output file is not filled little by little.
> 
> I think it comes from this change : 
> 
> +        @records = map {
> +            my $record = _get_record_for_export( { %{$params}, record_id =>
> $_ } );
> +            $record ? $record : ();
> +        } @{$record_ids};
> 
> For a huge export the @records object will take too much RAM.
> 
> Without this path the records are build one by one.
> 
> Please make some tests to confirm.

Good catch, didn't think about that aspect, I fixed this by using an iterator instead. I don't think we need to think about the performance cost of running GetMarcFromKohaField multiple times as it's only called for records with encoding errors, which should be a very small proportion of the total records.
Comment 111 David Gustafsson 2025-02-24 18:37:50 UTC
Created attachment 178618 [details] [review]
Bug 20551: Use iterator for retrieving records to save memory
Comment 112 David Gustafsson 2025-02-24 18:39:15 UTC
Created attachment 178619 [details] [review]
Bug 20551: Use iterator for retrieving records to save memory
Comment 113 Arthur Suzuki 2025-02-25 10:35:57 UTC
Hello,
In some condition the code could fail :
in the sub _get_deleted_biblio_for_export :
----
    # Creating schema is expensive, allow caller to                                                                                                                                                                 
    # pass it so don't have to recreate for each call                                                                                                                                                               
    my $resultset       = $params->{resultset} || Koha::Database->new()->schema()->resultset('DeletedbiblioMetadata');
    my $marc_flavour    = C4::Context->preference('marcflavour');
    my $biblio_metadata = $resultset->find(
        {
            'biblionumber' => $biblionumber,
            'format'       => 'marcxml',
            'schema'       => $marc_flavour
        }
    );
    my $marc_xml = $biblio_metadata->metadata;
    $marc_xml = StripNonXmlChars($marc_xml);
----

It seems that an empty result set could lead to querying an empty $biblio_metadata object, resulting in a crash.
Probably should warn about "no deleted records" and early return in case of empty resultset?
Comment 114 Arthur Suzuki 2025-02-25 13:22:15 UTC
Also I had some errors in the record_iterator sub.
Fixed by explicitely using $deleted_record_id instead of $_ :

        my $records_iterator = sub {
            if (my $record_id = pop(@record_ids)) {
                return _get_record_for_export( { %{$params}, record_id => $record_id } );
            }
            elsif (my $deleted_record_id = pop(@deleted_record_ids)) {
                return _get_record_for_export(
                    {
                        %{$params},
                        record_type => 'deleted_bibs',
-                       record_id   => $_,
+                       record_id   => $deleted_record_id,
                        resultset   => $deleted_resultset
                    }
                );
            }
            return;
        };

I guess the previous error was due to no biblionumber being passed to the _get_record_for_export sub.
Comment 115 David Gustafsson 2025-02-25 13:31:56 UTC
Created attachment 178679 [details] [review]
Bug 20551: Use iterator for retrieving records to save memory
Comment 116 David Gustafsson 2025-02-25 13:32:00 UTC
Created attachment 178680 [details] [review]
Bug 20551: Cache resultset instead of passing to sub
Comment 117 David Gustafsson 2025-02-25 13:32:05 UTC
Created attachment 178681 [details] [review]
Bug 20551: Warn and return if deleted biblio not found
Comment 118 David Gustafsson 2025-02-25 13:32:09 UTC
Created attachment 178682 [details] [review]
Bug 20551: Use timetstamp of deletedbiblio_metadata instead of deletedbibio
Comment 119 David Gustafsson 2025-02-25 13:41:46 UTC
(In reply to Arthur Suzuki from comment #114)
> Also I had some errors in the record_iterator sub.
> Fixed by explicitely using $deleted_record_id instead of $_ :
> 
>         my $records_iterator = sub {
>             if (my $record_id = pop(@record_ids)) {
>                 return _get_record_for_export( { %{$params}, record_id =>
> $record_id } );
>             }
>             elsif (my $deleted_record_id = pop(@deleted_record_ids)) {
>                 return _get_record_for_export(
>                     {
>                         %{$params},
>                         record_type => 'deleted_bibs',
> -                       record_id   => $_,
> +                       record_id   => $deleted_record_id,
>                         resultset   => $deleted_resultset
>                     }
>                 );
>             }
>             return;
>         };
> 
> I guess the previous error was due to no biblionumber being passed to the
> _get_record_for_export sub.

I don't think that could occur as the deleted biblionumber is always retrieved from the database before and thus always exists. But I agree it should be added to be consistent with (In reply to Arthur Suzuki from comment #113)
> Hello,
> In some condition the code could fail :
> in the sub _get_deleted_biblio_for_export :
> ----
>     # Creating schema is expensive, allow caller to                         
> 
>     # pass it so don't have to recreate for each call                       
> 
>     my $resultset       = $params->{resultset} ||
> Koha::Database->new()->schema()->resultset('DeletedbiblioMetadata');
>     my $marc_flavour    = C4::Context->preference('marcflavour');
>     my $biblio_metadata = $resultset->find(
>         {
>             'biblionumber' => $biblionumber,
>             'format'       => 'marcxml',
>             'schema'       => $marc_flavour
>         }
>     );
>     my $marc_xml = $biblio_metadata->metadata;
>     $marc_xml = StripNonXmlChars($marc_xml);
> ----
> 
> It seems that an empty result set could lead to querying an empty
> $biblio_metadata object, resulting in a crash.
> Probably should warn about "no deleted records" and early return in case of
> empty resultset?

I don't think that could occur as the deleted biblionumber is always retrieved from the database before and thus always exists. But I agree it should be added to be consistent with _get_biblio_for_export and future proofing the code. You are right it was triggered by me forgetting to replace $_ with $deleted_record_id in the previous patch. I have fixed this now. Also changed the table deleted biblios is selected from in export_records.pl from deletedbiblio to deletedbiblio_metadata as biblio_metadata is used for non deleted biblios. The deleted biblios resultset is now also cached instead of passed as an argument.
Comment 120 Arthur Suzuki 2025-03-26 08:34:38 UTC
Tried to apply patches to current master but got the tests failing :
'''
kohadev-koha@kohadevbox:koha(main)$ prove t/db_dependent/Exporter/Record.t 
t/db_dependent/Exporter/Record.t .. 1/7 
    #   Failed test 'Export XML: 3 records should have been exported'
    #   at t/db_dependent/Exporter/Record.t line 200.
    #          got: '1'
    #     expected: '3'
    # Looks like you planned 4 tests but ran 2.
    # Looks like you failed 1 test of 2 run.

#   Failed test 'export xml'
#   at t/db_dependent/Exporter/Record.t line 211.
Can't call method "subfield" on an undefined value at t/db_dependent/Exporter/Record.t line 202, <$fh> line 3.
# Looks like your test exited with 11 just after 3.
t/db_dependent/Exporter/Record.t .. Dubious, test returned 11 (wstat 2816, 0xb00)
Failed 5/7 subtests 

Test Summary Report
-------------------
t/db_dependent/Exporter/Record.t (Wstat: 2816 (exited 11) Tests: 3 Failed: 1)
  Failed test:  2
  Non-zero exit status: 11
  Parse errors: Bad plan.  You planned 7 tests but ran 3.
Files=1, Tests=3,  4 wallclock secs ( 0.03 usr  0.01 sys +  3.58 cusr  0.33 csys =  3.95 CPU)
Result: FAIL
'''
Comment 121 David Gustafsson 2025-03-26 23:02:43 UTC
Created attachment 179748 [details] [review]
Bug 20551: Return 0 instead of undef from _get_record_for_export not to break iterator loop
Comment 122 David Gustafsson 2025-03-26 23:04:03 UTC
(In reply to Arthur Suzuki from comment #120)
> Tried to apply patches to current master but got the tests failing :
> '''
> kohadev-koha@kohadevbox:koha(main)$ prove t/db_dependent/Exporter/Record.t 
> t/db_dependent/Exporter/Record.t .. 1/7 
>     #   Failed test 'Export XML: 3 records should have been exported'
>     #   at t/db_dependent/Exporter/Record.t line 200.
>     #          got: '1'
>     #     expected: '3'
>     # Looks like you planned 4 tests but ran 2.
>     # Looks like you failed 1 test of 2 run.
> 
> #   Failed test 'export xml'
> #   at t/db_dependent/Exporter/Record.t line 211.
> Can't call method "subfield" on an undefined value at
> t/db_dependent/Exporter/Record.t line 202, <$fh> line 3.
> # Looks like your test exited with 11 just after 3.
> t/db_dependent/Exporter/Record.t .. Dubious, test returned 11 (wstat 2816,
> 0xb00)
> Failed 5/7 subtests 
> 
> Test Summary Report
> -------------------
> t/db_dependent/Exporter/Record.t (Wstat: 2816 (exited 11) Tests: 3 Failed: 1)
>   Failed test:  2
>   Non-zero exit status: 11
>   Parse errors: Bad plan.  You planned 7 tests but ran 3.
> Files=1, Tests=3,  4 wallclock secs ( 0.03 usr  0.01 sys +  3.58 cusr  0.33
> csys =  3.95 CPU)
> Result: FAIL
> '''

Thanks for spotting this! The issue should now be fixed.
Comment 123 David Gustafsson 2025-03-26 23:11:01 UTC
Created attachment 179749 [details] [review]
Bug 20551: Return 0 instead of undef from _get_deleted_record_for_export
Comment 124 Andrew Fuerste-Henry 2025-04-04 16:13:30 UTC
Created attachment 180648 [details] [review]
Bug 20551: Add option for including deleted records in export_records.pl

Add option "--include_deleted" to include deleted
biblios in export_records.pl as marc records with
record status (in leader) set to "d" and
"--deleted_only" to export only deleted biblios.

How to test:
1) Run tests in t/db_dependent/Exporter/Record.t
2) All tests should pass
3) Delete a biblio record in Koha
4) Run the export script as:
   `export_records.pl --date=<date-just-before-deletion>
   --include_deleted`
5) Open the generated koha.mrc, and verify that contains
   the recently deleted record, and that the record header
   has record status "d".
6) Perform the same check with:
   `export_records.pl --date=<date-just-before-deletion>
      --include_deleted --format=xml`
7) Run:
   `export_records.pl --date=<date-just-before-deletion>
      --deleted_only --format=xml`
    and verify that only the deleted biblio was included
    in the export.

Sponsored-by: Gothenburg University Library

Signed-off-by: Frank Hansen <frank.hansen@ub.lu.se>
Signed-off-by: Andrew Fuerste Henry <andrew@bywatersolutions.com>
Comment 125 Andrew Fuerste-Henry 2025-04-04 16:13:34 UTC
Created attachment 180649 [details] [review]
Bug 20551: Fix option name in pod

Signed-off-by: Frank Hansen <frank.hansen@ub.lu.se>
Signed-off-by: Andrew Fuerste Henry <andrew@bywatersolutions.com>
Comment 126 Andrew Fuerste-Henry 2025-04-04 16:13:37 UTC
Created attachment 180650 [details] [review]
Bug 20551: Rename "marcflavour" field to "schema"

Signed-off-by: Frank Hansen <frank.hansen@ub.lu.se>
Signed-off-by: Andrew Fuerste Henry <andrew@bywatersolutions.com>
Comment 127 Andrew Fuerste-Henry 2025-04-04 16:13:40 UTC
Created attachment 180651 [details] [review]
Bug 20551: Improve documentation and make compatible with more options

Signed-off-by: Frank Hansen <frank.hansen@ub.lu.se>
Signed-off-by: Andrew Fuerste Henry <andrew@bywatersolutions.com>
Comment 128 Andrew Fuerste-Henry 2025-04-04 16:13:44 UTC
Created attachment 180652 [details] [review]
Bug 20551: Add missing subroutine import

Signed-off-by: Andrew Fuerste Henry <andrew@bywatersolutions.com>
Comment 129 Andrew Fuerste-Henry 2025-04-04 16:13:48 UTC
Created attachment 180653 [details] [review]
Bug 20551: Fix tests

Signed-off-by: Frank Hansen <frank.hansen@ub.lu.se>

Works fine for me.
Signing off

Signed-off-by: Andrew Fuerste Henry <andrew@bywatersolutions.com>
Comment 130 Andrew Fuerste-Henry 2025-04-04 16:13:52 UTC
Created attachment 180654 [details] [review]
Bug 20551: Fetch deleted biblios through _get_record_export

Signed-off-by: Andrew Fuerste Henry <andrew@bywatersolutions.com>
Comment 131 Andrew Fuerste-Henry 2025-04-04 16:13:55 UTC
Created attachment 180655 [details] [review]
Bug 20551: Refactor unmotivated partially applied validator function

Signed-off-by: Andrew Fuerste Henry <andrew@bywatersolutions.com>
Comment 132 Andrew Fuerste-Henry 2025-04-04 16:13:58 UTC
Created attachment 180656 [details] [review]
Bug 20551: (QA follow-up) perltidy and test fixes

Signed-off-by: Andrew Fuerste Henry <andrew@bywatersolutions.com>
Comment 133 Andrew Fuerste-Henry 2025-04-04 16:14:02 UTC
Created attachment 180657 [details] [review]
Bug 20551: Move some code that might be reused into functions

Signed-off-by: Andrew Fuerste Henry <andrew@bywatersolutions.com>
Comment 134 Andrew Fuerste-Henry 2025-04-04 16:14:06 UTC
Created attachment 180658 [details] [review]
Bug 20551: (QA follow-up) don't read record content if there is an error during decoding

Signed-off-by: Andrew Fuerste Henry <andrew@bywatersolutions.com>
Comment 135 Andrew Fuerste-Henry 2025-04-04 16:14:09 UTC
Created attachment 180659 [details] [review]
Bug 20551: Get biblionumber safely in validator function

Signed-off-by: Andrew Fuerste Henry <andrew@bywatersolutions.com>
Comment 136 Andrew Fuerste-Henry 2025-04-04 16:14:12 UTC
Created attachment 180660 [details] [review]
Bug 20551: Use iterator for retrieving records to save memory

Signed-off-by: Andrew Fuerste Henry <andrew@bywatersolutions.com>
Comment 137 Andrew Fuerste-Henry 2025-04-04 16:14:16 UTC
Created attachment 180661 [details] [review]
Bug 20551: Cache resultset instead of passing to sub

Signed-off-by: Andrew Fuerste Henry <andrew@bywatersolutions.com>
Comment 138 Andrew Fuerste-Henry 2025-04-04 16:14:19 UTC
Created attachment 180662 [details] [review]
Bug 20551: Warn and return if deleted biblio not found

Signed-off-by: Andrew Fuerste Henry <andrew@bywatersolutions.com>
Comment 139 Andrew Fuerste-Henry 2025-04-04 16:14:23 UTC
Created attachment 180663 [details] [review]
Bug 20551: Use timetstamp of deletedbiblio_metadata instead of deletedbibio

Signed-off-by: Andrew Fuerste Henry <andrew@bywatersolutions.com>
Comment 140 Andrew Fuerste-Henry 2025-04-04 16:14:26 UTC
Created attachment 180664 [details] [review]
Bug 20551: Return 0 instead of undef from _get_record_for_export not to break iterator loop

Signed-off-by: Andrew Fuerste Henry <andrew@bywatersolutions.com>
Comment 141 Andrew Fuerste-Henry 2025-04-04 16:14:29 UTC
Created attachment 180665 [details] [review]
Bug 20551: Return 0 instead of undef from _get_deleted_record_for_export

Signed-off-by: Andrew Fuerste Henry <andrew@bywatersolutions.com>