I think this bz needs more precision.

The attached file has been exported in MARC-8, which is why we want to select MARC8 for character encoding in the import page (in test plan).

By searching the code and the database, I found that Koha converts the imported file from MARC-8 to UTF-8, but in the 'import_records' table, in the database, the columns 'encoding' remains as MARC-8.

In 'Koha::Import::Record.pm', it seems the statement :
  my $record = MARC::Record->new_from_xml($self->marcxml, $self->$encoding, $format);
is converting a UTF-8 encoded file from MARC-8 to UTF-8 again, resulting in '�' symbols for special charaters.

Created attachment 168236 [details] [review]
Bug 37184 - Special character encoding problem when importing MARC file from the Acquisitions module

1. Go to Cataloging > "Stage records for import".
2. Upload the file "ExportMemento2024061010532869Marc8.mrc" or a MARC8 encoded file.
3. In the form, select the options :
   - Record type: Bibliographic
   - Character encoding: MARC8
   - Format: MARC
4. Click "Stage for import".
5. Find or create an open basket in the Acquisitions module.
6. Click "add to basket".
7. Select "From a staged file" and select the previous staged file.
9. Click "Select all" to check all records.
10. In the item information tab, choose the Document type.
11. Verify the titles of the records in the basket:
 --> Koha replaces accents with symbols �

12. Apply the patch.
13. Do the same from step 1 and notice there is no more encoding issues.

That seems like an odd place to be making the change: if we took a MARC-8 record, converted it to UTF-8, and then stored it in a database table with an encoding column, why did we set that column to MARC-8 instead of UTF-8? And out of the five non-UTF-8 choices while staging, is MARC-8 the only one we convert to UTF-8, or would this need to just set UTF-8 for everything, ignoring $self->encoding?

Certainly sounds interesting but not sure about this fix

Oh, the encoding param in MARC::Record->new_from_xml is "what encoding do you want for the record I return?" out of a choice of UTF-8 or MARC-8. So setting $encoding to "UTF-8" if it's "MARC-8" is the same as just passing the literal string "UFT-8", which is what we do in almost every case elsewhere in the codebase, with some odd exceptions. C4::ImportBatch sometimes calls ->new_from_xml( StripNonXmlChars( $row->{'marcxml'} ), 'UTF-8', $row->{'encoding'} ) or ->new_from_xml( StripNonXmlChars($rowref->{'marcxml_old'}), 'UTF-8', $rowref->{'encoding'}, $marc_type); which seems a bit odd, not sure how passing two encodings to something that wants an encoding and a marc flavor is supposed to work.

I'd be happy to sign off a patch that skips setting $encoding and just passes new_from_xml the string "UTF-8", and let QA worry about what on earth someone was thinking passing $self->encoding.

(In reply to Phil Ringnalda from comment #5)
> Oh, the encoding param in MARC::Record->new_from_xml is "what encoding do
> you want for the record I return?" out of a choice of UTF-8 or MARC-8. 

I don't think that's quite right. The encoding param there should be the encoding of the incoming XML. So if the XML is MARC-8 it would unset the UTF-8 flag in the leader and make sure to output MARC-8 bytes (which would be a weird thing to do but that seems to be how it works).

But in any case... OP is uploading a .mrc file which will run through C4::Import::RecordsFromISO2709File, which will convert MARC-8 records into UTF-8 records, which OP sort of notes is the encoding Koha uses internally.

(In reply to Phil Ringnalda from comment #5)
> I'd be happy to sign off a patch that skips setting $encoding and just
> passes new_from_xml the string "UTF-8", and let QA worry about what on earth
> someone was thinking passing $self->encoding.

I think you're right about just passing in the string "UTF-8" because at this point the records should be UTF-8 encoded.

Honestly looking at C4::ImportBatch::_create_import_record and friends...I have no idea why $encoding is even stored because it's useless at this point. Maybe if it was stored as "source_encoding" but the source file isn't there anyway. Who knows. 

Looking at 01d78e1ec71 it seems like $self->encoding was used here by accident.

Created attachment 170742 [details] [review]
Bug 37184: Special character encoding problem when importing MARC file from the Acquisitions module

1. Go to Cataloging > "Stage records for import".
2. Upload the file "ExportMemento2024061010532869Marc8.mrc" or a MARC8 encoded file.
3. In the form, select the options :
   - Record type: Bibliographic
   - Character encoding: MARC8
   - Format: MARC
4. Click "Stage for import".
5. Find or create an open basket in the Acquisitions module.
6. Click "add to basket".
7. Select "From a staged file" and select the previous staged file.
9. Click "Select all" to check all records.
10. In the item information tab, choose the Document type.
11. Verify the titles of the records in the basket:
 --> Koha replaces accents with symbols �

12. Apply the patch.
13. Do the same from step 1 and notice there is no more encoding issues.

Created attachment 170924 [details] [review]
Bug 37184: Special character encoding problem when importing MARC file from the Acquisitions module

1. Go to Cataloging > "Stage records for import".
2. Upload the file "ExportMemento2024061010532869Marc8.mrc" or a MARC8 encoded file.
3. In the form, select the options :
   - Record type: Bibliographic
   - Character encoding: MARC8
   - Format: MARC
4. Click "Stage for import".
5. Find or create an open basket in the Acquisitions module.
6. Click "add to basket".
7. Select "From a staged file" and select the previous staged file.
9. Click "Select all" to check all records.
10. In the item information tab, choose the Document type.
11. Verify the titles of the records in the basket:
 --> Koha replaces accents with symbols �

12. Apply the patch.
13. Do the same from step 1 and notice there is no more encoding issues.

Signed-off-by: Phil Ringnalda <phil@chetcolibrary.org>

Created attachment 173527 [details] [review]
Bug 37184: Special character encoding problem when importing MARC file from the Acquisitions module

1. Go to Cataloging > "Stage records for import".
2. Upload the file "ExportMemento2024061010532869Marc8.mrc" or a MARC8 encoded file.
3. In the form, select the options :
   - Record type: Bibliographic
   - Character encoding: MARC8
   - Format: MARC
4. Click "Stage for import".
5. Find or create an open basket in the Acquisitions module.
6. Click "add to basket".
7. Select "From a staged file" and select the previous staged file.
9. Click "Select all" to check all records.
10. In the item information tab, choose the Document type.
11. Verify the titles of the records in the basket:
 --> Koha replaces accents with symbols �

12. Apply the patch.
13. Do the same from step 1 and notice there is no more encoding issues.

Signed-off-by: Phil Ringnalda <phil@chetcolibrary.org>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>

Looking at the history, I concur.. 

I would be nice to have a unit test here though

(In reply to Martin Renvoize (ashimema) from comment #10)
> Looking at the history, I concur.. 
> 
> I would be nice to have a unit test here though

+1

Pushed for 24.11!

Well done everyone, thank you!