Re: [gentoo-user] locating non utf-8 characters

[Michael]

On Tuesday, 3 November 2020 05:46:41 GMT thelma@×××××××××××.com wrote:
> I'm using sql-ledger and while making backup it uses stardard gzip program:
> $gzip = "gzip -S .gz";
>
> The backup works with some dataset but one data set us giving me an
> error while trying to perform backup:
>
> Wide character in print at SL/AM.pm line 2044.
> Content-Type: application/file; Content-Disposition: attachment;
> filename=dataset_3-3.2.6-20201101.sql.gz
> ãù&ü_1604265628.dataset_3-3.2.6-20201101.sqlÏ\Yo;ñ~œØ –
>
> Since sql-ledger file are standard utf-8 files, I was thinking using:
>  grep -axv '.*' file
>
> would find all not utf-8 characters. And it did. I use "nano" to remove
> them but I'm still getting the same error while performing backup.
>
> Any ideas?

I have not used sql-ledger, but have come across the following two symptoms 
which may be relevant to your problem.

1. A SQL database which was created with an MSWindows application was using 
UTF-16 instead of UTF-8.  This added some UTF-16 null character at the start 
of the SQL dump which messed up the output.  The offending character was 
obvious as a block when inspecting the dump with 'less' in Linux with its 
default UTF-8 character encoding and could be deleted with a text editor.  I 
don't think this relates to your problem, but I am mentioning it for 
completeness.

2. The word "print" in the error reported gives a hint you should follow up.  
Perl which is used by sql-ledger, converts bytes to characters and can be set 
to use UTF-8 encoding.  However, it's conversion algorithm does not get things 
right every time and when it concatenates strings it can mistranslate them.  
You could fix this by setting both input *and* output encoding characters to 
UTF-8.  A good explanation of the problem and suitable solutions are described 
here:

https://www.ahinea.com/en/tech/perl-unicode-struggle.html