Re: [gentoo-portage-dev] [PATCH 1/2] fetch: Use real os.walk() to avoid unicode issues with Portage

[Zac Medico]

On 10/21/19 1:43 AM, Michał Górny wrote:
> Use real os.walk() when getting filenames for FlatLayout.  Unlike
> the wrapped Portage module, it return str output for str path parameter,
> so we don't have to recode it back and forth.
>
> Signed-off-by: Michał Górny <mgorny@g.o>
> ---
>  lib/portage/package/ebuild/fetch.py | 3 ++-
>  1 file changed, 2 insertions(+), 1 deletion(-)
>
> diff --git a/lib/portage/package/ebuild/fetch.py b/lib/portage/package/ebuild/fetch.py
> index cedf12b19..be277f1a3 100644
> --- a/lib/portage/package/ebuild/fetch.py
> +++ b/lib/portage/package/ebuild/fetch.py
> @@ -11,6 +11,7 @@ import io
>  import itertools
>  import json
>  import logging
> +import os as real_os
>  import random
>  import re
>  import stat
> @@ -270,7 +271,7 @@ class FlatLayout(object):
>  		return filename
>
>  	def get_filenames(self, distdir):
> -		for dirpath, dirnames, filenames in os.walk(distdir,
> +		for dirpath, dirnames, filenames in real_os.walk(distdir,
>  				onerror=_raise_exc):
>  			return iter(filenames)
>
>

The real_os.walk will trigger UnicodeEncodeError if distdir can't be
encoded with sys.getfilesystemencoding(). It's an edge case, but
generally I prefer to handle it.

We can continue to use portage.os for the os.walk call, and turn
get_filenames into a generator method like this:

    for filename in filenames:
        try:
            yield portage._unicode_decode(filename, errors='strict')
        except UnicodeDecodeError:
            # Ignore it. Distfiles names must have valid UTF8 encoding.
            pass
-- 
Thanks,
Zac