Re: [gentoo-dev] Locale check in python_pkg_setup()

[Brian Harring]

On Fri, Jul 30, 2010 at 01:16:42AM +0200, Arfrever Frehtes Taifersar Arahesis wrote:
> --- python.eclass
> +++ python.eclass
> @@ -355,6 +355,8 @@
>  	# Check if phase is pkg_setup().
>  	[[ "${EBUILD_PHASE}" != "setup" ]] && die "${FUNCNAME}() can be used only in pkg_setup() phase"
>
> +	local locale
> +
>  	if [[ "$#" -ne 0 ]]; then
>  		die "${FUNCNAME}() does not accept arguments"
>  	fi
> @@ -407,6 +409,16 @@
>  		unset -f python_pkg_setup_check_USE_flags
>  	fi
>
> +	locale="$(python -c 'import os; print(os.environ.get("LC_ALL", os.environ.get("LC_CTYPE", os.environ.get("LANG", "POSIX"))))')"

You're using python to get the exported env.  Don't.  Use bash (you're 
invoking python from freaking bash after all)...

> +	if [[ "${locale}" != *.UTF-8 ]]; then
> +		eerror
> +		eerror "Currently used locale '${locale}' is unsupported and can cause build-time or run-time"
> +		eerror "problems (usually UnicodeDecodeErrors or UnicodeEncodeErrors). Bugs caused by this locale"
> +		eerror "will be closed as invalid. It is recommended to use a UTF-8 locale to avoid problems."
> +		eerror "See http://www.gentoo.org/doc/en/utf-8.xml for information on how to fix locale."
> +		eerror

For cases such as this, ewarn, not eerror.  It's not an actual error, 
it's a potential source of problems people may see.

The more I look into this issue, the more I'm convinced it's not user 
settings that are problem- the problem is in the code, not in user 
env.  You've stated in a couple of places that "C/Posix locales are 
not supported", which frankly is very whacked- that's not really a 
proclamation you can make on your own for python, and you're actually 
ignoring that this problem would just as easily rear it's head with a 
latin-1 encoded file.


Take a look at 302425; the traceback in that is a classic example of 
where they *should* be using bytes mode (they don't need to interpret 
the data, just write the script across, thus bytes).

bug 328047 is induced by a patch we add (it's not in upstream python).  
The code in question also is invoking fricking ldd a few steps prior 
which is questionable in multiple ways: either way, relevant chunk is
+            os.system("ldd %s > %s" % (do_readline, tmpfile))
+            fp = open(tmpfile)
+            for ln in fp:

So... roughly, it invokes os.system, which will pass the environment 
straight through to it, meaning locale gets passed down.

Then it open's the file.  Note it specifes *NO ENCODING* nor is their 
actually an enforced locale best I can tell , thus ascii being the 
default.  The screwup here is in our patches- said patches should be 
forcing posix locale for the ldd call (resulting in ascii).  If you 
think through this bug, we've seen this multiple times in grep/sed 
calls- this is literally no different.

bug 287439 is a screw up in the programs source... should've been 
using bytes (non arguable).  Matter of fact, while generally I think 
Tarek knows what the hell he's doing, the skip they added to the 
tests ignored an actual valid bug in setuptools/distribute- shebangs 
from the standpoint of the kernel need to be consistant.  Thus reading 
the shebang line itself should be done in bytes, than converted to 
ascii and interpretted- they tried opening the file (in whole) in 
bytes, meaning they tried enforcing ascii across the whole buffer- 
not just the first line.  Program bug.

These bugs I got via searching for 'ALL python locale', and 
identifying the ones that were actually locale related.  I've at this 
point looked into the source of 3 bugs- meaning literally, 3 bugs 
checked into, 3 instances where the code was wrong.

I'll leave it as an exercise for others to keep digging, but the point 
here is that the programs themselves screwup their locale handling- 
trying to force all systems to use a utf-8 locale for the env is just 
a hack instead of fixing the actual issue.  A pretty bad hack 
considering I've spent all of 30 minutes digging into this and rooting 
out the actual flaws in the src I might add.

For shits and giggles, lets add one more bug in- one that has the 
potential of rearing its head in random consuming pkgs, bug 322425 
(docutils's build_html being flawed), their encoding handling is 
intrinsically flawed.  The encoding of a file their 
installing/parsing should be determined by the file itself- not 
attempting to arbitrarily force it to whatever locale the user happens 
to be running (which is exactly the first thing buildhtml.py attempts, 
literally `locale.setlocale(locale.LC_ALL, '')` at line 20).  The 
issue is not people using ascii locales, the issue is that these tools 
do not handle encoding correctly.

Recall, one of the purposes of py3k going bytes vs text (aka unicode) 
was to make clear that textual data's encoding need be known.  All of 
this code isn't actually forcing/handling the encoding for the data 
they deal in- meaning these are literal bugs, exposed purely due to 
py3k actually enforcing encoding in normal file opens.

So... this is a big -1 on adding such a warning (especially 
considering it doesn't actually resolve the raw issues, it just 
sidesteps a couple of cases).

Fix the actual problem instead...

Finally, cc'ing QA since this is a class of bugs they should be aware 
of with py3k.  This is a bit of a sign that a lot of source isn't 
really py3k ready yet either imo, but so it goes...

~harring