News for hackers. Stuff that matters.
Re:How to Use UTF-8 with Python by grifaton on Friday December 02, 2011 @12:48PM (#3304873)
This was published in 2005. Unicode in python has come a long way since then, especially with respect to python 3 [1], [2].

[1] http://docs.python.org/release/3.0.1/whatsnew/3.0.html#text-... [2] http://docs.python.org/release/3.0.1/howto/unicode.html

Re:How to Use UTF-8 with Python by joeyespo on Friday December 02, 2011 @01:49PM (#3305140)
Python 2 is still the status quo though.

One of my favorite things about Python 3 is its new string handling. It is fantastic and marginally more intuitive. Having an explicit (and a more general) immutable `bytes` class, no longer sharing a base class with `unicode`, and having Unicode instead of a potentially-encoded string as the default are all wonderful decisions and moves Python forward by miles in my opinion.

Until Python 3 becomes the norm, we're still stuck with the confusing `str` and `unicode` constructs.

Re:How to Use UTF-8 with Python by js2 on Friday December 02, 2011 @12:31PM (#3304795)
For decoding a byte string, I prefer bytes.decode("utf8"). Note that decode takes an optional second argument that says what to do if an invalid byte sequence is encountered - by default an exception is raised.

Also, I think http://docs.python.org/howto/unicode.html may be a better reference than this article.

Re:How to Use UTF-8 with Python by oconnore on Friday December 02, 2011 @02:16PM (#3305251)

    $ python3
    Python 3.2.2 (default, Nov 21 2011, 05:01:42)
    [GCC 4.4.5] on linux2
    Type "help", "copyright", "credits" or "license" for more 
    information.
    >>> print('hello world') # <-- Unicode!
Re:How to Use UTF-8 with Python by sukhbir on Friday December 02, 2011 @12:29PM (#3304785)
To add to this, Kumar McMillan's talk, "Unicode In Python, Completely Demystified" (http://farmdev.com/talks/unicode/) is really good.
Re:How to Use UTF-8 with Python by jrockway on Friday December 02, 2011 @12:59PM (#3304925)
First, you can place a UTF-8 byte-order marker at the beginning of your file, if your editor supports it. Secondly, you can place the following special comment in the first or second lines of your script:

# -- coding: utf-8 --

Does Python actually parse comments, or is this just to get your editor to do the right thing?

Re:How to Use UTF-8 with Python by js2 on Friday December 02, 2011 @01:12PM (#3304987)
Python supports writing Unicode literals in any encoding, but you have to declare the encoding being used. This is done by including a special comment as either the first or second line of the source file:

  # -*- coding: latin-1 -*-
The syntax is inspired by Emacs’s notation for specifying variables local to a file. Emacs supports many different variables, but Python only supports ‘coding’. The dash-splat-dash symbols indicate to Emacs that the comment is special; they have no significance to Python but are a convention. Python looks for coding: name or coding=name in the comment.

http://docs.python.org/howto/unicode.html

Re:How to Use UTF-8 with Python by rbanffy on Friday December 02, 2011 @03:01PM (#3305435)
It will also tolerate Vim users:

  # vim: set fileencoding=utf-8 
http://www.python.org/dev/peps/pep-0263/
Re:How to Use UTF-8 with Python by stock_toaster on Friday December 02, 2011 @01:43PM (#3305114)
python specifically looks for that in the first two lines of source (to make allowance for #! stanza on the first line).

See: http://www.python.org/dev/peps/pep-0263/

Re:How to Use UTF-8 with Python by krosaen on Friday December 02, 2011 @04:56PM (#3305867)
some useful helpers from tornado's escape.py:

    def utf8(value):
        if isinstance(value, unicode):
            return value.encode("utf-8")
        assert isinstance(value, str)
        return value

    def _unicode(value):
        if isinstance(value, str):
            return value.decode("utf-8")
        assert isinstance(value, unicode)
        return value