Basically, we needed to support a large existing Python 2.7 codebase. See discussion here: https://github.com/google/grumpy/issues/1
> It's a hard-code compiler, not an interpreter written in Go. That implies some restrictions, but the documentation doesn't say much about what they are. PyPy jumps through hoops to make all of Python's self modification at run-time features work, complicating PyPy enormously. Nobody uses that stuff in production code, and Google apparently dumped it.
There are restrictions. I'll update the README to make note of them. Basically, exec and eval don't work. Since we don't use those in production code at Google, this seemed acceptable.
> If Grumpy doesn't have a Global Interpreter Lock, it must have lower-level locking. Does every built-in data structure have a lock, or does the compiler have enough smarts to figure out what's shared across thread boundaries, or what?
It does fine grained locking. Mutable data structures like lists and dicts do their own locking. Incidentally, this is one reason why supporting C extensions would be complicated.
What about stuff like literal_eval? Or even just monkeypatching with name.__dict__[param] = value ?
> It does fine grained locking. Mutable data structures like lists and dicts do their own locking. Incidentally, this is one reason why supporting C extensions would be complicated.
Would there be a succinct theoretical description of exactly how that's implemented anywhere? What about things like numpy arrays.
literal_eval could in principle be supported I think. name.__dict__[param] = value works as you'd expect:
$ make run
class A(object):
pass
a = A()
a.__dict__['foo'] = 'bar'
print a.foo
bar
Also, NumPy is implemented completely in C and Python, and makes extensive use of CPython extension hooks and knowledge of the CPython reference counting implementation, which is part of the reason why it is so hard to port to other implementations of Python.
namedtuple will have to be implemented differently. I think it can be accomplished by defining the class with type()? Maybe with a metaclass...
I've done it using more or less that method. The code is in the "coll" sub-package of my plib.stdlib project; the Python 2 version is here on bitbucket:
If you like writing in functional style, namedtuples are much more natural than dict or classes, and more efficient to boot.
from collections import namedtuple
class Foo(namedtuple("Foo", "a b c")):
@property
def sum(self):
return self.a + self.b + self.c
f = Foo(1,2,3)
print f.sumThanks so much for bringing it up.
Namedtuples are a way to preserve the data unless the consuming code _really_ wants to change it, which is sometimes legitimate.
I'm not totally sold, as in some cases dictionaries or classes would add nice value. But namedtuples have a rigidity that makes you think twice before tampering with retrieved data.
1) You know before hand that the number of items won't be modified and the order matters since you are handling records. So it is a simple way of accomplishing that constraint.
2) Because they extend tuple they are inmutable too and therefore they don't store attributes per instance __dict__, field names are stored in the class so if you have tons of instances you save a lot of space.
Why creating a class if you just probably need a read-only interaction? But what about if you need some method? Then you can extend your namedtuple class and add the functionality you want. If for example you want to control the values of the fields when you are creating the namedtuple you can create your own namedtuple by overriding __new__. At that point it is worth it to take a look at https://pypi.python.org/pypi/recordclass.
(2) Apparently setting a __dict__ key works; they could be implemented like that.
I'm guessing pretty much the entire AST module is a no-go?
I managed to run into 2 trying to build a 5 line program :-)
$ cat t.py; ./tools/grumpc t.py > t.go;go build t.go;echo '----';./t
import sys
print sys.stdin.readline()
----
AttributeError: 'module' object has no attribute 'stdin'
$
$ cat t.py ;./tools/grumpc t.py
c = {}
top = sorted(c.items(), key=lambda (k,v): v)
Traceback (most recent call last):
File "./tools/grumpc", line 102, in <module>
sys.exit(main(parser.parse_args()))
File "./tools/grumpc", line 60, in main
visitor.visit(mod)
File "/usr/local/Cellar/python/2.7.12/Frameworks/Python.framework/Versions/2.7/lib/python2.7/ast.py", line 241, in visit
return visitor(node)
File "/Users/foo/src/grumpy/build/lib/python2.7/site-packages/grumpy/compiler/stmt.py", line 302, in visit_Module
self._visit_each(node.body)
File "/Users/foo/src/grumpy/build/lib/python2.7/site-packages/grumpy/compiler/stmt.py", line 632, in _visit_each
self.visit(node)
File "/usr/local/Cellar/python/2.7.12/Frameworks/Python.framework/Versions/2.7/lib/python2.7/ast.py", line 241, in visit
return visitor(node)
File "/Users/foo/src/grumpy/build/lib/python2.7/site-packages/grumpy/compiler/stin visit_Assign
with self.expr_visitor.visit(node.value) as value:
File "/usr/local/Cellar/python/2.7.12/Frameworks/Python.framework/Versions/2.7/lib/python2.7/ast.py", line 241, in visit
return visitor(node)
File "/Users/foo/src/grumpy/build/lib/python2.7/site-packages/grumpy/compiler/expr_visitor.py", line 101, in visit_Call
values.append((util.go_str(k.arg), self.visit(k.value)))
File "/usr/local/Cellar/python/2.7.12/Frameworks/Python.framework/Versions/2.7/lib/python2.7/ast.py", line 241, in visit
return visitor(node)
File "/Users/foo/src/grumpy/build/lib/python2.7/site-packages/grumpy/compiler/expr_visitor.py", line 246, in visit_Lambda
return self.visit_function_inline(func_node)
File "/Users/foo/src/grumpy/build/lib/python2.7/site-packages/grumpy/compiler/expr_visitor.py", line 388, in visit_function_inline
func_visitor = block.FunctionBlockVisitor(node)
File "/Users/foo/src/grumpy/build/lib/python2.7/site-packages/grumpy/compiler/block.py", line 432, in __init__
args = [a.id for a in node_args.args]
AttributeError: 'Tuple' object has no attribute 'id'1. Lambda tuple args are not yet supported -- I actually didn't know that was a thing :\ -- https://github.com/google/grumpy/issues/17
2. Even if that worked properly, sorted() is not yet implemented: https://github.com/google/grumpy/issues/16
def func((a,b)):
return b
mytuple = 1,2
print func(mytuple)
def func(t):
a,b = t
return b
This is probably the cleaner way to write that:
key=operator.itemgetter(1)Couldn't they supported with a slower runtime implementation? I mean I still love the idea and actually like the idea.
Python 2.7 is what's running at Google. Not really surprising they're looking at this considering the fast approaching end of (core dev) support for Python 2.7.
Write an interpreter in another language and programatically port modules to Go. Seems pretty sensible to me.
I'm not sure; as a Go developer, I kind of like the idea of having access to the Python library ecosystem from Go, without being forced to create an IPC bridge and building up the requisite release-management and deploy-time goop.
Plus, I'm just not a Python developer; in the case where the only library that exists to do something is written in Python, I'd much rather write Go that calls that Python library than Python that calls that Python library.
Eg: python code (from blog post)
from __go__.net.http import ListenAndServe, RedirectHandler
handler = RedirectHandler('http://github.com/google/grumpy', 303)
ListenAndServe('127.0.0.1:8080', handler)I'd like to see an example of this, because from the blog post I get the impression that this mostly allows accessing the Go ecosystem from Python, rather than the other way around. For example, how would Python classes be handled from Go?
I'd prefer that all new Python tools that need to support 2.x also support 3.x. It's an additional development cost, but IMHO, a worthwhile investment in the future.
Python 3.5 with uvloop+sanic can be faster than node.js without any JIT:
The fact that Google has started this project to migrate away from Python to an AOT compiled language, shows where the performance wins are.
Here you go:
https://morepypy.blogspot.com/2011/08/pypy-is-faster-than-c-...
https://morepypy.blogspot.com/2011/02/pypy-faster-than-c-on-...
For more examples, just search "pypy faster than c".
Also, here is an article from the Python wiki about why speed doesn't really matter a lot of the time:
https://wiki.python.org/moin/PythonSpeed
And, my own two cents:
Speed is relative. Does every piece of code need to be as performant as possible? No. I would argue that, in most cases, speed of development is far more important than speed of execution. This is, of course, not true for things like drivers or statistical analysis.
Writing a web application? Speed isn't that important as the whole process is i/o bound anyways.
Writing a machine learning algorithm? Depends.
Web scraping? I/O bound, speed not really important.
Image processing? Speed matters at least a little bit.
Writing networking glue for distributed systems? Speed probably doesn't matter.
It's all relative. If it needs to be fast, it needs to be fast. Most things don't really need to be fast. For the things that don't need to be fast, why build them with C/C++/Rust/Go when you could spend half the time building them in Python/Ruby/js/etc?
I usually ignore it when talking about Python, because that is what the community does, by gathering around CPython.
> For the things that don't need to be fast, why build them with C/C++/Rust/Go when you could spend half the time building them in Python/Ruby/js/etc?
Because one can use languages like OCaml, Haskell, Lisp, Scheme, Racket, F#, C#,... thus having both the productivity of a REPL environment and the execution speed of native code.
I thought we were talking about Python implementations exclusively. My mistake.
>OCaml
I have no experience with OCaml, so I won't make any comments regarding it's efficacy.'
>Haskell
Well thought out language. I like the purity, but it's too academic for real world use outside of scientific computing. FP isn't for everyone, and my personally belief regarding it is that it is better used as a tool alongside other paradigms than all by itself as the only paradigm.
>Lisp
Lisp is useful for a lot of things. It's also not very popular for new projects as far as I've seen. There are also a ton of different versions, so I don't know if "Lisp" is really a good descriptor.
>Scheme
As far as I know Scheme is the defacto teaching language for most compSci programs.Or, at least it was for a long time. Once again, FP is not for everyone. A lot of people also dislike Lisp style syntax, myself included.
>Racket
Same issues as Scheme.
>F#
F# is a fantastic language. There's not really a whole heck of a lot to complain about other than the .NET implications. The only detriment relative to Python is F#'s much smaller ecosystem and community.
>C#
Once again, some people just don't like .NET stuff. A lot of people also see static typing to be a detriment in many use cases.
Relative to Python, these languages also share several other problems when it comes to real world application: lack of competent developers, stagnating ecosystems, lack of third party libraries, ecosystem lock in, and cross-platform comparability issues. In the case of languages like Haskell, they could even be considered "esoteric".
I'll give you that many of these languages are more "pure" or "logical" than Python. I'll even give you that most of them are designed much better than Python. None of that changes the fact that Python is overall easier to read, easier to learn, easier to write, has a better ecosystem, is platform and file system agnostic, has a very non-restrictive license, and is, overall, very pleasant to work with.
It's likely code using lots of C-extensions will continue with CPython2 and new code will be written in Grumpy (pure Python2).
Grumpy is pretty much what most everyone would actually want out of a new Python and may have arrived just in time.
> The users never arrived
That's the way it used to be a few years ago - it changed a lot the last few years. Pretty much all libraries are ported and many new libraries are Python 3-only.
asyncio is nice.
All Python devs I personally know moved to Python 3. Porting is a lot less painful than it used to be.
[0]http://lucumr.pocoo.org/2016/10/30/i-dont-understand-asyncio...
I'm like really new to programming and I'm still just learning the basics, but I see this little addendum a lot from people who say everyone should be writing Python 3.
And if Google can make an interpreter for 2, then sooner or later, one for 3 will pop up. Since Google made some restrictions on what Grumpy supports from python 2, I'm sure someone somewhere will be able to do the same stuff for three.
I ask because I started to learn to code with Python 2 because that's what was preloaded on my system. Is one over the other a big enough deal at a beginner level that I should switch to 3 now? How much of a learning curve am I in for?
No, at a beginner level it's not, there are many guides that explain the differences at a beginner level, and you can go through those in a few hours at most, for example http://python-future.org/compatible_idioms.html
But, if you start working now on a Python 2 project and that project starts growing significantly, then it will be hard to convert the codebase. That's why you can see people saying that they didn't switch yet, it's not that they don't know Python 3, it's that upgrading large legacy code bases is hard (not only in Python).
Now the rest of upgrades were a bit painful (I was using some functional programming stuff, httprequest libraries, etc.)
My biggest program is like 100 lines of code maybe. So I will go ahead an switch now. But it's like 10pm where I'm at so here's hoping I don't play too much...
We're on Python 3 for all new stuff, and are migrating the old whenever we can.
From a new features perspective, the other reply's Placeholder is fascinating. (I haven't looked into it thoroughly yet.)
You'd be surprised.
If anything, the numbers show the opposite. The vast majority of Python codebases, legacy or new, are 2.7 or older.
For 2.7 Pypy reported 419,227,040 downloads for 2016.
At the same time, for ALL 3.x versions combined (up to 3.6) there are just: ~52 million downloads.
That's 1/8th of the Python 2 downloads.
The message I would take from those statistics is that needing a fresh download of Pypy is less common among 3.x users than among 2.7 users, who apparently needed to reinstall from the web at least a few times a day during 2016.
Occam's Razor would suggest that there are fewer Python 3 users
Those are not downloads of PyPi, but of packages. It's not like "number of downloads == number of individual developers". Those are packages, including package updates. A single developer can download 50 deps across his codebase, and update them to later versions 2-3 times a year.
As if individual developers are the reason behind the bulk of the downloads. I wonder how many downloads Travis alone counts for?
Your hate of Python 3 in every discussion about it is frankly baffling.
Travis runs/tests user projects, so there's nothing about it that's especially partial to Python 2 over Python 3.
>Your hate of Python 3 in every discussion about it is frankly baffling.
Or, you know, my pragmatic assessment of its popularity.
That you'd even use the word "hate" (when in fact, I like Python 3 over 2.7, even if its mostly tame updates over what 2.7 offers) shows that you're probably too partisan. I was enthused with Python 3 even when it was only a vision called Python3K back in 2000-ish. My personal preference has nothing to do with whether I see more people using it or not.
The situation is not unlike the perennial "next year is when Linux dominates the desktop", which has been every year since 1999.
> The situation is not unlike the perennial "next year is when Linux dominates the desktop", which has been every year since 1999.
Your bias is showing, as it does in every comment section on this site regarding Python 3, as you make comment after comment about how inferior Python 3 is and how nobody is using it at all because your sample of 2 companies shows this and how it personally hurt your family or whatever. You don't stop. Either you hate it or you hate something else and use Python 3 as a vent.
>as you make comment after comment about how inferior Python 3 is
Actually I've never made any such comment. In fact, tame updates" means that IT IS an update over 2.x, only not that much as it could be. Which most people I've read agree, or at least agreed until the async stuff.
>and how nobody is using it at all because your sample of 2 companies shows this and how it personally hurt your family or whatever.
Notice how I never said that, but actually gave concrete numbers that place those using it in much less (up to 1/8 less) of those who use 2.x?
So why the lie? Less is not the same as "nobody at all", and doesn't fix by itself just because you really really wish more people used 3.
>You don't stop.
Yeah, I continue expressing my opinion and my argumentation. I should stop because you happen not to like it?
Please don't bring "the feelz" into technical and community discussions. It cheapens the argumentation. If anything, it's you who are biased: 80% of your submissions on HN are for Python stories.
One can acknowledge that D is way less popular than Golang or that Perl 6 failed to gain traction over 5, without hating Perl 6. Ditto for Python.
From what I've seen of his posts, he's only talking about the reality of the situation.. not "how it should be".
Go look at the stats on PyPi and other metrics. Python3 failed, there is a cutoff time for adoption. It's no different than the first 24 hours of a missing person report. You don't get eternity to see if something is going to pan out or not. We're past that point for Python3. It may survive as it's own (smaller) thing, but Python2 isn't going to die either and that's more assured than Python3's fate.
And coldtea is right, but we're not going to do your research for you. What I'm saying needed to be said to you, but you need to find better ways to contribute than just rebutting everyone who has something to say about Python3. Talking about how he hates "something else" and using Python3 as a vent is just ridiculous.
Our legacy Python 2 build pipelines that we're actively moving off of hit PyPI far more often than our Py3 processes.
Maybe in your case, but from what I've seen, I seriously doubt use of Docker or Devpi makes any dent in newer Py3 codebase dependency downloads. Besides, tons of new codebases for greenfield projects are still done in 2.x Python.
Not sure how it is in scientific computing area, but for enterprise/web apps, any company that has legacy 2.x code and libs in production (which is most of them) will continue to write new parts (including new projects) in 2.7 for compatibility with their Python production setup.
3.x is either from companies that didn't already have significant 2.x Python code in production (generally newer companies that for some reason went with Python instead of Node or Go that the cool kids use) or new programmers that just get started and start with 3.x.
That's why packages like supervisor and graphite - which aren't libraries - are among the top downloads.
Those would exist for both 2.x and 3.x so it's not a differentiating factor.
Does it really imply many restrictions? Common Lisp, for example, is probably more dynamic than Python and it's been a compiled language for ~20 years.
Common Lisp was designed for interpretation and compilation from day one. The first implementations from 1984/85 had already compilers.
> Common Lisp, for example, is probably more dynamic than Python
Some parts are more dynamic than Python, some not. For example everything that uses CLOS+MOP is probably more dynamic. Also some stuff one can do when using a Lisp interpreter may be more dynamic. CL is more static, where one uses non-extensible functions, type declarations, static compilation, inlining, ... The parts where a CL compiler achieves good runtime speed may not be very 'dynamic' anymore.
When I wrote "append from two threads .. as expected" I meant "two items will be added, which one first is unspecified", and the GIL certainly takes care of that.
I agree it does not protect you from your bad threaded code - but then, nothing short of STM does (and even STM doesn't guarantee starvation in the general sense - nothing can).
Nobody uses the features of Python which make it a dynamic language? Google must write some really weird Python if their compiler is that strict.
Grumpy doesn't even seem to try to implement that. That's a good thing. If you restrict Python a little, it's much easier to compile.
Isn't that more or less what RPython does? https://rpython.readthedocs.io/en/latest/architecture.html I mean, I know that starting with a full-fledged(?) Py27 codebase rules out _actually_ using RPython for the stated goals of Grumpy, but I think the two projects agree in principle and differ about the definition of "restricted" :-)
Python has TONS of dynamicity besides those (eval and co), who are seldom used by anyone anyway....
If you think eval is what makes Python dynamic you're doing it wrong...
Frameworks do lots of such dynamic tricks in order to provide nice DSLs for building apps.
> getattr/setattr, or dynamically building classes with type()
"Upgrade to Python3" is the usual defense to that, but it's not really practical for large companies with software such as YouTube completely written in Python 2.x.
- It's a hard-code compiler, not an interpreter written in Go. That implies some restrictions, but the documentation doesn't say much about what they are. PyPy jumps through hoops to make all of Python's self modification at run-time features work, complicating PyPy enormously. Nobody uses that stuff in production code, and Google apparently dumped it.
- If Grumpy doesn't have a Global Interpreter Lock, it must have lower-level locking. Does every built-in data structure have a lock, or does the compiler have enough smarts to figure out what's shared across thread boundaries, or what?