And then I thought "why not use it for something web-like". My experience was dreadful. After coming from Python+Django, it felt like writing Java 1.4 (not a good thing). Especially for returning JSON objects with lots of mixed types.
So I switched to Clojure and never looked back (Python would have been an equally valid choice).
Go's a great language, but it's being applied in some very odd (and potentially unsuitable) places.
One problem I have is that the application is extremely CPU intensive. I can't get past 35-40 requests per second with 500 concurrent users (at 4 CPU cores, 14 GB RAM), which seemed too expensive economically. (I cached as much data as possible, both at the Nginx tier and with Redis) and tuned the number of uWSGI worker processes.
Do I have to try other languages or do you think I have more room for optimization with Python?
Then I'd suggest you use Cython for a speed up or try using PyPy which can be 200% faster without any changes.
When I used Python and worked with financial data, every live algorithm would be recoded as a Cython extension.
Having done web development for over a decade. My experience is that algorithm is far more important than language. After that data structures trumps language. After that doing work you don't have to do trumps language.
First look for really ugly nested loops in your code. I can't tell you the number of times this type of things ends up in your codebase, especially since dev environments often have a small subset of data. Even an O(n!) algorithm is fast for small values of n.
Next look to see what kinds of datastructures you're using. For instance, dicts are a seductive way to store data, they have a theoretical O(1) lookup, but they have a drawback of randomizing memory lookups, which means that for subsequent looks for a large dict, you'll end up with lots of cache misses, can make each lookup 1000 times slower. Meanwhile, a tuple has O(n) lookups, but if you're finding that you need to iterate over the data, you're going to benefit from memory locality. So know your tradeoffs. Also code that had one usage profile at launch, can often have a very different usage profile a year later, so it doesn't hurt to revisit this occasionally.
Last look for code that isn't doing anything. Are you calling the database twice in a row, asking for the same data, and not using it? It sounds obvious, but over time these kinds of things have a way of showing up in a codebase, and they can really add up.
I also should mention caching at this point. It can be incredibly hard to get caching right, but the performance savings are pretty big. It goes under the bucket of not doing things that you don't need to. Do whatever you can to do page caching. Putting an nginx cache in front of the webserver serving python for the majority of your pages could easily get you to hundreds of requests per second.
How did you like Clojure? Does the ecosystem it compare well against Django? I tried it once during an on-site interview but was put off by the multi-minute startup times.
So, comparing the languages, I tried Clojure because a colleague was using it and to me it feels like the best language I have ever used professionally (comparing to C#, C, Obj-C, Python, Java, JavaScript, Ruby, Go) and I'm very happy using it. Great functional programming features, great libraries, I really like the LISP syntax and the stress I'd been having with Python kind of melted away (things like state in objects and the type system (which kind of doesn't really exist most of the time with Clojure)).
In terms of doing web type things, the Liberator and Ring libraries feel a lot like a big chunk of Django's routing and middleware functionality. Selmer does a good job of templating.
For most user interface work I'm using React.js (which I also really like) so my Clojure code is only serving up JSON on an HTTP API. But for simple templated HTML, Selmer works well.
One question is "Django is an entire framework. How does it compare to lots of little libraries?". I have to say that Django is probably the most well constructed pieces of software I have worked with and it hangs together very well. Clojure with a few libraries is very good, but beating Django is a very high bar to reach for any framework or group of libraries.
Startup time is an over-done argument. Once you're running a REPL you don't need to restart it, so development isn't slowed down. Once you're running a server you don't need to restart it, so production isn't slowed down.
Since you work with React, how do you like JS? I'd imagine that, coming from Clojure, you'd be frustrated with its inconsistencies, as I am, coming from Python.
I've tried ClojureScript (without React) and it didn't quite click -- some of JavaScript's weird behaviour still shines through.
Hint, PHP is 100% one of those. The bad rap is either because of hipsterism or for issues that have been long solved.
The rest of its warts are no worse than the kind of BS any language has -- in fact before JS got in fashion the same things was said all the time for it too (for its bizarro coercion rules, only fp arithmetic, bs scope rules etc).
It's almost an irrational fear, but I remember working on plugins for applications where plugin hooks were non-existent, and the canonical approach was to patch code. I remember applications where there was no frontend server, but rather a sprawl of PHP files each doing one thing and hooking into the 'main' system via a 'require settings.php'.
Apart from that, working within PHP still obliges you to do a lot of hairy things---like working with Drupal. Now Drupal is certainly better architected than say Mambo/Joomla ever was, but it's still a better CMS than it is a web framework and for some odd reason lots of companies insist on building things into Drupal even today.
Now ... all of my recent knowledge is second hand, I've been out of PHP and into Python for at least 5 years now
I'm not snarking; I honestly don't know. I tried using PHP years ago, threw up my hands in horror, and have never touched it since. I felt a little bad about that, but a few years back Eevee's "PHP: A Fractal of Bad Design" [1] persuaded me I wasn't missing much, and that my "learn a new language" time was better spent on Scala and then Python.
So what's changed in the last few years that makes it 100% as good as Ruby or Python for web development?
[1] https://eev.ee/blog/2012/04/09/php-a-fractal-of-bad-design/
Without exactly knowing what exactly bothered you and what you use instead, we can't answer.
As for "PHP: A Fractal of Bad Design", I find the article quite shallow. One could say the same things comparing something like Python to Smalltalk, and yet Python is fine as it is.
> So what's changed in the last few years that makes it 100% as good as Ruby or Python for web development?
That seems like it should be pretty straightforward.
> One could say the same things comparing something like Python to Smalltalk, and yet Python is fine as it is.
If you've already dropped the standard from "100% as good as" to "fine as it is", then I think you've answered my question. Nobody's denying that PHP is a perfectly cromulent language; things get built in it. The servers hum, pages render, revenue is made. For its audience, it's fine as it is. fine But as far as language quality, developer experience, or aesthetic quality goes, I think PHP's kind of a mess. I don't mind that people don't care about the things that Eevee raises. Different strokes for different folks, after all. But I care about them.
Personally, having spent the last year writing Python, I agree that it's fine, that it's adequate. But I think it's often no better than adequate. The object orientation is bolted on and a bit clumsy, there are 70-or-so global functions that mostly should go objects, the inclusion stuff is kinda broken, there's a bunch of stuff that looks like Java envy, etc.
Given that's how I feel about Python, which is honestly fine, you could imagine why I think PHP looks like a novice in the 90s ate a half-dozen random O'Reily books and then threw up in a compiler. For people who do almost all their work in that language, have high detail memory, and are on the naturalist end of the naturalist/theorist spectrum, I think that's ok; they'll get by happily. I'm just none of those things.
Actually there's no logical inconsistency being "100% as good as" to "fine as it is". Obviously something "fine as it is" could be "100% as good as".
>But as far as language quality, developer experience, or aesthetic quality goes, I think PHP's kind of a mess.
If we're comparing it to something like Scheme or Haskell or Smalltalk, yes. So, compared to the "aesthetic quality" of what? Javascript? C++? Perl? Go? Java?
As for the developer experience, can we get any actual qualified statements? Because the average developer experience of getting lost in the JS framework-du-jour land, or the Java developer churning BS enterprise factories upon factories is probably worse. So what exactly is bad with the developer experience? Certainly not the tools, from Composer to IDEs catering to the language. And I'd say not the fact that "it just works" without elaborate setup (or do people love configuring stuff in other environments?).
>Personally, having spent the last year writing Python, I agree that it's fine, that it's adequate. But I think it's often no better than adequate. The object orientation is bolted on and a bit clumsy, there are 70-or-so global functions that mostly should go objects, the inclusion stuff is kinda broken, there's a bunch of stuff that looks like Java envy, etc.
All of points being to it being a pragmatic language that developed from early versions.
>* For people who do almost all their work in that language, have high detail memory, and are on the naturalist end of the naturalist/theorist spectrum, I think that's ok; they'll get by happily. I'm just none of those things.*
Well, I don't do most of my work on the language (I mostly work with JS these days. I've also been using Python professionally since before 2.0, with early Zope et al). But I still find it perfectly fine.
If one has an issue with "70-or-so global functions", for example, I can't imagine how he would feel with JS "global by default" scoping -- which unlike PHPs is a core language issue.
> there's no logical inconsistency being "100% as good as" to "fine as it is". Obviously something "fine as it is" could be "100% as good as".
One phrase I take meaning "adequate", the other "good". Of course, if the thing you're comparing it with is merely serviceable, then yes. If all you're saying is that PHP is no worse than Javascript as a language, then sure, fine. I also don't think Javascript is a particularly good language.
I thought you were saying that PHP's issues "have been long solved", so I thought perhaps I was missing something since my last encounter. But given the tone of your responses here and your repeated evasion of the question "So what's changed in the last few years that makes it 100% as good as Ruby or Python for web development?" I'm just going to keep running with my theory that I'm not missing much.
http://www.brightball.com/golang/go-from-a-php-perspective
TLDR: Go is good at the things PHP is bad at. PHP is good at the things Go is bad at. They actually compliment each other very well.
And PHP still has more vendors, support in all kinds of hosts, even in third world facilities, and more companies offering commercial support. Plus there's stuff in PHP that don't exist with the same level of maturity/adoption/community in Ruby/Rails, e.g. Wordpress.
And even a path to static typing, through Hack.
EDIT after reading the article: I do have to agree that concurrency in PHP is a bit of a pain, but those 3d party tools are pretty easy to use and maintain.
> Without significant forethought to several levels of caching; such as in memory caching, op caching ...
This is so trivial to setup it's not even worth mentioning.
> Sure, PHP has PSR standards etc, but they're fairly new and developers have been slow to adopt them.
Everyone I know and every library I use uses those standards. PHP is pretty standardized now, especially compared for example to js.
PHP frameworks work well out of the box and contain solutions for most use-cases.
Go is nice but when I tried it I really missed something like composer. "go get" and commit your dependencies to the repository? Seriously? That's wrong on so many levels. Meanwhile I think composer is one of the best package managers around.
I still don't like it but the landscape definetely changed, and it keeps getting better.
> Meanwhile I think composer is one of the best package managers around.
Apart from the fact that it uses gigabytes of ram on real projects. At work we ended up using it outside virtual machines, to later copy back the dependencies.
I used did a quick test (having run `composer clearcache` each time) on one of my projects:
Without composer.lock: Memory usage: 75.96MB (peak: 87.5MB), time: 38.54s
With composer.lock: Memory usage: 7.47MB (peak: 8.99MB), time: 28.6s
I use composer every day, and have 0 problems when doing a `composer install` on production.
I also recommend setting up Toran Proxy, which will allow you to ensure you're not boned when Github goes down.
One of not negligible size. They usually have many dependencies. I can see on our composer.json around ~60 dependencies, and the number will only grow (apart some lucky circumstances).
https://github.com/composer/composer/issues/1898
> I also recommend setting up Toran Proxy, which will allow you to ensure you're not boned when Github goes down.
Yes, that's a good idea
It is, but the overwhelming majority of tasks in a web app don't happen concurrently, so it doesn't matter.
When it does matter just use something else.
But is that cause, or is that effect? The dominant web programming languages for the last 20 years have all been single-threaded, so of course the web stack looks single threaded. Personally, I find there's a ton of things that ought to be concurrent but aren't. I mean, most other network apps need to do things concurrently, why not web?
Once you start looking at it this way, the staggering number of hacks employed to pry concurrency out of fundamentally single-threaded languages is astonishing.
Go is not the only solution to this, there are many. (Go is one of the more Algol-ish language choices that doesn't involve learning a new paradigm, though, and can make it a relatively easy sell where "Let's all learn Lisp!" or "Let's all learn immutable functional programming!", where "all" may be dozens+ people, may not happen. [1]) But I think in five or ten years, "the overwhelming majority of tasks in a web app don't happen concurrently" is going to sound very 200x-ish.
[1]: Incidentally, for those who are still baffled by Go's success, notice how all the alternatives involve trying to teach everyone a brand new paradigm. While that has its advantages, when it comes to language popularity, not requiring that has its advantages too. I'm a polyglot myself, so I get how Haskell is nice and Clojure is nice and how they open your mind to new paradigms etc. etc., no sarcasm, fully mean it, if you only know C-style languages I fully recommend you go fix that soon, but when you're trying to get a team of people on the clock to switch languages for a nicer concurrency story, cutting the requisite training time by 90%+ raises the bar very high for those other languages, in relative terms. (Even moreso than it sounds; not only is training costs less, but the risk of training failure is also that much less. It compounds.) I do not mean this as "advocacy", I mean it as explanation.
What sort of things? I've been writing web software for ~20 years, and the vast majority of things that I've made boil down to "request comes in > validate input data > transaction to database > check transaction worked > write a response". Concurrency has never really been a problem, or even something I've wanted.
Just this week I met with a friend who wanted an app for community meetings that would let people express their preferences in real time, changing their expressed opinions as discussion happens. That way everybody could see how the consensus was shifting in response to discussion without people having to say it out loud.
I quickly whipped together a prototype, and I intentionally didn't use a database because I wanted this to be dynamic. Whenever one person changed something, everybody else needed to see it immediately. That's very easy in a context where you have all your data hot in RAM in one process with easy concurrent-ish access to it: you have a bunch of observer objects that get notifications each time you change something. In database-land, it's a pain.
And it works for me at size as well. Years ago I led a team where we were building a new web product. We were big on test-driven development, and databases are a notorious pain point. We started out without one, figuring that we'd pull one in when we needed it. We ended up never needing one. Our system had incredibly low rendering times (<5ms, even on 2005 hardware), and its 1800 unit tests ran in a couple of seconds. Starting with concurrency saved us so much development pain.
Another common example of where concurrency is valuable is in notifications. Quora and Facebook both have excellent notification systems: dynamic and instantaneous. Twitter's are much rougher, and it's clear they've put a lot less thought into concurrency.
Games, of course. Imagine trying to build agar.io without concurrency. Or look at Google Docs: concurrency is at the heart of what makes them valuable in an office setting.
There's nothing wrong with CRUD apps. There's nothing wrong with all the mainframe batch-processing apps that preceded them, either. But there's more that we can do.
That's what I do!
(Actually, I do that from the start of the project, so that adding something as pedestrian as concurrency doesn't involve a complete port to another language.)
Why is this a pb? Makes your app is more resilient to external outages and no need to git clone all the repos with historical data.
But I've used it in the web app scenario. Replacing Java (jetty/tomcat servlets with pure Java code) with Golang. My own discovery was it could replace the jetty/tomcat with servlets part seamlessly, just with a bunch of http url handlers. I know it could be possible still more easily in other languages/stacks which are "zillion" times more productive.
Now, we all know that nothing comes free. For e.g. C++ templates cause compilation to take more time; have heard people complain about Ruby/ROR (including one on this page); I myself hated the Java configuration of the app servers, even in minimalist use of just a Servlet (the least needed for a web app thing), not to forget the SOAP-Servlets (Axis et al) before JSON ( good common sensical structure IMHO from the XML schemas/dtds (any one remember them?!)) became fashionable.
So the bloat/overhead can be in terms of useless jargon (e.g. the latter part of above para) which are too verbose and mean something very simple. Slowness e.g. Ruby/Rails. Or simply memory bloat (Which means more servers for the same amount of app processing capability).
So please do understand that there could be reasons other than "just can't get by without the iron hand of the compiler", where people use Go. :-)
People who have come to realize they wasted a decade chasing promises whether they were called OOP, EJBs, SOAP, XML, App Server, etc. Only to have been left with mysterious crashes because of "out of memory exceptions" (heck, jetty/tomcat are open source, but does any app developer should be debugging them. No, right? )
So there are also people disillusioned by the promises they fell for earlier. Who now prefer the minimalist approach. And actually very much appreciate when the wise people developing Golang, are hesitant to add a thing like Generics, despite the pressure from many quarters.
To summarize, why I use it:
Don't try to make it too easy for me, and pinch me at wrong times (when I start to get more users for e.g). Just Give me speed and give me transparency baby. I know how to make it work.
Edit: Typos and minor rephrase
In the Tomcat/Jetty/<any other Java app server> world. If we want to have a web app (HTTP) we need to code servlets and add them into web.xml config, matching urls to Servlet Names. And within the Servlet we have coded, we need to call other Java code (the logic).
The container/app server takes care of the TCP/HTTP layers. Thread pools (which you can configure etc.).
Now in the Golang world. We just use the http package[1] for all that.
To listen to url:
http.Handle("/doSearch", &MySearch{"MySearch"})
Where MySearch is a struct which just needs to implement the interface: func ServeHTTP( w http.ResponseWriter,r * http.Request)
Within this function you can put all the code which you will put in a java Servlet.And to make this a service in your main package/program you will also do:
err := http.ListenAndServe("localhost:10001", nil)
So these 3 pieces are the ones needed to have your own micro service or any web service running.In our case, the similar functionality when moved resulted in lower memory usage and much higher stability (very less crashes, in the rare event of the service needing memory, which OS doesn't have. So the OS will kill that service anyhow - tech independent. We check using dmesg | grep <program name>)
Another benefit of this move, from our experience is getting rid of that webapp folder in any Java Servlet container. You will often end up plugging along multiple smaller apps, alongside your main one. And all reside in the same process.
That's not ample segregation, in our use. Its better you have different processes. So even if one dies, the others are independent.
This is also easier to manage. Deploy new ones - restarting just the ones changed.
Now we all know hot loading (without restart) is offered in several techs including some Java containers. But in practice, there always is some silly reason for you to restart. And sometimes the reason is psychological :-), we are not convinced that there is some baggage until we restart.
So hope my brief experience share is of some use. All the best.
Edit: Just want to add You must review this package page, if you are considering it
Also, it's relative newness combined with the fact that it is on the lower level of things means that basic tasks that have been iterated to precision on other platforms are still very raw around the edges and require quite a bit more boilerplate, which is not at all suited to almost all "web development".
I know, I know. "But it's faster!" I'd take a clean, cohesive system that can scale horizontally even if it takes me 5 times the servers of my entire-system locking clusterfuck that makes me guess about data.
There is probably an interesting metaphor to be made that is corollary to the adage about the hammer, something about how if you give that man a screwdriver he starts looking for nails to pound with it?
When I learned Ruby I came from a Java background, and being able to search for things like "java interfaces in ruby" was incredibly helpful. It is a very succinct way of defining what you are trying to do in the new language, and often leads to a useful post explaining how to do what you want to do (even if that means doing it a completely different way in the new language).
I suspect this is also why you see frameworks like Revel gaining so much popularity in Go. Developers coming from Rails, etc all want something they are familiar with, and Revel looks familiar. It may not be the best way to build web apps in Go (I don't know), but limiting what you have to learn has its benefits.
I personally view this as a positive thing. It makes it easier for developers to learn Go and find out what it is useful for so that when they can benefit from it they realize it.
Hey, Haskell and Scala exist!
What stacks are you referring to exactly?
I mean it does feel like superpower when applied to tasks you'd previously have to use C for. But have you tried any of the modern web development stacks? They are like a zillion times more productive. Hint: PHP is not one of those.
I guess some people just can't get by without the iron hand of the compiler :)