Such a great question. I spent a lot of time working with ExtJS having been lured in by all those great, straight out of the box components which had their own healthy dose of added magic. Fast forward a year or two and suddenly when I need to do anything that is non standard then reality kicks in.

I now need to buy-in to the entire (framework) mindset to progress which slows things down (because it doesn't necessarily match my own way of doing things).

Had the same experience with cakephp. All those freebies which eventually caught up with me.

Ties in well with the whole libraries vs frameworks debate. I'm now in the libraries camp.

I wonder if Meteor packages are decoupled from each other so that I can pick and choose which ones I'd like to use in my project.

Turns out there really is no such thing as a free lunch.

> I now need to buy-in to the entire (framework) mindset to progress which slows things down

That's interesting, that idea could be folded into the technical debt metaphor.

So, by taking on a library or framework, I get a head start by just using it, but take on the knowledge-debt of not really knowing how it works. And when you get to the edge of what it can do, you either pay off the knowledge debt by learning how it works, or you throw the whole thing out and use a different library.

Yeah, said much better than I could. I don't regret any of the time spent with these frameworks because it really broadens your horizons and makes you think in new ways. Lots of the techniques I've seen now make regular appearances in the versions I roll myself. Other ideas I've completely rejected. Eg. ORMs. A few years ago they seemed like a godsend. In reality they simply shielded me from the basics of SQL and were inflexible black boxes.

I have to say that I feel like Rails' ORM does a magnificent job of saving me time. Migrations allow me to write database changes that can be undone more easily.

An ORM also seems to lower the amount of configuration it takes to get development databases synced. It's not as much of an issue for an experienced dev, but a designer or new team member would need help.

I have had to learn AREL, the relational algebra used by ActiveRecord, in order to do more advanced queries. That's analogous to learning SQL in more detail, but I'd still take that in a heartbeat over writing raw SQL. The ORM automates things like tersely expressing the object associations I've built, leaving room for fewer syntax mistakes.

I've not used the Rails ORM so I can't comment on how good it is or how easy it is to debug queries (peer behind the magic).

*The ORM automates things like tersely expressing the object associations I've built, leaving room for fewer syntax mistakes.'

Maybe it depends on the system you're working on however mitigating syntax errors seems like a small benefit. For me the SQL for most projects is fairly static i.e. once a given set of queries has been defined and tested they can lie there untouched, so once I nail a query and it performs the way I like I hardly ever need to go back and touch it again. However the performance penalty of sitting behind an ORM is ever present, for each query (at least for a cache miss). Personally I just don't like having 100s of lines of code sitting between:

model->get(('model.field' => 'value'));

and actually receiving the data.

It just seems so.... unnecessary.

of course YMMV, and perhaps the benefits kick in when you're working in a team (I'm not).

great point! i like to share this previous experience. years ago while working in large os development teams my approach was it was not enough to write tools/automate such that less experienced folks could perform tasks that had usually been done by senior folks -- the implementation had to empower the folks with less experience to control their own destiny -- which meant simplifying the implementation to the extreme. This also empowered the senior folks too as often the tools are not a primary focus, they just want to get some done and on with it

Like the reference to Rails—so easy and at the same time so inaccessible.

In the earlier days without powerful frameworks I was really developing: figuring out which pattern/algorithm did the best job. Nowadays when working with frameworks I feel dumber and dumber—before thinking myself I just google API calls and put them together. I am not trying anymore because I don't know where to start and everything is already solved on Github. This magic feels sometimes so meta and boring because I am more into consuming specs, tutorials, screencasts just gluing things together instead of real programming.

Just diving into JS and Node.JS to get this "go-kart" feeling back. But I guess the more magic involved the faster the development and Meteor admittedly looks revolutionary.

It's like the industrial revolution. Long ago, things (like furniture) used to be made by craftsmen. They had to choose the materials, they had to get a picture of what the final product should look like. They used to work by hand, using only simple tools. It was hard. To become a skilled craftsman you had to be an apprentice for several years and learn a lot from your master.

Then came factories. Factories were super-effective, they produced a lot of things fast, and cheap too, and the level of skill required from a factory worker was in no way comparable to the skill of a proper craftsman.

I notice the same tendency in software development today - we, the software developers, are more similar to factory workers mindlessly sticking together parts than "ninjas" or "rockstars". Of course, the process of software development today is not entirely like a factory, but I sense it's moving towards there. And if it is, programmers will become something cheap and fungible. Imagine hiring illegal immigrants to write an iPhone app :)

You're only seeing a part of the development task. Yes, the knowledge requirements for doing a task X (let's say a given CRUD app) are declining. But:

- How do you guarantee that all those pieces you just glued together will work with high availability? That they'll feel easy to use and consistent if exposed to the end user, or easy to maintain (future developers) and operate? Quite some work is often needed to have all that.

- User expectations are getting higher. You could get away with a certain quality and usability of software in 2005 that you couldn't today. Had a desktop software in 2000? In 2006 it was a desktop software and a website, now it is a web app, an iphone app, an android app...

- What you are describing is putting together a solution. Might make sense in many end user scenarios, web apps, etc. For things like automotive and other, low level programming is still needed - though abstraction is also slowly making its way in there also.

That's actually the sort of programming I like best! I don't want to care which sort algorithm is most appropriate for my dataset, I just want to call sort() and move on.

I love gluing things together that do the nitty-gritty stuff for me because it lets me focus on the big picture of making things happen, which is what I love about programming.

I know the feeling you're talking about. It's the difference between writing your own logic and simply learning what functions exist that someone created to handle the task you've been given.

Define "real programming".

"this is nothing short of revolutionary."

Oh FFS, this is not revolutionary. It is an attempt to wrap things in a friendlier package, while at the same time making something horribly insecure as a default install. Ruby on Rails did that a long time ago.

Better examples of revolutionary:

Manned Flight:

http://en.wikipedia.org/wiki/Wright_Flyer

http://en.wikipedia.org/wiki/Apollo_11

In computers:

http://en.wikipedia.org/wiki/Computer#First_general-purpose_...

Or even in programming:

http://en.wikipedia.org/wiki/C_(programming_language)

Try not to throw words like "revolutionary" around.

I get your point, but just because you can name examples of other technological advances that are more revolutionary, does not mean that this (in its own small world) cannot be described using that term. Revolutionary (according to the Oxford English Dictionary) simply means "involving or causing a complete or dramatic change". A technology that dissolves the boundaries between client-side and server side development is a pretty big thing in web development and it may very well cause a dramatic change in how we go about cranking out web apps.

I guess the problem is just with calling it revolutionary now, before we wait and see if it actually does cause any drastic change. There have been lots of frameworks that seem promising, only to learn partway in that they are still missing some really key functionality. It's unfortunate to use revolutionary to describe everything that could, someday cause a change.

http://www.webtoolkit.eu dissolves boundaries between client and server, and has been available for years. You shoult take a look.

I'd like to see some additional detail on this particular idea: "In Meteor, your server code runs in a single thread per request, not in the asynchronous callback style typical of Node. "

Seems one of Node's primary style advantages (async non-blocking style) has been eschewed.

Fibers take care of the "callback pyramid of doom" and makes it easier to write thick logic layers. However, it still would have the fairness and starvation issues that Node has with callbacks. It's still single-threaded and non-preemptive like "stock" node. I wonder how dependent Meteor is on Node? In other words, why Node? Is it for Node module compatibility? If it isn't, I'd love to see Meteor running on something else. I bet this thing would really fly on a lua backend with coroutines. Luvit.io?

I actually don't mind the callback pyramid so much for what it gives you.

One of the answers to "why node" question might be because it's all about Javascript. It's on the client, on the server, in the api, talking to the database...everywhere javascript. So node is a great choice for that. And it's very fast. I see luvit.io claims a 2 to 4 times faster than node performance, but it breaks the convenience of one language everywhere.

Lua's coroutines have the same issues that you mention for node's callback (single threaded and non-preemptive)

Coroutines and fibers in general, regardless of the language avoid the "pyramid callback" that can - depending on the developer - in spaghetti code.

Lua on its own right, is a bit special. Fast, small, easy syntax - almost similar to Javascript when you think about it. However, "Coroutines in Lua are not operating system threads or processes. Coroutines are blocks of Lua code which are created within Lua, and have their own flow of control like threads. Only one coroutine ever runs at a time, and it runs until it activates another coroutine, or yields (returns to the coroutine that invoked it). Coroutines are a way to express multiple cooperating threads of control in a convenient and natural way, but do not execute in parallel, and thus gain no performance benefit from multiple CPU's. However, since coroutines switch much faster than operating system threads and do not typically require complex and sometimes expensive locking mechanisms, using coroutines is typically faster than the equivalent program using full OS threads."

You avoid the code spaghetti problem that you can see with multiple levels of callbacks, but you still have the problem that doing a blocking operation in a coroutine (even something cpu bound like a long calculation) will block the entire lua process.

I wonder what's the performance hit of having a thread per request. I liked everything I say, but yes, having a single threaded node removes a lot of overhead.

In ASP.NET, a thread pool is maintained to offset the cost of spinning up a new one. There is some default throttling that controls how quickly new pooled threads will be created, but this can be tuned if you expect sudden bursts of traffic. Not sure how Meteor does it.

At the end of the presentation, I noticed they depend on a fiber library.

Joel Spolsky coined a term for this type of problem. The "leaky abstraction". See here: http://www.joelonsoftware.com/articles/LeakyAbstractions.htm...

It doesn't nessicarily need to be a leaky abstraction, in the case of rails (1) there was just a few miles between the path a beginner took and a pro took, which makes it real hard to stop being a beginner

I can't upvote this enough. When I saw the screencast, I said "Good,we are looking for that". But I want to know how this work, and how I can hack the core. At some point of building my app, I'll need finer degree of control.

The code is on GitHub: https://github.com/meteor/meteor

Another part about this kind of magic IMO is that it requires you to start from the ground up within this system. Otherwise the magic is either not doable or very hard. I know it's a tired argument but it's a reality - enterprise will have a hard time reaching for this kind of thing because it's so full-stack. Not easily separated without breaking all the magic. Maybe I've missed something about it, but it appears this way.

“Any sufficiently advanced technology is indistinguishable from magic.” —Arthur C. Clarke.

For an user of a product/service, this is bliss. For a hacker, it might get tricky. Now, considering the fact that this is all open source, the huge wall you’re talking about is just a matter of perception. You’re free to debug and go with it as low level as you need.

Easier said than done. The code of Rails was also publicly available, but getting from a hello world app to diving into a hard core codebase such as that of Rails is a huge step. Understanding framework code is quite different from understanding app code as well.

I agree. I didn’t say it would be easy, though. ☺ In fact, understanding “magic” is bloody difficult. It it weren’t so, then it wouldn’t be “magic”.

It’s also worth observing that Meteor is not presented as a “framework”. The comparison with Rails was introduced here, on HN. They say it’s “a set of new technologies”. I guess this implies loose coupling, which is a healthy paradigm when applied to components/modules/objects of a development ecosystem.

If only Meteor were nothing more than a proof of concept, it would still be mind-blowing.

This was my first concern too. I didn't have time to study the documentation, but I hope it is easy to integrate custom solutions both around and on top of the magic whenever it's necessary.

API question answered here: http://news.ycombinator.com/item?id=3826814