Angular has been one of the easiest to pick up javascript frameworks I've worked with.

Easy to start, sure, but try building a complete non-trivial application from scratch.

What would you conclude to be the single most important issue/hurdle/constraint about Angular when moving from simple app to a non-trivial application? (Genuine question)

Inertia. The best practice for many reasons is to structure your app as lots of little components, and doing that in Angular is too hard, mostly because every component is at least three files, likely spread out in different places in the folder structure. The overhead of refactoring a component into smaller pieces is huge, since you have to independently split the template, the scope, and the dependencies.

All of the things that seem useful about Angular become a burden at scale. ng-repeat is my favorite example: try repeating more than a few hundred non-trivial components and you'll quickly choke the browser. You end up having to spend most of the application development time figuring out how to do things in a way that doesn't make Angular too upset.

Going from a simple single-controller to multiple controllers, services, and the rest was a hurdle I never managed to cross. I understand the principles involved, but actually going from "Here's my complete data model, handled by this controller" to "Here's an angular application with a number of controllers and services" isn't something I've managed to do. (Although this is also not something I've had to do professionally so it's not been a focus for me.)

I like Angular a lot, and didn't find it too difficult to pick up.

I had spent a couple of years prior working with WPF which is MVVM based, so I was already familiar with that way of thinking, which maybe helped me out.

I think Angular 2 is going to catch a lot of people by surprise, if they haven't been keeping up though. It's all component based so I've been restructuring my apps so that everything is a component, which means that an upgrade shouldn't be too painful!

I've seen a number of Angular apps though where directives aren't even used, and those people are going to find upgrading to 2.0 pretty much impossible without an entire rewrite I think! Maybe I'm wrong!

I couldn't agree more. It was my first JS framework, and after 10-12 hours of tutorials I was already building a rather large application with it.

The process was also easy for my coworkers, granted they had someone to ask questions from.

I honestly cannot understand this criticism of the framework being hard.

It's one of angular's chief strengths that it's easy to pick up. That's my perception anyways. I think that statement about difficulty to master applies more to Ember.js than anything.