This has been codified in the Rule of Least Power: http://en.wikipedia.org/wiki/Rule_of_least_power
Indeed. Bitcoin transactions all contain validation scripts that are run through a stack-based interpreter. This Forth-like language does lack loops but, even after being there for years, the current client still only accepts a small set of standard templates.
This shows up all the time in the myriad Haskell eDSLs. Monad-based interfaces are obvious but throw away information that is useful for optimization (see http://homepages.inf.ed.ac.uk/slindley/papers/unembedding.pd... for how to get some of it back). Applicative interfaces are really amenable to optimization, though (see Facebook's Haxl project: http://www.reddit.com/r/haskell/comments/1le4y5/the_haxl_pro...)
Applicatives were the example that I had in mind. See also http://gergo.erdi.hu/blog/2012-12-01-static_analysis_with_ap..., or think about how regular expressions' limited power makes parsing with them a linear affair. (Incidentally, regular language parsers are Applicatives, too.)
As an example, SQL (ignoring the crazy bits) is so amenable to query optimization precisely because it is not Turing complete.