Having written code to both leverage a site's (private) API and to scrape that same site (when the private API stopped existing), I would much, much rather use an API. Yes, the scraper works, but the scraper's code is much messier, and JSON keys, for example, provide some documentation in their own right. Looking at the scraper months later, there's a lot more headscratching. Scraping will work, but it also will leave you searching for many little pieces of information that would be exposed by an API but aren't by a static site.
Still, I will agree that scraping is much easier; I've used jsdom (Node.js) extensively, and, for my use cases, it feels like working in a browser (full DOM, scripts, etc.).
http://mechanize.rubyforge.org/
http://phantomjs.org/ or "its easier to deal with cousin" http://casperjs.org/ for very client heavy sites.
FWIW, you can get away with HTML only scrapers most of the time, you just need to look harder to find all the data. Totally recommend using "View page source" as that would always give you the original HTML vs the possibly altered DOM (after JS has run on the page) that you might see with Dev Tools/Firebug.
I tend to use Mechanized until I can't, then switch to Watir. Over time, I've found myself just strait up picking up Watir as it runs your browser directly and supports javascript rendering as a result.
I've found Selenium scripts much easier to comprehend, modify, and maintain over time than the PhantomJS scripts.
1. Python - BeautifulSoup
2. Ruby - Nokogiri (use in conjunction with Watir if you're scraping a very client-heavy website).
3. C# - HtmlAgilityPack in conjunction with ScrapySharp (there's a nuget package for both) - I highly recommend ScrapySharp because it allows you to query elements using a very familiar Css selector type similar to how you query dom elements in jQuery. :)
Scraping online content is so simple these days, if a website doesn't offer an API you still have alternatives ;)