This is a very simple and very pathological example that's easily ferreted out with a few more summary statistics (median, min, max) but it's a good illustration of the blind application of statistics. Short of visualization, non-parametric statistics really help with such things. Correlation is a fragile, linear measure, and things that are obviously correlated by inspection can easily appear mathematically uncorrelated -- points on a unit circle, for example. Likewise, the mean of any skewed distribution tells you very little, but that's the statistic that's always cited. Quantiles, medians, and non-parametric measures of correlation such as rank correlation are simple and often overlooked. They do a good job screening for pathological data sets like Anscombe's quartet and real world ones.

It's also worth mentioning "dumbbell" data sets. Two clusters of data, each of which have a independent, meaningful correlation in them, can easily leverage a linear regression into a meaningless line passing through the two clusters. That's a pretty common issue with high dimensional data (obviously you can see it in a 2D scatter plot), and it's not easily caught short of looking at regression diagnostic statistics.