No, it isn't. It's illegal to sell it across state lines, or in some places to the general public. But in most states, you can buy it at a farm, or other licensed facility. It isn't like heroin, or even pot, and you won't get in trouble for possession.

http://www.farmtoconsumer.org/raw_milk_map.htm

Looks like I'm wrong, but it also looks like in most of the country where you can buy it you can only buy it from the farmer directly. So it's still not easy to get.

Not really. There are many states that allow farmers to sell raw milk; some only for pet consumption. The illegal part is generally about 'smuggling' raw milk across state borders.