In late 2015, I spent a couple weeks researching GIS data conversion utilities, Cesium's formats, and potential improved source datasets we could use to generate higher-quality imagery and terrain data. I can't haul out the relevant writeup atm, but I concluded that the EarthEnv DEM90 dataset appeared to be highest-quality free terrain source I could find ([0]), and the TrueMarble collection was the best free imagery source ([1]).
I generated our earlier terrain dataset using the Cesium-Terrain-Builder utility ([2]). It generates Cesium's "heightmap" format, but does not yet support the "quantized-mesh" format. I have seen a couple other tools that appear to deal with quantized-mesh tiles, though. On my list to dig into at some point.
As a related note, using the MBTiles format ([3]) for storing and moving tile datasets around is great. Multiple gigabytes in a single SQLite database file (or possibly a few files split by zoom level), rather than trying to move around millions of individual 10KB images on disk. Highly recommended.
[0] http://www.earthenv.org/DEM.html
[1] http://www.unearthedoutdoors.net/global_data/true_marble/dow...
(Of course I'm only talking DEM here, since you mentioned them, whereas Lidar proper kinda covers details such as houses and other structures as 3d points.)
[1] http://viewfinderpanoramas.org/dem3.html seemed to be the best in the "free" (for prototypes / MVPs and such) space when I last played in this space on-and-off from 2011 through 2014~ish.