There are no good libraries to build open source CAD on top of really. CGAL doesn't support fillets or chamfers, and you'd really want a geometric constraint solver. I think building code first is a good first step.