How about using CMake to download and cross-compile the 3rd party dependencies? I've worked on a couple of robotics applications (C++). I wouldn't depend on Ubuntu for packages. You'll lose the flexibility to choose package versions, apply patches etc.