Among the difficulties we encountered, I recall the need to use an offline optimiser to do trivial things like CSE and constant folding. CUDA and OpenCL will do this during kernel compilation, but Vulkan implementations seems to be designed as non-optimising (in order for loading to be faster, I assume). This is a perfectly understandable design, but it means it's a little more awkward to use. For direct programming (as compared to a compilation target), it's a lot more awkward.
Another issue was that basic Vulkan is very restricted, as I assume it's supposed to be usable on simple hardware. Additional functionality can be enabled through a collection of extensions. One particular problem I recall is that our code generator assumes the existence of general pointers, for example such that the same memory can be used for different types of values at different times. SPIR-V has/had a very strict notion of pointers, and no way to cast between different types without an extension, and that extension was IIRC not implemented by NVIDIA or AMD. There were lots of these kinds of issues.
In the end, Vulkan is usable for compute - after all, a single master's student (admittedly one who's very bright) managed to implement a Vulkan-targeting compiler backend in half a year. However, at the time I concluded that it's not as mature or as practical as CUDA or OpenCL, for reasons that seem perfectly solvable. However, for direct programming, the SPIR-V that is needed for shaders is completely inaccessible. It would be like writing machine code (not assembly) by hand. I assume graphics programmers have some layer on top to provide a sane interface, but since we were writing a compiler anyway, it wasn't a big deal for us.
Even though compute in Vulkan appears as a first class citizen instead of the weird tacked on feature it is in OpenGL, the design is still focused very much on embedding compute steps in a process that is designed to deliver images to a screen. And that uses a lot more opaque object representations like textures and samplers. GLSL has also always been a very restrictive language and that carried over into SPIR-V.