No, it is not that difficult, at least if you are fine with not having the most efficient weapon. You take a metal pipe, put a lump of Uranium and some explosive at each end and to a first approximation you are done. The USA did not even test their gun type design before dropping it onto Hiroshima because it was so simple that they were confident that it will just work.
Now, a dirty bomb, which is just a conventional explosive together with radioactive material - designed to spread radioactive material instead of releasing the fission energy into one giant explosion - is easy to create, but not that much more effective than just a conventional bomb. It would trigger a massive cleanup effort, and it would surely give some people cancer later down the line, but it wouldn't be that spectacular.