Assuming you're a C-programmer, read the libsodium docs first. https://download.libsodium.org/doc/public-key_cryptography/s...
If you're using higher level language, use a library that provides bindings for it https://download.libsodium.org/doc/bindings_for_other_langua...
By using libsodium, you're not rolling your own crypto. Rolling your own crypto would mean
-trying to find new one way functions for public key crypto -trying to implement RSA from textbook -trying to implement RSA-OAEP from papers, RFCs, books etc.
Using a library is not anywhere near those. There are other ways to fail cryptography too, from not doing public key authentication, to storing private keys in insecure places.
So it's highly recommended you take time to read a book on the topic. The best modern book currently availalbe is https://www.amazon.com/Serious-Cryptography-Practical-Introd...
That's debatable. Libsodium does not have a proper authenticated key exchange. The key exchange does the job somewhat, but it has worse security properties than a properly crafted interactive protocol.
Problem is, designing your own key exchange protocol is more delicate than most construction. I know, I designed my own, and made several serious mistakes in the process (one of which voided an important security property).
Granted, rolling your own constructions is generally less error prone than rolling your own primitive. But it still shouldn't be done without at least having followed and fully understood an introductory course in cryptography (I recommend https://www.crypto101.io/). I mean, I did quite a bit more than that, and I still don't fully trust myself.
There are still some dangerous things you can do (e.g. mismanage keys, reuse nonces).
I wrote a library called Halite for PHP developers that wraps libsodium and makes it even harder to misuse. My philosophy was, "You shouldn't even need to know what a nonce is to use it securely."
https://github.com/paragonie/halite
From elsewhere in the thread, PyNaCl takes a similar approach. So how dangerous "just using libsodium" is, with respect to the "don't roll your own crypto" guidance, depends a little bit on which binding you're using.
Also, a lot of tasks might require a specific protocol (PAKEs, VPN protocols, searchable encryption, etc.) that libsodium isn't suitable for.