ListenAndServe blocks forever. If you have 10 calls to ListenAndServe, it's implied that you're in 10 different, concurrently running processes (goroutines). A panic, in Go, does not cross a goroutine boundary. If, in imaginary parallel universe Go with exceptions, exceptions did cross goroutine boundaries, then one goroutine could crash another goroutine.
The point of Go is to be able to sensibly write stable, concurrent programs with compile-time safety. Exception-based error handling would be a mess with Go's concurrency model.
In practice, you would only call ListenAndServe 10 times if you were listening on 10 different ports, because you had 10 different projects on the same server. That's a perfectly realistic situation. Now consider that one site has an error and throws an exception. Should all the other sites go down?
Error handling in Go is admittedly tedious, but when you work with the language for a bit you'll start to understand why they chose to design it in that manner.
Which is why there are other nice ways of communicating such states in go that go hand in hand with handling errors at their occurrence. If you find an error in a goroutine, you can signal that on a channel, return and wait for a new goroutine to be restarted in place of the errored one.
The benefits of a properly designed (not Java-like, but C# or PHP like) exception system stems from the fact that in the wast majority of cases you don't need to catch exceptions right away. Most exceptions can be handled in a few centralized locations higher up the stack. If you have 10 calls to ListenAndServe, this means having 10 times less exception handling code.
Moreover, when you design a library and the users actually care about errors, exceptions save you from needing to pass your error codes up the call chain from deeply nested methods.