You are probably using one of the IRC bridges that is on the matrix.org server. The matrix.org server is having a lot of problems with being DDoSed as well as not being able to keep up with the normal traffic. The next generation homeserver written in Go is supposed to help with this a lot.

There is nothing inherently broken with Matrix <-> IRC bridges, it is just that one server. Other servers are bridging to IRC perfectly fine and you are welcome to host your own.