This usually happens when you have colour pallet increases. If you had a simple animated Gif with only 256 colours and convert it to any video you instantly have more data to compress. The better use case is where you start with video and convert down to gif (like when you use gif.ski)