Try running it on a similar Windows or OS X system. Or try Chrome. Worlds of difference in UI responsiveness.

Make sure you have OpenGL off-main-thread composition enabled. I think it's still disabled by default on X11 systems, because they don't trust the drivers yet. Check about:support for "GPU Accelerated Windows".

about:config → layers.acceleration.force-enabled = true, layers.offmainthreadcomposition.testing.enabled = true, gfx.xrender.enabled = false.

Also, some "beta quality" features: browser.tabs.remote.autostart = true (e10s multiprocess) + layers.async-pan-zoom.enabled = true (exactly what it says, async scrolling like in Chromium, IIRC it requires e10s).

With all these enabled in stable Firefox on FreeBSD, the experience is excellent.

I have neither enabled and Firefox+X11 is the smoothest Firefox on the same machine. May depend on drivers, versions, and config, obviously.

May depend on CPU performance :) On my 3.4GHz 4-core desktop, it is pretty much always smooth. But on my 1.7GHz 2-core laptop, async pan&zoom makes a noticeable difference.

FWIW I'm using it on an old Core2Duo and a mobile Sandy Bridge Dual Core, so nothing fast by today's standards.

For me Firefox runs smoother on Linux+X11, but machines and software versions and configuration can differ, so I believe you when you say it's better on Windows. At least for me Windows Firefox locks up more frequently, perhaps due to Windows' coarse-grained locks in its subsystems.

I know it's anecdotal, but my experience lies with krakensden. I've tried some of the settings lower in the thread, hopefully it'll help.