Oops, you are right. For some reason I thought using the If-None-Match header was the "right" way of doing this, it just adds flexibility to how you cache stuff.

http://developer.yahoo.com/performance/rules.html#etags

"If you're not taking advantage of the flexible validation model that ETags provide, it's better to just remove the ETag altogether. The Last-Modified header validates based on the component's timestamp. And removing the ETag reduces the size of the HTTP headers in both the response and subsequent requests."