Not doing so results in weird looking shading, which has a very 90s/early 2000s feel about it.
I think it's worth noting here that linearizing-after-load is often not good enough, since the renderer most often has already automatically filtered (think MIP pyramid/etc) the texture values by the time you get hold of them. You always want the values in a linear color space (gamma=1.0) during filtering. Some renderers try to support this so you can load a jpeg without thinking about it too hard but myself I would prefer to include an explicit linearization step for extra control and peace of mind. You may end up wanting to use a specific LUT to linearize with for example, and the errors that creep in if gamma is not 1.0 can be quite subtle and hard to put your finger on, but can really sap photorealism in a nasty way.
Not really what you asked for but it helped me understand how massive of an impact colour grading can have on how you perceive a scene.
https://developer.nvidia.com/gpugems/gpugems3/part-iv-image-...
This is about working with linear-to-light values on GPUs however the issues in VFX are largely the same.
Kinda seems like it might be quite nice in a web/css world to be able to similarly work in some kind of fairly neutral 'linear' color-space in the code, and let the platform and hardware translate that as well as it can?