Long context is a little bit different than extra email storage. Having 1 gb of storage instead of 50 mb has essentially no downside to the user experience.
But submitting 1M input tokens instead of 100k input tokens:
- Causes your costs to go up ~10x
- Causes your latency to go up ~10x (or between 1x and 10x)
- Can result in worse answers (especially if the model gets distracted by irrelevant info)
So longer context is great, yes, but it's not a no-brainer like more email storage. It brings costs. And whether those costs are worth it depends on what you're doing.
But submitting 1M input tokens instead of 100k input tokens:
- Causes your costs to go up ~10x
- Causes your latency to go up ~10x (or between 1x and 10x)
- Can result in worse answers (especially if the model gets distracted by irrelevant info)
So longer context is great, yes, but it's not a no-brainer like more email storage. It brings costs. And whether those costs are worth it depends on what you're doing.