therealpygon
I kid, but also not. If the smart model already has a smart plan, odds aren’t great more smart models are going to create an even smarter plan. Not entirely true because that is the goal of parallelizing discovery attempts, but the point is there is more room to gain benefit from adding additional inference-time compute with a smaller model than one that already had a ton of compute to begin with.
It seems like your counter-point is “they already use everyone’s data, why didn’t you just assume they would”, which is a bit weird from a victim-blaming perspective but valid nonetheless.
That makes the original statement even less accurate.
The lack of AI mental model between conversations or even within the conversation is a definite frustration. “No, I didn’t just decide to entirely change everything I’ve planned based on one ambiguous word…”
Your use of a separate session for q&a is actually something I find interesting in the new Granite Switch architecture’s KV Cache Normalization, where the kv cache is essentially frozen while the model performs certain other tasks in order to prevent the mental ‘conversation’ from getting polluted. However, I’m also assuming what you mean by “helps grounding” is that it helps keep the agent grounded in the context by not introducing a bunch of external information, which is certainly beneficial.
Outside of major players like Amazon/Google which have compatible offerings available, I believe together.ai, fireworks.ai, and deepinfra.com are, as best as I can tell, US-based (if that matters in your situation) SOC2/HIPPA/ZDR. Each of them offer serverless model hosting with caching, which is going to be the cheapest overall. Most of them also offer private AI deployment which is more expensive but your entirely own instance. Some of them more heavily quantize their models (deepinfra) so take that into your workload considerations. I included deepinfra because they tend to have some the new models sooner than others or have models the others don’t (like mimo). IMO, together > fireworks > deepinfra.
However, you’ll have to keep in mind that for true ZDR you give up caching which increases your inference costs, so it depends on how “Zero” you need the zero to be and on what timeframe. If you just don’t want your data being trained on, normal use is fine imo. If you are working on something actually sensitive (like, because of NDA agreements and such), then they might not be the right choice and you may be looking at more expensive offerings or building a machine.
One note: you have to pay attention to the source of the inference. Many (similar to open router) offer models that are pass-throughs to a providers API, but they are pretty clearly marked. Just make sure the model is marked as serverless when evaluating new ones. This is where a proxy comes in handy so you can prevent your LLM from accessing a different models from the specific ones you’ve vetted.
It used to be most of our spam came from Google accounts, and some still does, but now that is far and away outpaced by spam coming from Microsoft who does not take those same steps to deanonymize users. Also, “bot traffic” (AI traffic) is definitely a very real problem sites deal with. Also, you know who else uses Gemini from VPNs?
All the things you complain about seem to have real, direct consequences when not done.
I’m with you, I don’t want to be tracked and don’t want log in to every website. Just saying the reasons are pretty compelling when the reality we live in these days is a connected world with advanced technology where your average teenage could potentially hack a power plant and your average scammer can now use far more advanced AI tools to trick people.
This will only get worse, which is the real reason for the quiet US/EU push to unmask users, require IDs, etc. and not because some kids go online to naughty websites with boobs on them.
…and are not only “okay” with that, but enjoy it so much as to go online to ridicule others who don’t do the same? Weird.
News outlets always seem to forget “again” in the title.
Effectively no reward at any risk level, regardless of whether one agrees if there is actually any risk.
(I joke, but not really…)
Btw, no one suggested people shouldn’t consider various forms of information. All I pointed out is that if an LLM happened to guess right, it doesn’t mean that is some sort of datapoint that others should believe validates an AI’s predictive abilities any more than a telephone psychic or webmd should be a trusted source because they are occasionally right.
It is exactly the tactic that makes snake-oil…I mean the vitamin industry…so profitable from people going around and telling others crap like “rubbing the oil from this tree on my foot made me sleep better”, and “I took this garnish as a supplement and it cured my blah blah”. Can people need vitamins? Sure. Does the average person need anywhere near the amount they take? Rarely. People sell each other a dream of feeing better; this would be cutting out the middle man and suggesting an AI take over.
It’s funny how pointing out the risk means people act like I’m suggesting it should be banned. We can see how many MLM Karens came out of the woodwork to defend AI in this, as though I somehow said people shouldn’t be allowed to consider feedback from an LLM by simply pointing out exactly how wrong — best case — it could have easily been from a system known to make errors. In fact, the only reliable part about AI right now is that it will make mistakes. Literally almost all investment in AI is about having it make less mistakes,. Every single metric we use to measure AI is about how many mistakes it makes. Every time we test how smart it is, we measure how much it got wrong and how many times it failed.
I understand it. Sick people will turn to just about anything as a cure or to make them feel better, or simply to find some hope. Sometimes they get better, whether that thing helped or not, and then loudly tell everyone that whatever thing did it regardless of realities. I’m not discounting anyone’s stories, I’m pointing out how they were lucky it didn’t go exactly the opposite, which is what the next person could experience by trusting and not considering the feedback from an LLM cautiously.
Also, can literally anyone point out where the comment I replied to said a word about “doing research”? Anyone? People immediately Karen in on how “it’s just part of research” — where was that said? Or did people just read what they wanted to? Too funny.
PS, I have MS… before any more idiots want to tell me that I don’t know anything about the problems with the US healthcare system.
(Because some laws are stupid. Please support right to repair and DMCA reform.)
- You didn’t have a thyroid issue and you spent thousands more on tests the doctor was right that you didn’t need.
- Your cat didn’t have that issue and you spent hundreds on unnecessary tests.
Are you essentially saying that it is better for an LLM to sell you an idea that sometimes might be right, selling a dream to anyone with and without the money for it?
So… a telephone psychic?
The thing about WebMD telling everyone they have cancer is that sometimes it will be right.
(Just for clarity, I’m really glad it was helpful for you.)