Contrast this with remote attestation, where they might show you the source code for everything but you're still powerless to do anything.
You have no idea what has been baked into the weights in the training process. In theory you could find biases and attempt to "patch" them out, but its a vastly different process vs. patching machine code.
Consider what would happen if Google's open weight models were best at writing code targeting Google's services vs. their competitors? Is this something that could be patched? What if there were more subtle differences that you only notice much later after some statistical analysis?
Local ai is not ready, and if you think it is, prove me wrong with a detailed guide running commodity hardware with complete setup steps that can use a decently sized model.
I spent 2 weeks trying to get anything running - 8gb RX550XT, 12gb ram, 8core cpu. I even tried turboquant to lower memory utilization and still couldnt even get a 3B or 4B model loaded, and anything lower wont suit my needs (3/4B are even pushing it).
You're like the kid showing up to a test without a pencil.
It's ridiculous for you to suggest that an advanced AI model needs to run on your budget 7 year old graphics card that is already out of date for even today's gaming. My parents spent $2500 on a computer in 1995 and that was a 166Mhz Pentium 1. If they spent that money today it would be $5261. Think of what you can get for amount of money. Then you're over here trying to say a budget graphics card needs to somehow compete with the bleeding edge of computer innovation.
You do, in fact, need to spend money on appropriate gear if you expect to participate.
Spending humongous amount of money to get machine that'll felt obsolete in 2 years? I don't know.
They're for sure impressive but I don't see how anyone can push them as "open" when they are literally binary blobs. Worse, because it's not practical for anyone to actually train LLMs that can even come close to competing with the ones corporations are pumping out.
Open weight models can be a big boost to building Open AI (cough). Progress comes from incremental improvements, -- and open weight models are a big advance in privacy, security, and autonomy over relying on hosted closed systems.
Source vs not is only one (important!) dimension, moreover in FSF land they define source as being the preferred form for modification, at at least for some kinds of modifications the weights are the preferred form.
Any source on that?
https://www.gnu.org/gnu/thegnuproject.html
> [...] the easiest way to develop components of GNU was to do it on a Unix system, and replace the components of that system one by one. But they raised an ethical issue: whether it was right for us to have a copy of Unix at all.
> Unix was (and is) proprietary software, and the GNU Project's philosophy said that we should not use proprietary software. But, applying the same reasoning that leads to the conclusion that violence in self defense is justified, I concluded that it was legitimate to use a proprietary package when that was crucial for developing a free replacement that would help others stop using the proprietary package.
> But, even if this was a justifiable evil, it was still an evil. Today we no longer have any copies of Unix, because we have replaced them with free operating systems. If we could not replace a machine's operating system with a free one, we replaced the machine instead.
Still leave open the the question of RMS personally using SunOS (as opposed to some other proprietary unix) but I think at this point I'd just go dig up very old GNU sources for evidence of that, but I suspect your question was primarily about RMS' ethical reasoning which is well answered above.
Although it seems to me that the comparison is somewhat fragile : it was not possible to develop GNU anywhere else, whereas we could completely build local models from scratch nowadays, unless I'm mistaken.
One observation is that the LLM is a next token predictor but if you train it on the internet/textbooks/etc you get a predictor of that--- but that isn't the behavior we actually want. None of these sources tend to contain "Solve this problem for me. OK, here is the solution:".
It wasn't physically impossible to start GNU the other way around, by bashing machine code into a system until you had a working operating system. But doing so would have been a lot less reasonable-- much more expensive, making progress much less quickly, etc.
This can never be the case.
Both the licensing and source aspects of the Free Software movement are aspiring to create high level of equality of access to a [software] work between both the original author and far downstream recipients. Obviously full and universal equality is impossible because part of the work is only in the author's mind and not everyone can obtain and use computers, but approaching that as closely as possible is important and it is important to think about how to achieve a high level of equality for each work in each context. What is "source" in any given context is a choice the author makes about what level of access they want to pass on to others.
In the case of AI, weights can never be the preferred form for modification because of the equality of access issue. The people who trained the AI (and hide its training data/code but published the weights) will always have more access than the people who only have the weights. Just like a binary can almost never be the preferred form, because the authors have access to the source but we don't.
There are also many ways to bias the model and insert backdoors or other suboptimal behaviours into it during training data selection etc.
> It turns out a significant (but hopefully decreasing) number of the population is easily coerced into anything when "security" is given as a justification.
The people who opposed Intel are now telling each other how hopeless and powerless they are. You can see it on HN, in this thread: No drive, outrage, and self-organizing response to these issues, but despair - 'nobody cares', 'there's nothing we can do', etc. Quitting is a sure way to lose.
I don't think those are the same people. I, for one, will continue this fight by telling everyone I know about the fact that Google is going for absolute control of the Internet, and by extension, everyone's lives. They have already become an unelected global government.
The problem isn't the TPM, but attestation. As soon as the TPM is required to not be under your control to get access to Y, bad things happen.
Hell, in actuality, the problem isn't even attestation, its policy. The EU Parliament (the one the people vote for, the Commission are cronies) might eventually force corporations into something more citizen-friendly. Neither Apple, Google or Microsoft is going to drop a market that big.
I -- literally -- do not care about a single "account" in any "service" I use aside from my email and bank account. Most people would add a few social media accounts to that list.
You don't need a "place to put secrets". Your iPhone app does not do anything important enough to require a "trusted chain" of cryptographic bullshit, just use a password and Google/Apple login.
The reality is that there is software dependent on the user being unable to modify it. This safeguards the server against fraudulent users.
And what actual applications did you have in mind that warrant throwing everybody under the bus? (by that I mean some applications (allegedly) need it, so it gets forced on everyone)
That the laziest of us don't mind and the worst of us want something is not a respectable argument for anything, ever.
We can have a discussion about FaceID specifically, but “convenience” is not considered trivial within the security sphere.
Second, I work for a (very large) bank, and you actually do not want to trust them with your biometric data directly. You can be absolutely assured of the privacy of your biometric data with the bank, better than with a Silicon Valley tech company. But I would not trust the bank’s data scientists to come up with a model that will not have an extremely high rate of false positives and negatives.
The reality is, if such an initiative was started at a bank, it would be shuttered after years of delays.
Anyway flawed implementation doesn’t mean that hardware attestation is a fundamentally useless primitive. Apple Wallet is responsible for millions of transactions a day.
Passkeys absolutely do not need TPM.
You can get passkey support in any browser with a simple 1password plugin without any TPM hardware.
The same way you could get a TOTP app on your phone without any TPM.
TPMs are just an extra security layer for most usages.
They are mainly a necessity for some shady business like DRMs.
They do not, but how does the service you’re using know your passkey is secure? For all they know you’re just some gullible user that clicks through every fishing email you get. You’re dumb, weak, helpless, they gotta protect you from this scary world out there, and maybe yourself as well.
They can’t do that if they allow your passkey to be stored anywhere you control. KeepassXC? The second you type in your master password the keylogger will snatch it, and your entire database with it!
Okay, maybe you’re some hot shot cryptographer, you’re using a TKey (think Yubikey, except you have full control), and there’s no way your secret key leaves it even if your main computer is fully compromised. Well, the service doesn’t know that. All they see is your public key and a matching signature.
So, sorry Mr. Security Researcher, we’re gonna have to be safe, and require you to use approved hardware only. Too many (wo)men children out there must be protected, we have no way to tell you’re not one of them, so it’s remote attestation or you’re out. What’ online buying worth for anyway, when you can just cross the ocean?
---
Just so we’re clear, I agree with you here. But don’t forget there are two kinds of passkeys out there: with or without the evil remote attestation. And many companies will push for the remotely attested kind, using the exact argument I used above, except with a straight face.
Or they will just present a false dichotomy: remotely attested passkeys on the one hand, short easy to guess reused everywhere passwords on the other.
Passkeys are non-phishable. That's part of their schtick. I'm not a huge passkey fan myself, but this is a real benefit.
When the passkey is protected behind an HSM (TPM, Yubikey, Tkey…), even a compromise of your main computer can’t steal it. Attackers can still temporarily log in on your behalf, but they can’t do anything with your passkey as long as your computer is turned off. Which means you can un-pwn yourself out of this situation by reinstalling everything (but do keep your HSM!).
Overall, we have several levels of security here:
- Weak password, (potentially reused everywhere). Fished once, pwned everywhere. Not to mention password database leaks.
- Very strong unique password from your password vault (KeepassXC). Note that with automatic login, password managers may provide good phishing resistance. Manual copy pasta is still vulnerable, but at least you only compromise that one account.
- Passkey stored in your password database. Phishing proof as you say, but falls to a keylogger.
- Passkey sorted in a hardware security module. Can’t be stolen ever, save for a vulnerability in the HSM itself, or, if you haven’t set up a password for your HSM, theft.
Clearly that last option is the most secure. Clearly it would be nice if everyone could do that, though we do need a way to recover from the loss or destruction of the HSM (which in the case of the TPM may mean something as mundane as changing your graphics card). Yet often, other ways are more convenient.
Still, I strongly believe companies should not force people into one method or another. Okay, I could maybe tolerate passkeys being forced on me, but not the remote attestation part. Let me manage my own security, with my own tools (preferably open source), thank you very much. There is one use case for which I may approve of remote attestation: work accounts. Because at this point it’s not about the safety of the customer, it’s about the safety of the company itself. It makes sense then that the company (or government agency) impose whatever stringent restrictions on how to access their network. They do have to provide any required tool (company laptop, company palmtop, company dongle…), same way many companies are required to provide individual safety equipment to any of their employees working in hazardous environments.
When it comes to the notion of requiring DBCs without also requiring remote attestation, how do you deal with solving the problem of virtualized credential devices, e.g. swtpm? If some application wants to leverage DBCs, it will make some DBC API call, e.g. call out to a TPM. However, without some sort of attestation scheme, there's no way to verify who/what is on the other end of that API call.
Maybe it's not important for applications to be able to require DBCs without attestation. But at first blush it seems like a valid thing to want.
It’s definitely something I would want, but as you hinted at yourself, if there’s no remote attestation, the user can just use a software TPM. So, a company using passkeys has two choices:
- Enforce DBC with remote attestation. This raises the security floor, but enforces device vendor lock-in, and prevent users from selecting unapproved, but potentially even more secure, devices.
- Do not enforce DBC. This lets users use less secure virtualised devices, but there’s no vendor lock-in, and those who want may use the latest most secure device ever.
Which alternative is appropriate is now a social & political problem. My opinion is that for general computers released to the general public, remote attestation is never legitimate. Even with the best of intentions it is fundamentally uncompetitive, and they make it way too easy to go full Evil Corp. Specialised appliances and employees however are different stories.
---
Anecdotally, I have worked on TPM provisioning a couple years back, and I had to warn my hierarchy that doing it the way they specified, the TPM could be impersonated: we checked the signature of the certificate, but failed to compare the certificate root with the manufacturer’s keys. My boss didn’t believe me, until I showed the production code happily provisioned a software TPM, without detecting the impersonation. (Actually, he didn’t believe me even then, I had to go over him to the security specialist.)
This was totally a case of remote attestation. But I believe this particular case was legitimate, because it was a specialised appliance (electric car charging station), that was meant to process payments, similar to a gas station terminal.
That's my business, not theirs. If my password gets stolen, that's my problem, not my bank's. Same deal if my passkey gets stolen. They're welcome to try to educate me on good security hygiene if they want, but what hardware I use to secure my credentials is not something they should get to decide.
Many people aren’t like us. Give them freedom to chose their password without mandating 2FA, and some will lose money to a password database leak & offline guessing. The policy maker knows this, at which point they have a choice: stricter annoying rules with fewer victims, or looser rules with more victims?
Yes, we can mitigate much of this with education, as can we limit vendor lock-in by mandating that the bank does not require any particular device they do not themselves distribute, for free, to their users. (My bank for instance gave me a little device that has a camera, a small screen and a key pad. Upon payment I use the device to scan some QR-code, the device gives me a one-time code that I type, and done.) My point is, some kind of tradeoff remains.
Also banks kinda have to deal with fraud, which presumably costs them money. Stolen passwords mean more fraud, increased costs… that may be incentive enough to enforce stricter rules. And to be honest I’m okay with that, as long as it is accessible. Which in my case means no phone app of any kind.
Come to think of it, there is one law I would pass: for important stuff like banks, no amount of security justifies a lack of accessibility. If I don’t have a smartphone, I should still be able to do online payments. Same if I’m blind. Or both. When I hear all around me about people being utterly unable to do banking, or worse, accessing government online services, without a locked down Android or iOS phone, I’m horrified.
Yep, there's a reason freedom vs safety (or libertarianism vs authoritarianism) is an axis on many political spectrum charts. This is a very common source of tension in politics. As you can probably guess, I usually find myself on the libertarian side of such debates. Freedom is worth the price.
> Give them freedom to chose their password without mandating 2FA, and some will lose money to a password database leak & offline guessing
To be clear, I have no issue with secure defaults. There's only an issue when you start trying to make it impossible for users to compromise their own security, because accomplishing that requires you to take away their freedom to make choices, which I don't think is an acceptable thing to do to mentally sound adults.
There's plenty of competition in the banking space, so normally I'd be fine letting banks and their customers sort this out on their own. But there's not a lot of competition in the OS space, and allowing banks to limit your choice of OS exacerbates that problem.
The fix I've been floating in my head for some time now for a lot of these types of problems in the digital space is some sort of software freedom law guaranteeing users the right to modify software running on devices they own. It would fix so many issues with the software industry, including probably this one, since many common uses of hardware attestation would probably fall afoul of such a law.
I generally lean towards that too, including for this issue. But we do need to own up to it. Explicitly ask ourselves, what kind of bad consequences, and how much of them, are we willing to put up with in the name of freedom?
Also, some framings make it difficult: the second someone speaks of protecting the children, all of a sudden freedom becomes secondary. Which leaves two counters, which are logically compatible, but tend to be rhetorically exclusive: denying that this new thing will actually protect the children; and asserting that the protection it allegedly provides is not worth the loss of freedom.
The second one is a hard sell, which is why we so often revert to the first one. Take age verification: sure it won’t stop determined underage teens from seeing images of bunny girls. But it will deter some of them. And assuming images of bunny girls are bad for teen health, it means age verification does "protect the children". A little. And voilà, we’ve destroyed the argument that age verification does absolutely nothing, mass surveillance for the win!
> […] which I don't think is an acceptable thing to do to mentally sound adults.
I haven’t thought of the psychological damage over-protectiveness may cause. That’s a bloody good point.
> There's plenty of competition in the banking space,
Given how people in some countries complain that it’s difficult to find a bank that doesn’t require a locked down phone for online payments, I would argue perhaps not plenty enough. I totally agree though that for any bank to require one of two OSes is not good, and for this reason would be tempted to outlaw such requirements (thus reducing corporate freedom, but I care more about individual freedom).
> some sort of software freedom law guaranteeing users the right to modify software running on devices they own.
That is very tempting indeed. Do understand though that such a law comes very close to mandating Free Software everywhere: for this right to be effective, users need access to the source code, and be allowed to let some professional modify that code for them. Any mass produce piece of hardware would effectively have to publish the full code source of their drivers for all to see. I would absolutely love that, but NVDIA would likely lose their marbles over this.
The way I look it is that when someone uses their freedom for evil, the consequences of that are that person's fault, not the fault of freedom itself. Responding to evil done by one group of people by curtailing the freedoms of everyone, including innocents who have done nothing wrong, is fundamentally unjust. Perhaps in some extreme cases it could be justified, but I'd use a standard similar to how the US supreme court defines "strict scrutiny" when evaluating such measures.
Patrick Henry once said "Is life so dear, or peace so sweet, as to be purchased at the price of chains and slavery? Forbid it, Almighty God! I know not what course others may take; but as for me, give me liberty or give me death!"
Unfortunately yes, for some this is a hard sell. I'm not sure how to convince others of the importance of freedom if they don't already consider it to be important, to me it's an almost fundamental belief that I hold.
>> […] which I don't think is an acceptable thing to do to mentally sound adults.
> I haven’t thought of the psychological damage over-protectiveness may cause
My point was more that taking away a person's freedom for their own protection is the kind of thing you do when they're either children or mentally unsound. Outside of those cases I don't think it's acceptable.
> it’s difficult to find a bank that doesn’t require a locked down phone for online payments
Some of this may be due to regulations making banks partially responsible for things they shouldn't be responsible for (like the customer's phone getting hacked). Responsibility and control go hand in hand. But mostly I think it's just due to lack of demand, which would be solved if running modded OSs were more common.
> Do understand though that such a law comes very close to mandating Free Software everywhere
Yes, exactly. I guess I've been radicalized by Stallman. ;) Though to be fair I do partially disagree with his definition of free software, in that I don't think software needs to be freely redistributable without payment (freedom 2) in order to be libre free.
I understand this a pretty radical proposal, and completely politically non-viable for the foreseeable future. It might even be a bad idea to do all at one even if that were possible. But I think probably there are some smaller steps that could be taken in that direction which would be beneficial even if I'm not entirely sure what those are yet.
A chip which you can write to and interact with but can't read is valuable; it lets you enforce conditions which you otherwise couldn't. For example, you can protect your sensitive data with a 6-digit pin, secure in the knowledge that the chip will erase the encryption key after 10 failed attempts. If you had full access to the TPM storage, you could brute force that PIN in seconds.
I had an idea to create blatantly insecure passkey browser extension. Maybe I should do that.
An old account with typical activity patterns can be extended some level of trust. If you sign up for an email address and immediately send a message with 100 recipients in CC, you're probably a spammer, so you get blocked. If you've used the account for years, ehh it's probably invitations to your high-school reunion or a donation drive for your Church, let's let this one through.
You can only extend this level of trust if you prevent your gullible users from constantly getting hacked; 2FA is one way to do that.
"I don't use a TPM in my computer so it shouldn't exist" has always sounded like a weird argument against the tech in my opinion.
Many Android phones have their secret storage implemented as a virtual machine rather than a TPM. The lack of a TPM doesn't suddenly give me any more freedom, although it does come with security downsides.
Does it? Why waste time on developing exploits when you can just call up grandma and get her give you the money by her "own" volition - using her secure device - by pretending to be the bank/IRS/her grand daughter using AI voice/etc.
TPMs are a fucking mess. TPM 2 at least, I’ve worked with it for a few months. I love me some hardware security module, but I want to control it. And if it must be a standard, please please to something like the TKey, so it can be both much simpler than current ad-hoc standards and future proof.
The problem lies in companies like Apple/Google/Microsoft rejecting attestation that they do not control.
People confusing big tech's policy choices with tech features have made "I want my laptop's auth token to only be usable on my laptop" a controversial opinion.
Once you have the script, that’s a couple actors in a classroom, a couple e-ink readers for props, the film crew… It can be shot with less than 10 people in a day, then one person for a couple days for cutting and post production. And that’s on the very high end for this scene.
Considering the reach this video would meant to have, avoiding AI would not be that expensive.
So, it looks like they were aware about such kind of issues and tried hard to mitigate them.
Then the "security" and Trusted Computing authoritarians continued pushing for TPMs and related tech, and contributed to the rise of mobile walled gardens. Windows 11's TPM requirements were another step towards their goal. The amount of propaganda about how that was supposed to be a good thing, both here and elsewhere, was shocking.
It turns out a significant (but hopefully decreasing) number of the population is easily coerced into anything when "security" is given as a justification.
The war on general-purpose computing continues, and we need to keep fighting.
Stallman was right, as always. Time to give his "Right to Read" another read. (If it hasn't been done already, an AI-generated short film of it would be a great idea...)
"Those who give up freedom for security deserve neither."