News for hackers. Stuff that matters.
Re:Salesforce enables ‘modify all’ in user profiles by BonesJustice on Friday May 17, 2019 @01:17PM (#19941589)
Oopsie. Some poor bastard is having the worst day of their career.

I find it concerning that it was even possible for this to happen, regardless of whether it was intentional.

Re:Salesforce enables ‘modify all’ in user profiles by finaliteration on Friday May 17, 2019 @01:51PM (#19941917)
This is just a reminder that no matter how big or successful you are shit like this can always happen. And it’s usually not the fault of a single person, but rather some lack of process/review/control that made it possible in the first place.

I feel terrible for whoever initiated this. I’ve been in that boat and it -really- sucks.

Re:Salesforce enables ‘modify all’ in user profiles by llamataboot on Friday May 17, 2019 @02:05PM (#19942044)
It's hard to make things totally impossible, but hopefully a good post-mortem will identify the systemic issues that led to it happening, fix them, and not throw anyone under the bus.
Re:Salesforce enables ‘modify all’ in user profiles by londons_explore on Friday May 17, 2019 @03:37PM (#19942809)
"I'll just ask engineering to fix your wonky account"

"Ah yeah - it got in a bad state somehow, let me fix it manually"

    UPDATE permissions SET allow = 1 WHERE user=671156 AND permission=16 AND org=101 OR 102;
Classic SQL blunder...
Re:Salesforce enables ‘modify all’ in user profiles by londons_explore on Friday May 17, 2019 @04:20PM (#19943197)
For anyone reading this who has login access to a production SQL database...:

* Change your account to readonly. Make a new admin account, and put its credentials somewhere hard to get (and audited!).

* Make a directory in git for 'one off sql statements'. Make them all go through code review and have an automated system run them on merge/deploy.

* Enforce style rules with a linter/test, like "UPDATE must have a LIMIT"

* Anything the above process is too burdensome to do should have an API or admin interface built for the purpose.

* Aim to eventually get rid of your readonly account. A leaked customer data dump could kill the company and shouldn't be available to any malware on your machine. You aren't as secure as you think you are.

Re:Salesforce enables ‘modify all’ in user profiles by jjeaff on Friday May 17, 2019 @11:23PM (#19945393)
I agree with everything you said except "a customer database leak could kill a company".

Do we have any example of any company going under due to a data breach? Unfortunately, there seem to be a lot of examples of enormous breaches that did essentially nothing. (like Experian)

Re:Salesforce enables ‘modify all’ in user profiles by londons_explore on Saturday May 18, 2019 @03:25AM (#19946199)
That Italian malware vendor closed down after all it's internal documents were leaked. As did the law firm behind the Panama papers
Re:Salesforce enables ‘modify all’ in user profiles by usmannk on Saturday May 18, 2019 @04:01AM (#19946292)
Hacking Team is still alive and kicking.
Re:Salesforce enables ‘modify all’ in user profiles by jjeaff on Saturday May 18, 2019 @09:06PM (#19950721)
Well yes, but I would attribute the first to simply doing illegal things and getting caught. The data breach just helped them get caught faster. The second, yes, although privacy was literally their number one product and I wouldn't categorize a leak quite the same as a breach.
Re:Salesforce enables ‘modify all’ in user profiles by isatty on Saturday May 18, 2019 @06:14AM (#19946638)
You forgot one thing, if you’re ever executing one off statements make sure they’re in a transaction if it’s an insert or update.

BEGIN; Statement COMMIT; or ROLLBACK

etc

Re:Salesforce enables ‘modify all’ in user profiles by megous on Friday May 17, 2019 @08:36PM (#19944860)
You can also do changes in the transaction and check that it did what you expected before committing.
Re:Salesforce enables ‘modify all’ in user profiles by SOLAR_FIELDS on Saturday May 18, 2019 @08:05AM (#19946946)
Fair warning about this: if your transaction isolation level in your database is set to serialized hanging out in mid transaction while you verify your changes will block other calls to the table(s)!

I always recommend that you are wrapping calls in transactions if you are touching prod data. I only left the WHERE clause of my update query off once in my life before I started doing that.

Re:Salesforce enables ‘modify all’ in user profiles by megous on Saturday May 18, 2019 @08:15AM (#19946987)
If you fear this, you can put it into script like this:

BEGIN; UPDATE... SELECT for verification ROLLBACK;

And then just change ROLLBACK to COMMIT, after you're sattisfied. There's no need to do it interactively.

Re:Salesforce enables ‘modify all’ in user profiles by PowerfulWizard on Friday May 17, 2019 @09:53PM (#19945116)
That is smart, I would add "limit 2" to a single row update, so if it returns 2 rows updated I know it was wrong.
Re:Salesforce enables ‘modify all’ in user profiles by throwaway287391 on Saturday May 18, 2019 @02:51AM (#19946104)
> * Enforce style rules with a linter/test, like "UPDATE must have a LIMIT"

As someone who has never had access to a DB with any serious number of users, can you explain this one further? What if you really do want to update every row? Do you just do LIMIT INT_MAX or the like, and just force people to write that so that they always know they're updating the entire table? Or are you saying you should only ever use UPDATE on a known finite (and small?) number of rows?

Re:Salesforce enables ‘modify all’ in user profiles by londons_explore on Saturday May 18, 2019 @03:19AM (#19946183)
Updating every row is usually a bad idea anyway. That means every row needs to be rewritten, which involves reading and writing all data in the table. That will take locks which mean old data needs to be kept (in case of a transaction abort), doubling disk usage. Since half of rows are now dead, all other queries to the table will take double the disk access time.

Basically, if your table is over a few gigabytes, updating every row in one query on a production instance is a really bad plan.

Re:Salesforce enables ‘modify all’ in user profiles by CydeWeys on Saturday May 18, 2019 @10:10AM (#19947445)
So how do you make the change then? Do SQL dialects provide built-in tools to do an UPDATE to a large table in a saner way, or do you end up having to write an application to do so?
Re:Salesforce enables ‘modify all’ in user profiles by groceryheist on Saturday May 18, 2019 @03:01AM (#19946132)
I think it might just be like a psychological thing to try to prevent mistakes. Like a dialogue box asking "Sure?" before you delete your harddrive.
Re:Salesforce enables ‘modify all’ in user profiles by joatmon-snoo on Saturday May 18, 2019 @03:29AM (#19946206)
It's the notion of a staggered rollout. You should never initiate a global change without first validating it on some small, controlled-impact subset.
Re:Salesforce enables ‘modify all’ in user profiles by gknoy on Friday May 17, 2019 @08:33PM (#19944844)
I'm sorry for being a SQL noob. Can you elaborate on what is wrong with this, and what would be the way to write it without causing the scale of problem described in the OP?
Re:Salesforce enables ‘modify all’ in user profiles by fazzone on Friday May 17, 2019 @08:41PM (#19944883)
The problem is that the condition is parsed as

  (user=671156 AND permission=16 AND org=101) OR 102
The right way is to use something like

  org IN (101, 102)
Re:Salesforce enables ‘modify all’ in user profiles by jimnotgym on Saturday May 18, 2019 @04:03AM (#19946298)
There are lots of good answers already posted, but if I can make a further suggestion...

It is often worth running a SELECT on the WHERE clause you are about to use for your UPDATE. That way you can make sure only a limited amount of data comes back before you launch something catastrophic.

Re:Salesforce enables ‘modify all’ in user profiles by brianwu02 on Friday May 17, 2019 @08:39PM (#19944870)
the last OR statement "OR 102" will evaluate true.

It should be

    UPDATE permissions SET allow = 1 WHERE user=671156 AND permission=16 AND org=101 OR org=102;
Re:Salesforce enables ‘modify all’ in user profiles by slavik81 on Friday May 17, 2019 @09:36PM (#19945054)
You and fazzone have posted different answers. I'm a little unclear on which matches the original intent, but your where clause is equivalent to:

  (user=671156 AND permission=16 AND org=101) OR (org=102)
Re:Salesforce enables ‘modify all’ in user profiles by ummonk on Friday May 17, 2019 @09:47PM (#19945092)
That is still wrong and would apply to everyone in org 102
Re:Salesforce enables ‘modify all’ in user profiles by conro1108 on Saturday May 18, 2019 @03:03AM (#19946137)
Applying to everyone in org 102 is the intended behavior no? The slip-up would be the 'OR 102' which would just evaluate to true.
Re:Salesforce enables ‘modify all’ in user profiles by larkeith on Saturday May 18, 2019 @05:01AM (#19946425)
The example query is supposed to match a single account, not an org.
Re:Salesforce enables ‘modify all’ in user profiles by snaky on Saturday May 18, 2019 @10:47AM (#19947671)
> In addition to the SQL-standard privilege system available through GRANT, tables can have row security policies that restrict, on a per-user basis, which rows can be returned by normal queries or inserted, updated, or deleted by data modification commands. This feature is also known as Row-Level Security

https://www.postgresql.org/docs/11/ddl-rowsecurity.html

Re:Salesforce enables ‘modify all’ in user profiles by GuiA on Saturday May 18, 2019 @04:46AM (#19946385)
As a manager, if you see a poor bastard having the worst day of their career because of something similar, it means you have inadequate safeguards in place and you’re not doing your job correctly.
Re:Salesforce enables ‘modify all’ in user profiles by idlewords on Friday May 17, 2019 @08:26PM (#19944811)
The worst day of their career so far.
Re:Salesforce enables ‘modify all’ in user profiles by KallDrexx on Friday May 17, 2019 @11:57PM (#19945501)
I can imagine it happening, but it's hard to imagine how they applied it to their disaster recovery backups before noticing the issue.
Re:Salesforce enables ‘modify all’ in user profiles by rc_kas on Friday May 17, 2019 @01:39PM (#19941804)
Right. Like add a unit test or something. Geez.
Re:Salesforce enables ‘modify all’ in user profiles by finaliteration on Friday May 17, 2019 @01:53PM (#19941931)
I’m more curious how a script that does this even made it through review. And if there wasn’t a review... why not?
Re:Salesforce enables ‘modify all’ in user profiles by gambler on Friday May 17, 2019 @01:32PM (#19941743)
Stuff like this is always possible wherever you have fully centralized architecture. I don't know how many massive cloud failures it will take for IT community at large to realize this.
Re:Salesforce enables ‘modify all’ in user profiles by ralph84 on Friday May 17, 2019 @01:39PM (#19941803)
Salesforce launched in 1999 before before cloud was a thing. It's basically just a big Oracle database.
Re:Salesforce enables ‘modify all’ in user profiles by mpeg on Friday May 17, 2019 @02:51PM (#19942396)
That's not very fair to a company that was one of the pioneers of a lot of the things we consider normal on today's software as a service.

Also incorrect, they have great cloud and devops practices. If anything it's likely this bug's impact would be limited due to how decentralised SFDC operates.

Still a massive fuck-up, I'm interested in seeing if they'll release any more detail on why it happened.

Re:Salesforce enables ‘modify all’ in user profiles by the_duke on Saturday May 18, 2019 @12:06AM (#19945525)
While a little unfair, and Salesforce is a decent product with nice dev tooling (apart from the weird ancient Java ish custom language), but under the hood, it really is just a Oracle database per org.
Re:Salesforce enables ‘modify all’ in user profiles by vips7L on Saturday May 18, 2019 @09:53AM (#19947377)
Weird ancient proprietary language.

Impossible to run locally.

No debuggers.

Virtually impossible to put an entire org in source control.

No package manager.

More undefined behavior than a C compiler.

Re:Salesforce enables ‘modify all’ in user profiles by moocowtruck on Saturday May 18, 2019 @02:45AM (#19946086)
> decent product with nice dev tooling

whaaaaaaaaaaaaaaaaat hahahaha i can't take that seriously; I've used it and it felt like a giant pit of despair

Re:Salesforce enables ‘modify all’ in user profiles by no_wizard on Friday May 17, 2019 @08:33PM (#19944845)
Well,

they do own heroku

https://www.heroku.com/

Re:Salesforce enables ‘modify all’ in user profiles by sbr464 on Friday May 17, 2019 @10:16PM (#19945191)
They purchased Heroku.
Re:Salesforce enables ‘modify all’ in user profiles by sb8244 on Friday May 17, 2019 @10:58PM (#19945315)
In fairness they did do long enough ago that they would have massively messed it up if they at least didn't understand something about running software.

SFDC hate is pretty common, maybe because of how big they are. I think that their tech is actually pretty impressive.

Re:Salesforce enables ‘modify all’ in user profiles by CPLX on Saturday May 18, 2019 @09:55AM (#19947384)
The hate is likely propelled in part because their sales process is the work of Satan.

Like seriously I would rather lick alcohol soaked razor blades than do a standard annual renewal of a Salesforce contract.

Re:Salesforce enables ‘modify all’ in user profiles by tastroder on Friday May 17, 2019 @02:10PM (#19942091)
Other than being pretty public at this point, this incident could have easily happened in an upgrade/rollout in on-premise settings as well (at least without a good staging environment and test process).
Re:Salesforce enables ‘modify all’ in user profiles by meddlepal on Friday May 17, 2019 @09:52PM (#19945111)
You get all kinds of new and exciting failure and fuck up modes in a decentralized architecture.
Re:Salesforce enables ‘modify all’ in user profiles by rightbyte on Saturday May 18, 2019 @07:59AM (#19946927)
If I add a service listening on port 1234 that pipes text input to a root terminal most likely no one will ever know.

That's the advantage of decentralized architecture. It's a disadvantage too though ...