But someone somewhere high up in the ivory tower ticked a box. I suspect the same here happened with Amazon and an insurer somewhere. Push the hell down to the bottom of the stack.
The funny thing in this case is the process holes are still so enormous that you can drive a tank on fire through it and straight into production. The net gain is zero and the outcome is overconfidence.
Company was primarily using Java for the backend. The Application Security team decided to start scanning all our deployable JARs and WARs with a static analysis tool. Well, immediately the tool started complaining about our libraries. For example, the official Elasticsearch library, for the version of the database we used, had multiple System.out.println calls. Well those were all flagged and the static analysis tool deemed our code unacceptable. We needed to use a logger, not stdout. So we had to go and take it up with AppSec. And explain, no thats not our code. Yes, it does run in production. No we can't use a different library.
The whole thing was exactly as you put it, a kafkaesque nightmare where I'm being punished for something I don't understand by people who don't understand. An no-one can tell me why.
At some point I came across a tool (I think it was white source) that was going through Github page of each library used and was checking if there are no bugs reported.
Obviously very often there were totally bogus tickets, support requests created as bugs, rants as bugs, duplicates of already fixed tickets, etc. so investigating all that crap was really hard.
Another problem was licencing. There are a lot of libraries that have dual licence, e.g. GPL and CDDL, obviously the tool was panicking that GPL is used, not noticing that there is a second "business-friendly" license.
Those static analysis tools are not bad as a concept, however ratio of false positives to real issues found is too large to make them truly useful unless one can invest a lot of resources into tuning them.
The seemingly very smart people designing these systems don't understand that their 1 in a billion edge case happens hundred of times every single day when you scale it to the world
Of course the most interesting is to use counter-AI to generate video of right behavior without repetition and with matching location/street view in case if the Amazon/Netradyne AI is smart enough to have geolocation based street view integration.
Only if you know what the AI cam sees, if it's like a blackbox to you then I doubt there's way that you could do to fool it.
"Your camera doesn't get correct image"
"Why?! What is wrong?"
"See for yourself"
Uber drivers used to look up time slots for big planes arriving at the airport. They would then all go offline on the app to trigger "surge pricing" and then all benefit from it.
No one knows exactly how the algorithm works, but they know how to fool it.
The thing to consider is that no actions are taking place in a vaccuum. The drivers gaming the system is not an isolated input that is occurring. Instead, there are a number of different inputs that are putting downwards pressure on the rates a driver gets to charge / how much money the driver actually takes home after costs, taxes, etc.
While it's not incorrect (or even unjust) to describe the behaviour as being potentially 'gray' (although the use of colour is a different matter, I understand the sentiment being expressed), by only highlighting and focusing on the least powerful members in the system, it unconsciously paints a picture that draws focus away from the other inputs into the system that have resulted in this situation occurring in the first place.
And even if we just limit ourselves to tech users - some companies have long been logging coders' Eclipse (mostly not in the 1st rate countries of course), and I suspect these days that may get fed to AI. And some day it will make into US and West Europe too.
While I acknowledge that some people will prefer to be a driver than to work in any of the places whose doors would open with this skill, I suspect this is a minority.
It sounds like a system that needs to be banned or thoroughly regulated. Naive driver distraction systems like this do not make traffic safer. The company should prove that this system is safe just like autopilot manufacturers have to.
Of course the unfairness is also a valid concern, and I'd love to see them both addressed.
You should interview at Netradyne. From your comment you seem like a good fit.
What the AI is attempting to solve is a complex problem. I’m willing to give the drivers here the benefit of the doubt ;)
I’ve worked on smart interfaces for cars by the way. My experience not only as a driver but also as a developer in this space tells me that AI described in the article needs further development and your presumptions in this discussion are BS.
TFA even points out drivers making excuses for not wearing a seatbelt. If they won't even take personal responsibility for their own illegal actions, of course their opinions on more subtle things like following distances aren't reliable.
They did mention it: not every driver has the AI installed.
Also you’re assuming the bonuses are awarded like prizes to the top performers. So if everyone is penalised equally then the same people still get awards. My impression was those bonuses were awarded to anyone who passes specific milestones. So even if everyone is penalised equally it’s still unfair to everyone.
> TFA even points out drivers making excuses for not wearing a seatbelt. If they won't even take personal responsibility for their own illegal actions, of course their opinions on more subtle things like following distances aren't reliable.
Yes, that is a fair argument. I interpreted that in a more charitable way saying “they’re acknowledging some bad practices on their side so at least trying to meet the AI developers half way”. But I will admit your interpretation is just as, if not more so, plausible.
Even so, that doesn’t justify being ignored to by Amazon and the AI developers. Any new tech needs debugging in the field. And this is more so true with AI and complex problems like driving assistants than it is in most other software specialties.
That’s the part of the article that stinks the most and the original point I was making yesterday. I couldn’t give a rats ass who is to blame because the biggest fault lies with the company for not working with the users to ensure the software is performing correctly. And this is especially important when people’s income are being directly effected by the software too!
WTF.
This seems like the real-world analogy of a similar trend I've noticed in software development where, to "improve code quality", humans are replaced with stupid tools that give false positives all the time, and to placate their warnings and satisfy the metrics of the management imposing them, writing code becomes perverted into trying to figure out how to stop the tool from complaining instead of focusing on the actual problem being solved.