News for hackers. Stuff that matters.
Re:Amazon Has to Disclose How Its Algorithms Judge Workers P... by Closi on Monday September 27, 2021 @10:37AM (#28671156)
> But to be a FAIR system, you need to be able to parse out what portion of the time to move distribution is based on random chance vs. individual agency.

Delays are actually accounted for in these systems - generally in multiple ways:

* In the case for instance that there is an out of stock, or you need to go to another picking location, the system re-calculates the required distance/time considering the out of stock and this is accounted for in your score.

* If there is, for instance, a mechanical issue or something stopping you from being productive, you move onto an 'indirect task' where your time is no longer tracked as productive time.

> It could likely catch people who are truly slacking off. But... so could a human manager with basic observational skills.

We are talking about observing staff in a 1 million sqft warehouse across 5 floors of mezzanine, in a maze of aisles, and also the manager and associates shift patterns could be different - it's just impossible. I was a department manager (not for Amazon) and it's just impossible to eyeball picker performance in any sort of fair way, you have to look at data.

Additionally, dependent on the laws in your region, disciplining/firing someone because 'a manager observed them being slow' likely isn't good enough if it comes to a tribunal.

I've got direct experience with how these performance management systems are configured and work, so if anyone has any questions let me know!

Re:Amazon Has to Disclose How Its Algorithms Judge Workers P... by spywaregorilla on Monday September 27, 2021 @10:59AM (#28671448)
I've never done warehouses, but I've done a lot of performance management systems in different contexts. My takeaway was that most of them just capture noise. Companies believed fiercely that their systems captured the best performers, but when you looked at the autocorrelation of scores year over year for the same people, there was nothing. That is to say, given you were a top performer in year 1, we can infer nothing about the likelihood of your being a top performer in year 2.

Maybe this is different? Hard to say. But if I were to guess, I would imagine the differences in individual performance that are truly due to employee agency are not that high.

> We are talking about observing staff in a 1 million sqft warehouse across 5 floors of mezzanine, in a maze of aisles, and also the manager and associates shift patterns could be different - it's just impossible. I was a department manager (not for Amazon) and it's just impossible to eyeball picker performance in any sort of fair way, you have to look at data.

And I tend to imagine (perhaps uninformed) that these systems don't catch any small details. Presuming these people do in fact have managers, the big picture assessment of whether they're totally slacking off vs. working normally is likely to be apparent without data. The amount of actual agency people have in a task like this, I'd imagine, is heavily skewed left. That is, very few people are noticeably better than the median worker. But many people just don't work at all. Absenteeism is a problem in a lot of jobs like these, including pre-COVID. A large part of it is just getting people to show up.

Re:Amazon Has to Disclose How Its Algorithms Judge Workers P... by Closi on Monday September 27, 2021 @11:22AM (#28671775)
> And I tend to imagine (perhaps uninformed) that these systems don't catch any small details.

The systems basically direct user work, so you have a pretty good level of detail, particularly as users will also often scan to tell the system when they are doing other stuff for instance breaks:

* Go to location X (User scans barcode) * Take 2 of item and confirm (User scans barcode on each of the two items) * Now go to location Y (User scans barcode) * Take 1 of item and confirm (User presses not-found) * Now go to location Z (User scans lunch break barcode) * Please scan lunch break finished when complete (User scans lunch break complete) * Now go to location Z (User scans location) * Take 1 of item and confirm (and so on)

Each prompt and response is recorded as a transaction in the WMS, and then the total time estimate is calculated afterwards based on those transactions.

> That is, very few people are noticeably better than the median worker. But many people just don't work at all

Absenteeism is a huge problem - you are absolutely right! Yeah, it can be easy to hide in a giant DC. Regardless of how good of a manager you are and how much you walk around, if an associate decided to spend their full shift hidden behind some pallets reading a book, it would be really hard to notice their absence without seeing that they picked nothing on the system (particularly when you have a team of 30!).

Re:Amazon Has to Disclose How Its Algorithms Judge Workers P... by spywaregorilla on Monday September 27, 2021 @11:34AM (#28671945)
Huh. That's a lot more low level than I would have imagined. So now I'm curious, what does the distribution of outcomes look like? How much better is the best worker? Do you feel confident that the tasks are accurately and fairly scoped?
Re:Amazon Has to Disclose How Its Algorithms Judge Workers P... by Closi on Monday September 27, 2021 @11:48AM (#28672153)
Distribution of outcome is probably somewhat skewed by performance management, but I would say the worst performers would be c30% less than the measured standard and the best performers c30% above.

Good note on the scoping of the tasks - it depends totally on the implementation! Ours was pretty good, but the difficulty was keeping it up to date as you have to change the walk routes and distances any time you move physical equipment on site, or change site rules/physical processes.

Re:Amazon Has to Disclose How Its Algorithms Judge Workers P... by spywaregorilla on Monday September 27, 2021 @01:17PM (#28673434)
That's interesting. How consistent are people's placements within that ~60% range? Do you vary comp by much on that spectrum? How often do you aggregate measurements of performance to assess people?

I've never gotten to work with an activity level system, just overall daily or larger outcomes.

Re:Amazon Has to Disclose How Its Algorithms Judge Workers P... by Closi on Monday September 27, 2021 @02:43PM (#28674394)
It’s been a while since I have been a department manager, but the calculation was live and performance stats would be reviewed daily, although formal performance management would only happen after someone has had consistently low performance below target across a number of weeks.

A lot of the conversations were bonus related too - “Hey, FYI you are scraping the bonus threshold so if you really smash it across the next 3 days you can pocket an extra $200 or so”.

Re:Amazon Has to Disclose How Its Algorithms Judge Workers P... by sixothree on Monday September 27, 2021 @10:41AM (#28671203)
> In the case for instance that there is an out of stock, or you need to go to another picking location, the system re-calculates the required distance.

There have been reports from workers to the contrary.

I don't mean to refute your arguments. I'm just suggesting workers don't (or haven't) believe this to be the case.

Re:Amazon Has to Disclose How Its Algorithms Judge Workers P... by Closi on Monday September 27, 2021 @10:47AM (#28671284)
I believe this is a misconception, just because we used to have the same mis-understanding with staff where I worked.

The misconception comes from being given a target-time which is based on a calculation assuming the task can be completed successfully with no delays, however afterwards the target-times are recalculated to account for the additional movement.

A common assumption is that performance is (Time it took / Initial target time) - especially because the users see the target time - however it is really (Time it took / Recalculated target time accounting for additional movements).

I'm not aware of Amazon's exact implementation - but all performance management suites I'm aware of will re-calculate afterwards based on 'actual' task rather than 'forecast' task.

Re:Amazon Has to Disclose How Its Algorithms Judge Workers P... by PoignardAzur on Monday September 27, 2021 @06:07PM (#28676630)
I have no experience whatsoever with warehouses, so this is just speculation, but I suspect a big problem of these systems is they don't degrade gracefully in new situations.

So maybe the system has gives you "bonus time" if an item is out of stock so you can fetch it at a different shelf... except for a special item that needs to be stored in special conditions. So if you need to fetch the item and it's out of stock, you need to wait five minutes for a colleague to fetch more, and the system classifies it as wasted time.

From what I've gleaned in articles and the like, it's the main problem with these systems: if they encounter a problem that was accounted for, things go smoothly. If they encounter a new problem, they default to assuming the employee / contractor is at fault.

Re:Amazon Has to Disclose How Its Algorithms Judge Workers P... by Closi on Tuesday September 28, 2021 @02:50AM (#28679996)
Eh, in my experience all situations are pretty much accounted for to be honest. The exceptions are pretty much all known, except for extreme circumstances like fires or injuries (but the system accounts for these too, as new indirect time).

For instance to take your example at this scale the system will always tell you to go do something rather than wait at the pickface for a replenishment (there are lots of options for how this situation can be handled, but the job of the WMS is to capture and handle these exceptions). Less refined warehouse processes might wait, but at the level of system sophistication that these tracking systems need this is likely already engineered out.

IMO in terms of being a fair measure of individual effort/performance - these systems are about as good as you can get in a warehouse and are pretty sophisticated and good at calculating the exceptions. If people disagree with using them as a management tool that’s fair enough, but IMO the calculation is much more thorough and fair than in other places where people get performance managed.

Re:Amazon Has to Disclose How Its Algorithms Judge Workers P... by PoignardAzur on Tuesday September 28, 2021 @12:47PM (#28684828)
I was thinking about a recent article about drivers who said they had to cross highways on foot because the app thought it was a road and would penalize them if they drove the long way around.

I dunno. Maybe the Amazon's warehouse system is more polished, or maybe the complaints from the drivers were anecdotes not representative of the industry. I haven't really seen recent articles/studies about the working conditions in Amazon facilities, so I don't know how up to date the old complaints are.

Re:Amazon Has to Disclose How Its Algorithms Judge Workers P... by Closi on Tuesday September 28, 2021 @01:43PM (#28685397)
The only thing I would say is that the accuracy of the performance management solution isn’t necessarily indicative of the working conditions

You can have a really poor performance management calculation and great working conditions and visa-versa.

Re:Amazon Has to Disclose How Its Algorithms Judge Workers P... by lozenge on Monday September 27, 2021 @10:50AM (#28671327)
Maybe the system just accepts you'll miss the target 2% of the time for reasons like that. However, the workers don't see that and get stressed (by design). Also, if the actual frequency of these issues is 1.9%, some workers will have bad luck and get penalised (also accepted as part of the design).