News for hackers. Stuff that matters.
Re:Amazon Has to Disclose How Its Algorithms Judge Workers P... by spywaregorilla on Monday September 27, 2021 @08:39AM (#28669923)
It seems likely to me that the use of algorithmic quotas and whatever is not helpful. Fear of the unknown quota might be, which is why this law is good.

But basically... let's assume tasks come to the warehouse on some sort of random poisson distribution, each with a value, and a time they must be completed by. Everything must be completed.

If these parameters are highly differentiated, a really smart system might be able to prioritize tasks such that high value things get done quicker than lower value things. But this isn't something that the worker should be figuring out.

If the parameters aren't highly differentiated, all you really want to do is move stuff through the queue faster. There's just one master metric in the "perfect" system which is, "how long did it take you to move this item through the queue vs. the expected time to move it through the queue". But to be a FAIR system, you need to be able to parse out what portion of the time to move distribution is based on random chance vs. individual agency. Delays are likely to be long skewed right. That is, most items will pass through with minimal delays but some items will have big logistical problems that completely slow down the process to no fault of the worker.

Measuring this is likely to be incredibly difficult and mostly picking up on noise. It could likely catch people who are truly slacking off. But... so could a human manager with basic observational skills.

Re:Amazon Has to Disclose How Its Algorithms Judge Workers P... by Closi on Monday September 27, 2021 @10:37AM (#28671156)
> But to be a FAIR system, you need to be able to parse out what portion of the time to move distribution is based on random chance vs. individual agency.

Delays are actually accounted for in these systems - generally in multiple ways:

* In the case for instance that there is an out of stock, or you need to go to another picking location, the system re-calculates the required distance/time considering the out of stock and this is accounted for in your score.

* If there is, for instance, a mechanical issue or something stopping you from being productive, you move onto an 'indirect task' where your time is no longer tracked as productive time.

> It could likely catch people who are truly slacking off. But... so could a human manager with basic observational skills.

We are talking about observing staff in a 1 million sqft warehouse across 5 floors of mezzanine, in a maze of aisles, and also the manager and associates shift patterns could be different - it's just impossible. I was a department manager (not for Amazon) and it's just impossible to eyeball picker performance in any sort of fair way, you have to look at data.

Additionally, dependent on the laws in your region, disciplining/firing someone because 'a manager observed them being slow' likely isn't good enough if it comes to a tribunal.

I've got direct experience with how these performance management systems are configured and work, so if anyone has any questions let me know!

Re:Amazon Has to Disclose How Its Algorithms Judge Workers P... by spywaregorilla on Monday September 27, 2021 @10:59AM (#28671448)
I've never done warehouses, but I've done a lot of performance management systems in different contexts. My takeaway was that most of them just capture noise. Companies believed fiercely that their systems captured the best performers, but when you looked at the autocorrelation of scores year over year for the same people, there was nothing. That is to say, given you were a top performer in year 1, we can infer nothing about the likelihood of your being a top performer in year 2.

Maybe this is different? Hard to say. But if I were to guess, I would imagine the differences in individual performance that are truly due to employee agency are not that high.

> We are talking about observing staff in a 1 million sqft warehouse across 5 floors of mezzanine, in a maze of aisles, and also the manager and associates shift patterns could be different - it's just impossible. I was a department manager (not for Amazon) and it's just impossible to eyeball picker performance in any sort of fair way, you have to look at data.

And I tend to imagine (perhaps uninformed) that these systems don't catch any small details. Presuming these people do in fact have managers, the big picture assessment of whether they're totally slacking off vs. working normally is likely to be apparent without data. The amount of actual agency people have in a task like this, I'd imagine, is heavily skewed left. That is, very few people are noticeably better than the median worker. But many people just don't work at all. Absenteeism is a problem in a lot of jobs like these, including pre-COVID. A large part of it is just getting people to show up.

Re:Amazon Has to Disclose How Its Algorithms Judge Workers P... by Closi on Monday September 27, 2021 @11:22AM (#28671775)
> And I tend to imagine (perhaps uninformed) that these systems don't catch any small details.

The systems basically direct user work, so you have a pretty good level of detail, particularly as users will also often scan to tell the system when they are doing other stuff for instance breaks:

* Go to location X (User scans barcode) * Take 2 of item and confirm (User scans barcode on each of the two items) * Now go to location Y (User scans barcode) * Take 1 of item and confirm (User presses not-found) * Now go to location Z (User scans lunch break barcode) * Please scan lunch break finished when complete (User scans lunch break complete) * Now go to location Z (User scans location) * Take 1 of item and confirm (and so on)

Each prompt and response is recorded as a transaction in the WMS, and then the total time estimate is calculated afterwards based on those transactions.

> That is, very few people are noticeably better than the median worker. But many people just don't work at all

Absenteeism is a huge problem - you are absolutely right! Yeah, it can be easy to hide in a giant DC. Regardless of how good of a manager you are and how much you walk around, if an associate decided to spend their full shift hidden behind some pallets reading a book, it would be really hard to notice their absence without seeing that they picked nothing on the system (particularly when you have a team of 30!).

Re:Amazon Has to Disclose How Its Algorithms Judge Workers P... by spywaregorilla on Monday September 27, 2021 @11:34AM (#28671945)
Huh. That's a lot more low level than I would have imagined. So now I'm curious, what does the distribution of outcomes look like? How much better is the best worker? Do you feel confident that the tasks are accurately and fairly scoped?
Re:Amazon Has to Disclose How Its Algorithms Judge Workers P... by Closi on Monday September 27, 2021 @11:48AM (#28672153)
Distribution of outcome is probably somewhat skewed by performance management, but I would say the worst performers would be c30% less than the measured standard and the best performers c30% above.

Good note on the scoping of the tasks - it depends totally on the implementation! Ours was pretty good, but the difficulty was keeping it up to date as you have to change the walk routes and distances any time you move physical equipment on site, or change site rules/physical processes.

Re:Amazon Has to Disclose How Its Algorithms Judge Workers P... by spywaregorilla on Monday September 27, 2021 @01:17PM (#28673434)
That's interesting. How consistent are people's placements within that ~60% range? Do you vary comp by much on that spectrum? How often do you aggregate measurements of performance to assess people?

I've never gotten to work with an activity level system, just overall daily or larger outcomes.

Re:Amazon Has to Disclose How Its Algorithms Judge Workers P... by Closi on Monday September 27, 2021 @02:43PM (#28674394)
It’s been a while since I have been a department manager, but the calculation was live and performance stats would be reviewed daily, although formal performance management would only happen after someone has had consistently low performance below target across a number of weeks.

A lot of the conversations were bonus related too - “Hey, FYI you are scraping the bonus threshold so if you really smash it across the next 3 days you can pocket an extra $200 or so”.

Re:Amazon Has to Disclose How Its Algorithms Judge Workers P... by sixothree on Monday September 27, 2021 @10:41AM (#28671203)
> In the case for instance that there is an out of stock, or you need to go to another picking location, the system re-calculates the required distance.

There have been reports from workers to the contrary.

I don't mean to refute your arguments. I'm just suggesting workers don't (or haven't) believe this to be the case.

Re:Amazon Has to Disclose How Its Algorithms Judge Workers P... by Closi on Monday September 27, 2021 @10:47AM (#28671284)
I believe this is a misconception, just because we used to have the same mis-understanding with staff where I worked.

The misconception comes from being given a target-time which is based on a calculation assuming the task can be completed successfully with no delays, however afterwards the target-times are recalculated to account for the additional movement.

A common assumption is that performance is (Time it took / Initial target time) - especially because the users see the target time - however it is really (Time it took / Recalculated target time accounting for additional movements).

I'm not aware of Amazon's exact implementation - but all performance management suites I'm aware of will re-calculate afterwards based on 'actual' task rather than 'forecast' task.

Re:Amazon Has to Disclose How Its Algorithms Judge Workers P... by PoignardAzur on Monday September 27, 2021 @06:07PM (#28676630)
I have no experience whatsoever with warehouses, so this is just speculation, but I suspect a big problem of these systems is they don't degrade gracefully in new situations.

So maybe the system has gives you "bonus time" if an item is out of stock so you can fetch it at a different shelf... except for a special item that needs to be stored in special conditions. So if you need to fetch the item and it's out of stock, you need to wait five minutes for a colleague to fetch more, and the system classifies it as wasted time.

From what I've gleaned in articles and the like, it's the main problem with these systems: if they encounter a problem that was accounted for, things go smoothly. If they encounter a new problem, they default to assuming the employee / contractor is at fault.

Re:Amazon Has to Disclose How Its Algorithms Judge Workers P... by Closi on Tuesday September 28, 2021 @02:50AM (#28679996)
Eh, in my experience all situations are pretty much accounted for to be honest. The exceptions are pretty much all known, except for extreme circumstances like fires or injuries (but the system accounts for these too, as new indirect time).

For instance to take your example at this scale the system will always tell you to go do something rather than wait at the pickface for a replenishment (there are lots of options for how this situation can be handled, but the job of the WMS is to capture and handle these exceptions). Less refined warehouse processes might wait, but at the level of system sophistication that these tracking systems need this is likely already engineered out.

IMO in terms of being a fair measure of individual effort/performance - these systems are about as good as you can get in a warehouse and are pretty sophisticated and good at calculating the exceptions. If people disagree with using them as a management tool that’s fair enough, but IMO the calculation is much more thorough and fair than in other places where people get performance managed.

Re:Amazon Has to Disclose How Its Algorithms Judge Workers P... by PoignardAzur on Tuesday September 28, 2021 @12:47PM (#28684828)
I was thinking about a recent article about drivers who said they had to cross highways on foot because the app thought it was a road and would penalize them if they drove the long way around.

I dunno. Maybe the Amazon's warehouse system is more polished, or maybe the complaints from the drivers were anecdotes not representative of the industry. I haven't really seen recent articles/studies about the working conditions in Amazon facilities, so I don't know how up to date the old complaints are.

Re:Amazon Has to Disclose How Its Algorithms Judge Workers P... by Closi on Tuesday September 28, 2021 @01:43PM (#28685397)
The only thing I would say is that the accuracy of the performance management solution isn’t necessarily indicative of the working conditions

You can have a really poor performance management calculation and great working conditions and visa-versa.

Re:Amazon Has to Disclose How Its Algorithms Judge Workers P... by lozenge on Monday September 27, 2021 @10:50AM (#28671327)
Maybe the system just accepts you'll miss the target 2% of the time for reasons like that. However, the workers don't see that and get stressed (by design). Also, if the actual frequency of these issues is 1.9%, some workers will have bad luck and get penalised (also accepted as part of the design).
Re:Amazon Has to Disclose How Its Algorithms Judge Workers P... by raxxorrax on Monday September 27, 2021 @09:03AM (#28670150)
Before defining technicalities, which are of course even more complicated, we should ask ourselves if we want to work this way. It would get a clear 'no' from me. Also the worker doesn't profit in any way if he works faster. Sure, there is very like some form of bonus, but in the end throughput doesn't matter to him. So why the haste?
Re:Amazon Has to Disclose How Its Algorithms Judge Workers P... by spywaregorilla on Monday September 27, 2021 @09:52AM (#28670630)
Your wording to me implies you assume the bonus is trivial. The whole point of incentive structures is to make the desires of the business match the rewards of the employee.

> It would get a clear 'no' from me

Seems kind of entitled to assume this view generalizes. Sure, I too have a cushy job and would prefer more time / less stress instead of more money. But that's a privileged place to be. It's a view derived from circumstances, not wisdom, even if a lack of wisdom may permit you to fail to realize this.

Re:Amazon Has to Disclose How Its Algorithms Judge Workers P... by raxxorrax on Monday September 27, 2021 @10:14AM (#28670890)
> But that's a privileged place to be.

All too true but I think sharing is caring.

I am aware of what a worker needs to achieve to be competitive in a first world nation. This has nothing to do with that, on the contrary, it just worsens working conditions for no discernible reason and surveillance software is unproductive baggage, but most importantly you imply that this productivity tuning would exempt the wage of workers being subject to market forces, which is quite a huge assumption and unproven I would say.

The success of Amazon certainly didn't rely on that.

Re:Amazon Has to Disclose How Its Algorithms Judge Workers P... by kahrl on Monday September 27, 2021 @10:19AM (#28670936)
Yes, I'm sure the sweat shop workers enjoy working in a sweat shop. We're privileged, so who even knows if the poors know about holidays. Best to leave them to their hellhole.
Re:Amazon Has to Disclose How Its Algorithms Judge Workers P... by jjk166 on Monday September 27, 2021 @11:46AM (#28672132)
If they chose to work in the sweat shop and are free to quit at any time but choose not to, then they probably do enjoy working in the sweat shop, or at least prefer it to the other options they have reasonable access to. Without understanding the nuances of their current situation, it is impossible to say what changes will actually improve the situation, and it is easily possible that well intentioned changes can make things substantially worse.
Re:Amazon Has to Disclose How Its Algorithms Judge Workers P... by FormerBandmate on Monday September 27, 2021 @12:46PM (#28672994)
They said the exact same things about Triangle Shirtwaist era factory workers and their lives demonstrably improved massively after unionization
Re:Amazon Has to Disclose How Its Algorithms Judge Workers P... by spywaregorilla on Monday September 27, 2021 @11:04AM (#28671508)
The working conditions of the poor and their low wages aren't especially relevant as to general idea of aligning compensation incentives to outcomes. It's fully possible to believe working conditions should be improved, base pay should be improved, AND that incentives for performance are reasonable.
Re:Amazon Has to Disclose How Its Algorithms Judge Workers P... by FormerBandmate on Monday September 27, 2021 @12:45PM (#28672980)
The fact that his cushy job is in large part brought about by circumstances instead of wisdom just accentuates the need to improve the lives of people who don't have those circumstances. Amazon warehouse workers can't stand up to this but we can
Re:Amazon Has to Disclose How Its Algorithms Judge Workers P... by jasode on Monday September 27, 2021 @09:38AM (#28670490)
>Also the worker doesn't profit in any way if he works faster. [...] So why the haste?

Yes, it's true that they're not paid piecemeal per box so working "faster" seems pointless but some workers still hustle because they want to keep their jobs.

A worker having a defiant attitude of "I don't get paid to work any faster" is fine -- up until the point they are replaced by another worker who is faster -- which means they're unemployed and make $0.

[dead]
Re:Amazon Has to Disclose How Its Algorithms Judge Workers P... by dang on Monday September 27, 2021 @04:15PM (#28675459)
Please don't do this here.
Re:Amazon Has to Disclose How Its Algorithms Judge Workers P... by kodah on Monday September 27, 2021 @10:10AM (#28670837)
> Before defining technicalities, which are of course even more complicated, we should ask ourselves if we want to work this way. It would get a clear 'no' from me

Oh, we don't already? There's a graph on my companies VCS that displays my own velocity, along with a subgraph that differentiates reviews, code writing, etc...

Velocity is literally the speed at which we move items through the queue.

We were the guinea pigs, they just took the illusion of choice (priority) away from Amazonians.

Re:Amazon Has to Disclose How Its Algorithms Judge Workers P... by consp on Monday September 27, 2021 @10:24AM (#28671001)
> Velocity is literally the speed at which we move items through the queue.

Like adding lines of useless code when measured by the LOC you will surely start to add useless items to the queue if you have any control over that?

Re:Amazon Has to Disclose How Its Algorithms Judge Workers P... by spywaregorilla on Monday September 27, 2021 @11:31AM (#28671908)
Velocity for knowledge work is probably a lot more difficult to measure, but also more.. real? Like the difference in capabilities of devs is likely to be much more varied than the varaiation among warehouse workers.
Re:Amazon Has to Disclose How Its Algorithms Judge Workers P... by gentleman11 on Monday September 27, 2021 @01:46PM (#28673779)
I worked in factories and warehouses once. I don’t give a shit whether you eggheads think warehouse workers are lazy. Stop micromanaging and tracking people and subjecting them to mistreatment like this. The engineering world is morally sick for being so keen to support this. Factory and warehouse workers are mistreated enough as it is
Re:Amazon Has to Disclose How Its Algorithms Judge Workers P... by withinboredom on Monday September 27, 2021 @01:33PM (#28673618)
I’ve written picker software and did something like this. No one (including the managers) could see their individual score, but each picking team could see their team score in real time. Many of them eventually worked out how to game it, which was entertaining but still made it hard to slack off unless the team worked together to cover them (I worked closely with these picker teams to implement the software). The owner was thrilled that metrics looked great and productivity increased. The teams were happy they had immediate feedback on their score and “picks per minute” to see how they were doing. The managers sent people home once they hit their target — with pay as though they were there the whole day.

It’s a shame that CDs and DVDs went out of style and they went out of business. It was a fun place to work.

Re:Amazon Has to Disclose How Its Algorithms Judge Workers P... by s1artibartfast on Monday September 27, 2021 @03:29PM (#28674997)
Time an volume take care of nearly all of those these concerns.

Instead of looking at a per package performance, you simply look at the average over time.

If the average target is move 100 packages per hour, you are looking at the average of 16,000 samples for one month of work.