I worked on a discount calculating service with no prior knowledge.It was supposed to replicate an existing system, so we went through the oroginal specs and general litterature to get a sense of what we were supposed to make.

First day of test with a employee in the field, and it appears that they have specific calculation rules that are so obvious to them that it didn't warrant to be written. Welp. And a few iterations after we were still hitting blocks of "what ? didn't you know that in case of X you do Y?" due to us being outside of their field.

Real world systems are hard. I'd still blame Amazon for not having caught these issues very early in the process, that's on them. But bugs and gross calculation errors are par for the course.

It's not exactly rocket-science either - you basically involve domain experts in the development proces directly, and codify their knowledge in a form that they can understand (be that very explicit models, or focussed domain-specific languages).

I think there can be some slack in this. I've had great success when product managers and UX designers worked closely tightly with engineers but were the only ones talking to customers/users. As long as their focus is on deeply understanding the domain and user needs it can work really well. I think XP describes this as something like a standin customer.

In the second case: that is a massive smell and time to find a new job. Which from the comments regards turnover at Amazon seems to be the case :-)

> It's not exactly rocket-science either

I'd imagine that occasionally "rocket science" is exactly what it is!

But it’s more than that. To name a few: * How does one raise the thrust temperature as high as possible without melting the engine to maximise thrust? (The burning temperature IS already higher than the engine material melting point) * how does one reduce the redundancy (both system wise and material wise) to minimise weight while ensuring the rocket won’t fail?

I studied physics in undergrad and in classical dynamics we covered “rocket science” in these terms. The term is more or less a misnomer as science implies trying to understand how the universe works. Building rockets is engineering but studying their motion does rely on both the equation I gave above and also on fluid dynamics. From the point of view of “science” it is pretty simple. From the point of view I’d actually building a rocket the engineering is very hard. Since that time I’ve always had a bit of a chip on my shoulder about the “it’s not rocket science”. It also doesn’t help that both my parents studied aerospace engineering in undergrad and grad school and tried to impart a decent chunk of it on my as a kid (which I resisted very hard).

I worked on a ballistic missile tracking software that would calculate areas of uncertainty given coverage of various defense mechanisms.

It very much involved rocket scientists from JPL.

You are wrong in every measurable sense of the word.

1. If HN had a friend system/social graph, you'd have a pending request from me based on this comment alone.

2. Fantastic domain name in your profile.

Sounds like I lost the asshole measuring contest :)

Does that unbaffle your baffles (oh, yes, rocket scientists do have to deal with baffles! You left that out) or should I elaborate? Bless your heart, but it isn't exactly rocket science.

It is that rockets that can send payload into orbit are operating at the limits of what materials are capable of. You have extreme forces, pressures, temperatures, and some of the nastiest chemicals, all that while being as lightweight as possible. There is something called the Tsiolkovsky rocket equation. The equation is simple, but the implications are that space rockets are not like fireworks, and it is what makes rocket science rocket science.

     It takes sixty-five thousand errors before you are qualified to make a rocket.
     -Wernher von Braun

Even just orbital mechanics has a decent amount of complexity. Transfer between celestial bodies in general requires solving the three-body problem (or four for transfer between planets). And it's one thing to solve for just the trajectory of an object given some initial conditions, and another to figure out the correct initial conditions to give to get the trajectory you want, all the while staying within a fuel budget. Then you have to contend with not having instantaneous impulse in real life.

A simplification here is that you can get a decent approximation to the three/four-body problem with patched conics, which is where you assume that the gravity outside of a celestial body's 'sphere of influence' is zero, and within that SOI, all of the gravity comes from that body; in this way, you can treat it as a series of two body problems, where you 'patch' together the solutions (which are conic sections) for the orbital trajectories of the all the bodies involved. This is by no means a perfect approximation, though, and in practice I would expect that one would want to check a given solution found with patched conics with a more complete n-body simulation.

Even simpler than this, though, at least mathematically, is orbital rendezvous. Here, you only have to contend with the gravity of a single body. Yet it's very difficult to get the timing right, and the first couple attempts by the USSR and the US failed, and Buzz Aldrin even submitted a doctoral thesis based entirely around orbital rendezvous (two spacecraft meeting in Earth orbit):

> In its first human spaceflight program Vostok, the Soviet Union launched pairs of spacecraft from the same launch pad, one or two days apart (Vostok 3 and 4 in 1962, and Vostok 5 and 6 in 1963). In each case, the launch vehicles' guidance systems inserted the two craft into nearly identical orbits; however, this was not nearly precise enough to achieve rendezvous, as the Vostok lacked maneuvering thrusters to adjust its orbit to match that of its twin. The initial separation distances were in the range of 5 to 6.5 kilometers (3.1 to 4.0 mi), and slowly diverged to thousands of kilometers (over a thousand miles) over the course of the missions.[1][2]

> In 1963 Buzz Aldrin submitted his doctoral thesis titled, Line-Of-Sight Guidance Techniques For Manned Orbital Rendezvous.[3] As a NASA astronaut, Aldrin worked to "translate complex orbital mechanics into relatively simple flight plans for my colleagues."[4]

> First attempt failed

> The first attempt at rendezvous was made on June 3, 1965, when US astronaut Jim McDivitt tried to maneuver his Gemini 4 craft to meet its spent Titan II launch vehicle's upper stage. McDivitt was unable to get close enough to achieve station-keeping, due to depth-perception problems, and stage propellant venting which kept moving it around.[5] However, the Gemini 4 attempts at rendezvous were unsuccessful largely because NASA engineers had yet to learn the orbital mechanics involved in the process. Simply pointing the active vehicle's nose at the target and thrusting was unsuccessful. If the target is ahead in the orbit and the tracking vehicle increases speed, its altitude also increases, actually moving it away from the target. The higher altitude then increases orbital period due to Kepler's third law, putting the tracker not only above, but also behind the target. The proper technique requires changing the tracking vehicle's orbit to allow the rendezvous target to either catch up or be caught up with, and then at the correct moment changing to the same orbit as the target with no relative motion between the vehicles (for example, putting the tracker into a lower orbit, which has a shorter orbital period allowing it to catch up, then executing a Hohmann transfer back to the original orbital height).[6]

> As GPO engineer André Meyer later remarked, "There is a good explanation for what went wrong with rendezvous." The crew, like everyone else at MSC, "just didn't understand or reason out the orbital mechanics involved. As a result, we all got a whole lot smarter and really perfected rendezvous maneuvers, which Apollo now uses."

An extreme example would be if you're trying to introduce a logistics management system in a shop that was exclusively using pen and paper. You'll need someone with a foot in both world to even get to the point where they can look at your models and/or documentation and say "yes, it matches the business".

Or one of the participant in the project will need to become that person (ideally the client, we can dream).

But really, the software is there to solve a problem, so you end up shoulder surfing while the "expert" uses both systems simultaneously, and they're asking you questions and you're thinking about how you may rearrange the UI from their feedback. It's extremely iterative. The "domain expert" also becomes an expert with the software as they learn about all the things you implement.

Wut? The development team is capable of dealing with that just fine. It’s the business unit throwing requirements over the fence and hoping it’ll magically be OK that’s the problem.

In 2020 the German government lowered VAT in a reaction to covid from 19% to 16%.

On the first day Amazon had reduced thousands of products by exactly 19% and only increased by 16% after a few hours. AFAIK they did honor every sale during that time.

I can totally see that happening. Who develops systems with short term VAT changes in mind?

Aldi iirc just kept the old prices and simply gave a 3% discount at checkout

I get your point, but they probably gave a discount of 1-(1.16/1.19) =~ 2.5%

Probably also rounded down in favor for the customer but I can’t remember the details

I suppose it's the same codebase as any other country's amazon, and there's a different VAT rate in every country.

So it's unlikely their German prices would just take the same input as eg Dutch prices and changed the VAT around.

And apparently fixed book prices (printed on the back) were up to every single publisher to decide about

Then you also have different VAT rules depending on the product category. Food is 7% in Germany.

Lastly I also remember that Amazon gift cards were weird for a few hours. Something like you pay 48.50 and got 50€ worth of credit. I can’t remember if you could then use that credit to buy new gift cards.

Lots of ways where this becomes quite a complex problem real fast

Uber also sold UK journeys to UK customers but put the fare calculation taximeter in a server in another country to avoid the 20% VAT. I believe they actually won that case as in the EU, VAT is calculated by the seller using their country's laws regardless of the buyer's country's VAT rates and laws.

VAT law is harmonized across the EU, which country's rate is applicable is based on a rather complex (and ever changing) set of rules, but nowadays it's mostly the buyer's country.

But yes, I expect if Amazon accidentally sold me an item for 1% of its actual value due to a typo they will honor it. I expect if they send me the wrong item they'll make it right. As a consumer, I'm not worried about them fixing any errors in their favor and ignoring any errors in my favor.

I just don't expect them to treat their vendors the same way.

I worked for Amazon as well, and I'd say it is greed, not in that the behavior was purposely designed to be malicious, but in the way they don't properly fund developers to work through their tech depts, and spend proper time and effort to make sure all the systems work, have been thoroughly tested, have lots of test coverage, that the teams manages to keep subject matter expert around, etc.

Amazon doesn't because it's too expansive to have dev resources doing all that, so most projects get released using the bare minimum, and get maintained using the bare minimum.

Another element of this that cuts against "greed" as the motivator in my mind is that it isn't like anyone who would implement the details here would give a hoot if Amazon went plus or minus ten million dollars (or whatever the amount is). If you are the leaf node in the org chart making the exchange of money happen then you will get paid the same regardless of what happens. There may be some big boss guy somewhere who cares about that number, but he doesn't have the ability to actually do the cheating, he'd have to order someone else to do it. Again, maybe that's what happened, but it just seems unlikely to me as it would be so easy for the people receiving those orders to go "Hmmm, no, I don't think so."

is greed and:

"Let's save money creating the services we provide, almost certainly at the cost of our customers in order to be more profitable"

is not?

"in my mind is that it isn't like anyone who would implement the details here would give a hoot if Amazon went plus or minus ten million dollars"

in that case I'd argue it's a massive management failure on Amazon to have a bunch of people working for them that do not care about the consequences of their actions. To labor my previous metaphor: somehow the radiation machine manufacturer finds a way to get its employees to care that their products work as intended.

And while I agree it would be morally worse if they were intentionally bilking customers (be it content producers or listeners), sins-of-omission are very real as well. You're right that $10mio are nothing to Amazon, but $10k could easily be the difference between wild success of an independent audio book creator and their financial ruin.

First, there's 0 chance at promotion doing maintenance work, so L4 (jr) engineers will bounce after a year or so as they have a clock running and have to move up or out. L5 engineers will stagnate and they will at risk for URA as they are supposed to keep improving. Frankly the work isn't L6 worthy, an L6 maintaining this as the only thing they are doing won't last a quarter.

So these services that just 'do the job' get bounced around from team to team and perhaps when they get old and creaky enough, an L4 can promo themselves doing an upgrade/refactor. Until then, they just exist and are a bane to some team where they only produce off hour pages, even if a whole publisher segment depends on them to get paid.

...and then you go on to explain in business terms that only apply to the handful of companies that have cargo-culted similar dev team designs and incentives.

I started writing a comment here about how my current employer structures and incentivizes maintenance but I've deleted it and decided not to give it away that info for free.

Kind of a major factor though.

If Amazon cared about creators, they would have built an automated system that monitors royalties and ensures that no matter what changes occur, creators are being paid fairly, correctly. They'd have a small product manager team who are experts in royalties because it's their job to be.

But that doesn't benefit a customer, so none of that exists. Because only people paying Amazon money are worthy of such attention.

You can't make mistakes like this any more than you accidentally burn one of Bezos' house to the ground "accidentally" trying to get proper restitution ringing his doorbell and it all going wrong somehow. It's just not an accident in any meaningful interpretation of that word. It's a level of incompetence that we don't accept as any kind of defense.

https://cdn-images-1.medium.com/max/1600/1*lFhFi-5JsP-zhW7Al...

There is no excuse for this. None.

Try on the picture of the bullet holes in planes when mounting a defense of reckless and culpable driving occasioning property damage while making your case to the judge that he hasn't heard about all the times the accused did not crash their car despite driving in a woefully incompetent manner...

There is ZERO excuse for this. You can't "accidentally" rob people and claim incompetence while pointing to other times you were incompetent and yet didn't rob people. It actually goes the other way. If you were incompetent, knew it and didn't fix the issue you're in deeper trouble.

Pictures of planes don't change that there is no excuse for this. None. Really.

Fun story. So I'm sitting in court with the other people who are going to fight their ticket. I overhear a woman talking about her case. Remarkably similar to mine. Night, raining, car spun out, reckless driving ticket. Might even have been the exact same location.

If I had known this earlier, I could have compiled a record of all the other people who crashed in the same area under the same circumstances. I in fact absolutely would have used the argument with the judge "look at all the times I didn't crash", along with "look at all the times these other people didn't crash", building to a crescendo of "what are the odds all these people with otherwise clean driving records crashed in this same spot under these same conditions."

Luckily I didn't need to compile the traffic court history of that particular off ramp. I won the case anyway. Got about a third of the way through my stack of papers that I retrieved via FOIL request and online search before the judge interjected. "J: Counselor, are we beating a dead horse here? DA: I think so your honor. J: Mr. IIAOPSW you don't have to prove anything. I'm convinced the prosecution has failed to show sufficient evidence for their case. You're free to go. Me: Thank you kindly your honor."

Don't tell me what would or wouldn't work as a defense before a judge son. I've kicked the DA's ass before and I can do it again.

Your arguments are hyperbolic and you've engaged in a bait and switch. Hyperbolic, because Amazon miscalculating a few cents on a royalty payment isn't nearly on the level of wrongdoing as negligent vehicular manslaughter. Bait and switch because you started from the premise that "the fact that the error always works in Amazon's favor is proof of intentional wrongdoing". But when it was pointed out that you don't actually have evidence that the error always works in Amazon's favor, you switched to "no excuse for this. None. Ever." However, if we agreed to the standard that there's "no excuse for this ever", what's your excuse for all the people who stole from Amazon by accepting over payment for royalties? Why didn't they look at their payment slips, notice they were overpaid, and kindly notify customer service so they could return the money? Remember, in your own testimony you said "there is ZERO excuse for this. You can't accidentally rob people and claim incompetence while pointing to the other times you were incompetent and yet didn't rob people."

Your argument would be ripped to shreds by any competent lawyer.

Would it also be "We eventually figured out something was wrong and went through a very annoying process of learning about royalties and how they interplay with prices, discounts, and other factors and then set things right" if Amazon's mistake was cutting their own fair share in half rather than their business partner's?

For example, if you follow this line of thought you'd never investigate murderous cops simply because their cameras where turned off due to their incompetence rather than purposeful, malicious action.

- Albert Einstein

Not talking specifically about this case, although it may apply, we just read a twitter thread.

If you were really honest about the meaning of the quote you should say "never assume malice" because that's the actual statement here.

I have another saying though: “Sufficiently advanced incompetence is indistinguishable from malice”.

Everywhere I've worked where there were defined formulae (or formulae to be defined) this kind of thing was meticulously checked for correctness. No one should have shipped the service without knowing if the calculations were correct!

Meanwhile both sides of the interaction (Amazon and the publishers) are using words they recognize, but with different meanings behind them! And both sides are so used to seeing the words that _of course_ it's the right thing (until it's not!).

Of course the reality is probably more subtle than this, but spec documents don't save you from domain knowledge (or lack thereof) 100% of the time. Definitely helpful as a reference document, of course! And it eliminates a large class of discussions.

A big part of the problem with the situation I'm discussing here is that the people who were using the tool were using it incorrectly. The people using the tool (which made offers available to specific customers in certain conditions) did not understand the nuances of the different price values on offers and how they connected with royalties.

I forget the specifics, but imagine someone working with publishers to setup special offers and entering prices into our tool and they just populate two of the three different "Price" fields with the same price because they don't know the difference between "list price" and "my price" and they put the right value in "discount price" (or whatever the terms were). Totally unknown to the person entering the offer, royalties are (incorrectly) calculated based on one of those columns that they have now put a technically incorrect value into. Even if there was a perfectly clear spec, and there wasn't, and it had been implemented perfectly, the tool still would've failed because the people using it didn't really understand the system either.

Most companies now-a-days don't write spec up front. That's called Big Design up Front (BDUF) and it was seen in this industry that this was prone to failure which could cause the whole project to fail. That's because it's impossible to know everything up front and often during the project the requirements will change as the stakeholders learn of the consequences of their decisions.

In fact most companies don't do a "spec document" at all they design their system in small iterations called "sprints" typically 2-weeks and the "spec" is a ticket in a system like Jira (others are available). This process is part of Agile Project Management methods

> Everywhere I've worked where there were defined formulae (or formulae to be defined) this kind of thing was meticulously checked for correctness

There are some industries where they still do that kind of thing (BDUF) for example government work, defence, oil and gas. These are older industries that just haven't changed the way they approach projects. Amazon here is a bit a front runner when it comes to updating it's practices to be more leading edge so they likely adopted an Agile approach here learning about what was required as they go through the project.

As you point out, the agile manifesto eschews documentation for working software. This is not about having documentation, it's about defining "working" for the software. The ticket could not be written or handled without consultation with legal or BA teams on how the formulae are defined, similarly to how if you're implementing any kind of system where the formulae are critically important for correctness you need to consult back and forth with the expert stakeholders to double check.

I don't know any other way to do this that isn't negligent, not just by the developer, but by the reviewer, the stake holders, and ultimately the PM that signs off on accepting the task as "done" when it hasn't been checked for correctness.

The contract will most likely be done and dev are not normally involved and it'll be written after the process of the software being made. I don't think in 16 years I've ever gone through a legal contract and translated that into a technical requirement.

It's normally done the other way round that some requirement is translated into legalese. Reading legal documents is not a skill I would expect of any developer and I would be extremely wary of building software using this technique. Reading contracts requires many years of training that almost all developers do not have. Also note that laws are not universal and different jurisdictions will often allow different interpretations. An example of this is USA vs UK law, they are quite different and so if you are trying to interpret a USA contract as some from the UK you'll probably need a lawyer from the US on hand to help explain what they mean by certain terms. Without this training it can be very easy to misinterpret what is actually being said in a legal document. I'd also say that as a dev/po/pm/qa or basically anyone involved in the actual development it's not in my job description to be able to read legal documents so I'd be quite within my rights to refuse to do this work.

The point I made which you seem to be ignoring is that from the original Agile Manifesto:

"Working software over comprehensive documentation"

So you don't write a spec document up front but discover the requirements as you progress through the project. I suspect for Amazon this is exactly what they are doing there is no "spec document" and there was no contract but as they progressed through the project they just built a overly simplistic approach to the Royalty calc. Afterwards they got a contract written and the lawyer either wasn't told what the software actually done or just wrote what was the standard industry contract for Royalties.

Please Note: I'm not advocating for Agile vs any other technique but I was responding to the comment that you should just read a spec and code that directly into code that's just not the way you do things in an Agile environment.

> "Working software over comprehensive documentation"

Working includes correctly working. As soon as you interface with the legal world this means to actually calculate whatever your business agreed to and not something else just because it's easier, produces some random numbers, and exists without printing to stderr. That's not working software.

This typically can't be worked out by development teams themselves, for the reasons you mentioned. And that's exactly why some kind of spec (in whatever form necessary - probably not a document of hundreds of pages but also not a one sentence user story) is needed to capture that. That's what I mean by translating and I'm not proposing to leave that to some product manager.

Of course this can also be fixed by adapting your contracts (which can be an enormous task) but for that you also need some documentation on whatever the system is doing (just as developers can't read contracts, lawyers can't read code) and will discover that many stakeholders won't agree to that and demand changes.

This is all not to be confused with doing a big design of the technical architecture and all features beforehand.

2. I am no less uncomfortable with monopolistic publishers who exploit creators out of incompetence after becoming the only game in town and making any ability to walk away from negotiations with them a Hobson's choice at best than I am with ones where all of that is true except the harm is due to greed/malice instead

3. Corporations aren't people and even if you believe (as I don't) that they can meaningfully have intentions in the same sense that people do, those intentions do not matter in a context where the behavior is on the scale of an organization, and so separating incompetence from malice is a distraction in the first place

This is one of the reasons for the uptick of new and experienced authors ditching (or making plans to ditch) Amazon KU exclusivity and going wide.

> Basically, of the people who wrote the tool, tested the tool, and used the tool, none of them knew exactly how royalties should be calculated

I really don't see how this is a valid excuse. "How the royalties should be calculated" is a lawyer's domain, an engineer should never have had anything to do with the decision, and any engineer who was tasked with coding the calculation without proper guidance should have pushed back.

Sounds like easy profit from a captive audience.

I would not be surprised to learn that there are harsh or exploitative terms for audiobook publishers. In that sense I think Amazon would consider that serving the customer means securing better prices. I would not be surprised to learn that some element of the people responsible for getting this working correctly were not doing so in some way (either because they didn't understand, bugs in the system, or they were too busy doing other stuff). I would be really surprised if they were intentionally cheating Audible publishers out of relatively small sums of money though.

I don't like this saying because it generalises too much.

For example, if you follow this line of thought you'd never investigate murderous cops simply because their cameras where turned off due to their incompetence rather than purposeful, malicious action.

That's pretty naïve way to see the world, especially in this day and age.

Reading the quote literally i disagree. There is miles of difference between "adequately explained" and "possible"

> That's pretty naïve way to see the world, especially in this day and age.

I would actually argue the other way is naive. Assuming that all bad things that happen in the world are because of an evil person is what happens in movies and fairly tales. The real world is almost never that black and white.

I now live here and am astonished by the amount of adult, supposedly life-experienced people going into full shock that always comes with comments like: "I would never expect our government to do such thing!".

It's time to grow up. I bet you think billionaires are good people and real justice takes place in the courts of law. Also, rich people face the same consequences as the rest of us. No?

You would be better served not posting this kind of reply.

In the cop example, if everyone is turning off their camera because the interface is terrible and if you don't do it just right - that problem requires a very different solution than if they were intentionally turning it off. If you assume it must be malice the fixes will be wrong and not accomplish anything. That doesn't imply that malice is impossible, just that you shouldn't assume it without investigation.