So the argument is that former members of the Clinton campaign -- highly competent individuals familiar with elections -- put together a company comprised of 'amateur app coders' paid by 'clueless amateur voting registrars'?

The founder, Gerard Niemira and Krista Davis, have a great pedigree. Niemira was formerly at kiva.org. Davis was on the Clinton campaign tech team. These are not incompetent individuals.

Now, you may be right that this is all incompetence. However, your contention that these are 'amateur coders', managed by 'amateur product designers', getting paid by clueless 'amateur voting registrars', is completely ridiculous. The leadership team is clearly professional and tech-savvy..

"The leadership team is clearly professional and tech-savvy.."

And yet they failed miserably.

I'm so sick of this attitude. Leadership never takes the fall, it is always the underlings who get the blame.

Their important high visibility project failed. The leadership failed. When you fail, that means you did not perform in a competent way.

I think you mistook me. I am clearly blaming the leadership team, but not for incompetence. For actual malice. They clearly knew what they should do, and decided not to. That should be something they are punished for, not their employees.

If you're claiming malice, what's the theory of malice? Who benefits from this?

I don't have a theory... I'm just noting that professional neglect at this level is malice, whether due to laziness or actual ill intent.

Like, if I sold someone a piece of software that I knew hadn't been tested and told them it worked great, that is malicious behavior, whether my motive was to make a quick and easy buck, to eat ice cream instead of doing actual work, or to ensure that my customer failed. The behavior is malicious independent of motivation.

Malice refers to intent, the idea of "malice... without ill intent" doesn't make sense. That's why Hanlon's Razor is worded as it is. By your definition, any shoddy work could be considered malicious.

I get your point though, I think we're just arguing semantics.

"negligence" is probably the word that's being sought here.

>Who benefits from this?

Probably one of the candidates who had a decent chance of winning the Iowa primary, but was not the most likely one to be picked. I don't like throwing accusations without concrete support for them, so I will just say that the whole involvement of Buttigieg's with the app, as well as his announcement of victory last night (way before votes got actually counted), seems at least suspicious.

Though it is just as likely as simple incompetence, probably even less, so i am leaning more towards incompetence, but I cannot fully dismiss the alternative in the light of all the recent DNC events.

Remember, however, this app was commissioned by the Iowa Democratic Party (IDP), not the DNC.

I mean they are clearly not as professional and tech-savvy as their "pedigree" would have us believe or they wouldn't have made so many amateur mistakes. But they clearly are good at convincing non-tech folks in a position to hire them that they are a lot more competent than they really are, so kudos on that?

Those of us who work in software understand that bugs happen even when you follow best practices. But if you don't even try to follow best practices for mission critical software, then roll out in production for the first time at a marquee event with millions watching, that's on you.

> Those of us who work in software understand that bugs happen even when you follow best practices. But if you don't even try to follow best practices for mission critical software, then roll out in production for the first time at a marquee event with millions watching, that's on you.

Exactly. And this kind of incompetence may be expected from someone who is actually an amateur good at selling themselves. However, the leadership is clearly not comprised of amateurs. Thus, this level of incompetence is professional, almost criminal, neglect.

Maybe. But I think it's more likely just another lesson in how far someone who is charismatic, well-connected, and self-promoting can advance, and how much credibility they can amass, while still essentially being a complete fraud who has none of the skills or experience they claim to have.

Some folks take "fake it 'til you make it" to an art form, and they keep right on faking it even after they've made it.

> the Clinton campaign -- highly competent individuals familiar with elections

I'm going to disagree with this premise.

I get the joke, but c’mon...

> So the argument is that former members of the Clinton campaign -- highly competent individuals familiar with elections -- put together a company comprised of 'amateur app coders' paid by 'clueless amateur voting registrars'?

Yes, that is exactly the argument. Just because you happened to be a member of some noteworthy organizations does not mean you are competent. People drift from golden parachute to golden parachute all the time, especially at an executive level where you get to benefit from people assuming "well he's an executive so obviously he knows what he's doing" and blame your underlings for your failures.

Highly competent is not a word heard that often about the failed 2016 Clinton campaign, especially not from people who realize that she lost under the exact same rules that was in place when Obama won.

What the involved people are is highly connected which is rarely the same as competent as demonstrated by the antics of the current crop of in office politicians, and it's a huge part of the reason why crazy outsiders like Trump have a base at all.

The stupid idea that people who's only achievement is climbing the rungs of an failed organization is super cometent and whose ultimate failure could only have been due to nefarious actions of supernaturally competent foreign enemies is silly and seen as such by an huge sway of the actual public as the kind of nonsense you would expect from North Korea or Russia.

> Highly competent is not a word heard that often about the failed 2016 Clinton campaign, especially not from people who realize that she lost under the exact same rules that was in place when Obama won.

One can be a competent presidential candidate and still lose. That is the point of democracy. Clinton's campaign could be competent while still more people in the right places want Trump to be president.

Yep but she lost which means that simply being an high ranking member of the 2016 Clinton's campaign cannot be proof of competency on it's own without the rational analysis of the campaign that's being successfully blocked by the emergence of a bunch of conspiracy theories recycled from the 1960ies as mainstream politics by the democratic party of America.

Competent people fail all the time for reasons outside of their control but it's also true that incompetent people can rise to the top of an organization based on luck, connections or by being associated with someone else. So you need something more detailed then having been a part of an failed bureaucracy to justify claims of extreme competency in running software projects.

Just because they were competent at something somewhat related, doesn't mean they were competent at this particular thing, or that they gave it the attention it needed.

A real-time vote transmission system that requires data integrity and validation is not quite the same as a consumer website, or an email marketing system.

> A real-time vote transmission system that requires data integrity and validation is not quite the same as a consumer website, or an email marketing system.

That's not at all what this was or sounds like. This was a data collection app, from everything I've read. It is meant to replace a phone call.

Ok. So, this is just a data collection app? Trying to have casual inexperienced users accept/trust a developer certificate? Both PINs and 2FA? This is beyond embarrassing.

"...Two people who work for Acronym, speaking on the condition of anonymity because they did not want to risk their jobs, acknowledged that the app had problems. It was so rushed, they said, that there was no time to get it approved by the Apple store. Had it been, it might have proved far easier for users to install.

Instead, the app had to be downloaded by bypassing a phone’s security settings, a complicated process for anyone unfamiliar with the intricacies of mobile operating systems, and especially hard for many of the older, less tech-savvy caucus chairs in Iowa.

The app also had to be installed using two-factor authentication and PIN passcodes. The information was included on worksheets given to volunteers at the Iowa precincts tallying the votes, but it added another layer of complication that appeared to hinder people..."

https://www.nytimes.com/2020/02/04/us/politics/iowa-caucus-s...

My assumption when seeing that is that those individuals you mentioned signed on to a company for profit and to use their connections thusly. Does not mean they were involved at all in the day to day.

> highly competent individuals familiar with elections

Is this a joke?

There's nothing suggesting competence here and everything suggesting casual nepotism.

You may be right, but this will always be the case with electronic voting. A normal child could understand and verify the chain of a non-electronic vote. A very technical adult could not do the same for an electronic vote.

This isn't electronic voting. This isn't even paper voting (though they also use paper backups). This vote is primarily done by people standing in a particular corner.

This is related to the TRANSMISSION of the vote count to the Iowa Dem Party, not the count itself. There was a separate issue from app issues where the different rounds of voting weren't matching up in some precincts. Paper backups of the vote still exist.

They're hiring!

https://shadowinc.io/jobs

"Client Success Representative" sounds like a fun job.

A lack of transparency combined with a reputation (earned or not) for guiding primaries to their desired outcome, it's not hard to see why some people might think this is an example of the DNC putting their finger on the scale in some way. There's no evidence either way, which means confidently claiming incompetence is fairly baseless, as well.

If we want to talk what things are generally true, sure, I'd put money on incompetence without any context, but there IS context to this.

There is no proof of either yet. What makes everyone think this is suspicious is there is proof of election meddling from overseas in the past. So there is proof elections can be modified and I doubt the IT skills groups like these to realize or adequately stop tampering. Without evidence everyone is assuming everything.

Not only that, the party itself has a history of meddling against this election's likely winner.

Did Russia actually meddle with (read: modify) the election? Or did they just buy ads? I thought it was only the latter.

I read there was unsuccessful attempts and not only by Russia, but if foreigners are willing to do it I absolutely would not put it past interested Americans to try either. The ads were meddling but in a different sense, just like the Russians meddled in Brexit with ads.

There is no evidence of changing votes by any other nation. Ad buying and fake advocacy accounts have been shown to exist for multiple nations. Also, multiple countries have given money to organizations in the US for non-election work that seems to end up on elections. The DC advocacy culture is a wonder to behold from money both domestic and imported.

Most people consider the illegal hacking and distribution of emails as "meddling" with the election and believe it had a greater impact on how people voted than the ad buys did.

If you're just arguing that it's irrelevant because vote totals weren't hacked, fine. That's a dispute over word choice of what it means to "meddle." Russia also hacked into and downloaded data from voting registration systems, which should cause us to re-evaluate our voting systems, but again, that didn't impact vote computations or anything like that.

I’m gonna break a rule, but, honestly, HN confuses me. I asked a genuine question and I’m being downvoted for it because (I assume) some people mistake it as trolling.

Anyways, I never said they didn’t do anything. I said, “here’s what I thought: x. Was I wrong?” My use of “(read: modify)” was to clarify that when I said meddle in that instance, I meant actual modification; I was referencing what the parent comment had said.

I didn't downvote you (I promise!). And I hope I answered your question.

For those who did, I think what happens is that it's hard to tell which people are and are not operating in good faith and sometimes people mistake one for the other. But it sounds like you genuinely weren't sure whether Russia had actually hacked into the voting machines. Only into the Podesta, the DNC, and into voter registration systems. There's no evidence to suggest they changed tabulations (and by that I mean they almost certainly didn't), so that's why people haven't made that claim.

Thank you. I like HN because the community is a lot nicer than others, but it’s frustrating being drive-by downvoted when you just asked a genuine question. It reminds me of other online communities where people just downvote because you said something they disagree with.

The only thing I want to add to this discussion is that even though there's no evidence any nation has ever hacked into our voting machines, we still need a paper trail. It could absolutely happen.

We have over 3000 different voting systems - one for each county. Some counties are staffed better than others. They use different types of machines. In some ways, that's a strength. But in others, it's a weakness. There should really be much more discussion of the issue nationally and what needs to be done.

Oh for sure. The issues with all digital elections are well known (in the tech community), but the governments ignore us. I think Russia was a wake up call for a lot of people that hacking our results is a real possibility, but it sadly wasn’t enough.

Thankfully(?), Georgia (the state) is in the middle of some lawsuits (somewhat) regarding their lack of a paper trail in their elections.

Agree that this is certainly the greatest example of Hanlon's Razor in recent memory. This includes their naming choice.

As anyone in this industry knows, this is very common even with larger, more established companies.

> * Sufficiently advanced incompetence is indistinguishable from malice*

I like what you did there.

I agree 99% with your observations.

As you pointed out, this is not just "normal human incompetence", these are layers and layers of incompetence.

I am a big Bernie Sanders supporter and am always eager to put on my tinfoil hat.

But I have to agree with you. The owner of Shadow's parent company has close ties to the Hillary campaign, a lot of the Shadow employees worked for the Obama campaign, Hillary campaign, and for the DNC.

The scandal here is a bunch of noobs got an extremely important contract through cronyism, and they bungled it horribly.

I'd surprised you're not more critical of Clinton and her cronies as a Bernie supporter given they abused superdelegates to award Hillary the nomination over Bernie in 2016

Oh believe me, I have plenty of venom for the DNC.

As far as I am concerned they stole the nomination. The DNC was biased against Bernie Sanders and they ratf*ed him every chance they had. Bernie had staff at most of the caucus locations and they kept their own independent tally to ensure there was no funny business.

The bias was so obvious. It damaged Hillary's campaign. I was so disgusted I was tempted to vote for Trump ( I didn't! ), and I am pretty sure a LOT of people tuned out after the circus was over, and didn't bother to vote in the election. It was so obvious the head of the DNC had to step down in shame.

And not to say the shadow bungle was not damaging. It robbed the candidate who won from being able to deliver a victory speech. Instead... everyone delivered a victory speech just in case xD

Yeah I have a similar story. I was a Bernie supporter but when all that went down I started paying attention to Trump and found out I actually agreed with much of what he was running on

The reports I've seen have the app budget as $60-70k, which seems super low budget for how important it was. If this is cronyism (and not just low bidding), they didn't do a very good job of it.

https://www.thestreet.com/politics/acronym-shadow-iowa-caucu...

I saw the same thing and I had the same thought: Wow that is low.

My suspicion: The company is all designers and product managers, and they shipped it overseas to the lowest bidder. That is unconfirmed, though.

It's funny because I'd bet most of the people that know how to program around here could write a similar app (and likely a betterone at that) in a matter of hours if they were so inclined.

The difference being that you didn't win the contract.

That's pretty immaterial. I was just pointing towards the fact that the simplistic nature of the application in the first place only serves to further highlight the incompetence required for such a catastrophic failure.

Got it. My comment was saying, wryly, the amount of thought you put in just now was probably the maximal level of project design and thinking they did.

Or maybe the situation was treated with seriousness, hence the outcome...

How would you test this type of thing at scale?

Disclaimer: not an app developer!

It takes a very dedicated plan to sufficiently test apps at scale when we're talking the scale of 300 million or billions... In this case the scale was 1700 (maybe off by an order of magnitude) when your scale is on the thousands or tens of thousands you can't do outright stupid stuff... but you can do reasonably dumb stuff and be fine - running a test with a hundred or so users should've clearly demonstrated these issues and there are companies out there that excel at resourcing a few minutes of a few hundred folks' time for just this kind of load testing.

That said, this was a pretty pathetic failure to scale, and I don't use harsh language like that lightly. I assume we're going to find out someone's brother or nephew runs the company because, for whatever reason, politics loves nepotism.

This is a small scale app by any sensible standards and probably not that complex compared to what's being routinely deployed by 2-3 man development teams in the world of business. i routinely handle spreadsheets with larger datasets and more complexity then whats involved here.

There is books written on how to test an new system/feature but in general you have 3 stages:

The first is an automated internal integrity test(sometimes called unit test) where you throw mock data at the systems independent component, in order to test for known classes of edge cases and race conditions.

The second step is implementation where the ops team tries to yank network and hardware resources away form underneath the app until something breaks in order to see if it fails in a sensible way that don't lead to irrecoverable data corruption.

The third stage is acceptance testing which on proprietary solutions is often the only one the end user is 110% in control of where you both test the app with actual user performing standard procedures and an team of specialist trying and break it using whatever knowledge they have on the systems design.

All tree stages have it's own skill set and is often performed by different teams and it's not uncommon to see specialist companies brought in doing part of the QA process for really important new systems but for the most part the entire things just fade into the the daily routine as an newer ending feedback loop.

This isn't really "scale."

It needs to record something like a dozen numbers from each of the ~1,700 caucuses in the state. You could have used Excel as the database here with no problem.

I was in the shower this morning thinking about how I would solve this, and something like this is exactly what I would have done. Use simple, familiar, battle tested technology to do the heavy lifting.

On production, apparently.

Iowa has 1,681 precincts. This is not a scalability problem.

Even the simplest CRUDs will have problems when you have 1681 users who must use it and you can't just bounce out the ones who don't get it and chalk it up to bad user fit.

If your CRUD can't handle 1681 users, then you're almost certainly doing something horribly, horribly wrong.

Like, that's "LAMP stack in 2001 hosting someone's blog about cacti wearing sweaters in Duluth" levels of scale.

"Scale" in this case is 1,681 precincts, each with a handful of users (the election officials). I'm not sure that was the issue in this case.

There are apps and libraries that can simulate load.

Something, something, monorail.

If Pete ends up winning (a significant "if" but possible), then he's the one who is most HURT by the delay in reporting.

It's not clear he'll win, I'd maybe even bet on a narrow-loss though we'll know for sure in 2 hours, but all campaigns agree was a strong showing. If not first place, likely second. His polling had him doing worse, so delays in him celebrating beating expectations are bad. And it's undisputed that this bug happened early in the process, so at a time when nobody had any idea who this would hurt/help.

(Also, the issues weren't all app-related. My understanding is that the non-tech issues related to counts took much longer to figure out.)

> Also, the issues weren't all app-related. My understanding is that the non-tech issues related to counts took much longer to figure out.

If I understand the IDP press release, the non-tech part was really just going back to the paper records as part of verifying the nature of the tech problem, which was that the app was gathering data correctly but not including all the data in its reporting function.

Could be. I was basing that part of my comment on the statement they put out last night ("We found inconsistencies in the reporting of three sets of results"), but there could be additional info on that today.

> I was basing that part of my comment on the statement they put out last night ("We found inconsistencies in the reporting of three sets of results"),

There are three sets of results data reported through the app (first alignment, second alignment, state delegate equivalents).

Right. I had understood that statement to mean that people were reporting bad data between those three results that were contradictory. That there was an issue on the user end, too. But it's vague enough that it could just mean that the app bug was causing that inconsistency.

The detailed press release seems to indicate that the apparent discrepancy was not do to misreporting by precincts but because the app's reporting functionality didn't include all recorded data in reports. But it doesn't explicitly state that there is no other source of discrepancy, so maybe there was something else.

It may be too late for you to ever see this, but it seems there are in fact other sources of discrepancies in the counts:

https://www.nytimes.com/2020/02/06/upshot/iowa-caucuses-erro...

There's also some dispute about whether satellite caucus rules are being interpreted correctly. (Currently, it's being interpreted in a way that's friendly to Sanders.)

What a complete mess.

Bottom line is that the quality of the app used in an election that could decide the president for the next 8 years was left up to high school level coding from an untested, uncontested company, with "software developers" with little to no credentials.

Oh yeah. Terrible decision-making. There shouldn't have been an app at all. And not only shouldn't it exist, it was poorly done. More details keep coming out about that:

https://twitter.com/AdrienneRoyer/status/1224796527392108545

I have bigger issues with caucuses in general and in Iowa specifically disenfranchising people from being able to vote at all, but that doesn't excuse that this was also terribly implemented.

The handling of this situation was terrible and, while Pete Buttigieg Buttigieg did pay the company money, I haven't yet seen any clarity on what service he was after. Your statement is almost pure speculation.

> conspiratorially, meaning speculatively

I'm only claiming that it is a convenient coincidence and I tend not to believe in those. I'm not talking absolute corruption. But I believe there is a buddy-buddy effect at the very least

I think one of the more dangerous aspects of the world we live in right now is how willing people are to immediately assume a conspiracy and making a sweeping a baseless statement only fuels that.

Your comment did call out the fact that you're uncertain, but it then continued to make baseless claims. There are interesting things that need to be dug into especially connected to Pete's payment to this app company and I'll be watching closely - but we simply have next to no information yet.

Again, I claimed no explicit corruption. I said that these events favor Pete and he had his hands in making these events occur. Why is evidence regarded as baseless?

He called in and pulled the poll that was released in a timely manner for 76 years. His adviser is married to the company's founder, and he bought 42k$ worth of "service" from them before. When app crashed, he claims victory like a rat and all of media announces his speech to the public.

Again I'm not claiming Iowa Dem party actively helped him. I'm claiming that he was allowed to help himself in multiple ways and noone stopped him, which is unfair to every other candidate.