This! As programmers, it is easy to list N reasons why spreadsheets suck and deserve to die. But the reality is that it is the most ubiquitous business software and the non-tech/business users would rather be in a spreadsheet than anywhere else. Companies run on spreadsheets, even if they have the best "data stack" and tools. Because few, if any, software can beat the

* flexibility (throw data anywhere and link it to each other, make edits, write notes),

* power (formulas, etc), and

* familiarity (the most underrated factor) of spreadsheets.

Nothing else allows the non-tech user to feel empowered like spreadsheets do.

Spreadsheet evolution has been slow though. Google Sheets added cloud + collaboration 10 years ago. We (https://coefficient.io) are adding the layer of connectivity to sheets so they can remain in sync with the actual sources of data (Cloud apps like Salesforce, DBs, BI tools, etc) so sheets actually become "live" (even though they have been in the cloud for a while) and to reduce manual work and increase trust/accuracy. There is so much more that can be done to leverage this largest software platform that is out there.

I used to work at in construction as an estimator and we used spreadsheets. No backups, no version control, a single value change took 10s of minutes to propagate through the many sheets in the file. We fixed errors when we saw them, but I always felt it was an exercise in pointlessness.

These were not small companies either, ~$800M in yearly revenue for one, and ~$150M for the other.

Then, afterwards, the end value would be massaged to what felt right. And yet, these sheets were seen as part of the "secret sauce" of the business.

It's one of the reasons I really wanted out.

Programmers have a reputation of being arrogant assholes, but I think this push-back and ridiculing other industries of using excel for stuff like this is completely justified. Excel spreadsheets let these people FEEL productive and like masters of their own fate with a bunch of numbers neatly encapsulated in their own little cells in a table, but their actual usefulness is questionable. For construction, it gives a rough feel for a project, but a lot of it is smoke and mirrors.

I can't remember where I read it but at one point Boeing was maintaining the entire bill of materials for one (maybe more) of their aircraft in a spreadsheet. I tried to google it but could not find the reference. I did find that Boeing actually had their own in-house spreadsheet software for a while, which was kind of interesting.

https://en.wikipedia.org/wiki/Boeing_Calc

> I used to work at in construction as an estimator and we used spreadsheets. No backups, no version control, a single value change took 10s of minutes to propagate through the many sheets in the file. We fixed errors when we saw them, but I always felt it was an exercise in pointlessness. These were not small companies either, ~$800M in yearly revenue for one, and ~$150M for the other.

Jesus... want to start a competitor to these guys!? Email me

I think I big mistake people make here is that they think they are far better at everything on a computer than a non-programmer. They don't realize how much you can do in Excel and how much they suck at it. If you can't use it without a mouse and/or if you don't know pivot tables, you are just as much of a beginner as a C++ programmer who doesn't know what a pointer is.

It's only after trying to convert an excel sheet to [insert your favorite language] that they realize it would take them 6 month with a team of 5 to replicate what a single accountant did in a week.

Entirely true. Many physicists also basically live in Excel. And replicating the functionality of pivot tables isn't at all trivial and they also have a lot of helpers by now. In many cases you just need to click it and have all column neatly formatted so that you don't have to do much anymore.

I also have seen Access applications with more features than a common Salesforce CRMs. I have seen it as a navigation system that could bring you to the nearest partners or shops that provide necessary parts. Yes, there is code of doom behind it and if those maintainers leave the company, problems arise. But the feature set was often extremely hard to beat.

And I am not sure how to respond to non-technical users about better alternatives. They use it as a front end for SQL manipulation and it isn't easy to come up with something better for that target group.

Even worse is SharePoint. It is a complete abomination, it can easily beat most horror novels. But the workflow engines are just extremely practical for corporate processes. Only recently some alternatives came to the market... I just wish I could kill it...

> Spreadsheet evolution has been slow though.

Excel has to contend with nearly 40 years of backwards compatibility (MultiPlan, the predecessor to Excel, was released in 1982) and a deep userbase that literally has decades of experience and muscle memory with the software. The Symbolic Link "SYLK" file format introduced in MultiPlan is still supported in recent versions of Excel, leading to the infamous CSV "ID" issue.

Many of our users still run very old versions of Excel and Windows (e.g. Excel 5.0 on Windows 95) because a change in a future version of Excel caused problems or gave different results.

> Nothing else allows the non-tech user to feel empowered like spreadsheets do.

I'd say it allows the same for technical users as well since it helps bridge communication/knowledge gaps and move things along. It's typically not the solution but often plays a critical supporting role.

You're correct, but this entirely misses the point of the article. It's not calling for the end of spreadsheets -- it's stating that there's an unspoken problem that demands our attention.

Spreadsheets aren't 100% reliable for use cases where you need to collaborate and share immutable health records. Especially during a time of global emergency when tensions are heightened.

Spreadsheets don't impose validation, schema correctness, constraints, etc. and can amplify human errors. They can also inject errors of their own (eg. turning March1 the gene [1] into a date) when they're simply trying to be helpful.

How do multiple people manage a spreadsheet? How do you safely merge spreadsheets? How do you keep records from being duplicated or replaced? How do you do double bookkeeping? Can you atomically identify individual records?

This article is saying that we have to realize spreadsheets can be a source of scientific error simply because of their design intentions and ergonomics.

The closing remarks estimate that 1,500 people died as a result of spreadsheet error. That's remarkable.

[1] https://www.proteinatlas.org/ENSG00000145416-MARCH1

You're also correct, but I think miss the point as well. The spreadsheet isn't going anywhere. Curse it all we might, blame it for deaths or attribute any aspect of malignment to it, and you'd be right. But the spreadsheet isn't going anywhere.

Spreadsheet mistakes have been directly attributed not only to thousands of deaths but billions of dollars of mistakenly wasted money. Heck, in the UK alone, they have attributed billions of pounds and thousands of deaths specifically to spreadsheet errors and flaws, and that's not even counting the COVID-related issues from the article. Check this scary article: https://theconversation.com/excel-errors-the-uk-government-h...

There have been hordes of purpose-built apps that have aimed to replace the common spreadsheet for any number of tasks from inventory management to financial planning to even tracking nuclear weapons (true story: our nation's nuclear weapons stockpile is tracked in a spreadsheet). And yet, the spreadsheet is still here. The spreadsheet is the cockroach of all apps. At once utterly adaptable and seemingly indefeatable.

The spreadsheet is a horror show with lack of control, schema, and collaborative management. We surely can do better. But we haven't. Google and Microsoft have the best developers in the world who can do anything. And they have produced more spreadsheets. But spreadsheets are the bane of our existence! And yet they are still here.

The spreadsheet has survived from the mainframe era to the cloud-based era in pretty much the same form with enhancements. The spreadsheet is not going away, even with all the complaints in the article. The points are well made and well founded. But unfortunately they miss the point. 20 years from now we'll still be cursing the tyranny of the spreadsheets because of their utility.

I close with a Haiku:

| Rows and cells, alas |

| What makes your simple structure |

| So very useful? |

It is very clear that they are useful but it is also very clear that they can be problematic. In Finance people check each number - multiple times per day - because there is no way to write tests in Excel. Sounds good? Until you can trust the people crunching the numbers, of course. Would you build a bridge like that? Or a building? I think it is worth to talk about this problem and, at the very minimum, inform people there are alternative ways to crunch numbers that are more reliable than Excel.

Lots of engineering calculations are indeed done in Excel, it's very common.

> Would you build a bridge like that? Or a building?

That's exactly what's being done. I work in mechanical engineering and Excel is the primary calculation tool all over the industry.

It would be hard, but I think a new kind of spreadsheet that was easier to validate, easily impose a schema and constraints and helps us identify human errors would be helpful.

I bet excel has some excellent features around this, but they are not front and center in the UI.

Lotus Improv was a bit like that I think. It's been a long time since I used it. Wikipedia says:

Conventional spreadsheets used on-screen cells to store all data, formulas, and notes. Improv separated these concepts and used the cells only for input and output data. Formulas, macros and other objects existed outside the cells, to simplify editing and reduce errors. Improv used named ranges for all formulas, as opposed to cell addresses.

https://en.wikipedia.org/wiki/Lotus_Improv

Like most of the software on the NeXT platform, it was ahead of its time.

> easier to validate, easily impose a schema and constraints and helps us identify human errors would be helpful.

yeah, it's called IF

Right.. at my company we had to purge spreadsheets from all processes due to these issues. Instead we allow import and export of spreadsheets but they aren't the source of truth.

> From the first Visicalc to Excel 2300 I don't think we'll see the end of spreadsheets for a long time coming.

We might even argue that spreadsheets were already used in 1295 BCE, as shown in Fig. 1.1 of [0].

[0]: http://uruk-warka.dk/mathematics/ER6%20tables.pdf

Spreadsheets are the private motor vehicles of computing.

I think the best approach in many cases is to bring data management into a specialised application, rather than using spreadsheets as a database. They can use the CRUD applications we all write or some off-the-shelf solution, but the important thing is that it provides validation, schemas etc.

Then you provide a spreadsheet export or direct connection that can be hooked into another spreadsheet with all the formulas, pivot tables and so on.

This way the core data is safeguarded and kept clean while not requiring constant development for every report requested.

Personally, I strongly prefer working with GSheets to Excel. It handles large sheets better and you never worry about the app crashing and taking your precious work along with it.

Instead one only needs to worry that one can’t get one’s work done.

I don’t think I’ve ever had a time when I wanted to use a spreadsheet for anything moderately complicated where gsheets was up to snuff. That’s not to say that excel doesn’t have problems or does things easily, but it doesn’t lead to quite the same level of frustration and hair-pulling, and indeed it’s usually possible to achieve things in excel when they are impossible in gsheets. Indeed that app even struggles at the ‘easy’ task of being able to fiddle around with table formatting. Its sole advantage is the collaboration. It can be used as a place to dump small amounts of varied tabular data to share but not really for calculation or analysis.

As far as competing anecdata, I found gsheets absolutely much better for joining together multiple sheets of data. The lookups are so much less touchy simply because "it's all in the cloud" (as long as you have permissions).

With Excel you have to ensure the file is there, and readable, not tampered with, etc.

This allowed us to create better multi-workbook "apps" that have a bit more modularity - you could even embed sheets into a webpage as a view-only client.

In the end, Excel is good for us only for interoperability with external groups.

I have. Especially before Excel released their dynamic array features. I love the sort function in gsheets. Combined with filters it allowed me to automate things that relied on vba in Excel.

Google Sheets has more useful formulas than Excel. For example query(), but also all the array formulas that they include.

Wait, seriously? I find the exact opposite! I would love it if Gsheets were as performant as Excel with large tables but scrolling gets bogged way down. If you add too many formulas, Gsheets will get stuck recalculating formulas endlessly. I love Gsheets because of the ability to link between documents easily, their comment system, filter views, and native regex… but performance?? It’s terrible!

Large sheets? Google Sheets has a much lower cell and row limit compared to Excel

I assume they mean performance wise. Excel could have larger hard limits but handle complex sheets worse.

Google Sheets is an unusable dumpster fire. I just tried to import a 3 million row CSV yesterday and it wasn't able to do it. INDEX/MATCH don't even work properly! Basically unusable unless you need it for scheduling parties or whatever people use Airtable for.

Have you considered using a different tool for the job? Processing 3 millions rows of CVS sounds like a task that is more fitting for a small script than Excel/Google Sheet.

Google Sheet excels at many other tasks, but it's not a replacement for specialized tool (and it should not).

Yes of course, I originally generated the file using Pandas but it's always a pain in the ass to remember the magical incantations to make pretty 2D graphs that aren't fugly. That being said, I have a knack for making great 3D graphs that are awesome! In any case, it's hard when my employer doesn't just gimme that juicy Excel subscription because cost-cutting and efficiencies.

In the end I put on my robe and wizard hat and used matplotlib. (╯°□°）╯︵ ┻━┻

Seaborn in Python to make matplotlib pretty by default + plot some uncommon graphs. `%matplotlib widget` to make matplotlib interactive. Plotly in Python if you want decent graphs that are also interactive by default.

Observable if you want an interactive, hosted visualization notebook (and all their libraries are open source, except their UI). Plotly, Observable's Plot library, or D3 if you don't mind JavaScript. Can bundle libraries + data + visualization into a single html file and deliver that as an interactive report; once you have the base template down it's decently ergonomic.

Plotly is my current default when I'm working with Jupyter, though. Sane defaults are nice.

I'd love to learn if I'm doing something wrong, but Sheets has terrible performance compared to Excel in my experience.

Not just sheet sizes, but Sheets feels much slower and has a vastly weaker toolset. Just the fact that the browser takes precedence for keyboard commands from the web app makes the usability suffer massively.

And controversial as this may be, JavaScript.

performant JavaScript exists apparently - I was shocked to learn that there are some new DeFi projects that are executing DEX arb bots and write all their code in TypeScript (lmao).

It will be extremely funny to see how Google does this whole "rewrite all our office products to use <canvas> instead of the DOM" project over the next couple quarters... he says, laughing, shaking his head at the wasted effort... Caesar wept for there were no more worlds to conquer

Newer versions of Excel have native JavaScript support https://docs.microsoft.com/en-us/office/dev/add-ins/referenc...