You don't need a domain-specific tool for every problem your business has. If a spreadsheet works, great. If it ain't broke, don't fix it.

In a similar vein, there are plenty of small software teams that use nothing more than a whiteboard for project management. Yet there are teams tooled up with Jira/Trello/Slack with much less productivity.

I have to argue against this, because spreadsheets are a real pet peeve of mine. It can be hard sometimes to see _why_ a spreadsheet is broken. Spreadsheets simply don't scale (and I don't mean lolwebscale or whatever.)

If two business processes interact through a spreadsheet, or more than two people edit, you're going to immediately run up against the rough edges. Multiple people editing the same sheet inevitably ends in error. It can take a really gross amount of time to update the data when you're pulling/pushing from two sources. It may seem small on the surface, but even 2 hours a week adds up to nearly 3 weeks of work over a year. Can you replace this spreadsheet in 3 weeks?

I've seen this play out many times at large and small scale. The first example that comes to mind is a 10-hour day trip my wife took to go out of town and manually update a whiteboard because somebody messed up the shared spreadsheet. She was very unhappy, and this wasted a precious day during a _very_ busy period. Initially, the whiteboard and spreadsheet worked great, so they didn't "fix" it. It didn't scale to the busy times and really screwed them over.

At a much smaller scale, I wrote a quick SQL query to get some business metrics for the CEO. He asked if I could run it for him every week. Since I didn't have a good place to put this query, I just ran it manually and emailed it to him at first. Then he wanted to share it with someone on the opposite coast outside our network, so he asked if I could start putting it in a shared spreadsheet. My initial thought was to bang out a page they could pull up instead, but I figured it wasn't worth it. Then he wanted another metric. And another that required a bit of logic that couldn't be done in SQL. Pretty soon I was spending an hour a week manually running SQL and copying the results in to a spreadsheet. (We're a Google Drive shop. No way to connect Sheets to DB2.) That's simply not worth my time. I should've listened to my first instinct.

The initial investment in a spreadsheet isn't large, but the ongoing maintenance can really rack up hours. I think it's very often worth investigating any spreadsheet that users have been maintaining/recreating for more than a quarter.

> We're a Google Drive shop. No way to connect Sheets to DB2.

FWIW, Google has an API that can be used to get data into and out of things like Sheets. The documentation for it is abysmal and there are only a few libraries out there, most of which aren't complete. But, I've done it, and have some rough (but easy-to-understand) code that can "import" Sheets into MySQL. Going the other way should be a matter of tweaking the Google account permissions and calling a couple of different functions.

It's in PHP, but it's not gross PHP, so you can probably translate it into whatever you're familiar with.

Here's a freebie that I wrote in like 15 minutes and use in multiple prod situations. Treat any Google Sheet like a JSON data source. https://gist.github.com/chrsstrm/3fb0ce6820acecf62c5490d220d...

Yes, the App Script API is kind of limited, but also extremely powerful for simple services or to glue services together.

There's another really cool way that I've started using lately - doPost[0]. You can set up a js (appscript) function with that name and then publish it for anonymous use (such that the long endpoint url is the secret key).

You can for example set up a cron job to post some json to that endpoint and have the js validate it and update a sheet accordingly (or even create a new one).

[0] https://developers.google.com/apps-script/guides/web

I use pydrive. If you're doing something reasonable once a week it'll easily be fine. I had problems trying to push a few hundred thousand things in a short time, but otherwise I highly recommend this as a first step for sharing data.

What's not worth your time? You're being paid for this all the same aren't you? 1 hr/week would be 52 hours for the whole year, are you able to create something that much better in less time? And will that maintain all the flexibility?

There's a reason Excel is so powerful, but it can't magically solve for people lacking a process and most software solutions solve it by using complex permissions, versioning and collaboration features. You can just easily lock down the shared spreadsheet to readonly and have 1 or 2 people who are designated editors and get the same effect, or just have the changes done on a new tab and compare that way.

Even the most amazing software can and will get screwed up by users who don't have a process.

We programmers are paid for the time we do at work, but side projects like this very often aren't taken into account for the things we are expected to do during the regular work week. That is it's own problem, but this happens to a lot of people. You are paid for doing "the big projects", but then someone asks you to do this little thing on the side that gets bigger and bigger, but it's taking away time and focus on the big projects. Your boss probably doesn't think the side thing is what your job is.

Then take ownership and communicate so that they know.

If there's a better way then suggest it, or at the very least document the time and effort spent on these reports. If they are that critical (which they usually are for decision making) then it shouldn't be an issue when planning future work.

I think I understand the situation you described.

I’m not sure how “global level” organisations are, but mostly you just can’t convince a typical layman CEO that something looks easy, is really not easy.

Then when you serve the “small request”, next he’s in doubt why adding a small feature could increase the complexity of your work by a few order of magnitude.

Maybe today, but its going to keep growing until you hear it groaning under the weight of features that have more and more risk, and generally an unknown amount because those Excel methods are getting creakier and creakier over time as you try to accomplish more with the same tools.

The problem isnt Excel or any useful tool, its the misapplication of the tool and seeing people driving that same road already extremely well traveled.

http://www.laputan.org/mud/

This, a million times.

I can't count the many times a business process is done (poorly) by inputing lots of data in an excel file, running a huge (poorly written) macro, and spending the file via email to the next person and so on...

> Spreadsheets simply don't scale

This means that if your computing problem doesn't need to scale, a spreadsheet may just be the perfect tool.

However, I agree with you. The next Facebook might be of whomever builds a spreadsheet that scales.

Check our Air Table. https://airtable.com. I’ve not used it in production yet but it looks like a good excel replacement for database type info. The UI is very slick.

I tried smartsheets but it’s more calc centric. Asana is great if you are task centric.

It never needs to scale - until one day you find out that it has overgrown this stage many months ago.

> Spreadsheets simply don't scale

You've never seen panic like a multi million dollar business discovering that excel doesn't allow more than some fixed number of rows (1m?).

The issue is the constant need for new features. Much more cost effective to add a tab vs. modify code.

I think something lost in this conversation is how spreadsheets are largely a part of general primary school curriculum. Generally people have a feeling of confidence around them that is lost with other tools. Particularly "easy database" applications that attempt to replace spreadsheets.

They don’t scale well, but you don’t want to over engineer a solution until you really need it.

Check out Google BigQuery. Not sure if you can do exactly what you need, but we use it as glue between some dbs and Google sheets.

DBVisualizer can export any query you just did to a spreadsheet...

> You don't need a domain-specific tool for every problem your business has.

The opposite is also true: You can make a decent living creating, customizing or even just maintaining domain-specific tools, no matter if the customer's business really needs them or not.

> You don't need

It's very easy to criticize something by saying you don't need it. There are very few things in life that you absolutely need.

Yet there are things that add convenience, greater efficiency, etc, and people will pay for them.

Completely agree with you here, but I see it most of the time from another angle: If you're working on a project where you can't just plug in external tooling for certain things, there might be tooling that's rather subpar and slows everybody down. And yet - when you try to convince management that someone should spend a couple of hours bringing them up to snuff the reaction usually is "meh, that already works.", ignoring the manhours you throw away every week because the thing is organized so terribly. You'd think from an expenses standpoint alone it would be pretty easy to decide to "fix" things like this if you do a napkin calculation of how much of a black hole for money the whole situation is...

> If a spreadsheet works, great. If it ain't broke, don't fix it.

A spreadsheet works until it doesn't.

Maybe you load in past, say, 150mb of data.

Perhaps the last person to touch it linked it to a data or content source they only have access to.

Perhaps the person editing it has it checked out from a Sharepoint and forgot to check it back in.

Perhaps the person editing it left it open on their computer and forgot about it being hosted on a shared drive.

Perhaps the business critical sheet is hosted on someone's desktop.

Fundamentally, spreadsheets are cocktail napkins. It's fine to sketch out a basic idea. It's a terribly risky approach to run a business from it.

I usually frame this as "there isn't any single problem for which excel is the right tool, but for surprising amount of problems it is usable tool"

And I've found out that having excel export of essentially any data in your line of bussines application is indispensable unless you expose some kind of excel-like querying and data manipulation mechanism to users as part of your application.

> Yet there are teams tooled up with Jira/Trello/Slack with much less productivity.

I work for a small company, we generally use Jira (which we all hate, some with more of a passion than others... ok mainly me).

More recently we sort of accidentally started using google sheets for managing issues on some long running projects: it's easier to get an overview, i can add and remove unnecessary attributes of issues and filter them appropriately and dynamically without having to define a whole new workflow and start over. It takes basically the smallest amount of time possible to log an issue... I can organise them however I want. Issue management is overrated, issues are simple, and having something as flexible as a spreadsheet is a surprisingly good fit for anything not customer facing.

Oh also, I don't need to _learn_ how to do all those things... it's a spreadsheet so it's obvious, but Jira - it's so hard to do anything unless you live by it, I don't want to live by an issue manager I want it to get the fuck out of the way so i can do work.

What scares me is the whole world runs on Excel. Name a data-intensive critical industry: deep-sea oil drilling? Power grid management? International finance? All powered by Excel at critical junctures.

It works until it doesn't work then you are Donald Ducked.

I one worked at a place where a key part of the finance system ran on a spreadsheet - turns out this SS had mess up or finances and was one of the reasons the company restructured.

Donald Ducked as in royally fubared?

>You don't need a domain-specific tool for every problem your business has. If a spreadsheet works, great.

If what you are doing is applying math formulas over small or medium sizes of data, then excel is OK.

Otherwise it will suck and will be a waste of man-hours at the Office.

The problem is external criterions like data recovery, data security, quality control, access control, auditing etc.

*twitches

I can't believe I'm about to sink so low, but... criteria!

Criterions are what one adds to soup ;)

> Criterions are what one adds to soup ;)

You're thinking of crouta.

Let me introduce you to a cool website:

http://www.dictionary.com/browse/criterion

That something incorrectly oft-repeated can take on a veracity all its own is a wonderful aspect of the English language, but it doesn't make things any less twitch-inducing and, bluntly, incorrect.

Besides, I wanted to make a somewhat self-deprecating comment, rather than - as you have here - regurgitate some half-arsed passive aggressive form.

One advantage of spreadsheets is that data can remain local. Even air-gapped.

So can most other things - I'm aware that it's fashionable to have More Cloud, but domain-specific, local toolchains have zero reason to be non-local. "Spreadsheet xor cloud" is a false dilemma.

Well, there's local, as in your DC. And then there's local, as in your laptop.

Still not unique to spreadsheet.

What do you have in mind?

That custom solutions do not inherently require a server - "either it's Excel or it doesn't run on the workstation" is something that's actually easier to disprove today, with containers and whatnot. Custom solutions can be fully local, i.e. no network access required.

OK, fair enough. And indeed, I use MySQL with MySQL Workbench in Ubuntu VMs. But still, Excel on equivalent Windows VMs pivots much faster.

Pivoting is a feature, not a solution; plus it might pivot well on tiny datasets, but for anything over 100k rows, it scales about as well as a brick.

It wasn't that I had too many lines, but too many columns. As I recall, a few hundred columns and maybe 50-100K lines. Even after increasing the field/column limit in MySQL, the query wouldn't complete. But in Excel on an equivalent VM, it computed in a few minutes.

> Looking for a SaaS idea? Ask random people in a target industry what kinds of things they do with Excel.

Strongly disagree with this based on personal experience in "traditional" spaces that rely heavily on spreadsheets. There's a reason most of the financial industry uses Excel spreadsheets and it isn't because they can't afford to pay for specialized software. Excel spreadsheets are incredibly versatile. Your data format can change a little bit over time and you'll still be fine. With a custom software solution, you're kind of stuck within the confines of what the developers/product managers gleaned from your working process.

I'll strongly disagree in turn. I personally know some successful founders whose business, in one important facet, could be summed as "get rid of those painful Excel spreadsheets". It's not the use or not of Excel. As with most things, and as TFA discusses, it's about the pain factor. Spreadsheets are, far and away, the greatest success in end-user programming to date. But like all software tools, their usage can grow into painful places.

Do the users have massive, error-prone and/or time-consuming workflows built around those spreadsheets? Is it very hard to bring new staff up to speed, or to keep existing staff compliant with process, or to meet day-to-day business goals with those spreadsheets? Is there no solution between "painful spreadsheet" and "$X,000,000 contract w/ bigcorp" to solve those problems? If 'yes' is cropping up in answer to those questions, then lifting the logic from ad-hoc process around spreadsheets into software may be a good idea and/or a good business.

I worked at a mid-size startup where all of the customer-facing people were ex-finance---I was consistently amazed by the stuff they did in Excel using just pivot tables, rudimentary graphing, and (VBA? I don't think you could use Python at the time) scripts to do data cleaning, ad-hoc CRM, even rudimentary procedural generation of web content. 90% were one-offs, so it was totally appropriate to not throw developers at the problems (recurring things eventually got built out as microservices).

The main downside was that the functions and transformations weren't sharable across documents, and often there ended up being several mildly tweaked versions of the same functions across a bunch of documents. But I think I learned something about human tool use: when motivated, people will find a way to use the general purpose tools they know to improve individual efficiency.

I generally agree with you, but you picked sort of an exceptional example. As far as I know, Finance/Accounting is one of the explicitly defined use cases for Excel. For finance people, it's actually the best-in-breed tool for what they're doing.

GP is talking about people outside Finance who use Excel as a database. Though he's being a bit facetious, he's actually not far off. For the store data/retrieve data/share data workflows that most of these folks have, there actually is a good SaaS business in putting Excel on a server, wrapping it with a database, putting a nice UI on it and adding a few industry specific features.

Excel is the best tool in the world for quantitative exploration because the data flow and logic are incredibly easy to change. This makes Excel spreadsheets extremely versatile but also extremely easy to fuck up.

The basic counter argument to this is software engineering tools. If you have a more than trivial SS you end up needing tests, version control with branches, merge, and rebase, and an automated process for applying both.

OP referring to all use cases outside of pure financial. Basically where excel being used for managing a list.

For the several people reacting negatively to this: the advice isn’t “replace every spreadsheet you find” which would indeed be silly, but “look for spreadsheets as a possible smell that there’s a gap that could be filled”.

Yeah, and to that I would add another nuance. Even if one does find spreadsheets that "ripe" for a more robust tool-- that still isn't enough.

One also has to consider the cost of engaging the IT department or others who can do and/or manage development within the organization. That's NOT cheap. Many businesses can barely manage to handle their own shit, let alone successfully do even a small unplanned project to port an excel app running in somebody's cube to something more robust.

> Looking for a SaaS idea? Ask random people in a target industry what kinds of things they do with Excel.

I understand where you're coming from, but this is just not good advice. Somebody (I can't remember if it was pg, Joel Spolsky, or someone else) once wrote an entire essay basically saying "DON'T make a startup that's just pivot tables. I have seen so many SaaS startups fail because they didn't realize they were just selling pivot tables at a much higher price than Microsoft wants for Excel."

Also take a look at this HN thread: https://news.ycombinator.com/item?id=12450802 where a whole bunch of people express bafflement of and loathing for people who unnecessarily replace Excel with half-baked software. It seems like a good idea; it's usually not.

> It seems like a good idea; it's usually not.

Not for a startup, no, but in my experience it works well as a lifestyle business. Companies pay well for this kind of work and it seems the number of spreadsheets they want to have replaced provide enough work for the next few decades to come. Better the replacement isn't half-baked though...

It may also have been in a blog post, but Joel Spolsky makes that remark in his presentation "You Suck at Excel" https://www.youtube.com/watch?v=0nbkaYsR94c

That was Joel in his "You Suck at Excel" video.

> Looking for a SaaS idea? Ask random people in a target industry what kinds of things they do with Excel.

True, but you also have to make sure that you can improve on the spreadsheet way enough to be worth potential customers' switching cost and money.

I actually wrote a post about exactly this only a couple of weeks back: "Your biggest competitor is a spreadsheet" - https://medium.com/@hjalli/your-biggest-competitor-is-a-spre...

Specifically you need to whittle it down to those uses of Excel where...

- one spreadsheet is shared among more than about 5 people (or they each have their own version of it... shudder)

- the sheet has more than about 10,000 rows

That's the point where Excel truly starts to be not the right answer.

There's also the case where the spreadsheet contains a whole bunch of endlessly duplicated and error-prone cell logic that nobody really understands, partly designed to work around limitations in the underlying cell-programming model.

When you say "shared" do you mean the Excel file is kept on some network drive and that multiple people can access it there?

That would be the best case! There's also the horror of emailing them back and forth.

I was actually toying with the idea of replacing our custom-made, rather limited shop system with something entirely based on Google Sheets.

We have rather low requirements, which is why I'm resisting existing shop systems. They all seem like overkill to me.

But from time to time, someone wants a some statistics, or needs to send emails to "everyone who bought in the last four weeks". These are trivial for me to compile in SQL, but a spreadsheet would allow others to quickly work with the data.

Unfortunately, the Sheets API at around one second per request was too slow for my tastes when I last checked it out.

You could always schedule regular exports of your data to sheets if you just want it to be easier for others to answer simple questions.

I had a super slow sheet script but I found it had a batch-all-my-changes type of workaround in the API which brought it back into reasonable territory.

Excel spreadsheets are a lot more flexible and easy to use than domain specific tools, particularly for someone already familiar with Excel.

I agree this is a good place to look. I'm not sure a new tool will be an improvement as it will always have limitations and a learning curve Excel doesn't have.

90% of the people who invent a CRM on a spreadsheet are better off using that flexible spreadsheet they know very well and can change, add or remove features, store anywhere they want etc. than using a stupid CRM software with 3 features, designed around its creator limited imagination only.

Yeah. Until they leave for another job and literally no one else in the company can understand the ad-hoc process unintentionally developed over five years of "let's just add this here".

Which is not to say that whatever replaces it is guaranteed to be better, but it will at least have a good start by using the right tools for the job.

The issue for the spreadsheet is always around data-integrity, people start typing random things in their Excel and then it's a mess to try to make sense of the data. I would not mind Excel much if it was strongly typed instead.

But being strongly typed would remove a lot of the usability that makes Excel the most widely used programming language among non programmers.

I think there would be space for a middle ground solution where you could define a sheet "type" to something like "datatable" that would enforce data types on columns, only allow data entered a row at a time and if calculated columns were there, would enforce same formula in each row.

Perhaps there's something to try in that space.

Isn't that what Microsoft Access was?

Microsoft Access used to be the real test. By the time they've added a UI and started to get frustrated with one user at a time it's time for a CRUD app and a database

How is a spreadsheet scalable for a large organization? The team or person might be better off but CRMs and other tools help the organization standardize their processes.

What happens when this person leaves? What if business starts booming and you need to scale up? Are you really better off sharing a spreadsheet between a 100 or more people?

> How is a spreadsheet scalable for a large organization?

Just gonna focus on this part here. The last offices I worked in (and the last places I even dealt with tech stuff beyond teaching it to primary-schoolers), "Track Changes" was simply a godsend. One person "owned" the spreadsheet and could allow use or not; and we had it hosted on OneDrive - of course we all had the OneDrive client on our work PCs and it could tie into Excel natively, so we were all using the same document.

If the entirety of a large org needs the same spreadsheet, you're probably looking at a scenario where they really need multiple sheets, but haven't figured out how to do references to other files and sheets yet. But using Track Changes and having one person designated as Owner so that that one individual makes a decision to approve/disapprove all potential changes makes it much easier to let multiple people work together.

> What happens when this person leaves?

That's a management / succession-planning issue. Ad-hoc solutions shouldn't be kept secret, and allowing employees to just go off and do as they please in that regard is a major issue. If all work-related documents are required to be stored in the company cloud, and you storing a new doc alerts your manager, etc., you'll be able to deal with this much easier. One place I worked for a few weeks on a project did a reset on employee's terminals every Friday at around 7PM, and anyone who claimed they lost data or progress from that on Monday was in trouble, because they didn't save company-related data to the document & data server (which obviously did not get reset) - meaning they'd done something against company policy that they'd been told about the day they were hired.

The person that invents the CRM spreadsheet is better off. However, that spreadsheet is usually harder to share with the rest of the company. If the spreadsheet has more than 1 user than it might be worth a SaaS.

Just use a google sheet and you’re good for a long time.

It depends. Sheets with multiple users is a bad smell to me.

If you can get them to use Quickbase/Gsheets/AirTable or similar instead there's a big improvement. Still hacky, but you know what's out there and roughly what it's doing. And better centralized access control and backup.

If that’s the case, put me in jail and throw away the key. But I think the opposite: between a spreadsheet and something engineered I will take the (google) spreadsheet every time.

I’m allergic to the over-engineered solutions so prevalent.

I encountered situations in corporate environment where a couple of questions with an answer selected from predefined list (radiobuttons/dropdown) were made in Excel... of course completely broken without the latest office on windows and outside of the intranet...

Where I work, I big Word document with a table in it, is the duct tape. Using Excel would be the right way.

I actually had a gig based exactly on this. There was a very complex spreadsheet that was used by an actuarial department of an insurance company that they finally decided to automate. There were .net libraries that could access the spreadsheet components and produce external reports, and put values into a database. Oddly, they kept the spreadsheet component as a the input mechanism.

Oddly? Why is it odd to not want to change from an input method that you already know like the back of your hand? If everyone who’ll be using this relies on the spreadsheet program for most of their work anyway, being able to keep a lot of the process in there makes sense. Learning a new tool is friction.

It was pretty clumsy in its realization. There were like six or seven regions on the one sheet and it was hard to work with. So they kept that part of it.

To be honest, I first learned about using spreadsheets because my mother worked in insurance - not as an actuary but as one of the people under the actuarial department. Lotus 1-2-3 was the norm not only for that office but the industry. As she was retiring, Excel was becoming big (because the standard OS moved from DOS to Windows), and Excel was just starting overtake Lotus. When it's been an SOP for more than 30 years to the point that they can reliably teach the basic use of these sheets in colleges/universities? You're not highly likely to shift it in an established corp.

A hundred times this. Many times the apps which we are building tend to be a simplified version of the spreadsheet. Business points is spreadsheet that is online, which is controlled by authentication and authorisation and has some customizations.

We sometimes even propose to build a native Excel based VBA app. The users can always use something like Dropbox sync or box sync on Google Drive sync to share their work with others.

My most popular project to date started out as copying videogame player names from a website running "SteamQuery", and pasting into Excel.

> Ask random people in a target industry what kinds of things they do with Excel.

Think of all the would-be SaaS businesses that have been disrupted by Airtable.

I run all my business logic on a computer that I built in minecraft.

When did spreadsheets, a tool that is generic and versatile, become worse than a n$/month SAAS with fancy bootstrap inputs?

When they became effectively impossible to audit. That is, at their inception.

I've once worked with a trader whose magic sauce was encoded a spreadsheet he had been developing and carrying around for 10 years. It had tens of thousands of formulas and references cross ~50 sheets.

He suspected he had introduced an error somewhere, but gave up after a couple of months of trying to, you know, verify the whole thing.

Yeah I'd say that If you have a hundred formulas and more than 10 sheets, it is time to change to another tool. But when someone is this far, they'll have accumulated so much technical dept that they will probably have to hire a programmer because the typical SASS won't solve all of their needs.

When each user has a different copy, and nobody knows what the most up to date version is.

When you have error-prone processes that you do hundreds of times a day based on those spreadsheets, that a bigger application can automate.

When ten of those spreadsheets could really be replaced with one.

As with most things, its strengths are its weaknesses. Versatility is great, but it's also the enemy of things like standardization, data security and data accessibility. The more people are collaborating, the more those things start to be important.

Perhaps google trends might be useful (is there some API?) for example "how do i sort a list of customers in excel?" How do i track shipping containers in excel?

Love this...