Re:Social Movements Are Pushing Google Sheets to the Breakin...
by ENOTTY on Thursday August 06, 2020 @10:51AM
(#24071418)
Sheets seems like a good way to get to a MVP quickly to prove the concept. If Sheets starts falling over due to demand, you know you've got a winner and its time to invest in something more reliable.
Re:Social Movements Are Pushing Google Sheets to the Breakin...
by ISL on Thursday August 06, 2020 @12:18PM
(#24072389)
The tricky part is the timescale.
For a crisis-response, an approach may reach maximum impact within days and really only be needed for a short duration (a week to a month). Migrating a system with intense user/volunteer churn (and keeping two databases synced as you transition) can be tricky. Handling permissioning at the same time is also a challenge.
At FindTheMasks, we expected the government to have supply chains and epidemiological response (testing/contact tracing) together within two or three weeks. When that didn't happen, we kept bolting on new tools and strategies to double the lifetime of the endeavor, expecting the government to get its poop in a group by then. 20 weeks later, supply chains are somewhat improved, though the entire situation is compromised by the lack of overwhelming success at the epidemiological scale.
Re:Social Movements Are Pushing Google Sheets to the Breakin...
by toomuchtodo on Thursday August 06, 2020 @12:09PM
(#24072284)
A spreadsheet is a database you haven't yet performed a migration on yet. I think the argument is less around better spreadsheets and more around migrating that data into real datastores once constraints are encountered (whether that's a desire to support large amounts of contributors, not run on Google, the dataset has grown too large, etc).
It's hard to go from Google Sheets to a web frontend to Postgresql and a snappy shared frontend. That is the problem statement to solve for imho (interface, schema, data portability).
Re:Social Movements Are Pushing Google Sheets to the Breakin...
by PeterCorless on Thursday August 06, 2020 @12:52PM
(#24072793)
No one has really solved the "Spreadsheet —> Database" migration painpoint. We don't have anything that is universally easy-to-use. The equivalent of a Hypercard or a Filemaker for the Web. (And no, I'm not talking about MongoDB or other NoSQL; I'm talking about the kind of thing you can hand over to a typical office worker.)
Re:Social Movements Are Pushing Google Sheets to the Breakin...
by toomuchtodo on Thursday August 06, 2020 @12:55PM
(#24072833)
Agreed. I have a hunch that any solution to this problem will look similar to https://github.com/simonw/datasette if it develops. Tied together with https://supabase.io/ and you've built a versatile, powerful, open data collaboration engine.
Re:Social Movements Are Pushing Google Sheets to the Breakin...
by ISL on Thursday August 06, 2020 @11:45AM
(#24071987)
With FindTheMasks, we found that tooling written around Sheets may also confront the 5 million cell limit.
As you tack on extra tabs and handle more fields, it isn't too hard to approach that limit.
If we do trip that threshold, we will be forced to degrade some functionality.
As a bonus, I don't believe that Sheets allows anyone to see exactly how many cells are in use, so one lives with a probabilistic risk of lockup.
If we were continuing to handle large volumes of intake, a switch to Airtable would be in the works. Fortunately, the PPE situation is at least somewhat better than it was in March, so we are simply keeping the lights on and wheels turning.
Re:Social Movements Are Pushing Google Sheets to the Breakin...
by ISL on Thursday August 06, 2020 @12:58PM
(#24072874)
If those records could have an arbitrary/large number of fields, that would suffice for our foreseeable needs. We're at ~6,000 individual entries at the moment and do not expect a 10x jump at this time. Our efforts now center on updates to and removals from our public map.
Airtable-wise, I'm parroting our devs on the subject. We had perpetual/nascent plans for an Airtable migration (and did so when FindTheMakers came to life) which have been tabled for now. We believe we'll make it to sunset on Sheets, but just barely.
Re:Social Movements Are Pushing Google Sheets to the Breakin...
by spacehunt on Thursday August 06, 2020 @03:14PM
(#24074413)
Airtable is closer to a database than a spreadsheet. Records are represented as rows in their default grid view, but there are other views available. They can also be retrieved as JSON objects through the API that Airtable automatically generate for you.
Re:Social Movements Are Pushing Google Sheets to the Breakin...
by snowwrestler on Thursday August 06, 2020 @11:20AM
(#24071708)
Not just social movements; I sat in a free career webinar and they posted a Google Sheet link in the chat for all the participants to share LinkedIn profile links. It crashed quickly and for the rest of the meeting the chat had a steady stream of comments like "it's working now", "I'm locked out, please help", "can anyone share the link to the sheet", "here's a link to another sheet we can use", etc.
I sat there thinking "I can't believe there isn't a better option for this sort of thing." Kind of surprised Zoom does not have a native feature for meeting participants to share contact info with each other during the meeting.
Re:Social Movements Are Pushing Google Sheets to the Breakin...
by scottlamb on Thursday August 06, 2020 @11:48AM
(#24072025)
The main problem with Google Sheets for that use case IIUC is that it's designed to be prepared for any of the users to edit arbitrary cells which can cause recomputations of other arbitrary cells, and it wants to give real-time feedback to all the other users about every edit (and cursor movement). That makes it harder to scale up to many simultaneous editors without compromises in the backend (due to the graph of recalculations) and UI (clutter, contention to claim the first blank line).
A custom web app for this purpose can scale better by being designed with the knowledge each user only edits their own row and nothing else needs to be recalculated. And of course it can have a tailored UI.
Unfortunately I don't know of any way for a non-developer to match that. There's no fundamental reason IMHO that there couldn't be a nice rapid, code-optional CRUD app generator, like Microsoft Access for the 21st century. Maybe this is what Microsoft Lists will do? AirTable is a general-purpose web-based RDBMS frontend, but I don't know if it scales well to many simultaneous editors either, I don't think you can tailor the UI as much as I'd like, and I hear folks complaining about database size limits.
You can get Google Sheets to scale better, but it's a bit awkward:
* Give folks a Google Forms link to edit their own information. This atomically claims and populates the row.
* Give folks a publish link (see File | Publish To the Web) to view everyone else's information. This link scales well because it doesn't let them update anything and (more annoyingly) it doesn't show changes until they hit refresh.
Re:Social Movements Are Pushing Google Sheets to the Breakin...
by joshvm on Thursday August 06, 2020 @11:31AM
(#24071822)
For surveying-type requests you should probably be using a Google Form which links to a Sheet (with limited access control). You shouldn't be giving everyone simultaneous edit access to the document.
Re:Social Movements Are Pushing Google Sheets to the Breakin...
by blakesterz on Thursday August 06, 2020 @11:10AM
(#24071604)
"A Google support page states that “up to 100 people with view, edit, or comment permissions can work on a Google Docs, Sheets, or Slides file at the same time” and has a section devoted to troubleshooting files that become unresponsive after being shared with many people, recognizing the common pitfall."
That's a big number, but it's not as big as I'd expect from something from Google. I'd assume the average number of people working on any Sheet is 1 or 2, but I'm surprised that haven't been able to scale it up to 1000 or more by now.
Re:Social Movements Are Pushing Google Sheets to the Breakin...
by pwinnski on Thursday August 06, 2020 @11:29AM
(#24071801)
Also in the piece, "Jonathan Rochelle, the lead product manager for Google’s suite of collaborative tools from 2005 to 2013, said that he and his team found that having more than 25 to 50 editors on a single document made it hard for people to collaborate effectively."
The technical limitation is 2-4 times what the former product manager believed was a human limitation already. Scaling beyond that doesn't seem like a priority.
Re:Social Movements Are Pushing Google Sheets to the Breakin...
by smolder on Thursday August 06, 2020 @11:41AM
(#24071933)
Allowing a very large number of people to edit a document at the same time isn't trivial in terms of the state management and network load. Everyones edits have to be sync'd and sent to everyone else. Higher numbers than 100 are certainly achievable but perhaps there are limitations in their infrastructure for processing power available to a given sheet.
Re:Social Movements Are Pushing Google Sheets to the Breakin...
by codenesium on Thursday August 06, 2020 @11:45AM
(#24071980)
This jumps out to me as the same thing as database contention except you have to distribute changes to every consumer. If everybody is only allowed to append I'm sure it scales to infinity like an event store. Handling updates just gets messy.
Re:Social Movements Are Pushing Google Sheets to the Breakin...
by robjan on Thursday August 06, 2020 @11:17AM
(#24071677)
I would think the document would quickly become unmanageable if more than a few dozen are editing it at the same time. Even if it's possible technically, we start to enter social problems.
Re:Social Movements Are Pushing Google Sheets to the Breakin...
by lukeschlather on Thursday August 06, 2020 @12:30PM
(#24072540)
I've been using Docs (and a bit of sheets) for stuff like this and haven't noticed too much issues, though I usually keep under 10 people. Honestly I hope Google doesn't try to address any of the complaints in this thread.
Google Docs/Sheets/Drive function a lot like Stardew Valley multiplayer, in that pretty much anyone can do anything, and this allows really fluid communication but it simply won't work with 100 people.
Sheets supports 100s of people to an extent (even more) but I wouldn't want them to optimize for this. I think if I were in charge of it I would be exploring ways that they can find UI paradigms that allow more seamless interaction and realize the dreams that I think Google had when they built Wave.
* What if you could embed live video/audio in a Doc?
* Sheets comments kind of suck.
* Is there a way to use comments that makes them more visible without ruining the sheet view? (Maybe not.)
* As this article says, people are often using a Sheet as a very hacky app dev platform. Is there a way to embed the sheet data in a Doc (or something Wave-like) that has a more seamless communication pattern?
* Is there a way to build small apps on the fly without messing with a bunch of macros?
Though I think a lot of this is about communication. I don't think Google should be thinking about scaling, they should be thinking about those use cases where you're talking to a small group of people and you need a little custom app that can't take more than 30 seconds to put together or just talking would be more efficient.
Re:Social Movements Are Pushing Google Sheets to the Breakin...
by Yhippa on Thursday August 06, 2020 @12:53PM
(#24072802)
Let's say they fix Google Sheets to be arbitrarily scalable. Now you have another problem: how can that many people edit the Sheet simultaneousy and still have it make sense?
Re:Social Movements Are Pushing Google Sheets to the Breakin...
by jlkuester7 on Thursday August 06, 2020 @06:20PM
(#24076042)
The technologically maximum number of concurrent editors that Google Sheets supports is perhaps the least important constraint when trying to crowdsource an information-base. At a certain point (well below the point where Google Sheets crashes) you run into community constraints around permissions, validation, how you organize the data, how you record changes, how you discuss changes.
That being said, this article seems to completely ignore the fact that there are already well-established solutions to this problem. Off the top of my head, wikis and GitHub/Lab pages offer very nice solutions to creating a scale-able information-base with built-in features to help address some of the challenges I raised above....
Re:Social Movements Are Pushing Google Sheets to the Breakin...
by disease on Thursday August 06, 2020 @11:33AM
(#24071841)
This is the second story on the front page about the weakness of modern spreadsheet software. It could be cool if someone stepped forward and showed us how to do spreadsheets using everything we now know, like what git did with source control back in the mid 2000s.
Re:Social Movements Are Pushing Google Sheets to the Breakin...
by WorldMaker on Thursday August 06, 2020 @03:30PM
(#24074544)
What about "Don't use spreadsheets"?
It's a flippant answer, sure, but spreadsheets are the index cards, sticky notes, and yarn approach to data organization. Certainly they are easy and approachable to use, but any sufficiently complex data set in general is going to be more "chaotic mind corkboard mess" than "organized".
Spreadsheets are very useful data hammer, but not all data is a nail; to borrow another aphorism.
That said, there's a wide gulf in the general user approachability between spreadsheet applications and more bespoke/"better" data organization options. Decades of experiences and mistakes learned from tools like DB2, FoxPro, and Access have shown us we have a hard time finding an "easy medium". AirTable thinks they have some modern ideas. Microsoft's PowerApps teams have their own modern approaches (and as almost the direct the inheritors of FoxPro/Access, presumably the most experience to learn from). Simon Willison's work with Datasette (SQLite DBs wrapped in simple view/management webpages) has had some interesting ideas. Though I don't think any one approach will solve it, it might take a whole shelf of tools, and at the end of the day maybe as a species we'll never have as clean of a user experience as the worst spreadsheet.
Re:Social Movements Are Pushing Google Sheets to the Breakin...
by aspenmayer on Thursday August 06, 2020 @11:32AM
(#24071833)
I feel like Sheets should gracefully fail much better than it does under load, especially for supposedly being Google-scale. With more users, instead of not loading at all, or starting then failing to load, it would be nice to see the most recent change in read-only mode, and show an edit timeline like the Internet Archive's Wayback Machine UI shows archive occurrences on a calendar. Just a couple ideas for how Sheets could break down gracefully while retaining usability.
For a crisis-response, an approach may reach maximum impact within days and really only be needed for a short duration (a week to a month). Migrating a system with intense user/volunteer churn (and keeping two databases synced as you transition) can be tricky. Handling permissioning at the same time is also a challenge.
At FindTheMasks, we expected the government to have supply chains and epidemiological response (testing/contact tracing) together within two or three weeks. When that didn't happen, we kept bolting on new tools and strategies to double the lifetime of the endeavor, expecting the government to get its poop in a group by then. 20 weeks later, supply chains are somewhat improved, though the entire situation is compromised by the lack of overwhelming success at the epidemiological scale.
It's hard to go from Google Sheets to a web frontend to Postgresql and a snappy shared frontend. That is the problem statement to solve for imho (interface, schema, data portability).
As you tack on extra tabs and handle more fields, it isn't too hard to approach that limit.
If we do trip that threshold, we will be forced to degrade some functionality.
As a bonus, I don't believe that Sheets allows anyone to see exactly how many cells are in use, so one lives with a probabilistic risk of lockup.
If we were continuing to handle large volumes of intake, a switch to Airtable would be in the works. Fortunately, the PPE situation is at least somewhat better than it was in March, so we are simply keeping the lights on and wheels turning.
Airtable-wise, I'm parroting our devs on the subject. We had perpetual/nascent plans for an Airtable migration (and did so when FindTheMakers came to life) which have been tabled for now. We believe we'll make it to sunset on Sheets, but just barely.
Is there some special meaning of "record" in their vernacular?
I haven't used either before but cryptpad seems solid. Here's a demo sheet https://cryptpad.fr/sheet/#/2/sheet/edit/DirOb8rZr+6gfi5PCrS...
I sat there thinking "I can't believe there isn't a better option for this sort of thing." Kind of surprised Zoom does not have a native feature for meeting participants to share contact info with each other during the meeting.
A custom web app for this purpose can scale better by being designed with the knowledge each user only edits their own row and nothing else needs to be recalculated. And of course it can have a tailored UI.
Unfortunately I don't know of any way for a non-developer to match that. There's no fundamental reason IMHO that there couldn't be a nice rapid, code-optional CRUD app generator, like Microsoft Access for the 21st century. Maybe this is what Microsoft Lists will do? AirTable is a general-purpose web-based RDBMS frontend, but I don't know if it scales well to many simultaneous editors either, I don't think you can tailor the UI as much as I'd like, and I hear folks complaining about database size limits.
You can get Google Sheets to scale better, but it's a bit awkward:
* Give folks a Google Forms link to edit their own information. This atomically claims and populates the row.
* Give folks a publish link (see File | Publish To the Web) to view everyone else's information. This link scales well because it doesn't let them update anything and (more annoyingly) it doesn't show changes until they hit refresh.
That's a big number, but it's not as big as I'd expect from something from Google. I'd assume the average number of people working on any Sheet is 1 or 2, but I'm surprised that haven't been able to scale it up to 1000 or more by now.
The technical limitation is 2-4 times what the former product manager believed was a human limitation already. Scaling beyond that doesn't seem like a priority.
Google Docs/Sheets/Drive function a lot like Stardew Valley multiplayer, in that pretty much anyone can do anything, and this allows really fluid communication but it simply won't work with 100 people.
Sheets supports 100s of people to an extent (even more) but I wouldn't want them to optimize for this. I think if I were in charge of it I would be exploring ways that they can find UI paradigms that allow more seamless interaction and realize the dreams that I think Google had when they built Wave.
* What if you could embed live video/audio in a Doc?
* Sheets comments kind of suck.
* Is there a way to use comments that makes them more visible without ruining the sheet view? (Maybe not.)
* As this article says, people are often using a Sheet as a very hacky app dev platform. Is there a way to embed the sheet data in a Doc (or something Wave-like) that has a more seamless communication pattern?
* Is there a way to build small apps on the fly without messing with a bunch of macros?
Though I think a lot of this is about communication. I don't think Google should be thinking about scaling, they should be thinking about those use cases where you're talking to a small group of people and you need a little custom app that can't take more than 30 seconds to put together or just talking would be more efficient.
That being said, this article seems to completely ignore the fact that there are already well-established solutions to this problem. Off the top of my head, wikis and GitHub/Lab pages offer very nice solutions to creating a scale-able information-base with built-in features to help address some of the challenges I raised above....
It's a flippant answer, sure, but spreadsheets are the index cards, sticky notes, and yarn approach to data organization. Certainly they are easy and approachable to use, but any sufficiently complex data set in general is going to be more "chaotic mind corkboard mess" than "organized".
Spreadsheets are very useful data hammer, but not all data is a nail; to borrow another aphorism.
That said, there's a wide gulf in the general user approachability between spreadsheet applications and more bespoke/"better" data organization options. Decades of experiences and mistakes learned from tools like DB2, FoxPro, and Access have shown us we have a hard time finding an "easy medium". AirTable thinks they have some modern ideas. Microsoft's PowerApps teams have their own modern approaches (and as almost the direct the inheritors of FoxPro/Access, presumably the most experience to learn from). Simon Willison's work with Datasette (SQLite DBs wrapped in simple view/management webpages) has had some interesting ideas. Though I don't think any one approach will solve it, it might take a whole shelf of tools, and at the end of the day maybe as a species we'll never have as clean of a user experience as the worst spreadsheet.
https://www.theverge.com/2020/8/6/21355674/human-genes-renam...
edit: HN link
https://news.ycombinator.com/item?id=24070385
I wonder if that's why the new version of macOS Numbers defaults to a read-only mode — to ease the burden when the sheets are in collaboration.
I would love to see Elixir based solution just to see how much performance can be squeezed out of it, whatsapp seems to handle huge load just fine
Archve.is link: https://archive.is/N00i5
This got me thinking what would it take to build a decent collaborative editor that would work for hundreds of people.
The low hanging fruit would be adding a read-only view of a static copy, because that's what people actually needed.