News for hackers. Stuff that matters.
Re:The first rule of Microsoft Excel: Don’t tell anyone you’... by azalemeth on Monday October 24, 2022 @10:52AM (#33317439)
Also, the sort of formula shown in that article makes my skin crawl. Not because it's hard – it's mostly a set of sums based on conditional lookups with rounding – but because by the time you've written something like that, Excel is almost certainly the wrong paradigm for your data. It's a terrible IDE; making minor changes to that...thing will half as happily pop an error message and junk it as it will do the right thing. Similarly, you can't exactly git excel formulae alone, and the mixing of data and analysis inevitably causes chaos at some point – just look at all of the examples of financial disasters caused by it. I carte-blanch refuse to touch anybody's attempts at doing stats with it (in a medical science context), unless it also comes with R, sas or stata script..

What the article's really trying to say is "Being good at Excel implies you have a logical mind and the ability to see a way through problems. This is something that others lack".

Re:The first rule of Microsoft Excel: Don’t tell anyone you’... by kqr on Monday October 24, 2022 @11:12AM (#33317768)
Though I have to reach the heights of that formula, I keep coming back to Excel for the same reason that I come back to Perl. Not because it's always the best tool for the job, but because it's the lingua franca in many contexts. If I create the thing in Excel, then I know I can hand it off to someone else and they will both know how to run it, and, in a pinch, how to change it.

With any other tool, there would eventually be an onboarding process not just for the thing I built, but also for the tool I built it in. Not with Excel – that will just run, whereever you go.

Re:The first rule of Microsoft Excel: Don’t tell anyone you’... by Justin_K on Monday October 24, 2022 @11:39AM (#33318134)
Just send management a docker compose that will auto deploy your perl script!
Re:The first rule of Microsoft Excel: Don’t tell anyone you’... by Scarbutt on Monday October 24, 2022 @11:25AM (#33317932)
Where is Perl the lingua franca?
Re:The first rule of Microsoft Excel: Don’t tell anyone you’... by masklinn on Monday October 24, 2022 @11:36AM (#33318089)
The nightmare corpse-city of R'lyeh, deep under the waves, far from the glare of the yellow sun.
Re:The first rule of Microsoft Excel: Don’t tell anyone you’... by thesuperbigfrog on Monday October 24, 2022 @11:44AM (#33318211)
>> The nightmare corpse-city of R'lyeh, deep under the waves, far from the glare of the yellow sun.

No. That is not Perl. That is C Recursion: https://www.bobhobbs.com/files/kr_lovecraft.html

Re:The first rule of Microsoft Excel: Don’t tell anyone you’... by hollosi on Monday October 24, 2022 @12:50PM (#33319130)
Perl is preinstalled on pretty much every Linux and MacOS since time immemorial, and it's pretty much guaranteed to stay preinstalled for the foreseeable future.

Python3 is up and coming, and eventually it will get there, but old machines must die before it can reach ubiquity.

Re:The first rule of Microsoft Excel: Don’t tell anyone you’... by masklinn on Monday October 24, 2022 @01:00PM (#33319255)
> Perl is preinstalled on pretty much every Linux and MacOS since time immemorial

awk, bc, and sh are POSIX base utilities, they are literally preinstalled on every unix derivative.

That doesn't make them linga franca, which is what Scarbutt was asking about.

Hell, ed is a POSIX base utility. Does that make ed's command mode the linga franca of text manipulation? Of course not, the thought is preposterous.

Re:The first rule of Microsoft Excel: Don’t tell anyone you’... by melagonster on Monday October 24, 2022 @08:07PM (#33324401)
Is not this actually a famous advantage of vim/vi in old editor war?
Re:The first rule of Microsoft Excel: Don’t tell anyone you’... by masklinn on Tuesday October 25, 2022 @03:59AM (#33327526)
Yes?

That it is an advantage does not naturally and unfailingly translate into being a linga Franca.

Re:The first rule of Microsoft Excel: Don’t tell anyone you’... by Ferkv on Wednesday October 26, 2022 @04:56AM (#33341293)
It's the other way around.

"vi" being so common (ie. "lingua franca") is what gave it an advantage.

Re:The first rule of Microsoft Excel: Don’t tell anyone you’... by Ferkv on Tuesday October 25, 2022 @09:10AM (#33329992)
Then what would, in your opinion, make it "lingua franca"?

I took it to mean "so common that it's available almost anywhere" (which is consistent with the context of the comment).

I've found it extremely useful to rely in tools such as the ones you just listed. Specially because they are often present even in light vms were perl might not.

Re:The first rule of Microsoft Excel: Don’t tell anyone you’... by kqr on Tuesday October 25, 2022 @02:37PM (#33334508)
I don't find it preposterous at all.

Yes, I would consider awk and sh lingua francae (?) alongside Perl – but in slightly different contexts, namely low-complexity jobs.

I find Perl slightly more suitable once things get a little complicated, mainly because it's footgun:power ratio is better than awk (few footguns, but also very low power) and sh (many footguns, not that powerful).

I have come across plenty of boxen where bc was not installed, so I don't put that in that bucket.

And yes, if I want to describe text changes in a highly portable way, I will use sed (or ed, depending on context) rather than, say, unified diffs. I could accept an argument that unified diffs are also a lingua franca for more complicated changes, but they are harder to write.

Re:The first rule of Microsoft Excel: Don’t tell anyone you’... by hedora on Monday October 24, 2022 @01:50PM (#33319947)
It will be a long time before it makes sense to use python where longevity/portability matter.

Python breaks backwards compatibility with old scripts every six months or so. In practice, this means you have to port scripts every time you switch machines. Virtual environments sometimes address this issue, but they are hit or miss.

In contrast, perl scripts from 1999 often work unchanged on clean 2022 OS installs.

Re:The first rule of Microsoft Excel: Don’t tell anyone you’... by uid1357 on Tuesday October 25, 2022 @04:22AM (#33327704)
Is there even such a thing as lingua franca in any given field?
Re:The first rule of Microsoft Excel: Don’t tell anyone you’... by marcosdumay on Monday October 24, 2022 @11:49AM (#33318282)
> I can hand it off to someone else and they will both know how to run it, and, in a pinch, how to change it

How to change it, yes. How to get the correct result, probably not.

But well, they will get a result. Many people are satisfied with that.

Re:The first rule of Microsoft Excel: Don’t tell anyone you’... by treis on Monday October 24, 2022 @11:58AM (#33318416)
It's too bad that browsers don't have a better story for local development. For reasonable reasons the browser has always been internet focused but it's the one thing that everyone has and runs anywhere these days. But if you try to write some quick JS/HTML thing it won't work when you point your browser at the file.
Re:The first rule of Microsoft Excel: Don’t tell anyone you’... by johannes1234321 on Monday October 24, 2022 @02:50PM (#33320732)
> then I know I can hand it off to someone else and they will both know how to run it, and, in a pinch, how to change it.

I'm not sure a receptive t will be able to successfully edit a spread sheet. So easy, to get interdependency between cells to a degree that it is really hard to follow, if you didn't create it yourself.

Or did I just describe Perl? Not sure.

Re:The first rule of Microsoft Excel: Don’t tell anyone you’... by 2rsf on Monday October 24, 2022 @11:02AM (#33317614)
I fully understand, but what would you recommended to an accountant? a business analyst without technical aptitude? a business owner, again with zero programming aptitude? Excel is intuitive to begin with, it gets ugly fast and almost impossible to debug but for small sets of data it is very easy to explain how to use it and you get results fast.
Re:The first rule of Microsoft Excel: Don’t tell anyone you’... by RalfWausE on Monday October 24, 2022 @11:14AM (#33317797)
Seriously: APL!

Yeah, it takes a bit of time to become really comfortable, but our accountant was (after the initial take-it-away! phase) really glad i have her introduced to it

Re:The first rule of Microsoft Excel: Don’t tell anyone you’... by capableweb on Monday October 24, 2022 @11:36AM (#33318094)
I'd hesitate recommend a language whose symbols are not easily accessible or understandable. Try showing the following snippet to even a seasoned programmer and they'll struggle to understand what it does:

    Avg←{(+⌿⍵)÷≢⍵}
Also, if you want a non-programmer to quickly be able to learn a language, the more learning resources they have available to them, the more probable it is that they'll be able to pick it up. APL probably don't have as many modern, free and online resources as other languages.
Re:The first rule of Microsoft Excel: Don’t tell anyone you’... by jxy on Monday October 24, 2022 @12:01PM (#33318448)
Yet you cherry picked an extension that's not in the ISO standard and only modern APL supports.
Re:The first rule of Microsoft Excel: Don’t tell anyone you’... by capableweb on Monday October 24, 2022 @01:14PM (#33319436)
Sorry about that, I don't APL well myself to know what parts belong to what. I simply searched for "APL hello world" and landed on https://tryapl.org/ which has that snippet on the left side.

The "Examples" section on the Wikipedia page for APL also contains a bunch of symbols I would have no idea how to enter on a computer: https://en.wikipedia.org/wiki/APL_(programming_language)#Exa...

Re:The first rule of Microsoft Excel: Don’t tell anyone you’... by jxy on Wednesday October 26, 2022 @11:00PM (#33352550)
The whole point is not whether you, without any learning, would have any idea how to read, speak, think in any specific language. You likely have forgot how long you had to learn English. Do you know how many people in the world have no idea what you had written here, yet they are still very comfortable in using a computer?

It's easy for an English speaking person to consider programming languages in English as the only kind that's usable. To be fair, English dominates in the tech world. But it would be narrow minded to dismiss a programming language solely based on its non-English symbols.

Mathematicians have long been using symbols for efficiency in its expressiveness. Why not programming languages? I'd like to keep an open mind.

Re:The first rule of Microsoft Excel: Don’t tell anyone you’... by capableweb on Thursday October 27, 2022 @02:43PM (#33362098)
I'm not arguing about English vs non-English programming languages, I'm talking about symbols available on the keyboard. When you start out programming, you're being met with symbol you probably haven't extensively used before, such as the pipe ("|") or even curly braces, as you usually don't come across those in prose. But you can look down on a keyboard and see how you can enter those symbols.

Contrast that to APL which has you enter symbols that are nowhere to be found on a keyboard. You need to manually look up how to enter the symbols, or run specific software to have it on-screen, and if you forget, you can't just look at the keyboard, you need to look it up again.

I'm sure it becomes second-nature after a while, just like entering pipe characters or curly braces. But it's definitely harder to get started if there is more to remember than just concepts.

I have not forgotten how long I had to learn English, as it's not my mother language and I had to learn it in school at a certain age. I'm also currently learning a fourth (speaking/writing/"human") language currently, so I'm well aware about that there are other languages out there.

I'm also not arguing against APL as a language as a whole, I'm personally curious about it as well and will probably give it a go as it's currently missing from my repertoire of languages, which is wide already but still missing things. I'm simply arguing against it as a first programming language for people to learn, compared to languages that don't contain symbols you cannot "normally" enter on a traditional keyboard.

Re:The first rule of Microsoft Excel: Don’t tell anyone you’... by ulucs on Monday October 24, 2022 @12:30PM (#33318846)
wait, doesn't APL have forks? Avg doesn't get more readable than (+/)÷≢ (sum over divided by tally) as long as you know what the signs do.
Re:The first rule of Microsoft Excel: Don’t tell anyone you’... by schwartzworld on Monday October 24, 2022 @12:39PM (#33318977)
Not using a tool because people might not know the tool already is an interesting take. Learning a few extra symbols isn't as much of a bottleneck as you think. Heck, people learn to read entire new alphabets, I'm learning one right now and that's a task much harder than learning some APL glyphs.

> APL probably don't have as many modern, free and online resources as other languages.

Not as many, but that can be a good thing too. Lots of languages have tutorials, libraries and articles that use horrible practices. I write React + TypeScript for work, and it's an especially big problem.

Re:The first rule of Microsoft Excel: Don’t tell anyone you’... by capableweb on Monday October 24, 2022 @01:12PM (#33319414)
> Not using a tool because people might not know the tool already is an interesting take

I guess it would, that's not what I said though. Most programming languages use symbols already visible and available on your keyboard without having to do anything extra. APL uses a bunch of symbols you either need to learn a new keyboard combinations to enter, or otherwise extra software you probably haven't used before unless you're a mathematician.

> I'm learning one right now and that's a task much harder than learning some APL glyphs

I agree. Learning new complex glyphs is probably easier than new complex concepts, but what's for sure is harder is learning both at the same time rather than just one.

Using a language like Python or any other mainstream language would at least make the person not have to worry so much about new symbols, but mostly just new concepts.

Re:The first rule of Microsoft Excel: Don’t tell anyone you’... by sudosysgen on Monday October 24, 2022 @11:50AM (#33318306)
There is a modern successor to APL often used in finance called Q from kx. A lot of analysts swear by it, but personally I prefer Julia as, like APL, it's a bit write only.
Re:The first rule of Microsoft Excel: Don’t tell anyone you’... by aquariusDue on Monday October 24, 2022 @01:05PM (#33319320)
The more I look into Julia the more interesting it seems. There's a YouTube series by doggo dot jl that showcases its many uses.
Re:The first rule of Microsoft Excel: Don’t tell anyone you’... by wrycoder on Monday October 24, 2022 @02:44PM (#33320674)
After you leave, and later the accountant leaves, that business is going to be in a world of hurt trying to find an accountant who understands APL (or an APL programmer plus an accountant who together can understand the previous accountant's version of APL program design).

Speaking as an old, enthusiastic APL programmer.

Re:The first rule of Microsoft Excel: Don’t tell anyone you’... by RalfWausE on Tuesday October 25, 2022 @01:14AM (#33326511)
We are a small-ish business with a really special park of custom build machines, using tons of (over decades) custom written software... THIS is the reason we train our own replacements and have at any time aprenticeships going.
Re:The first rule of Microsoft Excel: Don’t tell anyone you’... by worthless-trash on Monday October 24, 2022 @10:04PM (#33325289)
I like the old langauges, probably more than the next guy.

Why did you choose APL over something like GnuCOBOL. Ive been pleasantly surprised with how teachable Cobol is to a range of people.

Re:The first rule of Microsoft Excel: Don’t tell anyone you’... by RalfWausE on Tuesday October 25, 2022 @12:59AM (#33326420)
COBOL would also have been a great choice, especially due to its ...ehm...'BUSINESS ORIENTATION'

I did chose to show her APL simply because i am somewhat firm in it myself, so it was more simple to teach it someone else.

Re:The first rule of Microsoft Excel: Don’t tell anyone you’... by ungamedplayer on Tuesday October 25, 2022 @07:43PM (#33337736)
That makez perfect sense. If you know it and can assist, why not.
Re:The first rule of Microsoft Excel: Don’t tell anyone you’... by golergka on Monday October 24, 2022 @11:11AM (#33317746)
They could just use Visual Basic right in Excel instead.
Re:The first rule of Microsoft Excel: Don’t tell anyone you’... by TheLoafOfBread on Monday October 24, 2022 @11:23AM (#33317917)
Or they could install Visual Studio and start writing .NET applications. Common, regular users often does not even know that Excel can do formulas and are filling single cells with calculator manually.
Re:The first rule of Microsoft Excel: Don’t tell anyone you’... by pjmlp on Monday October 24, 2022 @11:35AM (#33318080)
Actually when I went back to desktop Windows development for a while on life sciences, that is exactly what many researchers did after they outgrew VBA, asking IT to get VB.NET on their desktops.

No R, no Python, VB.NET it was.

Re:The first rule of Microsoft Excel: Don’t tell anyone you’... by CorrectHorseBat on Monday October 24, 2022 @11:40AM (#33318148)
If that makes your skin crawl, how would you feel like if I told you it's very likely that you a have a chip made with Excel in your pocket?
Re:The first rule of Microsoft Excel: Don’t tell anyone you’... by thfuran on Monday October 24, 2022 @11:56AM (#33318387)
I would tell you that I don't even know what it means for a chip to be made with excel and ask you to leave before you ruin that.
Re:The first rule of Microsoft Excel: Don’t tell anyone you’... by dahart on Monday October 24, 2022 @12:08PM (#33318542)
Can confirm parent’s comment; some hardware design teams run on Excel, for a variety of core planning activities. They aren’t drawing traces in Excel, but they are designing pipeline scheduling and doing Tetris to fit all the reusable different compute units together, among other things.
Re:The first rule of Microsoft Excel: Don’t tell anyone you’... by giaour on Monday October 24, 2022 @11:16AM (#33317818)
The way Excel lays out data and intermediate states visually is pretty intuitive (compared to, say, using a step-through debugger), but the formula editing pane has an atrocious UX. I would love to see Excel but with a scratch/snap-like interface for authoring formulas.
Re:The first rule of Microsoft Excel: Don’t tell anyone you’... by nerdponx on Monday October 24, 2022 @11:28AM (#33317980)
The more I use Excel, the more I feel that its primary limitation is its UX for working with formulas.
Re:The first rule of Microsoft Excel: Don’t tell anyone you’... by willhslade on Monday October 24, 2022 @12:21PM (#33318709)
Check out the new LET function.
Re:The first rule of Microsoft Excel: Don’t tell anyone you’... by rbonvall on Monday October 24, 2022 @01:07PM (#33319344)
LET is a great syntactic improvement, but the UI is still the same. I'd love if the formula box supported the basic things one expects from a code editor, like syntax highlighting, formatting, indentation, comments, etc.
Re:The first rule of Microsoft Excel: Don’t tell anyone you’... by criddell on Monday October 24, 2022 @11:44AM (#33318202)
The formula editor is pretty bad. If Microsoft fixed it and added GitHub integration easy enough for non-programmers, I think a lot of concerns about usage of Excel could be dealt with.
Re:The first rule of Microsoft Excel: Don’t tell anyone you’... by Xeamek on Monday October 24, 2022 @11:58AM (#33318406)
This is actually great idea. If microsoft can add block-building programming for minecraft edu edition, why wont they do that to their flagship tool that would also help them in their competition against Google sheets.

Especially since pretty much every year we hear another story where some small Excell formula bug cost company millions of dollars. Redesining Excel formula pane should be their number 1 feature in 'todo'.

...unless they still care to add big features at all, and they aren't just in maintenance mode like most of their stuff

Re:The first rule of Microsoft Excel: Don’t tell anyone you’... by rahimnathwani on Monday October 24, 2022 @12:12PM (#33318595)
> I would love to see Excel but with a scratch/snap-like interface for authoring formulas.

Long/complex Excel formulae should usually be broken into smaller chunks, with intermediate results stored in separate cells. For example, if you're calculating two numbers, and then calculating their ratio, it would be better to use three formulae instead of one. That way:

1. Each formula is shorter.

2. Each formula has a single purpose which can be understood.

3. The outputs of the intermediate steps in the calculation are obvious.

This applies also to complex nested IF() statements. Instead of calculating all the conditions inside the IF(), calculate them outside, and then reference the TRUE/FALSE cell values in your IF statement. When you need to debug why you're not getting the result you want, you can easily look at the intermediate calculations all at once, without needing to step through the calculation and check it in the order of calculation.

Re:The first rule of Microsoft Excel: Don’t tell anyone you’... by giaour on Monday October 24, 2022 @12:46PM (#33319072)
Totally agree. This is exactly what I would do to debug a complex expression in a program: break out chunks into smaller expressions, assign their results to intermediate variables, then echo out or inspect those variables in a debugger. Once it clicks with users that an Excel cell is a variable, they tend to write more state out to the sheet so they can inspect it.
Re:The first rule of Microsoft Excel: Don’t tell anyone you’... by simulo on Monday October 24, 2022 @12:16PM (#33318640)
Bas Jansen and Felienne Hermans created a block based language, XLBlocks. Read their paper here: https://www.researchgate.net/publication/336187404_XLBlocks_...
Re:The first rule of Microsoft Excel: Don’t tell anyone you’... by sicp-enjoyer on Monday October 24, 2022 @11:23AM (#33317910)
Everything else in their business ecosystem is in Excel, and that's where people are trained. It's maybe less convenient to make it work in that environment, but the fact that they can, despite those constraints is very valuable.
Re:The first rule of Microsoft Excel: Don’t tell anyone you’... by habibur on Monday October 24, 2022 @11:21AM (#33317882)
We could write VisualBasic script for excel. Write this whole formula in a function, indenting and line breaking as is common, and then call that function with parameters from that excel sheet.

Not sure if the current web/JS based excel still supports those things.

Re:The first rule of Microsoft Excel: Don’t tell anyone you’... by HyperSane on Monday October 24, 2022 @11:05AM (#33317646)
Python + SQLite should replace a huge amount of Excel uses.
Re:The first rule of Microsoft Excel: Don’t tell anyone you’... by FredPret on Monday October 24, 2022 @11:31AM (#33318027)
With respect, and as a guy who learnt Python to work with Pandas on datasets too large for Excel, I'm not so sure.

Excel allows non-programmers, of whom there are very many, to visually explore and play with their data. I was already a programmer when I learnt Python; it took many years to learn those skills.

Non-programmers usually do not even want to think about their data in an abstract and general way anyway.

Re:The first rule of Microsoft Excel: Don’t tell anyone you’... by lofatdairy on Monday October 24, 2022 @12:27PM (#33318810)
I completely agree, drag and drop is a hard habit to break for a lot of people who have never coded before. I will say Pandas is pretty easy to learn (obviously mileage may vary but I found it far more syntactically consistent then it's cousin Numpy), but not being able to see what you're doing makes it hard to just jump in and work, you probably will start on small toy datasets and work up to working blind on larger datasets.
Re:The first rule of Microsoft Excel: Don’t tell anyone you’... by eddieroger on Monday October 24, 2022 @11:22AM (#33317890)
That's really dismissive of the learning curve of learning two things versus one, as well as having the expertise to install Python and maybe SQLite versus Office being installed on most computers already. Python and SQLite also have no portability as compared to emailing someone an Excel sheet. I love writing SQL whenever I have the chance, but "should" is a really strong word here.
Re:The first rule of Microsoft Excel: Don’t tell anyone you’... by D13Fd on Monday October 24, 2022 @11:31AM (#33318031)
It's not just learning two things vs. one. It's learning a text-, command-, and script-based interface vs. a mostly visual one.

You could likewise say "a huge amount of file operations should be performed with rsync" and just ignore the fact that it's much harder to learn than a drag-and-drop setup.

Re:The first rule of Microsoft Excel: Don’t tell anyone you’... by CJefferson on Monday October 24, 2022 @11:29AM (#33317998)
I would love that to be true, but I'll be honest -- I consider myself a good programmer, but a couple of times I've tried Python + SQLite and ended up just giving up and using csv or JSON.

Is there a nice library which makes it easy to use SQLite in Python even close to as easy as Excel? Or even as easy as Python lists? Anything which requires me to fill my Python code with SQL strings is (in my opinion) disqualified.

Re:The first rule of Microsoft Excel: Don’t tell anyone you’... by erikcw on Monday October 24, 2022 @01:36PM (#33319741)
I recently inherited a codebase that uses Pony ORM[0]. Might be worth taking a look if you want your interactions to look as similar to vanilla python expressions as possible.

[0] https://ponyorm.org/

Re:The first rule of Microsoft Excel: Don’t tell anyone you’... by lofatdairy on Monday October 24, 2022 @11:18AM (#33317841)
I personally vote R + Tidyverse only because I know a ton of older people in data/stats background who have done work in SAS and R, but struggle a bit with knowing what to do in Python. If all you're doing is tabular data all day, R handles that pretty well by default and RStudio is a much better IDE than even a Jupyter notebook for a more interactive paradigm.
Re:The first rule of Microsoft Excel: Don’t tell anyone you’... by oogway8020 on Monday October 24, 2022 @12:29PM (#33318830)
Tidyverse is awesome, RStudio too
Re:The first rule of Microsoft Excel: Don’t tell anyone you’... by yuliyp on Monday October 24, 2022 @11:47AM (#33318251)
Hardly. I can often do in Excel in 5 minutes what would take me an hour or two with Python.

I've got some output from something or a table of data I can copy-paste into Excel, quickly do some text-to-columns or string manipulation (tweaking it instantly if I see things don't look right) to get the data into a usable format, then go and build up some formulas, sorts, and plots to answer whatever question I had / get something I can share with others.

Each of those steps is doable with Python + SQLite, but takes more time: I have to write my own data parser (yes, Python's string utils are very good for this, but generally Excel is faster to use, despite using Python for years for my job. I have to then create a schema for the data, and then I have to iterate on my analysis functions, and then finally write my own output / futz around with a graphing library, rerunning whenever I want to tweak something (instead of just changing a cell or chart parameter and seeing it instantly reflected on screen.

There is a crossover point where the complexity of the analysis, amount of data, or otherwise makes Excel less practical, but it absolutely has a place as a super-helpful tool, even for those capable of using more powerful general purpose programming languages.

Re:The first rule of Microsoft Excel: Don’t tell anyone you’... by simonw on Monday October 24, 2022 @12:00PM (#33318440)
"I have to then create a schema for the data"

You might find my sqlite-utils Python library interesting - one of its key features is that it can create the right schema for you automatically to fit the data (as a Python dict or list of dicts) that you pass to it: https://sqlite-utils.datasette.io/en/stable/python-api.html#...

Re:The first rule of Microsoft Excel: Don’t tell anyone you’... by HyperSane on Monday October 24, 2022 @04:11PM (#33321945)
I didn't say ALL uses. Just the ones where Excel is being used as a crappy combination of programming language and database.
Re:The first rule of Microsoft Excel: Don’t tell anyone you’... by LeifCarrotson on Monday October 24, 2022 @12:00PM (#33318438)
If I'm going to be the one to use and maintain it? If it's a black box to my customer? If I expect considerable complications, extensions, scope creep, performance needs? Absolutely. Maybe I'd use C# instead of Python to simplify installation on the non-developer Windows machines I'm surrounded by, but that depends on the context. A little Node app on the LAN that everyone can access with their desktop or mobile browser can be nice too, though browsers keep breaking stuff in the name of security so I've stopped doing that much. I'm also a fan of domain-specific language config files written in Lua; there are many better ways to build powerful, extensible, useful applications than an overloaded Excel sheet.

But if those things are not true, if I'm going to be the one to test out an idea and hand something off to my coworker or customer, building in Excel means they're likely to figure it out instead of their eyes glazing over when I explain that they need to install Jupyter, or a text editor, or Visual Studio, or to modify the Lua config file, or invoke something from the command line...they're not going to do any of those things, they're calling me when ~~it breaks~~ their inputs and requirements change.

It's similar in my primary role building industrial manufacturing control systems. Rockwell Automation is a terrible company with asinine licensing fees and outdated technology...but they're the safe choice if you want to transfer system ownership and not get calls about that one machine with the "Linux NUC" for the next 20 years.

Re:The first rule of Microsoft Excel: Don’t tell anyone you’... by dahart on Monday October 24, 2022 @12:16PM (#33318643)
Hehe, learning how to use Pivot Tables should replace a huge amount of Python uses. I say that as a huge advocate for Python!

Of course, this really comes down to using the right tool for the job. Python should be used for automating batch processes, and using Excel manually should be used for data exploration. Jumping into automation before you know exactly what you want is premature, and doing something manually over and over again is wasting time. Us programmers sometimes reach for code writing and automation before we’re actually ready, and waste time writing the wrong code. (I am guilty of this.)

Re:The first rule of Microsoft Excel: Don’t tell anyone you’... by lelanthran on Monday October 24, 2022 @02:18PM (#33320322)
> Python + SQLite should replace a huge amount of Excel uses.

It won't even replace the edge-case uses, n/mind the majority of uses.

Re:The first rule of Microsoft Excel: Don’t tell anyone you’... by somat on Tuesday October 25, 2022 @05:36AM (#33328287)
Excel combines two things that most environments keep separate. storing data. and calculation/logic.

I have this theory that a relational database management system is basicly a grown up spreadsheet. and it is, it does great, far better that excel at storing and querying data. And you have better choices for logic, from views to full blown stored procedures(most will probably use an external program instead). all of which keep your logic out of your data.

Re:The first rule of Microsoft Excel: Don’t tell anyone you’... by dd36 on Monday October 24, 2022 @11:05AM (#33317658)
It really needs a debug formula feature that allows you to look at a formula in a more readable format.
Re:The first rule of Microsoft Excel: Don’t tell anyone you’... by cultofmetatron on Monday October 24, 2022 @11:08AM (#33317706)
prolly a bit outside the mainstream but -> https://gtoolkit.com/
Re:The first rule of Microsoft Excel: Don’t tell anyone you’... by ricardonunez on Monday October 24, 2022 @04:15PM (#33322008)
I took a quick look and it reminds me Observable.
Re:The first rule of Microsoft Excel: Don’t tell anyone you’... by spywaregorilla on Monday October 24, 2022 @11:43AM (#33318190)
At a glance it just looks amatuerish too. You can see they're constantly checking the cell D2 and switching on whether it falls under a value 1-5 or so.

This should be 5 cells calculating an output uniquely for each possible value of D2. Then picking the appropriate one later on.

Re:The first rule of Microsoft Excel: Don’t tell anyone you’... by BitwiseFool on Monday October 24, 2022 @11:44AM (#33318204)
Excel is not an IDE, but I find it to be the most accessible thing for the average office worker who needs to manipulate data. Not optimal, mind you, but in terms of someone being able to jump in and start working with data it is unmatched.
Re:The first rule of Microsoft Excel: Don’t tell anyone you’... by driscoll42 on Monday October 24, 2022 @01:23PM (#33319576)
Don't forget just about everyone has it installed. Want to send a report to your boss? Email it. Build out something in an ID using .NET or something else? Well if they want to make changes hope they have it installed and are familiar with the syntx.
Re:The first rule of Microsoft Excel: Don’t tell anyone you’... by 4oo4 on Monday October 24, 2022 @11:50AM (#33318304)
This 100%. My hell for a couple years was being an Excel forms "developer" as part of my job duties, and maintaining an extremely complex Excel spreadsheet (that should have been a web application) that I inherited from my predecessor. I wish I had taken screenshots but I think some of the most complex formulas in it had about a dozen levels of branching in them, and to make it worse it was used for a business critical process related to documentation requirements for consumer mortgage loans. The formula shown in the screenshot is representative, though to make it more manageable you can hide and label intermediate calculations in hidden columns and sheets. It looks like they may have done some of that based on the cell references, but they could still offload a lot more of that to different places.

There were days I spent several hours clicking "Evaluate Formula" button close to 100 times in a row to debug things like this. The syntax colorization definitely helps, but I would have killed for just the ability to pretty print a formula or set a breakpoint. I'm always shocked it doesn't have features like this that programmers depend on, but I wonder if part of that is because the people that would write something this complex in Excel have never experienced all the tools you get with "real" programming languages.

At home I was getting deeper into Linux and bash/python, and that made me resent Excel even more. I finally moved into the IT department and got really into PowerShell and training myself as a sysadmin, but I would still get requests about that damn spreadsheet for at least a year or two until they finally stopped asking. I used to enjoy problem solving challenges with Excel for a while, but in the end it became a parasitic relationship because of the time needed to fix issues and the complete lack of ownership and knowledge within the department that supposedly owned the form and process.

Critical decisions are made based on Excel formulas like these, (people getting fired, financial decisions, etc.), maintained by people that barely understand programming. Yes, it's nice that it's helped democratize automation and coding for non-technical people, but Excel just wasn't designed for things this complex. When your Excel formulas and macros start to look like this, you need better tools to manage the complexity and trust the maintenance to tech people instead of business people that don't fully grasp the monsters they're creating when they just keep using the only tool they know.

Re:The first rule of Microsoft Excel: Don’t tell anyone you’... by HPsquared on Monday October 24, 2022 @12:25PM (#33318770)
There are some tools available to pretty print Excel formulas, e.g. https://www.excelformulabeautifier.com/
Re:The first rule of Microsoft Excel: Don’t tell anyone you’... by la64710 on Monday October 24, 2022 @11:54AM (#33318354)
Microsoft excels in encouraging their users to be dumb and dumber. Just out of the kindness of their hearts.
Re:The first rule of Microsoft Excel: Don’t tell anyone you’... by la64710 on Saturday October 29, 2022 @02:59PM (#33386962)
Not sure why it got downvoted. It’s not necessarily a negative comment. Their business runs on their customers being dumber than yesterday.
Re:The first rule of Microsoft Excel: Don’t tell anyone you’... by counttheforks on Monday October 24, 2022 @12:00PM (#33318435)
I feel like Excel is almost there. If only you could save a sheet as a folder, where each formula was its own file and Excel would hot reload the changes made to those files. Then you might even be able to track the formulas in git.
Re:The first rule of Microsoft Excel: Don’t tell anyone you’... by HPsquared on Monday October 24, 2022 @12:27PM (#33318813)
Microsoft actually does offer a diff tool in some versions of Office, "Spreadsheet Compare", you can filter to look at e.g. formula changes (or any other area of differences between two Excel files).

Annoying it's not offered in most versions.

Re:The first rule of Microsoft Excel: Don’t tell anyone you’... by pbhjpbhj on Monday October 24, 2022 @12:10PM (#33318575)
I'd rather a "dual view" sheet, where each cell may shows the formula in the top half and the result in the bottom half. Save the CSV of the formulae & data view to your versioning system.

I guess you could have a third sub-cell for style?

Re:The first rule of Microsoft Excel: Don’t tell anyone you’... by HPsquared on Monday October 24, 2022 @12:20PM (#33318703)
A bit of indentation would help a lot. I tend to do this by hand, but there are automatic tools also, e.g. https://www.excelformulabeautifier.com/
Re:The first rule of Microsoft Excel: Don’t tell anyone you’... by Tagbert on Monday October 24, 2022 @02:29PM (#33320482)
Yes, that formula did give me the heebie-geebies.

When I need to do a complex calculation like that in Excel, I break it up into multiple cells each doing a subset of the calculation, then I have a result cell that brings the results together at the end. that way I can work on manageable formulas and can check the results of each. It’s probably less efficient from a processing and memory standpoint but I rarely work on large data sets and the gains in reliability are well worth it.

Re:The first rule of Microsoft Excel: Don’t tell anyone you’... by joeman1000 on Tuesday October 25, 2022 @04:33AM (#33327803)
The biggest issue is just how badly you can fuck something up by propagating a bad formula, then changing it somewhere and not updating what was propagated. It obscures the most important thing: the logic behind the calculations. Who in gods name wants to see just columns and columns of data? For what reason? I really want excel/spreadsheets to die a horrible death. They’ve ruined engineering as far as I’m concerned. Try to do an engineering degree without 12 different people forcing you to wade through their hare-brained idiosyncratic fuckfest of an excel file when what they want is probably two lines in python. I want to smack people who can’t just into dataframes. For gods sake.