News for hackers. Stuff that matters.
Re:Finance Pros Say You’ll Have to Pry Excel Out of Their Co... by d--b on Thursday November 30, 2017 @05:02PM (#15819998)
Finance dev guy here.

Excel's dominance in the field is because it is an _application container_ that _non_ dev people can use.

The workflow is this:

- old trader guy says to his junior guy: "hey can you look into xxx."

- junior trader guy says: "sure I'll make a spreadsheet for it"

- old trader guy: "great your model is all I need, let's trade"

- several weeks later, IT guy says: "hey you're running a $100m book out of a spreadsheet, we'll make you a nice system for it, cause your stuff will blow up."

- several months later the IT guy comes back with a web app that does the same thing as the spreadsheet.

- old trader guy says: "hey I can't copy shit around, my shortcuts aren't working, I need to be able to do basic maths on the side, I can't save my work, etc."

- IT guy: "ok I'll make you an export-to-Excel button"

Seriously I've seen this happen over and over again.

The issue is not how to get rid of Excel, it's how do we make a better spreadsheet...

Re:Finance Pros Say You’ll Have to Pry Excel Out of Their Co... by btilly on Thursday November 30, 2017 @06:53PM (#15820757)
At a former job I had to build a reporting system. And had the trust to build it however I wanted. But it was my job to make it sufficiently useful that everyone would use it.

The smartest thing that I did was make it accessible from Excel. You could build a spreadsheet off of my report. Refresh the spreadsheet, the report ran, you got updated data.

I got essentially 100% adoption, and the rest of my job was spent finding people who needed data and adding it as an option to the reporting system.

Usually you expect a complex reporting system to have features like graphing, pivot tables, etc, etc, etc. My answer to all of that was, "You can already do that in Excel. I could spend a lot of time on it but I'm not going to do it as well as what you already have."

Re:Finance Pros Say You’ll Have to Pry Excel Out of Their Co... by inopinatus on Thursday November 30, 2017 @10:06PM (#15821601)
The best non-core feature I ever added to my SaaS service remains live import/export to Google Sheets.

My customers are not finance people, but they were using ad-hoc spreadsheets extensively and most of them through G suite. By implementing bidirectional sync for both master data and outputs they have the best of both worlds.

The Google Drive and Sheets APIs (and especially the on-change push notification callbacks) are not a pretty sight but once the twisty maze of nested data structures, webhooks and OAUTH2 and what-have-you were successfully navigated we got an efficient, complete and stable solution that everyone's happy with.

Re:Finance Pros Say You’ll Have to Pry Excel Out of Their Co... by sdrinf on Thursday November 30, 2017 @09:14PM (#15821417)
This sounds an incredibly useful capability to have. May I ask which data connector did you used to pull data, and how did you made it available in that format from the app side?
Re:Finance Pros Say You’ll Have to Pry Excel Out of Their Co... by emidln on Thursday November 30, 2017 @09:55PM (#15821561)
Excel has a feature for pulling data stored in HTML tables into a sheet called "Web Queries"[0]. It also has a feature for automatically building these in the form of .iqy files. When I worked at Abbott Labs, it was a big deal to be able to offer export-to-Excel in a way that Excel could refresh automatically (or manually). This made it a breeze since we could register an .iqy serializer for a dataset and just give a download link to it in place of a CSV, JSON, or XML file.

[0] https://support.office.com/en-us/article/Get-external-data-f...

Re:Finance Pros Say You’ll Have to Pry Excel Out of Their Co... by btilly on Friday December 01, 2017 @03:15PM (#15826954)
That was exactly how I did it. The only major downside was that I had to be sure to keep the html consistent, and couldn't require login.

Since it was a system for internal use, that was deemed acceptable.

Re:Finance Pros Say You’ll Have to Pry Excel Out of Their Co... by cm2187 on Friday December 01, 2017 @02:52AM (#15822586)
If you can code, I suggest creating a custom addin with something like ExcelDNA. You offer an excel function to load the report in memory based on a given date, name or whatever. And a few functions to query that loaded report, that take as parameter the value returned by the load function to identify the report.

So that gives the user the ability to query for a certain date, or sum for a date range, or to get a breakdown by some meta data. Being able to load multiple reports allows them to do deltas. And switching to a newer version of the report just takes changing the argument of the load function and press F9.

Worth also giving the users sample spreadsheet, or having a way to export the full report with all these formulas in Excel so that they can learn by example.

The only thing to watch for is memory clean up. You need somehow to have a way to unload all reports otherwise your memory blows up over time.

Re:Finance Pros Say You’ll Have to Pry Excel Out of Their Co... by protomyth on Thursday November 30, 2017 @10:47PM (#15821761)
Yep, the most useful program I ever wrote for business was a Perl script that ran SQL against a Sybase database and sent the output as a spreadsheet. Multiple return sets went on multiple sheets. Took the column names from the return sets and did decent formating of data. Then the user would do all the fun stuff.
Re:Finance Pros Say You’ll Have to Pry Excel Out of Their Co... by cm2187 on Friday December 01, 2017 @02:39AM (#15822546)
That’s a very good approach. I found the best way to do that it to be able to query the report with Excel formula (which is probably what you did). It gives an incredible flexibility to the business to build tables and charts for whatever presentations they need to do.
Re:Finance Pros Say You’ll Have to Pry Excel Out of Their Co... by PeachPlum on Friday December 01, 2017 @04:51AM (#15822940)
Similar here.

My Julia code creates Excel sheets including charts and sparks

I use this shiv to Python's XlsxWriter

https://github.com/lawless-m/XlsxWriter.jl

With a bit of effort you can even make it set up scenarios for Excel solver

Re:Finance Pros Say You’ll Have to Pry Excel Out of Their Co... by intended on Thursday November 30, 2017 @11:47PM (#15821987)
As unpopular as it would be to say - Stop trying. Excel is a phenomenal accomplishment, and while everyone may talk about a better spread sheet, the market really doesn't want it or need it. Its basically craigslist.

The only people who think it can be or needs to be done better are primarily not the users.

Excel is the human condition, messy, varied, capable of doing many things and most of the features aren't used, and results are often horrifying. But it works.

That said, I want regex based search in excel, or better data cleaning tools.

Re:Finance Pros Say You’ll Have to Pry Excel Out of Their Co... by dredmorbius on Friday December 01, 2017 @01:54AM (#15822408)
The problem with Excel is that whilst it's an easy development environment, and a vastly flexible one, it's absolute death to debug.

At Uni, decades ago (and using Lotus-1-2-3 rather than Excel), we were given a set of spreadsheet templates for a business management exercise (the project ran much the duration of the course). I discovered looking at these that there was an error in the spreadsheet double-counting all inputs (rows and columns were added and the row-totals included in the grand total). Best I can tell, finding that bug was not in fact part of the course goal....

At an early gig doing analytics we hired an old fart who actually introduced some rigour to our project practices, including code walkthroughs. Some of this involved more obviously programmatic tools, but the process was also applied, over the strenuous objections of the analyst involved, to a project largely done in Excel. Numerous errors were found, and corrected. The spreadsheet was also significantly rationalised by having "dumb" data-entry area, and then relying on a sheet with VBA code rather than in-cell logic for the computations.

The analyst continued to object to the code review, after the errors were found and corrected, insisting that it was inappropriate because he "wasn't a programmer". (I looked him up recently, he's continuing to work on analytic projects, two decades on.)

This was also about the time that I first heard of Ray Panko, at the University of Hawaii, who'd found reliably that 88% of spreadsheets had errors. This still holds largely true.

http://panko.shidler.hawaii.edu/

I'm not saying "get rid of all Excel". But I am saying that the tool makes for very, very, very buggy processes, and this is itself a problem.

I'd love to see work toward alternatives that retain the flexibility and versatility of Excel (or other spreadsheets) but reduce the possibility of bugs and errors.

Re:Finance Pros Say You’ll Have to Pry Excel Out of Their Co... by flukus on Friday December 01, 2017 @02:10AM (#15822457)
> I discovered looking at these that there was an error in the spreadsheet double-counting all inputs (rows and columns were added and the row-totals included in the grand total). Best I can tell, finding that bug was not in fact part of the course goal....

Multiple times I've come across issues like this when developing the replacement for an old access system. The numbers between the new an old system don't match, so you have to dig through both to find the error. If the error is in the new version that's great, but when you tell the boss that the error is in the old one and they've been off by a few million dollars over the years...

Re:Finance Pros Say You’ll Have to Pry Excel Out of Their Co... by beobab on Friday December 01, 2017 @03:25AM (#15822688)
Been there: "No. The old is correct. Make the new way match."
Re:Finance Pros Say You’ll Have to Pry Excel Out of Their Co... by johnbellone on Friday December 01, 2017 @07:34AM (#15823436)
A mistake plus keleven gets you home by seven.
Re:Finance Pros Say You’ll Have to Pry Excel Out of Their Co... by GFischer on Friday December 01, 2017 @07:23AM (#15823403)
Yep, Excel is an easy way to lose a few million dollars.

However, development costs are so expensive they might have broken even when factoring the cost of creating a system.

Re:Finance Pros Say You’ll Have to Pry Excel Out of Their Co... by teekert on Friday December 01, 2017 @01:22AM (#15822306)
I agree and I have stopped trying as well. I'm in bioinformatics and I do the big data, Linux cluster, data pipeline stuff. Then in the end I dump it all in Excel and let the very smart and knowledgeable (lab) biologists play with it. It's important that they play and have flexibility to massage data as they see fit, they understand the data deeply. I started out constantly trying to figure out what they want and build it into the pipeline... It doesn't work because often they don't know what they want exactly when starting.

Now I just offer handy things like pre-sorting, filtering (some boolean columns to allow them to quickly filter stuff), better column names or reports with extra info. I try to figure out the things they find tedious and automate them once and for all. But I hold back as much as I can.

Re:Finance Pros Say You’ll Have to Pry Excel Out of Their Co... by lucozade on Friday December 01, 2017 @06:49AM (#15823290)
> Excel is a phenomenal accomplishment

Couldn't agree more. It's also a huge, latent liability. In my experience, mostly because it's an opaque data store.

To use the parent's example. Turn's out the $100mm business is actually only worth $80mm because someone had accidentally coded GBPUSD directly into the spreadsheet. Before Brexit was announced. Say goodbye to senior trader.

This sort of thing happens though not as much as it used to as we don't allow Excel in the path that hits the ledger.

Re:Finance Pros Say You’ll Have to Pry Excel Out of Their Co... by lxchase on Friday December 01, 2017 @11:14AM (#15824735)
There are a few regex based add-ins for excel that might work for your needs. (I haven't ran these myself however).

http://www.codedawn.com/excel-add-ins.php http://blog.malcolmp.com/2010/regular-expressions-excel-add-...

Re:Finance Pros Say You’ll Have to Pry Excel Out of Their Co... by skgoa on Friday December 01, 2017 @07:54AM (#15823493)
Yep and that's the same reason Matlab et.al. stay in their dominant positions. Sure, these tools can be a pain to work with and a lot of alternatives do a lot of things better, but most heavy users are happy doing the things they need to do.
Re:Finance Pros Say You’ll Have to Pry Excel Out of Their Co... by shostack on Thursday November 30, 2017 @06:26PM (#15820608)
>"I need to be able to do basic maths on the side"

I'm a marketer who echos the "pry it out of my cold dead hands" statement, and this is an interesting aspect of it that I hadn't thought as much about before.

There's really something to be said for having ALL of the data and model at your finger tips for quick sidebar calculations, and there's something to be said for formatting tools to make it easier to read and work with. Often times I'm building a model or report, and need to do some quick checks against different assumptions, or have a little scratch pad of notes or placeholder values somewhere.

Being able to just toss those into somewhere a few cells over so I can see things in context while I work is a godsend. If the model was built as something with a front-end that didn't expose everything, I'd have to manually copy that info out and just do it in a spreadsheet anyway.

I get that there are times when it absolutely makes sense to not have things live in a spreadsheet, and what the risks are around that. But there's also risk in slowing things down, or not having certain things immediately jump out at you because you're seeing all the numbers change, and can quickly modify things as needed to check yourself.

Re:Finance Pros Say You’ll Have to Pry Excel Out of Their Co... by djtriptych on Thursday November 30, 2017 @05:07PM (#15820030)
I think pragmatic businesses that employ finance pros / quants are wise to include Excel as an essential part of their product development workflow. Excel is a great prototyping tool, and people are going to use it whether you want them to or not. These folks are motivated by money and don't give a shit about your tool if (they think) it costs them money (or time). Besides that, how many enterprise applications really offer opportunity for prototyping and running small experiments? Excel is to modeling as wireframing is to UI design.

Someone should write a book about productionizing excel models. It's not going away (and shouldn't).

Re:Finance Pros Say You’ll Have to Pry Excel Out of Their Co... by d--b on Thursday November 30, 2017 @05:22PM (#15820154)
> Someone should write a book

Nah, this is not gonna work... The only way out is to make something that's somewhat like a spreadsheet but with source control, a nicer language etc.

I was writing it and then the firm I was working out threatened to sue me for bullshit IP infringement...

Re:Finance Pros Say You’ll Have to Pry Excel Out of Their Co... by toomuchtodo on Thursday November 30, 2017 @07:26PM (#15820960)
Sounds like Excel needs a git extension!
Re:Finance Pros Say You’ll Have to Pry Excel Out of Their Co... by yosyp on Thursday November 30, 2017 @08:12PM (#15821180)
A quick google search shows some people are already working on that, including https://www.pathio.com
Re:Finance Pros Say You’ll Have to Pry Excel Out of Their Co... by uridium on Friday December 01, 2017 @02:57AM (#15822599)
Schematiq - https://www.schematiq.com also handles source control for spreadsheet and a lot more. The platform is about making sheets enterprise grade applications - regression, testing, CI etc and introduces new features to simplify spreadsheet logic.
Re:Finance Pros Say You’ll Have to Pry Excel Out of Their Co... by ccozan on Friday December 01, 2017 @07:04AM (#15823341)
I remember I wrote not long ago a VB Macro for Excel, where all the VB code inside was imported/exported, git pull/commit+push, etc, all of a button push.

It became an instant hit in that company and I short of a medal :).

Re:Finance Pros Say You’ll Have to Pry Excel Out of Their Co... by fzumstein on Friday December 01, 2017 @07:21AM (#15823394)
Note that Pathio has been relaunched as https://www.xltrail.com - it supports the standard Git workflow and every Git system (GitHub, Bitbucket, Gitlab etc.).
Re:Finance Pros Say You’ll Have to Pry Excel Out of Their Co... by jimmcslim on Thursday November 30, 2017 @06:14PM (#15820536)
Or seek greater adoption of standards around spreadsheets, such as FAST [1].. Flexible, Appropriate, Structured, Transparent.

[1] http://www.fast-standard.org/

Re:Finance Pros Say You’ll Have to Pry Excel Out of Their Co... by opportune on Thursday November 30, 2017 @06:51PM (#15820754)
Ehh I doubt most quants are interacting much with Excel except when they need to work with non-quants. There's not much of a reason to use Excel when you could use something like Pandas instead, plus the Python libraries/environment will make everything non-tabled better to deal with anyway.
Re:Finance Pros Say You’ll Have to Pry Excel Out of Their Co... by djtriptych on Thursday November 30, 2017 @08:51PM (#15821335)
it's different now but historically quants were not well versed in software engineering. If anything it would have been R 10-15 years ago.
Re:Finance Pros Say You’ll Have to Pry Excel Out of Their Co... by hardlianotion on Friday December 01, 2017 @07:13AM (#15823367)
10-15 years ago it was excel excel excel for quants in most investment banks.
Re:Finance Pros Say You’ll Have to Pry Excel Out of Their Co... by kgwgk on Friday December 01, 2017 @02:28AM (#15822517)
R was almost irrelevant in finance 10 years ago, Matlab was much more widely used. R may have more momentum now, but I’m not sure it has overtaken Matlab yet. And C++ was, and probably remains, king.
Re:Finance Pros Say You’ll Have to Pry Excel Out of Their Co... by vostok on Friday December 01, 2017 @03:06AM (#15822610)
I think a lot of the discrepancy is explained by the fact that quant is a very widely applied title. There are quants who spend most of their day in Excel and quants who spend most of their day in C++.
Re:Finance Pros Say You’ll Have to Pry Excel Out of Their Co... by MrMattWright on Friday December 01, 2017 @07:37AM (#15823444)
Just for fun I have a couple of nice Excel based stories from my time in Investment Banking.

Story 1: I Inherited a system once that had 1000s (yes thousands) of excel spreadsheets checked into SourceSafe (yes Source Safe) each sheet represented an Equity Derivatives trade, which was checked out and used by traders when pricing a deal. Also...now it gets fun...there were common functions for calculating the present value and risk metrics for each trade...so when we needed to calculate the risk the traders were running (intraday and end of day) we had a huge compute grid that would check the spreadsheets out and run them on virtualized windows boxes and sum the results to produce the official risk metrics for the bank's trading desk. Absolutely not making that up. I should really write a blog about it...it's truely terrifying.

Story 2: What if I told you that Excel has this little know function called RTD (real time data) that let's you stream data into sheets in well....real time. So you can see all your prices ticking away without refreshing the sheet. I also took over a system that had hundrends of sheets that did this (at this point you must be thinking this guy is a sucker for punishment) one of the problems here was latency...so each trader's sheet used their local machine to get prices...and the analytics library we used to calculate prices to an 'indeterminant' amount of time to execute. Oh...and one of the sheets published the banks prices out to Bloomberg...for trading. This whole mess was sorted out by eventually pricing everything centrally and pushing consistent numbers to bloomberg, the spreadsheets and the risk systems.

My overall summary is that excel is "what is dead but may never die" and you'd better embrace it if you want to deal with front office systems. That and I think people would be surprised to see how much systemic risk some of these places are running (i'm talking about a Top 15 bank here).

I sort of came to love it as a tool, but that's what people with Stockholm syndrome say right?

Re:Finance Pros Say You’ll Have to Pry Excel Out of Their Co... by osullivj on Friday December 01, 2017 @08:01AM (#15823524)
Story 1 sounds totally nuts, but Story 2 is a common setup in my experience. Using Bloomberg prices on the desktop is a lot cheaper than BPipe - the server side solution for BBG market data. And quoting indicative prices on BBG from Excel is common too. An interesting new startup that I follow - pricingmonkey.com - is doing BBG driven IRD pricing in the browser with an IR pricing library written totally in JS. There are more details on my blog at etrading.wordpress.com
Re:Finance Pros Say You’ll Have to Pry Excel Out of Their Co... by MrMattWright on Friday December 01, 2017 @08:21AM (#15823607)
Yep, that was pretty much it. Price wars over BPipe vs Retuers. "You mean I have to pay twice? Once for my desktop and once for this BPipe thing, and that costs whhhhhaaat? How much?" Also it's not quite as terrifying as firm live pricing, it's only indicative with a good Bid/Offer spread. Enjoyed your blog....I like the Pricing Monkey idea. Would they be picking up the prices from the local machine though? Because surely any other way would mean the same large fees?
Re:Finance Pros Say You’ll Have to Pry Excel Out of Their Co... by osullivj on Friday December 01, 2017 @08:49AM (#15823754)
Thanks for the kind words on the blog. Yes - PricingMonkey gets the prices via the BBG API. The dev told me he'd coded it in Python.
Re:Finance Pros Say You’ll Have to Pry Excel Out of Their Co... by oldandtired on Thursday November 30, 2017 @08:40PM (#15821289)
For many years I was the contractor that came in to work with the Subject Matter Experts (SME) using Excel and Access, bypassing the IT department. My role was to review with each user their Excel and Access usage and make sure that we could make life simpler for them. If this meant automating some process they did with Excel then that's what I did.

Pharma company - took the manual Excel process and automated the living daylights out of it so that what the departing analyst took days to do would now be done in minutes.

Petrochemical company - took the reports being done in Excel by a petroleum engineer (each taking 2 hours to do each) and automated to allow him to do the reporting for 50 client companies in under a minute.

Whether Excel or Access, these tools will be used by the SMEs and nothing any IT department does will change this. What IT needs to do, is make sure that the models used are accurate and that the spreadsheets and databases are correct.

Most people who use either Excel or Access make interesting blunders in their creations and end up basing decisions on incorrect output from their creations. It takes an experienced and clue-in developer (IT person) to speak with and then analyse the creations to make sure they actually match what is required.

For those of us who do this kind of work, we must understand that the SMEs are those trying to use Excel and Access and that we are there to help them achieve the SME's desired outcome. The SME's need to understand that they are NOT SME's when it comes to Excel and Access, which most seem to never get this point.

Re:Finance Pros Say You’ll Have to Pry Excel Out of Their Co... by jimmcslim on Thursday November 30, 2017 @06:11PM (#15820514)
> The issue is not how to get rid of Excel, it's how do we make a better spreadsheet...

Or how do we better support custom tools integrating with Excel.

Excel has a number of extensibility models... maybe too many;

* VBA,

* XLL add-ins (which is a C++ API, but check out the excellent Excel-DNA [1] which is a C# wrapper around this),

* Visual Studio Tools for Office, which doesn't support the ability to implement your own custom functions (e.g. =MyAddIn.GetStockPrice() unlike the previous two options, and

* The more recent 'Apps for Office' extensibility model across the whole Office suite, which is basically an embedded web browser instance in your Office app which gets a Javascript API for interacting with your Office document. This is also getting custom functions support in Excel [2]

At the energy consultancy I work for we have built a suite of Excel add-ins that allow analysts to get the data that they want easily into their sheets, but don't impose any (unreasonable) restrictions on how they work with the data.

[1] https://excel-dna.net/ [2] https://dev.office.com/docs/add-ins/excel/custom-functions-o...

Re:Finance Pros Say You’ll Have to Pry Excel Out of Their Co... by d--b on Thursday November 30, 2017 @06:42PM (#15820698)
Better addins only means you get to do more stuff with Excel.

Instead of doing one model, you may now be able to do a time series of models, or calculate The VaR (Value at Risk).

There is no end to what people can ask from data. The only limitations are technological. And whatever extension you may have, your users will always push Excel to its limits.

Re:Finance Pros Say You’ll Have to Pry Excel Out of Their Co... by cm2187 on Thursday November 30, 2017 @06:54PM (#15820762)
I am a typical excel power user, and have started to increasingly do work outside of Excel (C# mostly). I find however that Excel is still a formidable GUI, particularly for tabular inputs.

One middle ground is libraries like SpreadsheetGear that give you a UI that has an almost perfect excel look and feel while giving you more control. You don't need to integrate your tool in Excel, you can integrate a quasi-Excel in your (.net) tool.

Re:Finance Pros Say You’ll Have to Pry Excel Out of Their Co... by rjbwork on Thursday November 30, 2017 @09:19PM (#15821433)
I'm a professinal C# dev. When I need to do any one off analysis of data or a running system, I just spin up linqpad, write some C# to grab my data, put it through a functional transformation pipeline with LINQ, then, optionally, graph the results using some WPF graphing components. I hit F5, and my results appear in the window below as either a table of data or a graph.

I don't even have excel installed on my windows desktops anymore. A 50 dollar tool from an upstart programmer is better for me than excel.

Re:Finance Pros Say You’ll Have to Pry Excel Out of Their Co... by cm2187 on Friday December 01, 2017 @01:45AM (#15822391)
Yeah but data analysis is only one of many usages for Excel.

Let me give you a practical example I ran last week. I want to construct a portfolio of loans starting with an existing portfolio, such that it follow a certain total balance and maturity profile through time, by replenishing it every month with new loans of various maturity (I am solving for the individual notionals of these new loans).

So inputs are

1. the existing portfolio, hundreds of entries with various parameters: start, end, notional, etc

2. the target maturity profile at every point in time

3. the target balance of the portfolio at every point in time

A simple LINQ query won't do. For the example above, you want to model it in C#, but the inputs are lots of tables. In that case I find the optimal way is for the script to accept spreadsheets as input files (wrote a "xlsx (de)serialiser") and output files. Effectively Excel becomes the UI of the script. Nothing beats the Excel UX in giving you the ability to build a table, change or copy/paste data in bulk, etc.

Re:Finance Pros Say You’ll Have to Pry Excel Out of Their Co... by rjbwork on Friday December 01, 2017 @10:36AM (#15824397)
Personally I'd just throw it into a number of SQL Lite table and run Linq to it via EF. Or just write SQL against it. Pretty much just text files.

If it were too big (probably not if Excel can handle it) I could also just load up the data into memory via a TSV/CSV deserializer, of which there are a few for .NET. If it comes as excel from the source, excel extraction is a piece of cake these days (the codebase I work on takes excel as raw input for data feeds).

As for nothing else letting you build/change/copy/paste data into a table, Sublime Text Advanced CSV package is fantastic for editing tabular data, and Sublime Text has no problem handling files many gigabytes in size.

Re:Finance Pros Say You’ll Have to Pry Excel Out of Their Co... by zgao on Thursday November 30, 2017 @08:57PM (#15821351)
Exactly this.

Shameless plug: I am a founder of AlphaSheets, a company working on solving all of these issues. It's quite scary (building a spreadsheet is like boiling an ocean) but our mission feels very meaningful, we're well-funded, and we are now stable and serving real users.

A big problem in finance workflows is that there is a tradeoff between several factors: correctness, adoption / ease-of-use, rapid prototyping, and power. We aim to solve several of these major problems. We've built a real-time collaborative, browser-based spreadsheet from the ground up that supports Python, R, and SQL in addition to Excel expressions.

Correctness is substantially addressed, because you don't need to use VLOOKUP or mutative VBA macros anymore. Your data comes in live, and you can reference tables in Python as opposed to individual cells. A lot of operational risk goes away as well, because the AlphaSheets server is a single source of truth.

We help with adoption of Python and adoption of correct systems as well. You can gradually move to Python in AlphaSheets -- many firms are trying to make a "Python push" and haven't succeeded yet because the only option is to move to Jupyter and that's too much of a disruption. It's less brittle than Excel. The important keyboard shortcuts are there.

And finally, the entire Python ecosystem of tools (pandas, numpy, etc.) and all of R is available, meaning that many pieces of functionality that had to be painstakingly built in-house in VBA and pasted around are simply available out of the box in well-maintained, battle-tested packages.

Our long term plan is to broaden our focus into other situations in which organizations are outgrowing their spreadsheets. We think there's a lot of potential with the spreadsheet interface but the Excel monopoly has prevented meaningful innovation from happening. For example, every BI solution tries to be "self-serve" and "intuitive" these days, but encounters resistance from users who end up sticking with spreadsheets due to their infinite flexibility and immediate familiar appeal.

We hope to bring the spreadsheet in line with the realities of the requirements of the modern data world -- big data, tabular data, the necessity of data cleaning, data prep / ETL, the availability of advanced tooling (stats, ML), better charting -- because we think there's a giant market of people waiting to move to a modernized but familiar spreadsheet.

If there's anyone interested, contact me, because I'd be very interested in chatting! I'm michael at alphasheets dot com :)

Re:Finance Pros Say You’ll Have to Pry Excel Out of Their Co... by apthnz on Thursday November 30, 2017 @11:20PM (#15821865)
I'm a Product Manager who uses Excel extensively at work and is also learning Python in the evenings.

This is the sexiest thing I've seen since I kissed my girlfriend goodbye this morning.

Re:Finance Pros Say You’ll Have to Pry Excel Out of Their Co... by gaius on Friday December 01, 2017 @03:40AM (#15822724)
Exactly this. I dunno if you have seen old movies where there is a guy at a desk and in front of him is a pad - built into the desk - with large stacked sheets of paper tucked in at the corners to hold them down, A3 size or larger. This was used to just do any quick bit of writing, make a note, do a calculation, sketch something out, whatever. When a page had been sufficiently dooodled on it was just peeled off the top and there was a fresh one underneath ready to go. THAT is the experience Excel replicates. A dedicated, single-function piece of software to do a particular task COMPLETELY misses the point.
Re:Finance Pros Say You’ll Have to Pry Excel Out of Their Co... by ghostly_s on Friday December 01, 2017 @12:44PM (#15825619)
Derail but, no, I've never seen such a thing as you describe. Any movies in particular that feature this?
Re:Finance Pros Say You’ll Have to Pry Excel Out of Their Co... by sls on Wednesday December 06, 2017 @10:03AM (#15861496)
If you search for "desk pad" on Amazon, Google images, etc. you'll see them. A large rectangle of paper, typically being held in a base at the corners, top and corners, etc., sometimes printed with a grid, calendar, etc.

My father's desk featured one of these, used exactly as described.

Re:Finance Pros Say You’ll Have to Pry Excel Out of Their Co... by sct202 on Thursday November 30, 2017 @05:28PM (#15820204)
Where I work, we get to the part IT guy part, and then the quote comes out to be like $200k+ (and 6 months) to duplicate a SAS program, Excel spreadsheet, or Access database that an analyst hacked together in their spare time.
Re:Finance Pros Say You’ll Have to Pry Excel Out of Their Co... by Declanomous on Thursday November 30, 2017 @06:07PM (#15820487)
Or it could be like the place I worked where IT made as many hurdles as possible to the installation of new software. So I, as a database marketing associate, linked 4 different CRMs used by different departments together using the magic of Access, Excel and VBA.

IT found out when they disabled a library (DAO) because "cryptolockers" could use it, only to break almost every single process at my organization. The IT director was pissed, but he had to reenable the library because everything ground to a halt.

Bonus points: he told my boss that VBA macros were bad because hackers could use them to take over the mouse and click on things.(Edit: I also distinctly remember the IT director accusing me of being a hacker when he found out I was using VBA.)

I still wake up in a cold sweat with nightmares that something broke in that system. The VBA editor in Office is an absolute nightmare when it comes to debugging. No version control either.

Re:Finance Pros Say You’ll Have to Pry Excel Out of Their Co... by agar on Thursday November 30, 2017 @06:50PM (#15820750)
Version control in Excel VBA is easy.

Just do a Save As..."Mission Critical Finance Model v2.xlsm", then "Mission Critical Finance Model v3.xlsm", then "Mission Critical Finance Model v3a (temp with changes for Jim).xlsm" and then "Mission Critical Finance Model v4 (maybe-trying to merge 3a back-DO NOT USE).xlsm" and so on...

Re:Finance Pros Say You’ll Have to Pry Excel Out of Their Co... by Declanomous on Thursday November 30, 2017 @07:10PM (#15820859)
That's true. Good luck doing anything with that though.

My main database was up to version 2051 by the time I left.

The class I have nightmares about is the "compile and backup" class I wrote, that decompiled the database, appended the date to each file, compared all of the non-data and non-binary components, recompiled the database with the most recent versions, and then made sure every database was linked to the most recent versions of other databases.

Why did I decompile everything? Well I learned that unless the file size changed markedly, the deduplication feature of the file replication system that our company used assumed the files were the same. Well Access databases can stay the same size for a long time unless you run a compact and repair on them. So I lost several weeks of work more than once because the system decided that an older version and a newer version of a database were the same.

Without that stupid "feature" I would have just had the production Access file be named Access-prod.accdb and updated it each day.

I honestly have so many nightmares about that job I'm intentionally doing easy work until I can convince myself that files don't magically delete themselves without a trace and administrators aren't deliberately evil.

Re:Finance Pros Say You’ll Have to Pry Excel Out of Their Co... by CydeWeys on Thursday November 30, 2017 @09:42PM (#15821521)
... wow. Instead of hashing the file contents, it just checked file size? Who came up with this algorithm?! There's plenty of applications that use files of fixed large sizes initialized at install, for performance reasons or just simply so if you're going to run out of space you do so during interactive installation, rather than silently breaking later on.
Re:Finance Pros Say You’ll Have to Pry Excel Out of Their Co... by Declanomous on Thursday November 30, 2017 @10:42PM (#15821744)
From what I can determine, the authoritative version was always the one with the later timestamp. This might have worked at least in theory, however there were always issues.

The foremost issue was that the timeserver was virtualized, and drifted to an unbelievable degree. This issue was compounded by the fact that we used citrix virtual machines for some applications, and our local desktops for other applications.

The latter issue wasn't a problem until they moved the citrix servers from an on-site server room to cloud provider about 40 miles away. (I think this is when the timeserver became virtualized as well) At that point the latency between the citrix sessions and the local desktops was such that I could create a document on the file share, edit it on my local desktop, close it, save it, and then open it on the remote desktop and it would lock the replication from my computer and lose all my work.

My department managed to wipe out the changes every other department had made to the yearly budget workbook because of this interaction. Of course part of the issue was using Excel as a shared document, but that issue was hugely compounded by the fact that our network infrastructure was bananas.

For instance, building access databases using the Microsoft recommended separate front-end/back-end files, with each individual user having their own front-end file actually managed to make the issue occur more frequently. If you used a single access database file on a shared drive the issue was substantially less likely to occur. I can only imagine this is because the dedupe was postponed until everyone was done using the file, and the file was more likely to substantially change in size when multiple people were using it.

The IT director had an incredibly fragile ego, and he attacked anyone who challenged him in any way. He hired a truck driver (with absolutely no IT experience) to work the IT help desk, which I assume was because he wanted a sycophant. He reported me to HR any time I did anything he didn't understand or which perceived to be a challenge of his power. He reported me to HR for using open-source software, with a description of open source software that made it sound like Wikipedia meets programming, and only used by hackers. He also reported me to HR after he heard I called our IT situation Kafkaesque.

As far as nightmares go, that might have been my favorite, because my defense was literally "it is Kafkaesque, and I can prove it." I managed to successfully convince HR that the IT situation was Kafkaesque using examples from Kafka's work, whose response was "well there isn't anything we can do about it, but you aren't wrong."

The sensible thing would have been to leave as soon as possible, but I really liked the field I was in, which was kind of specialized. Looking back I stayed way too long.

Re:Finance Pros Say You’ll Have to Pry Excel Out of Their Co... by DCoder on Friday December 01, 2017 @01:57AM (#15822423)
This would make a great story for The Daily WTF: https://what.thedailywtf.com/
Re:Finance Pros Say You’ll Have to Pry Excel Out of Their Co... by intended on Friday December 01, 2017 @12:15AM (#15822086)
sounds like talesfromtechsupport material.
Re:Finance Pros Say You’ll Have to Pry Excel Out of Their Co... by agar on Thursday November 30, 2017 @07:14PM (#15820879)
Just to clarify, I was being sarcastic about it being easy (or actually being version control, for that matter).

But thanks for the horror story. What a nightmare.

Re:Finance Pros Say You’ll Have to Pry Excel Out of Their Co... by TeMPOraL on Friday December 01, 2017 @07:13AM (#15823368)
> and administrators aren't deliberately evil.

Stories like that, and my experience, tell me that IT staff in large companies and institutions ensure everything works by making it unusable. After all, users won't break stuff if they are not actually using the infrastructure.

Re:Finance Pros Say You’ll Have to Pry Excel Out of Their Co... by cosmie on Friday December 01, 2017 @12:02AM (#15822055)
I was once on a team piloting a new inventory planning model in Excel, testing it out at a handful of distinctively different manufacturing sites to stress test it before paying SAP's ransom to bake it into the official system.

When I started, I was backfilling someone who created the Excel model and was responsible for batch exporting data from SAP, crunching the numbers in Excel, and sending that out to the sites every morning. My onboarding consisted of being given his laptop and his login credentials[1], and being met with a desktop full of dozens nested folders of Excel sheets in that exact version control format. I still have nightmares of that week.

[1] My manager had successfully kept IT from being able to get the laptop for re-imaging. But my senior VP located on a different continent had to get involved to get IT to reset the guy's password for us.

Re:Finance Pros Say You’ll Have to Pry Excel Out of Their Co... by hexane360 on Thursday November 30, 2017 @07:20PM (#15820926)
Mission Critical Finance Model v4 FINAL_revised (2).xlsm
Re:Finance Pros Say You’ll Have to Pry Excel Out of Their Co... by agar on Thursday November 30, 2017 @08:44PM (#15821299)
Stop looking at my file system!
Re:Finance Pros Say You’ll Have to Pry Excel Out of Their Co... by jrs235 on Thursday November 30, 2017 @08:50PM (#15821332)
Mission Critical Finance Model v4 FINAL_revised (2) - Copy (2) (2).xlsm
Re:Finance Pros Say You’ll Have to Pry Excel Out of Their Co... by tmzt on Friday December 01, 2017 @11:21AM (#15824800)
Or you could:

   unzip *.xlsm
   git add .
Probably with some custom diff scripts for the xml files.
Re:Finance Pros Say You’ll Have to Pry Excel Out of Their Co... by Declanomous on Saturday December 02, 2017 @12:55PM (#15833063)
I did most of this in Access.

Also, I wasn't allowed to install version control of any sort, so that's really the bigger problem.

Re:Finance Pros Say You’ll Have to Pry Excel Out of Their Co... by manigandham on Thursday November 30, 2017 @08:48PM (#15821321)
Box, Dropbox, etc solve much of the problem
Re:Finance Pros Say You’ll Have to Pry Excel Out of Their Co... by gaius on Friday December 01, 2017 @03:50AM (#15822743)
The funny thing is you can save your spreadsheet into Sharepoint which does have version control and people still do it that way in Sharepoint too...
Re:Finance Pros Say You’ll Have to Pry Excel Out of Their Co... by fma on Thursday November 30, 2017 @09:54PM (#15821555)
I work in a large manufacturing company that gets lots of data about the manufactured product...The business people received a single excel reports from a database. They wanted a fancy UI to export the data (more queries, pull from other databases, etc) into Excel because they do all their analysis in there.

I have no problems about it...less work for me, and they can do their work easily.

Re:Finance Pros Say You’ll Have to Pry Excel Out of Their Co... by smallnamespace on Thursday November 30, 2017 @10:08PM (#15821607)
GS a decade ago had Excel connectors to its internal quant IDE/ORM/trading system.

It was pretty hacky, but you could pull prices for any security or book in the system, or call any options pricer, as long as the user had the right permissions.

Since the ORM had bindings to stuff like convex quadratic optimizers (IMSL), you could build spreadsheets that let users input linear constraints (e.g. I want to spend less than $50mm and buy between 0-5 units of X and 2-10 units of Y) and then plop the solution vector right back in their workflow. Basically magic to the user, and much better than Excel's 1D root-finder.

Most quants are notoriously bad at building UX, so this was often a much better solution than trying to write a dedicated tool from scratch.

Re:Finance Pros Say You’ll Have to Pry Excel Out of Their Co... by rukuu001 on Thursday November 30, 2017 @07:41PM (#15821031)
"How do we make a better spreadsheet..."

Some things I'd like to see:

- "Git for Excel" Diffs, tags, push changes to individual sheets or regions.

- Snapshots: Create 'Time Machine' type interface for spreadsheet regions.

- Collaborative editing.

- Access control.

- First-class support for model verification (folks roll their own at the moment).

Re:Finance Pros Say You’ll Have to Pry Excel Out of Their Co... by listenallyall on Thursday November 30, 2017 @08:02PM (#15821135)
Excel should have a built-in, top-notch SQL editor/IDE. Existing SQL IDEs return resultsets in a grid, thus, Excel is a natural fit. Excel would allow developers to go beyond a SQL IDE by subjecting the results to the rest of the spreadsheet's functionality. Would work great for exploratory analysis or for production reporting.

Currently, running SQL and returning the results to a sheet is possible, but very cumbersome now: either .ODC files, VBA, or PowerPivot, none of which are flexible or developer-friendly.

Re:Finance Pros Say You’ll Have to Pry Excel Out of Their Co... by NamTaf on Friday December 01, 2017 @12:06AM (#15822063)
It's not great, but there's an existing SQL IDE called Microsoft Query. You can find it in the Data > Get External Data > From Other Sources area of the ribbon. It's straight out of the Win3x era graphics-wise but it lets you do drag-and-drop style query structures. It does fall apart horrendously once you get to anything approaching complex datasets with many tables, joins, etc. but the precident is there for a graphical IDE baked into Excel.

I used ODBC connections to pull in data for further manipulation in Excel to run a ~$200m project for a couple of years. It saved a great deal of time compared to the manual system that was in place prior whilst presenting the data in the familiar excel system which didn't scare away the old baby boomer who was doing all the manual work beforehand. Once I set up the queries and showed him how to refresh the data from the ribbon, he was able to do it without breaking it and whilst still feeling at home enough to get the job done on his own initative.

edit: I note that the above all falls apart somewhat when you need R/W access to your DB. I've only ever done SELECTs because I want to know info from a DB, crunch numbers, then feed decisions into another system. I don't know if you can UPDATE, INSERT, etc. via Excel/MS Query as I've simply never tried it.

Re:Finance Pros Say You’ll Have to Pry Excel Out of Their Co... by bigger_cheese on Thursday November 30, 2017 @11:49PM (#15822001)
One of my biggest concerns about Excel is it integrates poorly with databases.

A notorious thing that happens in my org is an expert user will extract a bunch of data from the source DB tables and dump the data into an excel spreadsheet. The spreadsheet will pass into the hands of an analyst who will make some tweaks maybe paste in some data maybe manually adjust a few cells here and there and so on and so forth. The spreadsheet will pass into hands of someone else and they will make their own 'adjustments' and so on down the line until finally I will get a phone call along the line of.

"The numbers I have look different to the numbers this other department has in their spreadsheet which is the 'real' value".

Eventually I'll track down the source database and lo-and-behold neither spreadsheet accurately reflects the original database number.

The more hands a spreadsheet passes through the greater the chance of original data being modified into something irreconcilable.

I've played around in the past with linking spreadsheets to DB directly and populating cells via SQL but it fell afoul of requiring various DB/2, oracle, OLEDB etc drivers to be installed on every PC you wanted to open the excel file on and each of those computers required having data source correctly configured via control panel - nightmare for places like my workplace because Windows PC's are all locked down so you'd need to submit a service ticket for every PC you wanted to be able to open spreadsheet on.

This was some years ago but I don't believe anything has changed with way database connection configuration works in Windows client world.

I much prefer the way Dedicated BI tools like COGNOS / SAS etc work. In that situation the user authenticates to the BI software (running on a central server) - the server has all of the DB connections configured and the server executes the DB query on behalf of users.

Our org is currently evaluating a newish Microsoft product called "Power BI" which uses something called an enterprise gateway more info here https://docs.microsoft.com/en-us/power-bi/service-gateway-on... I don't know all the details but seems promising solution for our org's data sharing issues at least.

Re:Finance Pros Say You’ll Have to Pry Excel Out of Their Co... by intended on Friday December 01, 2017 @12:19AM (#15822101)
Oh hey, how is the Power BI eval going?
Re:Finance Pros Say You’ll Have to Pry Excel Out of Their Co... by lowkeyokay on Friday December 01, 2017 @05:51AM (#15823120)
We're currently trying out QueryStorm [1]. It supports SQL through built in SQLite, but you can connect to any DB and get the full SQL feature set of your DB. It's also possible to manipulate data, and automate stuff using C#. So far it's been quite good. Everyone can view spreadsheets created using QueryStorm, but the data will just be plain data unless they have QueryStorm installed. That's the only drawback.

I really thought PowerPivot could be this excellent data tool, but the DAX formula language is just such a pain. I really dislike it. Using SQL og C# is just much more intuitive - especially SQL.

1: https://querystorm.com

Re:Finance Pros Say You’ll Have to Pry Excel Out of Their Co... by rukuu001 on Thursday November 30, 2017 @11:18PM (#15821854)
Heh, I didn't mention that because of this: https://news.ycombinator.com/item?id=13114421
Re:Finance Pros Say You’ll Have to Pry Excel Out of Their Co... by lzhou on Thursday November 30, 2017 @10:10PM (#15821619)
Assuming your result set isn’t massive, it’s not very hard to have a 30-40 line array function (that takes a query) and returns a sql resultset.
Re:Finance Pros Say You’ll Have to Pry Excel Out of Their Co... by starik36 on Thursday November 30, 2017 @08:34PM (#15821264)
There are a number of add-ins that do just that. I've never used any of them, so I can't speak to the quality.
Re:Finance Pros Say You’ll Have to Pry Excel Out of Their Co... by jchw on Friday December 01, 2017 @03:59AM (#15822770)
I mean personally, I'm OK with just continuing to use Excel. If anything it'd be nice if they made newer APIs. I actually like COM for some reason, but it does feel dated at this point. I bet there are some other APIs, but I'm not in the biz of writing Excel plugins, so I'm not up to date on it. I did it once for fun and never spoke of it again.

But basically, I kind of just want a simple API for interacting with workbooks and providing functions, maybe using embedded JS or Python scripting. That, to me, sounds like it would make Excel a lot more palatable toward developers like it is toward end users. (Honestly, I think what I'm describing is a bit like WebExtensions but for Excel workbooks. I'd dig it.)

Re:Finance Pros Say You’ll Have to Pry Excel Out of Their Co... by drumhead on Friday December 01, 2017 @07:33AM (#15823433)
So True!

Spend a few million on a web based reporting system. Allows self service reports using real time data. As it runs on a browser literally anyone can use it.

Only problem - formatting and printing is for shit and users want it in excel so they can manipulate and format data how they want, on the pages they want, in the colors and font sizes they want.

I agree with Excel not being the most stable or error free platform for reporting, but is there any easy to use alternative? Whatever we try we keep coming back to Excel/spreadsheets.

Re:Finance Pros Say You’ll Have to Pry Excel Out of Their Co... by SolarNet on Friday December 01, 2017 @02:33AM (#15822527)
I think part of the problem is the mindset of devs that try to tackle this. Excel is the dynamically typed python to the statically typed and compiled C++ of most web apps. The power of excel is that people can build their own little apps in it as needed. Any replacement to Excel will require those features first, and then you can build specialized integrations.
Re:Finance Pros Say You’ll Have to Pry Excel Out of Their Co... by arkh on Friday December 01, 2017 @05:05AM (#15822991)
The column / line filter menu is hard to beat and one of the little things making Excel impossible to replace.
Re:Finance Pros Say You’ll Have to Pry Excel Out of Their Co... by jimmcslim on Thursday November 30, 2017 @06:17PM (#15820557)
I also wonder if there is a middle way here... in encouraging greater use of interactive notebook computing such as that espoused by Mathematica, Jupyter Notebook, IPython, etc.
Re:Finance Pros Say You’ll Have to Pry Excel Out of Their Co... by analog31 on Thursday November 30, 2017 @07:15PM (#15820892)
Man, I'd love to be able to edit tabular data within a Jupyter notebook. Every few months I look to see if this problem has been solved in a tolerable way.
Re:Finance Pros Say You’ll Have to Pry Excel Out of Their Co... by d--b on Thursday November 30, 2017 @06:36PM (#15820666)
I strongly believe these are almost there. The interface is what sucks with notebooks. Code should be hidden, cells should be smaller so you can pack a lot more info in them. 2D organization of data is a lot more effective that a long 1D list of things.

I believe OnservableHQ has a good shot at it.

Re:Finance Pros Say You’ll Have to Pry Excel Out of Their Co... by codingdave on Thursday November 30, 2017 @05:40PM (#15820304)
Excel is old-school Business Intelligence. We don't need to make a better spreadsheet, we need to make a better toolkit for data-driven decisions. There is very little reason that all users need to engage with their data in a grid of cells. When BI tools are as usable and flexible as Excel, that is when Excel will dwindle.
Re:Finance Pros Say You’ll Have to Pry Excel Out of Their Co... by ng12 on Thursday November 30, 2017 @05:56PM (#15820427)
> we need to make a better toolkit for data-driven decisions

Why? Excel is ubiquitous, powerful, and accessible. It only really falls apart for huge data sets -- at which point you build a cluster and feed the output back into Excel anyway.

I'll never understand why software engineers feel like everybody else is a caveman banging rocks together. Excel replacements are a hard sell because, by and large, they're not needed.

Re:Finance Pros Say You’ll Have to Pry Excel Out of Their Co... by twic on Thursday November 30, 2017 @06:11PM (#15820515)
Excel sheets aren't very maintainable. You get a big grid, plus named ranges. There aren't good tools for generalisation or abstraction, so big sheets have all the usual problems of bad software: hard to understand, hard to change, full of inconsistencies, etc. There are also no good options for source control, modular reuse, etc.

I work on a desk which has a lot of vital calculations in Excel spreadsheets. They work, and we make money, but it takes a lot of human effort (expensive trader effort!) to keep them working properly. If we want to scale up - more people, more products, etc - we need to move more and more of those calculations into software.

Improvements to or replacements for the spreadsheet as tools for non-programmers would certainly be helpful. But better tools and techniques for programmers to port functionality from spreadsheets to 'real' software would also be helpful.

Re:Finance Pros Say You’ll Have to Pry Excel Out of Their Co... by roenxi on Thursday November 30, 2017 @08:31PM (#15821255)
Excel spreadsheets aren't maintainable, but in context there are some subtleties. If the process is upgraded to a full-blown piece of 'real' software, now you need a full-blown software dev to maintain it. A better result from more resources isn't a clean improvement, it is just a change.

Straddling the line between 'technically best outcome' and 'cheapest workable outcome' is what businesses strive to do. In most mature businesses change should be approached extremely cautiously; everybody loves changing things and the costs are frequently big.

Re:Finance Pros Say You’ll Have to Pry Excel Out of Their Co... by vorotato on Friday December 01, 2017 @08:51AM (#15823770)
If your risk is less than it costs to build the software then that's totally acceptable, but what is unacceptable is billion dollar businesses using spreadsheets to manage critical business operations.
Re:Finance Pros Say You’ll Have to Pry Excel Out of Their Co... by vorotato on Friday December 01, 2017 @08:48AM (#15823750)
Probably because studies show that people who use excel to manage finance end up coming up with different numbers. JP Morgan lost 6 billion because of excel, and these are preventable problems. It's because you aren't codifying your process, you're just memorizing it and trusting that your memory won't change. Trusting that you won't miss a step. The moment you start doing serious calculations on your data you would be better writing some form of software which can actually handle the amount of complexity that you have in your job. Excel is opaque, hard to manage, easy to diverge. The agency I'm at have made a tremendous reduction in errors by migrating away from excel. Sure it's more work nailing down every process, but they are actually nailed down and you can see them and evaluate whether they are actually correct. You won't have multiple competing standards within a business because we can share our processes. If you think excel is enough you're either doing very simple or low risk job, or much more likely you are floating a lot of totally unnecessary and expensive risk. In finance CENTS matter.
Re:Finance Pros Say You’ll Have to Pry Excel Out of Their Co... by noitsnot on Thursday November 30, 2017 @09:13PM (#15821415)
Two opposite ends of the spectrum. The engineer who has to constantly keep up on the latest technology, then there are people who use excel, which is a program from the beginning of time/Windows.
Re:Finance Pros Say You’ll Have to Pry Excel Out of Their Co... by d--b on Thursday November 30, 2017 @06:39PM (#15820683)
I'm not saying we should make a better grid of cells, I'm saying we should make a better application container that non-dev people can use.

The big grid of cells is the weakest part of Excel by far!

Re:Finance Pros Say You’ll Have to Pry Excel Out of Their Co... by agar on Thursday November 30, 2017 @06:57PM (#15820779)
People have tried to improve on that big grid of cells since VisiCalc in 1979. Considering it's the weakest part, it has been remarkably difficult to find something better.

As someone that spent years being buried in complex spreadsheets for hours a day (and supported and trained others who spent literally 12 hours a day building models), I can't think of a better combination of functionality, capability, and usability than that big grid of cells.

Re:Finance Pros Say You’ll Have to Pry Excel Out of Their Co... by sanityUnbounded on Thursday November 30, 2017 @09:49PM (#15821542)
And it makes sense why it works so well. On a 2D screen, seeing data in linear rows and columns is the best option because it follows the same format we have been taught verbatim since birth. The first type of data processing a person does is learn to read. Left to right, each letter (column) represents a different sound (value), these letters build a picture, and the more of them there are the more information the story contains (debatable). Then we learn to build our own information by writing.

Then we move on to math, and we still do it with paper, almost as an extension of writing. When doing calculations in class it is natural to keep calculations and values in some type of invisible grid pattern that makes sense to the student. Then we get to the computer, and that format still follows.

Except that the first information processing a person does isn't reading at all. It's learning sounds and words, and we do that by listening and watching others do it. And the first thing we create isn't words on a page, it's likely something physical, in the form of building blocks or some other toy. This type of learning is great, but it can't be replicated well in a school system because it is expensive and doesn't scale well to the real world. If you're going to engineer and design a bridge, you're not going to start by making a bridge. So we're taught to make lines on paper that represent a bridge.

This is where (I hope) AR/VR becomes disruptive. An infinite amount of objects that can represent any piece of data on any scale in non space.

Think about how successful minecraft is. When I first played it made me realize how much creativity I had lost to pen and paper through no fault of my own. I was building logic gates without even realizing what formal logic was until I came to college.

Now make it engulf me, make it blank, and give me a wand and an in game terminal.

Re:Finance Pros Say You’ll Have to Pry Excel Out of Their Co... by intended on Friday December 01, 2017 @02:16AM (#15822474)
I always pour water on the AR VR user input ideas - The calorific need is too high - and thinking in 3D results in an unnecessary cognitive load.

Since this thread probably has more Finance users than usual - most people have seen a bloomberg terminal. Its ugly, and filled with arcane short cuts.

Its extremely extremely good at getting its job done, and keyboard short cuts and commands are the fastest system once you gain expertise.

Keyboard short cuts and memorized lists of invocations will regularly be the fastest way of activating commands on a system for experts. Especially once it becomes muscle memmory.

For most abstract symbol manipulation, most jobs can get away with a stack of 2D pages.

If you are working in 3D, then you are doing a pretty unique set of modifications, and the average case will not apply to you.

Which is to say that AR VR will be only for a very niche set of uses.

Re:Finance Pros Say You’ll Have to Pry Excel Out of Their Co... by bonesss on Friday December 01, 2017 @07:19AM (#15823387)
To bolster you point even further: take something like air-traffic control.

Operators are so in tuned with the incoming data that they intuitively make 3D maps in their head, while requiring 2D data for quick observation, comprehension, and real-time monitoring.

Working in a 3D space makes sense for blocking out movies, exploring physical architectures, or certain kinds of research. For everything else there universe has given us an amazing parallel processor connected to eyes that work really nicely on tabular data.

Re:Finance Pros Say You’ll Have to Pry Excel Out of Their Co... by sanityUnbounded on Friday December 01, 2017 @10:09AM (#15824190)
I think you are right. But while I think AR/VR is good for the types of research you mentioned at first glance, I have not seen any comprehensive prototype for working with elementary mathematics in 3D space taught in schools. So I hope you are wrong, and there is room for a hands on/physical type of math.

If something comes out that can help students who struggle with math visualize those concepts in a more (to them) natural way I will be thrilled. To me these concepts were always very visual, and allowed me to pick up concepts in class by thinking that way. I only have experience in the U.S. education system, and in my experience it felt that my teachers always taught in a way that emphasized problem memorization and notation over actually understanding, mentally, the abstract concepts and problem solving. This seems like a barrier for the average student, and leaves a lot of people feeling left behind or "not smart" even though it is likely a fault in the way they are taught.

If something comes out that will allow those mental barriers to be broken, I will consider it a win.

Re:Finance Pros Say You’ll Have to Pry Excel Out of Their Co... by cm2187 on Thursday November 30, 2017 @06:34PM (#15820651)
Data analysis is only one of many use cases for Excel.
Re:Finance Pros Say You’ll Have to Pry Excel Out of Their Co... by osullivj on Friday December 01, 2017 @09:15AM (#15823914)
I've seen that repeating pattern many times too, and I've been the IT guy as well. IMHO the solution is to serverize and automate the Excel models. See my profile for links to my product that does just that, even for sheets with VBA and addins.
Re:Finance Pros Say You’ll Have to Pry Excel Out of Their Co... by hyuuu on Friday December 01, 2017 @01:42PM (#15826100)
this is a very accurate observation of the state of excel. I have always thought that the solution is the have an optimized portion of spreadsheets, specific to the use case, in your example a model that is deployed as a web app. Making a better spreadsheet is a very good point, I wonder, if there are attempts already to solve it from this angle?
Re:Finance Pros Say You’ll Have to Pry Excel Out of Their Co... by Kluny on Friday December 01, 2017 @12:44AM (#15822180)
I think the spreadsheet is fine, it just needs a source control and debugging framework on the site.
Re:Finance Pros Say You’ll Have to Pry Excel Out of Their Co... by odonnellryan on Thursday November 30, 2017 @08:39PM (#15821287)
Export to excel is a great solution.
Re:Finance Pros Say You’ll Have to Pry Excel Out of Their Co... by 1_2__4 on Friday December 01, 2017 @09:11AM (#15823890)
In business individual productivity is not necessarily the most important thing. Companies are about collaboration and communication and if the tooling doesn’t support that then it’s at odds with the company’s needs. Maybe those two finance guys are less productive but now it’s auditable, regulatory compliant, redundant/backed up, and resilient to either of those two people leaving. That seems like a win for the company even if it’s not a win for employee productivity.