Sometimes the spreadsheets are the output of one system, but the input to the next: the spreadsheets of the Office of National Statistics in the UK are the final, and official output of a long chain of number wrangling, but when it comes to re-consuming them to power their new data-driven website it's a real challenge -- I wound up working on a project to allow programmatic consumption of these complex spreadsheets.
(https://github.com/sensiblecodeio/databaker for the top-level project; https://github.com/sensiblecodeio/xypath for the library I created).
Get in touch if you're interested -- [email protected]
Oh fantastic, I want to do more with the data but the spreadsheets are designed for people rather than machines. Will have to check this out, thanks!
See also DataWrangler.
No enterprise application is complete until it gets Export to Excel feature.
Re:Data Organization in Spreadsheets
by andyidsinga on Monday August 20, 2018 @03:00PM
(#17802816)
I've seen and heard from colleagues that there is a lot of data being stored and shared in CSV too ..and then analysis done in excel.
that seems to change as an automation and IT culture starts to develop in an organization; interestingly, IT can sometimes be a barrier to data organization (too slow, cross-org costs etc) and cause people to continue to use CSV and excel.
fwiw - we built a service called "yukon data solutions" that solves these classes of problems for people trying to transition from CSV and spreadsheets into automated and reproducible analytics and reporting.
FYI Yukon was the codename for Microsoft SQL Server 2005. Dunno if you care.
Re:Data Organization in Spreadsheets
by andyidsinga on Monday August 20, 2018 @08:26PM
(#17805594)
oh - that's cool, didn't know that :) Our's is based on the Canadian Air force CC-106 Yukon.
While I don't necessarily agree with the author there, I don't think there is contradiction between their and your statements. Tools are not necessarily commonly used for what they are best suited for.
Also the context is bit different; research vs businesses
Spreadsheets are also great UIs for configuration files to be used for models. Perhaps not for the programmer (though you can write your excel serializer/deserializer) but for users yes. If you have to make large changes on many entries, try updating a json or xml file by hand, or spend two hours clicking through buttons and drop downs in a UI vs a few seconds on a spreadsheet for any decent excel user.
in every organization I've worked in, its usually the exact opposite - spreadsheets are used to do analysis and create reports while the source data is coming from other systems (SAP, Salesforce, quickbooks, etc)