News for hackers. Stuff that matters.
Re:How to share your data effectively by yebyen on Monday April 08, 2013 @03:17PM (#5513841)
There is lots of bad advice in this article. OK maybe the article is shorter than I thought. Here, this is bad:

Use short variable names with no whitespace. Underscores are usually a safe bet, so instead of "Number of new tuberculosis cases" use "incident_tb"

While it might be clear to the domain experts, I had no idea that "incident_tb" meant "Number of new tuberculosis cases" other than that it was placed next to that text.

I think the message is right, but honestly if they are sending you data to parse in PDF, you better just give up now. That person will most likely not have seen a README file before in their life, they're probably not even trying.

Re:How to share your data effectively by krichman on Tuesday April 09, 2013 @03:18AM (#5516857)
I think the rest of the points are good suggestions but "use short variable names" is a terrible suggestion. "incident_tb" is one of the worst possible names that could be used while still being relevant to the data represented.

It's 2013, software can handle titles longer than 8 characters. Even something like "n_new_tb_incidents" (which is shorter than I'd like) buys a lot more clarity at the expense of a few characters. I'd take that bargain any day.

Re:How to share your data effectively by mrtriangle on Monday April 08, 2013 @03:21PM (#5513872)
I agree with you, I actually had a client who demanded that the data in their project be done in excel (with multiple files ranging from 1-3 gb in size!). Both myself and my partner tried exhaustively to use a database or even JSON/XML solutions but the client kept repeating "in the real world business is done in excel!" Needless to say it was a nightmare.
Re:How to share your data effectively by dav-id on Monday April 08, 2013 @03:29PM (#5513935)
I don't have the full picture here but by moving away from an Excel format you are also potentially taking away a lot of control they have themselves over managing the data and working the data which Excel is extremely good at with little to no programming.
Re:How to share your data effectively by omni on Monday April 08, 2013 @05:53PM (#5514888)
Your point is true generally, but I have yet to use a version of Excel that can handle multiple gigabytes of data without serious performance degradation and probably a lot of crashes.
Re:How to share your data effectively by cosmie on Monday April 08, 2013 @07:57PM (#5515540)
Which would matter, if businesses cared. I spend 4 days every month compiling a 1.5GB, Excel-based report. 80% of that time is spent with Excel frozen up and me playing on my phone, hoping that when Excel unfreezes it doesn't crash.

I suggested porting the process to Access with an Excel-based front end, which would cut the compilation time down to 4 hours at the most. I was denied due to the fact that whomever supports it after me will more than likely not have Access experience, and told to continue with the 4 days of hours-long freeze ups.

Corporate life for you, I guess.

Re:How to share your data effectively by ambiate on Monday April 08, 2013 @03:34PM (#5513965)
This is seen all the time in Accounting departments. GAAP and STAT financials in spreadsheets with millions of linked documents and macros. A large amount of this deals with business logic being tied directly in Excel. Their data and the constraints around it are embedded into the spreadsheet. Segregating the data from the spreadsheet means adapting the business logic from Excel and maintaining it in two places. (Source: I work with an accounting department that regularly deals with spreadsheets that take anywhere from 30 minutes to 4 hours to load/process/filter).
Re:How to share your data effectively by rayiner on Monday April 08, 2013 @04:38PM (#5514405)
Excel is absolutely standard in accounting circles. It has the benefit of allowing accountants to do all sorts of things without knowing how to program.
Re:How to share your data effectively by duncans on Monday April 08, 2013 @03:34PM (#5513964)
Maybe your client was wanting ubiquitous Excel-compatible export/import - not for the back end to actually be Excel? Making any table in your app exportable to at least CSV is pretty straightforward.
Re:How to share your data effectively by mrtriangle on Monday April 08, 2013 @08:18PM (#5515644)
I think that is perfectly logical, the project was basically a series of command line tools. I tried to swap out excel for CSV or at least TSV, the client demanded .xls in .xls out. Believe me we tried every suggestion and that was as far as the client would negotiate. We got it to work and I died a little inside.
Re:How to share your data effectively by coldtea on Monday April 08, 2013 @04:48PM (#5514488)
Obligatory "Excel as a database" link (you'll thank me for it):

http://wyorock.com/excelasadatabase.htm

Re:How to share your data effectively by yebyen on Monday April 08, 2013 @03:29PM (#5513933)
That person is obviously bat-shit crazy. I am very surprised to hear that Excel or any other spreadsheet program will open multi-gigabyte files without exploding.
Re:How to share your data effectively by jeffasinger on Monday April 08, 2013 @03:36PM (#5513975)
This.

When moving from a mostly broken .xls data export system to a CSV file with the same data, we got lots of complaints from customers that it was too hard to use.

Re:How to share your data effectively by yebyen on Monday April 08, 2013 @03:43PM (#5514024)
CSV files ARE broken in Excel. Have you ever tried importing a column of data with numbers that contain leading zeros / are not meant as numbers? There are at least 5 ways to do it wrong.

I know how to do this in LibreOffice (requires an extra step above and beyond double clicking) but honestly I do not know how to do it in Excel, and I would not attempt to learn or teach this knowledge to a person who does not have experience with a different spreadsheet tool first.

I will send them .xls formatted files instead.

Re:How to share your data effectively by jQueryIsAwesome on Monday April 08, 2013 @03:25PM (#5513905)
"Well, that explains why the business world is so broken"
Re:How to share your data effectively by TsiCClawOfLight on Monday April 08, 2013 @03:22PM (#5513883)
This can be helpful advice for many people. Why bash him? It's good advice. I would understand "incident_tb" in health context, and I'm 17. So a readme should be plenty, right?
Re:How to share your data effectively by yebyen on Monday April 08, 2013 @03:27PM (#5513926)
Yes, honestly I read down to that point and began writing my comment, assumed there would be lots more bad advice to follow, then the article went and ended on me. The README is the correct solution in this case.
Re:How to share your data effectively by crntaylor on Tuesday April 09, 2013 @07:28AM (#5517396)
> Why bash him?

*her