Re:For ‘Big Data’ Scientists, Hurdle to Insights Is ‘Janitor...
by deathanatos on Monday August 18, 2014 @01:13AM
(#8191506)
> Like a farmer reaching down to touch the soil, my first step is exploring the rows and columns with little tests here and there to find weaknesses within the information at hand rather than paving over them with instantly flawed reporting.
This needs to be done by more people, more often. I've worked with more than one (often SQL, but it doesn't matter¹) database whereby someone will claim that X is always true. It X is something that can be put into a DB query, then you should do that, and run the query. Every time. In my experience, if it isn't enforced by constraints in the database software, then it isn't true.
¹The nice thing about SQL is that you can but constraints on the data, and use the database to keep various invariants invariant.
Often, you'll find people making assumptions about the data based on the business logic, not the constraints in the database. Such thinking is flawed: you cannot reason about what you think the data is, you must reason from what the data actually is.
Worse, you'll find people making assumptions about what they think the business logic is.
This needs to be done by more people, more often. I've worked with more than one (often SQL, but it doesn't matter¹) database whereby someone will claim that X is always true. It X is something that can be put into a DB query, then you should do that, and run the query. Every time. In my experience, if it isn't enforced by constraints in the database software, then it isn't true.
¹The nice thing about SQL is that you can but constraints on the data, and use the database to keep various invariants invariant.
Often, you'll find people making assumptions about the data based on the business logic, not the constraints in the database. Such thinking is flawed: you cannot reason about what you think the data is, you must reason from what the data actually is.
Worse, you'll find people making assumptions about what they think the business logic is.