News for hackers. Stuff that matters.
Re:Nasdaq Acquires Quandl to Advance the Use of Alternative ... by claytonjy on Tuesday December 04, 2018 @02:23PM (#18602817)
We're not talking about data cleaning, but about data validation. I can fix a weirdly formatted field (cleaning), but I can't reliably impute most kinds of missing data. I can detect errors, but can't fix most of them without additional information...which is exactly what I'm paying the provider for.
Re:Nasdaq Acquires Quandl to Advance the Use of Alternative ... by vasilipupkin on Tuesday December 04, 2018 @02:24PM (#18602822)
you can, there are ways to do it. interpolation, etc. Sometimes the data is missing just because it's not available, you still have to handle that case. proper way of filling in this missing data will depend on what you are using it for - so for provider to do it would be kind of wrong actually.
Re:Nasdaq Acquires Quandl to Advance the Use of Alternative ... by claytonjy on Tuesday December 04, 2018 @02:45PM (#18602988)
I think we're talking past eachother here. I don't expect the provider to do imputation for me, but I shouldn't have to bug them to get the best version of the data they have. Sure, sometimes missing is missing, but in my experience with Quandl/Zacks, its usually an error on their end. The price jumps are sometimes because they conflated two different tickers. If they divided instead of multiplying (split factors), I have to have external information to even detect the error! Same goes if they get a date wrong somewhere.