Re:Selling Data to Hedge Funds (2017)
by conformist on Monday November 02, 2020 @04:22PM
(#24973554)
Absolutely. Ideally, you'd want both, "append only" and a vintage of the corrected historical data at every point in time.
And good metadata.
Re:Selling Data to Hedge Funds (2017)
by fractionalhare on Monday November 02, 2020 @03:59PM
(#24973274)
Yep. You typically need to incubate your data for several quarters to demonstrate a real correlation. And you have to balance how you prove this against the fact that some firms will try to reproduce the data collection internally once they learn how you get it.
Definitely agree. I'm not on the level of these hedge funds but I do love using data for trading.
Re:Selling Data to Hedge Funds (2017)
by jkinudsjknds on Monday November 02, 2020 @06:56PM
(#24974874)
What do you mean by this? I'm imagining, say, rainfall per day, with a column for date, and a column for rainfall. What's useful about getting the data as it was vs. just having a total dataset with accurate dates.
For something as simple as a sensor reading, you're right, the most natural way to store it is basically an append-only time-series database so it doesn't require much special care.
But say the Rainfall table had a foreign key to a Sensor table that had the coordinates of the sensor. If the physical sensor were moved, the most straightforward data schema would call for the coordinates for the Sensor table to be updated, but then the historic data would have the wrong coordinates if you did a join.
The ideal solution is to design a bitemporal schema so that rows are only ever inserted, not updated, but failing that, regular automatic database snapshots are a good start.
Otherwise, any time you update the data in place, you make it impossible to reconstruct what the data "looked like at the time" for a backtest.