Re:We don't need data scientists, we need data engineers
by dragonwriter on Monday January 18, 2021 @06:49AM
(#25820789)
> The second you use pandas with a custom data type
That's a pandas (and maybe numpy) issue, not a dynamic language issue. (If you want to generalize from the specific libraries more accurately than “dynamic language”, it's “using a low-level library whose type system doesn't match the host language type system” issue.
> Quants tend to use fixed precision types. It is like a float in every way, except base 10 instead of base 2 so there is no floating point error.
No, a type that is like binary floating point in every way except base 10 instead of base 2 would be decimal floating point, not fixed point. Decimal fixed point is different from binary floating point in more ways than base.
Re:We don't need data scientists, we need data engineers
by hntrader on Thursday January 14, 2021 @03:58PM
(#25782403)
Quants don't care about floating point precision in research. It's just applied stats
Re:We don't need data scientists, we need data engineers
by proverbialbunny on Thursday January 14, 2021 @07:36PM
(#25785287)
I do, because the results from my research varies when I'm validating the model.
You would be surprised. The second you use pandas with a custom data type (let alone any other library you'd want to use) it can randomly auto convert it to a float. Furthermore identifying when it randomly converts the type on you is a pain.
>so floating point basically must be used.
Quants tend to use fixed precision types. It is like a float in every way, except base 10 instead of base 2 so there is no floating point error.