News for hackers. Stuff that matters.
Re:Dear AI startups: Your ML models are dying quietly by ohazi on Tuesday April 23, 2019 @02:00AM (#19725964)
I don't see how this has anything to do with AI or ML. It's a great description of why you might want to prefer strongly typed languages, avoid "the data is the schema" key-value monstrosities like mongodb, and maybe think about writing some sanity-check-tests that need to pass before deployment, though.

No system should ever fail silently if a required field suddenly goes missing or has the wrong type/unit.

Re:Dear AI startups: Your ML models are dying quietly by nabla9 on Tuesday April 23, 2019 @04:39AM (#19726621)
The problem with numerical programming is that all data is the same type. If you access wrong row, multiply across wrong axis, function uses wrong parameters, it's semantically wrong, but it can still work and system can learn something.
Re:Dear AI startups: Your ML models are dying quietly by amelius on Tuesday April 23, 2019 @05:18AM (#19726798)
Isn't there a typesystem that can help with this?

E.g., give rows a type, give columns a type, and then if you multiply two matrices, see if the type of the columns of the first matrix and the type of the rows of the second matrix match.

Also, the typesystem could use units, just as they are used in physics.

Re:Dear AI startups: Your ML models are dying quietly by chpatrick on Tuesday April 23, 2019 @06:40AM (#19727231)
We use this at our startup. We have a Haskell math library where every vector/matrix/rotation knows the spaces they deal with (World, Camera, etc), and using them with incorrect spaces is a type error. This is enormously useful because in 3D computer vision/graphics almost everything is 3 or 4-dimensional, so it's very easy to use them the wrong way. When you read C++ computer vision code a datatype often has a Matrix3 or worse yet a double[] in it and you have no idea what coordinate system (or even matrix order) it represents. With this, it's all self-documenting and difficult to get wrong.
Re:Dear AI startups: Your ML models are dying quietly by em500 on Tuesday April 23, 2019 @07:29AM (#19727483)
In pandas you could/should use dataframe index and column labels that way, it automatically aligns by column and index names before arithmetic, unlike raw numpy arrays which aligns by position. It takes more discipline and some performance hit though.

You could even do it with SQL field (column) names and natural joins. The performance hit is even bigger and discipline/retraining is even bigger, as few SQL users are used to thinking about column names (rather than types) as type definitions.

Further, this would only catch a class of bugs that are relatively easy to spot manually. The difficult case is when the processes that generated the numbers have slowly changed in time (e.g., you have a very different mix of visitors/customers now than during model training).

Re:Dear AI startups: Your ML models are dying quietly by nabla9 on Tuesday April 23, 2019 @05:44AM (#19726928)
You would like to have 'unit system' that would keep track of the units of measure in general and the ability to use concepts like dimensional homogeneity to avoid semantic errors while programming. Alas, they don't exist as far as I know.
Re:Dear AI startups: Your ML models are dying quietly by alkonaut on Tuesday April 23, 2019 @06:16AM (#19727105)
F# has units of measure. A frontend is (usually) not F#, but even for a more loosely typed system such as your typical web frontend, it should be possible to represent scalars as tuples of a number and an identifier. Eg. { qty: 12, unit: "kgs" }.
Re:Dear AI startups: Your ML models are dying quietly by anonymou2 on Tuesday April 23, 2019 @06:07AM (#19727055)
Re:Dear AI startups: Your ML models are dying quietly by AKrumbach on Tuesday April 23, 2019 @07:36AM (#19727548)
A search on Google turns up this Common Lisp code, which appears to define a number of SI units (both base and derived) in the sort of manner described.

http://www-ksl.stanford.edu/knowledge-sharing/lib/unit-conve...

Given the way this appears to be defining some sort of class-based hierarchy, with the correct type requirements this code should perform exactly the sort of unit/type checking the grandparent post wanted, but I didn't go digging into the rest of the project files to confirm this code defined things in the correct manner for that to work.

Re:Dear AI startups: Your ML models are dying quietly by carlmr on Tuesday April 23, 2019 @06:16AM (#19727106)
Re:Dear AI startups: Your ML models are dying quietly by alehander42 on Tuesday April 23, 2019 @11:43AM (#19729691)
Nim also has it
Re:Dear AI startups: Your ML models are dying quietly by dual_basis on Tuesday April 23, 2019 @08:02AM (#19727723)
There is a namedtensor library that tries to fix this.
Re:Dear AI startups: Your ML models are dying quietly by ayazhan on Tuesday April 23, 2019 @12:33PM (#19730197)
Thanks for your feedback ohazi! You are right that “No system should ever fail silently” and there are many ways to address this. However, we’ve seen cases where not enough attention is put into input data pipelines and similar mistakes are being made by new and even well-funded AI startups. One of the reasons this is happening is because some of these startups are launched by people with little experience in machine learning / data science. For example, we’ve seen doctors, web developers, lawyers and accountants moving into AI space. They can learn the basics of machine learning and maybe hire someone to help them, but often don’t know about the best practices or things to watch out for when building a data driven company. Unfortunately, there are few guidelines or best practices available online. With this article, we were hoping to help these new AI startups by bringing awareness to the importance of data pipelines.
Re:Dear AI startups: Your ML models are dying quietly by bartimus on Tuesday April 23, 2019 @02:30AM (#19726111)
Good point. Or perhaps it was that nobody had ownership over the recommendation system. There should've been a performance indicator on a dashboard somewhere.
Re:Dear AI startups: Your ML models are dying quietly by jimmyechan on Tuesday April 23, 2019 @11:07AM (#19729346)
Yes! It could actually be both. Ownership over the recommendation system needs to include not just designing it and training it but also making sure that the upstream data that feeds the model remains good. And I agree that a performance indicator is a potential solution, it just needs to be able to catch these kinds of problems.