bertomart

I agree, but you're looking at everything through an automated ETL workflow which sorta invoke the idea of using a tool like airflow. But generally you don't need that. What you need is a specific set of functionality (training, tuning, testing) so you can either kick off manually when you wanna do things interactively or hook up into a pipeline when you wanna automate things. And yes you can track everything using MLFlow on databricks. Matter of fact Delta Lake is one of the most powerful features as you track not just migrations but data changes, so you can track end-to-end lineage and so far it's the ONLY platform (that I've tested) that allows this with ease.

They (Databricks) are not advertising themselves as a best practices way of doing anything, they're just a platform for doing data science and analytics. It's up to the data scientist to use the tool properly and to have proper methodologies. A platform is simply a set of tools that helps a target audience, and that's what databricks is. Now the issue here is that a lot of datascience either cannot code/do engineering or just don't want to. But in my view people like that are just glorified analysts. Data Science is built on the foundation of software engineering (+ stats, viz, math, etc...); this is why it's so complicated. If you cannot code or don't understand the SDLC or best practices you're basically a carpenter who can't hammer or saw.

Also there's a level of automation for model tuning (experiments) especially if this is an update to an existing model (you still need to do model validataion). For the initial model there's gonna be automation but a lot of time will be spent validating the model with business KPI, not just some model metric like accuracy. Databricks makes this automation and tracking pretty simple. You don't have to do this in notebooks. Once you're done with exploration, build out the pipeline in your "framework" and kick off jobs to execute it on databricks. Yes you can use ETL tools here to, I dunno, train every x hours...whatever

Can't disagree with you more here. Databricks is a tool. It's on you to create your frameworks or whatever to optimize your workflow. I've been using Databricks and it's such a great platform, at least compared to what I've used in the past (EMR/Sagemaker). Simply build a framework where you offload the notebooks and exploratory stuff to databricks and build out productionable pipelines locally using sample data from dbricks, then test on databricks. That's what we've been doing and it's working very nicely. Furthermore you can remotely kick off jobs on databricks, so once you have your pipeline worked out you really don't need to interact with notebooks if you don't want...just use it as a scalable backend, dashboard, model tracker, whatever fits in your workflow.

I feel like instead of using the tool a lot of folks have specific expectations and if the tool doesn't fit exactly into how they work then it's just written off.

Does it have a question/answering component that's trainable/finetune-able on custom datasets?

Databricks is not an RDBMS...it's a Unified Analytics Platform (their words i know), but that really describes it well. It's been really liberating to having easy access to versioned data at your fingertips when iterating on models...and also being able to track models and share work. It does have some warts but it's much better than any other single platform I've used for data science/ML

What are you trying to say? over 16 people are just lieing! Whatever dude, you're privilege to not have to face the bs. Then why don't you go get the details before you start talking smack

this is pretty much a every company thing...at every company i've worked at...not surprised

just in-time for AMD's computex keynote...nicely played

i don't watch tv anymore due to the excessive 'spin' and sometimes straight out lies. usually they start off by saying "allegedly" and by the time they're done you'd think it's one of Newton's laws of motion

I like it, pretty nice! Just used it