idomi

I think it really depends on your need, if you're working on your local and trying to get something to work, caching, scale etc, might be an overkill.

Pretty cool, it's about time we got some facelift for Shiny.

Come join us!

Feel free to open an issue if you're looking for new features/something's missing. Also we'd love to chat more about your Azure Data Studio!

When data scientists are training locally on small datasets, they have to deal with dependency installation, parameterization, and provisioning infrastructure. Once they want to train the model for production on a full load dataset, the complexity increases, and a new configuration must be considered.

Today we're launching a GUI feature in ploomber to solve this issue, by allowing our users to drop notebooks and execute them on the cloud without spinning up clusters or worrying about any infrastructure.

The service is based on our open-source software and have a free-tier that allows to scale multiple models before depleting the quota.

Would love to hear thoughts and impressions of it! You can also reach out to me directly at [email protected]

+1 I also think it's faster that way on both environment setup and ad hoc rapid experiments, from my experience using the library in a team doesn't scale well, it becomes pretty slow.

Pretty interesting. I think this is part of this notion to release half baked products, like some of the stuff in there are really cool, just enough to get you in but it doesn't scale and usually is complex to deploy/use.

How many data scientists that use Databricks for modeling do you know?

Hi, we’re Ido & Eduardo, the founders of Ploomber. We’re launching Ploomber Convert today, a web-based application that allows data scientists to convert notebooks to PDF, no setup required.

As data scientists, we have to share our work with non-technical colleagues to communicate results. To allow them to read our findings, we use nbconvert, which enables us to export notebooks to PDF or HTML. Unfortunately, nbconvert requires Pandoc, TeX/XeLaTeX, Pyppeteer, Chromium, and other packages, which is complicated. Ploomber Convert provides the nbconvert functionality without installing a single package.

Ploomber Convert is built on top of AWS and runs all the necessary packages in a docker container. Since notebooks often contain sensitive information, we do not store any notebooks or PDF files.

Ploomber Convert is free to use. Go to https://convert.ploomber.io, drop your Jupyter Notebook to convert, hit ‘Convert to PDF’, and save it.

We want to make Ploomber Convert the go-to tool for data scientists to turn their notebooks into shareable reports. We’re working on adding support for Quarto, custom CSS templates, export to HTML, and other features. Let us know what else you need!

We’re thrilled to share Ploomber Convert with you! So, if you had difficulties exporting your Jupyter Notebooks or if you have any feedback, please share your thoughts! We love discussing these problems since exchanging ideas sparks exciting discussions and brings our attention to use cases we haven’t considered before!

I doubt it, they got too much traction at this stage.

First you can always research on your own, there's tons of resources online and in git. If that doesn't work I find a teammate or a friend with the right domain expertise. There are also some useful slack communities for instance on MLOps etc.

There are so much stuff you could do with it right?

How is this different than turbotax for instance?

Thanks for sharing! Yeah definitely a problem, we've been trying to build it mainly on the feedbacks we're getting from the open-source solution.

You're right to some extent - at least on some of the concepts. We don't focus on a specific ML use case, like computer vision. In addition, we're oriented towards notebooks and this notion allows us to break it into smaller tasks, cache the results and execute in parallel (locally or via this new cloud service). BTW, we tried talking to the founders but couldn't get a hold of them, if you or anyone know them - we'd love to chat!

Yes, you should follow best practices and isolate each job to the smallest task possible (and then reuse components). We have this functionality in 2 flavors, you can define hooks as part of your pipelines(https://docs.ploomber.io/en/latest/api/spec.html#id2), in addition you can define this dependency as part of your jobs DAG (https://docs.ploomber.io/en/latest/get-started/basic-concept...), e.g get the data, clean it, train the model and test it.

Make sure to checkout ploomber, our support is seamless tons of docs (https://docs.ploomber.io/) and we take our users seriously. P.S. We integrate with airflow and other orchestrators if you still need to tackle those.

When it comes to scale and DS work I'd use the ploomber open-source (https://github.com/ploomber/ploomber). It allows an easy transition between dev and production, incrementally building the DAG so you avoid expensive compute time and costs. It's easier to maintain and integrates seamlessly with Airflow, generating the DAGs for you.

Ploomber actually can work with shell script so you could export you js code into a pipeline. But besides that not that I know of.