News for hackers. Stuff that matters.

sync_jeff

Re: on Thursday April 11, 2024 @12:07PM (#40003727)
Understanding why your job performance is changing is critical to both optimize costs and prevent crashes. In this post we show a new visualization timeline features to help engineers find root causes quickly
Re:Rethinking Serverless: The Price of Convenience on Monday February 12, 2024 @07:33PM (#39352672)
Thanks for reading!
Re: on Wednesday February 07, 2024 @11:57AM (#39290953)
We provide our own perspective on the high level comparison between Databricks and Snowflake
Re: on Tuesday November 28, 2023 @09:09AM (#38445824)
We built an ML-powered Databricks cluster optimization tool to help remove the tedious work of tuning Jobs clusters to lower costs.

We recently shipped a huge upgrade to our product, Gradient, that can automatically improve your job cluster settings to hit your business goals.

This just came out of private preview where we had awesome results with early testers. We're looking for more feedback and would love to have folks here give it a spin!

Check out our demo video here: https://www.youtube.com/watch?v=i09x6B6mtjg

To get started and try for yourself, see our docs!

Re: on Wednesday October 18, 2023 @09:03AM (#37928047)
We ran a $12K experiment to test the cost and performance of Serverless warehouses and dbt concurrent threads, and obtained unexpected results.
Re: on Tuesday October 10, 2023 @07:55AM (#37830947)
Databricks provides two different ways to monitor usage and costs. In this post we look at the pros and cons of both solutions.
Re: on Monday September 11, 2023 @06:55AM (#37465406)
Realities of company and cloud complexities require new levels of control and autonomy to meet business goals at scale
Re: on Sunday September 10, 2023 @01:51PM (#37458218)
Realities of company and cloud complexities require new levels of control and autonomy to meet business goals at scale
Re: on Monday August 21, 2023 @07:16AM (#37207952)
In this blog we run the TPC-DS benchmark to see the cost and performance benefits of Photon and Graviton instances on Databricks. The results are intriguing and reveal a techno-economic struggle!
Re: on Tuesday July 11, 2023 @09:47AM (#36680917)
Our new tool that enables users to programmatically optimize their Databricks clusters.
Re: on Tuesday June 27, 2023 @08:46AM (#36491982)
In this blog you'll learn how to integrate Gradient, an optimization service, into Airflow + Databricks to lower costs and hit runtime goals.
Re: on Friday June 23, 2023 @10:10AM (#36447049)
In our blog we discuss our data science work on how we built a system that optimizes Databricks clusters.
Re:Sync Autotuner Reduced Our EMR Cost by 25% on Tuesday April 04, 2023 @03:16PM (#35444587)
Customer blog on how the Sync Autotuner API integrated with their Airflow infrastructure helped to optimize their ApacheSpark clusters.
We perform a cost and runtime evaluation of Graviton instances for Apache Spark on EMR AWS using the TPC-DS 1TB benchmark.
In this blog we analyze the impact of varying the worker instance size on cost and performance of Apache Spark in EMR using the TPC-DS 1TB benchmark.
Re: on Wednesday January 25, 2023 @02:03PM (#34522672)
We take a deep dive into the costs performance of Databricks's autoscaling algorithm with the TPC-DS benchmark.