iblaine

Is the before/after backwards?

https://docdiagram.com - A tool for creating technical diagrams. Basically an upgrade from mermaid. Load it into your MCP, prompt it, you'll get a clean diagram binary file in svg/png/etc.

https://anomalyarmor.ai/ - A tool for managing data observability and data anomalies.

Used daily by customers but also for my own work. These tools fill a gap that helps me get sh*t done.

AnomalyArmor (http://anomalyarmor.ai) is a data observability platform. I got tired of being priced out of existing tools and hand-configuring per-table thresholds for every new dashboard. An AI agent sets up schema-drift, freshness, and row-count monitoring on Snowflake/Databricks in minutes, and you can ask natural-language questions when something breaks.

$5/table, about half what incumbents charge. Happy to trade a 30-day Professional trial code for honest feedback. Reply here or [email protected].

FastMCP is great and this post is overdue. What did FastMCP solve? Lots of code reduction, reduced complexity and the ability to streamline auth. Offloading the auth was a huge benefit. With FastMCP I could replace all my custom JWT validation and get an auth workflow with fewer steps.

Boggles my mind that this is actually a thing that still needs to be solved. Just remote into your computer (I prefer TeamViewer). That is it. One step.

I had a bad experience with greptile due to what seemed to be excessive noise and nit comments. I have been using cursorbot for a year and really like it.

Remind me in 5 years. AI tools don’t hide the fact that the output needs to improve. Calling out the obvious problems is a lazy take.

This is common. Big company buys smaller company. Smaller company execs stay on and establish the idea of independence. Eventually smaller company execs move on and Big company fills the ranks with their employees. Nothing sinister. Just how it goes.

Having been aquihired three times by FAANG+, the biggest take away is have accelerated vesting. To do that you got be lucky or in the C-suite. Being bought out usually sounds better then reality for everyone but a few that get that accelerated vesting clause.

Is there a difference between implicit and explicit joins? No, but use an explain plan to be sure.

(Saved you a click)

An IRC bot that would kindly insult project managers should they join our engineering chat channel while doing a deployment.

The correct answer is I don’t like makefiles when they are abused. They have no state yet people try use them as such and create pain for others.

I used to be bullish on creating ERDs, often using them as an onboarding exercise, even for databases with 100+ tables. However, comprehensive ERDs are becoming rare, and that's okay. Their value is short-lived due to the high cost of maintenance. While polished ERDs can be nice to have, they aren't essential.

For creating ERDs as code, tools like dbdiagram.io and eraser.io are popular options. ERWin is the original tool for UML/ERD diagrams, but it's expensive.

Looking at this reminds me of Apache Pig. That’s not a compliment.

Low code ETL tools (informatica, Appworx, talend, pentaho, ssis) were the original services for ELT/ETL. A lot of progress was made to go towards ETL-as-code starting with Airflow/Luigi. Going back to low code seems backwards as this point.

(I have used all of the above tools in my 15+ yr career. Code as ETL was a huge industry shift)

It’s funny that indexing used to be unavailable when processing big data on Hadoop. Now that we have options to do so we now care. As tech evolves we adapt.

  Location: Santa Barbara, CA
  Remote: Yes
  Willing to relocate: No, will consider sunny tropical islands
  Technologies: Python, PySpark, Spark, Databricks, Snowflake, Terraform, CI/CD, AWS, Postgres, Scala, Kubernetes, SQL, gRPC
  Résumé/CV: https://bit.ly/iblaine-resume
  LinkedIn: https://www.linkedin.com/in/iblaine/
  Email: iblaine+ynews--at--gmail.com

Looking for Staff Software Engineer - Data, Data Engineer, Platform Engineer, tech leadership roles for data platforms/pipelines that operate at TB/PB scale. Ex-LinkedIn, One Medical/Amazon, Abnormal Security.

I'm in the early phases of interviewing. No contractors please.

https://bit.ly/iblaine-resume

My OCD approves of this idea. Let’s also add, IDs cannot start with 0 or O.

One of the best gifts you can ever gift yourself.

Is crypto or the pastor more to blame?

TL;DR; do the easy things fist, in this case it was to fix bad SQL

Given the options to optimize SQL, move read operations to replicas, shard data or go towards micro services, optimizing SQL is the easy choice.

A walking bike with a wheel in the front and 6 Theo Jansen type legs in the back

https://www.thedailybeast.com/judge-aileen-cannon-comes-out-...

Agreed. The job title of Data Engineer gets abused. Some DE's are Analysts, others are Backend SWEs. Their goals are aligned, to create value out of data, and their methods may differ greatly.

The Data Engineering track seems to attract data analysts and non-engineers. I’m one of them, have worked with many others in my same bucket. What makes Data Engineers excel, I think, is to have all the skills of a software engineer + the ability to think like a data analyst.

SQL is definitely part of that journey, and more skills should be mastered if you’re going to do things like use Kafka, airflow or spark.

I saw the kickstarted version of this at a maker fairy and was blown away. The product was great for what it did.

Idea for a new linter; throw a warning if it sees extreme non-intuitive uses of Python. Given the trajectory of Python improvements over the past few years, this linter may not be far off.

It’s monumentally frustrating to be forced to give estimates too soon in the planning process. The compromise, I hope, is to end up in a place where most of the problem is known and some is unknown, and you get there by breaking down the original problem. Leave it up to the “buyer” to digest your best estimate into their planning process.

Same thing happened at LinkedIn after the Microsoft acquisition. LinkedIn was using gmail, Google was quick to let the g suite contract die, so the timeline to move from gmail to Outlook was accelerated. Engineers at LinkedIn were livid about being forced to move off gmail.

If I were building a tool to test SQL, then I'd try to load the SQL into a dataframe, then test it by mocking the tables and the output. This is a tough problem to solve. If testing is important, possibly move away from SQL and towards ORMs.

CTEs still give me anxiety due to their inability to be potentially not be optimized. I know many database engines today optimize for this type of thing, I'm thankful for it, but there was as time 10+ years ago when CTE expressions could take down databases. I also blame DBT for convincing people that CTEs are good. CTEs are only good if used correctly.