Peteris

An often overlooked framework used by NASA among others is Kedro https://github.com/kedro-org/kedro. Kedro is probably the simplest set of abstractions for building pipelines but it doesn't attempt to kill Airflow. It even has an Airflow plugin that allows it to be used as a DSL for building Airflow pipelines or plug into whichever production orchestration system is needed.

Exactly - this release train metaphor that Spotify uses explains this visually:

https://medium.com/pm101/spotify-squad-framework-part-i-8f74...

One of the reasons smart contracts work so well is that they retain some of the properties of libraries yet are monetizable.

If you are feeling overwhelmed with yet another machine learning pipeline automation framework, you should check out Kedro (https://github.com/quantumblacklabs/kedro).

Kedro has the simplest, leanest, functional-programming inspired pipeline definition and also spits out AirFlow and other formats readily + comes in with an integrated visualisation framework which is stunning & effective.

Anyone else came here expecting a Haskell post?

Ditto.

You should try http://roamresearch.com/ and follow #RoamCult on Twitter - they are way ahead on note taking.

The author is forgetting that Google is a search monopoly.

AI was (and continues to be) the biggest threat to their success in search so they invested heavily in that (it was called ML back then).

Clearly they are still doing well even with subpar cloud software, so this has proven out.

This seems to lend more credit to ReasonML. Compile times are faster and it doesn't bring the associated complexity of object types while still retaining the benefits of static typing.

Paywalled.

Tell them you're looking to join a company precisely to get mentorship, see how a successful company operates, learn to work in larger product lines, etc. That way you're pre-emptively going around the Founder damage issue.

One way to get around this is to use Kedro https://github.com/quantumblacklabs/kedro, which is the most minimal possible pipeline interface, yet allows you to export to other pipeline formats and/or build your own exporters.

Airflow is an incredibly powerful framework to use in production, but a little unweildy for anything else.

You can use something like Kedro (https://github.com/quantumblacklabs/kedro) to get started building pipelines with pure Python functions. Kedro has its own pipeline visualiser and also has an Airflow plugin that can automatically help you generate airflow pipelines from Kedro pipelines.

https://github.com/quantumblacklabs/kedro-airflow

Use Roam Research.

I had a similar problem until I knew I had to work on a start-up and pick a market to go help. It took me ~2 years to choose that market but I haven't looked back (crypto developer tools).

Look back, not forward.

Take all the things you have spent the past 5-10 years being interested about. For me, it was coding, programming languages, developer tools, design & minimalism, products, B2B, mathematics, economics, investing, productivity. Crypto dev tools is the intersection of those - a new field allowing me to keep doing the things I'm already provably interested in rather than making bets about the future.

Side note: the best way to build Airflow pipelines is through Kedro https://github.com/quantumblacklabs/kedro-airflow.

Haskell is not boring.

Kedro puts emphasis on seamless transition to prod without jeopardizing work in experimentation stage:

- pipeline syntax is absolutely minimal (even supporting lambdas for simple transitions), inspired by the Clojure library core.graph https://github.com/plumatic/plumbing

- sequential and parallel runners are built-in (don't have to rely on Airflow)

- io provides wrappers for existing familiar data sources, but directly borrows arguments from Pandas, Spark APIs so no new API to learn

- flexibility in the sense you could rip out anything, for example, the whole Data Catalog replacing with another mechanism for data access like Haxl

- there's a project template which serves as a framework with built-in conventions from 50+ analytics engagements

5-6, then iterate.

I'm interested in trying Rascal, but it's meta-meta: https://www.rascal-mpl.org/.