smif

Kubernetes 1.25 (in Danny Glover voice): I'm getting too old for this shit!

I interpreted their notion of loyalty as like a sort of trust that things will work out in the end, "maybe this is just a rough patch", etc.

Re: neglect, I think there's a more subtle gradient there. Lets say your company decided to implement a dumbass policy X, and you rail against it, but since it's highly popular it's like you're pissing against the wind. So you kind of resign yourself to it's inevitability and make a long bet that it's going to fail. This is in the hopes of a brighter future where it has already failed and been replaced with something better. You might even make decisions during this time that set you up to be better positioned for when it does eventually fail. This is a kind of negative semi-passivity, but it's only negative in the sense that you're betting against something that you think is a bad idea and you're trying to improve your position within that potential better future after the bad thing is gone.

That's not strictly the same thing as sabotage which would require malicious actions on your part to undermine the effort. You don't even need to hope that it fails, you can just make impartial/rational decisions to prepare for it's eventual downfall without being heavily emotionally invested in that same downfall. And on top of that, you could rationalize all of that as setting the company up for future success after it gets through this rough patch, so it's not all about self-interest.

> ORMs don’t fix everything but it does help constraint the ‘degrees of freedom’ and help keeps logic repeatable and consistent, and generally better than writing your own string-manipulation functions. An idea I had I continued (I wrote the post early last year) was to use static analysis tools like Meta’s UPM to allow refactoring of tables / DAGs (keep interfaces the same but ‘flatter’ DAGs, less duplicate transforms).

I get what you're saying, but think about a large org with a lot of different teams and heterogenous data stores - it's gonna be pretty hard to implement a top-down directive to tell everyone to use such and such ORM library, or to ensure a common level of ownership and craft. This is where SQL is the lingua franca and usually the native language of the data stores themselves and is a common factor between most/all of them. This is also where tools like Trino / PrestoSQL can come in and provide a compatibility layer at the SQL level while also providing really nice features such as being able to do joins across different kinds of data stores / query optimization / caching / access control / compute resource allocation.

In general it's hard to get things to flow "top down" in larger orgs, so it's better to address as much as you can from the bottom up. This includes things like domain models - it's gonna be tough to get everyone to accept a single domain model because different teams have different levels of focus and granularity as they zoom into specific subsets so they will tend to interpret the data in their own ways. That's not to say any of them are wrong, there's a reason why that whole data lake concept of "store raw unstructured data" came in where the consumer enforces a schema on read. This gives them the power to look at the data from their own perspective and interpretation. The more interpretation and assumptions you bake into the data before it reaches the consumers, the more problems you tend to run into.

That's not to say that you can't have a shared domain model between different teams. There are unsurprisingly also products out there that provide the enterprise the capability to collaboratively define and refine shared domain models, which can then be used as a lens/schema to look at the data. Crucially the domain model may shift over time, so this decoupling of the domain model from the actual schema of the stored data allows for the domain model to evolve over time without having to go back and fix the stored data because we have not baked in any assumptions or interpretations into the stored data itself.

Sure thing! I'd say first off, the solutions may look different for a small company/startup vs. a large enterprise. It can help if you explain the scale at which you are solving for.

On the enterprise side of things, they tend to buy solutions rather than build them in-house. Things like Informatica, Talend, etc. are common for large enterprises whose primary products are not data or software related. They just don't have the will, expertise, or the capital to invest in building and maintaining these solutions in-house so they just buy them off the shelf. On the surface, these are very expensive products, but even in the face of that it can still make sense for large enterprises in terms of the bottom line to buy rather than build.

For startups and smaller companies, have you looked at something like `dbt` (https://github.com/dbt-labs/dbt-core) ? I understand the desire to write some code, but often times there are already existing solutions for the problems you might be encountering.

ORM's should typically only exist on the consumer-side of the equation, if at all. A lot of business intelligence / business analysts are just going to use tools like Tableau and hook up to the data warehouse via a connector to visualize their data. You might have some consumers that are more sophisticated and may want to write some custom post-processing or aggregation code, and they could certainly use ORM's if they choose, but it isn't something you should enforce on them because it's a poor place to validate data since as mentioned there are different ways/tools to access the data and not all of them are going to go through your python SDK.

Indeed in a large enough company, you are going to have producers and consumers that are going to use different tools and programming languages, so it's a little bit presumptuous to write an SDK in python there.

Another thing to talk about, and this probably mostly applies to larger companies - have you looked at an architecture like a distributed data mesh (https://martinfowler.com/articles/data-mesh-principles.html)? This might be something to bring to the CTO more than try to push for yourself, but it can completely change the landscape of what you are doing.

> More broadly is the issue of the gap of what you think the role is, and what the role actually is when you join. There are definitely cases where this is accidental. The best way I can think of to close the gap is to maybe do a short-term contract, but may be challenging to do under time constraints etc.

Yeah this definitely sucks and it's not an enviable position to be in. I guess you have a choice to look for another job or try to stick it out with the company that did this to you. It's possible there is a geniune existential crisis for the company and a good reason why they did the bait-and-switch. Maybe it pays to stay, especially if you have equity in the company. On the other hand, it could also be the case that it is the result of questionable practices at the company. It's hard to make that call.

It sounds to me like the author got thrown to the wolves in an environment of what data engineering looked like before "big data" and ML took off (and before it was even really called data engineering). There are a lot of enterprises that are still working in this mode because they are not Google and they don't have the same level of sophistication and automation when it comes to this stuff.

There is some bad information no doubt in the article, but if we're being charitable, it feels like it's someone who took a wrong turn somewhere and is struggling to find their feet in an unfamiliar place without the proper guidance and mentorship and that's a bit admirable at least that they're trying on their own.

There is no direct bearing on ETL in the article, aside from the focus on SQL queries and data validation hints that they might be talking about ELT (Extract-Load-Transform) as the level beyond ETL, but it's not clearly explained. It's clear to me that they are at the start of their journey and they are gonna learn things the hard way without guidance from someone more experienced.

I guess I'm just having trouble visualizing how this problem gets solved for the average joe consumer in a world where hypothetically the hardware is open. Who pushes the security patches out to the devices? All of that has a cost in terms of bandwidth, maintenance, etc. If it's a community effort, what happens when the device gets old enough where no one is really working on it anymore, no more community updates, people have moved on, etc. How does liability work in a world with community-driven updates? What happens if a buggy community update is pushed and the smart fridge malfunctions and causes a flood / damages? What about supply-chain attacks and such?

I guess for the code-literate subset of consumers, they can just go to the github repo and see exactly what is changing and where, but for the non-code-literate consumers, how do they know what kind of updates they are getting from the community?

What about a middle-ground option? Where towards the end-of-life for the product, you are asked a question if you want to switch to a different update channel than the manufacturer default, and if there is no response recorded after X amount of days or whatever, the device just bricks itself?

This is basically what I'm arguing - no matter how accurately our theories line up with observation, we can never be sure that we have reached "the final theory" AKA the truth. I think this is where a lot of misunderstanding and mistrust for science originates. It will never deliver to us the truth - if it did, how would we ever know?

It is a method of getting closer and closer to what we believe is the truth. But there is still a gap there, however small it might be in the case of quantum mechanics. The scientific method by it's very construction is unable to bridge that gap.

Still, to date there does not exist a more effective method we know of as a species at getting closer to what we believe to be the truth. I think the above is a maybe subtle distinction there that is worth pointing out and educating people on. Just sort of making that distinction between the process and the results. That it is the best process we have, but even so, it cannot cross that gap and definitively say "this is the truth". That that is a gap we have to choose whether to cross ourselves with a leap of faith (or sometimes a very tiny hop of faith in the case of quantum mechanics). I think that might help people cement their faith in the process even if they dont necessarily place their faith in the results (in the case of questionable psych studies for example).

No, flat earth "theory", if you can call it that, has close to zero supporting evidence and AFAIK has no actual predictive power. Stick with consensus science if you want actually useful theories, but that is very different from claiming they are giving you objective truth.

Let me ask you this, when a theory that was previously accepted as consensus science loses support in light of new evidence and gets supplanted by a new and better theory, does that mean that the objective truth changed?

This is a pretty bold claim and you would have to do a bit of work to make it more convincing. Besides, it's not really how science works. Different theories wax and wane over time all the time. What we're really doing with science is coming up with ever better models that give us greater predictive power.

You could argue that at the macro scale we're asymptotically approaching some kind of objective truth with the whole endeavor of science, but you can't simply tunnel across that gap and make the leap to say that we know there is an objective truth.

The best that we can do is probably say something along the lines of "these are the best methods of getting closer to the truth that we have available - if anyone claims to have better methods, they are very likely wrong", but you still need to have the humility to accept that even the best models that we have to date are not infallible. They do not give us the objective truth, nor do they promise to, but they are the most effective tools that we have available to us at the current time. This is critically not the same as claiming that they give us the objective truth.

You could say that for all intents and purposes/everyday usage, "sure, these are objective facts about reality" - but I don't actually think that helps anyone and it serves to foment mistrust towards science and scientists. I think the best chance we have at preserving the status of science and scientists in our society starts by being honest about what it actually is giving us - which is quite a lot, but let's not oversell it for the sake of convience or whatever.

Doesn't seem to really answer anything. Isn't this just a really fancy way of saying "we don't know the solution to the Drake equation"? It could be a 99.999999...% (<100%) chance of being alone, it could be a 0% chance, or anything in between.

Given the title of the paper, this is a very loose definition of "solution" for the Drake equation ("it could be anything!").

In this way you could be led to a kind of inverted Pascal's wager, where you can't reasonably go down the nihilist route because everything might just might matter, but you just don't know. You also don't know in which ways it might matter if it does, so you don't really have a conclusion to draw about where to go from here.

ie., they're saying that since understanding is relative to a perspective so is what is understood. This is the irritating fallacy popular amongst those who go around abusing the term 'relative'.

I'm not sure if I find this particularly convincing. I think most lay people are actually talking about the thing itself rather than their understanding/perspective on the thing.

Maybe this is something that a study could shed light on, if we're able to construct a survey that can give us some insight into how lay people are using/abusing these terms.

I've actually found the opposite - people will often use "subjective" when they really mean "relative". They often unwittingly imply the relation to culture. I think it's far more rare to find a lay person using the term "subjective" to precisely refer to the philosophical meaning of the word.

> And lay people use 'objective' to mean 'universal', which it doesn't. Cultural relativism is consistent with an objective metaethics.

It's true there is a distinction there between "objective" and "universal", but do you feel that in most contexts, when someone says "objective" they are also implying "universal" and the onus is on them to elaborate on the distinction when that implication doesn't hold? (I guess asking for both the case of lay people and the one for philosophers).

It's kind of like how when we (programmers) say "functional programming" we have a very specific concept in mind. If a lay person read and interpreted that phrase literally, they would probably not have the same concept in mind. They might even ask "What is functional programming? Isn't all programming functional? How can there be non-functional programming? etc."

What we mean when we say it is a style of programming that is often contrasted against object-oriented or procedural programming styles. We have supplanted the ordinary English meaning of these words and attached additional connotations and meaning to them. It's the same thing happening with the terms above in philosophy.

When using the terms(relativism and objectivism) more generally, it means something like that dichotomy of "there exist an objective set of univeral X" vs. "there is no objective set of X".

Note that in other contexts these terms can have other meanings (notably Rand's Objectivism political philosophy means something quite different).

With tech debt, you can ask several engineers and get very different answers about how much tech debt there is on a particular project. It's not well defined, subject to interpretation, and so doesn't fit well with management wanting quantify the objective value of doing something. I think this is the real reason it's often disregarded.

It requires a degree of trust from management to the engineers to properly address, and that trust can be abused. All sides are aware of this and I'm sure I'm not the only one that has used tech debt as an excuse for some minor padding here and there.

I would say if their society can grant them the safety to turn down work for cheap swords at their discretion without the risk of having their life ruined through a down period that is lacking high-quality work orders, then maybe you might have a case for also holding the blacksmiths accountable to a higher standard of work.

Accepting and taking on the risks associated with highly selective work orders has it's own intrinsic value and should be rewarded somehow. If it's not, then why would anyone do it?

Also, while what you say is true in general for modern DB's, there are some implementations like old Oracle versions where the only way to create the effect of an inner join was in terms of a Cartesian product.

He's not gonna be the one using the sword and he can make it clear up front about the tradeoffs. If it shatters in use and the customer gets maimed, oh well, they were warned. Sometimes the market is flooded with people who just want cheap swords made fast and the blacksmith needs to pay the bills too right?

Maybe one line of inquiry that might be able to give us some clues is if we're able to actually have people learn something like a physics equation with no prior knowledge of it while they aren't conscious of the learning of it. That might be a hard study to do with possible ethical concerns but if there's a way to design it, I could see some hope for getting closer to understanding the role/structure of consciousness.

One maybe plausible explanation might be that there's some sort of hierarchy of pattern matching there, where first you need to understand language, then mathematical language, then finally the physics equations. If you take that view, then the equations are really just extremely sophisticated pattern matching constructs. They don't have to actually _be_ the code that makes the universe-computer run so to speak, they just need to give us a precise enough predictive model that we're able to do useful things with it.

Maybe another way to put it is, we're not actually uncovering the source code of the universe, we're just looking at the functioning of the universe and extrapolating our own patterns that yield the greatest predictive power. On that view at least there are some plausble avenues for explaining strange artifacts like the time-reversibility of some equations - i.e. our patterns show that this will give predictions even if time flowed backwards, but the universe's time doesn't flow backwards because this pattern that we found is not the same thing as the actual universe. In that sense we'll never be able to actually uncover this source code so to speak, but that is just how science works - we're always looking for a more accurate theory - we'll never reach "the final theory", and even if we did, how would we know?