blr246

Hi beforeolives—

I like your breakdown, and I've observed similar things in my experience as an engineering focused data person! I've had many discussions with my colleagues about how to manage effectively these different blends of roles and skills.

I'm looking for someone for an engineering type of data role right now. Is there a way to get in touch with you about it?

Our product helps companies listen to their customers by unifying natural language feedback across various channels, applying signals using various natural language modeling techniques, then aggregating them to help teams deliver better outcomes using more relevant information.

Hope to hear from you (brandon at frame.ai) :)

edit: forgot to share agreement for your breakdown

We are agreeing that NYC is not at a moment of urban collapse. The processes that drives away the tax base includes policies and social and market forces that erode the city's effectiveness as a sustaining economic and social hub.

The 1960s and 1970s crisis had a lot to do with the end of NYC's industrial epoch. Suburban development and globalization eliminated manufacturing and pulled workers and residents out of the city. The recovery of NYC was bringing high-value services, retail, and tourism back along with arts and culture.

In the time since, NYC has become increasingly a luxury experience, which is indeed part of its strength but also its weakness, since it accelerates decline when people can up and leave without having roots.

>Instead, what kills cities is a long period in which their leaders fail to reckon honestly with ongoing, everyday problems—how workers are treated, whether infrastructure is repaired. Unsustainable, unresponsive governance in the face of long-term challenges may not look like a world-historical problem, but it’s the real threat that cities face.

The feels correct to me.

I lived in New York City for 15 years. Until last year. I've thought about this theme all year. Decades of policy supporting foreign investment and developer speculation gutted the chance for even affluent upper middle class New Yorkers to afford housing and setup a home base, and so many left. The situation has been incomparably more challenging for low income residents.

I agree the urban collapse meme is much easier to spread than a thoughtful discussion about policy and priorities and how to balance the economic strength of a city's major players with the daily priorities of everyday citizens. I hope the New York remainders shift priorities and initiate a different kind of prosperous era than the one I got to enjoy.

I had the same initial thought based on the title. Unfortunately, the answer is no.

The article discusses a low-dimensional KNN problem. The curse of dimensionality guides intuition that the methods here likely will not apply to extremely high-dimensional problems.

faiss actually comes with a lot of excellent documentation that describes the problems unique to KNN on embedding vectors. In particular, for extremely large datasets, most of the tractable methods are approximations that make use of clustering, quantization, and centriod-difference tricks to make computation efficient.

See https://github.com/facebookresearch/faiss/wiki/Faiss-indexes and related links for more information.

At Frame.ai, we are using both PostgreSQL and faiss (and other tools) in our stack to do several different kinds of inference tasks on semantic representations of text to help companies understand and act on customer chats, emails, and phone call transcripts.

We've frequently had the same dream of adding more native support for nearest-neighbor type queries, since that is the workhorse of so many useful techniques in the modern NLP stack.

Right now, we have lots of dense vectors stored in massive toast tables in PG. It's faster to fetch them rather than recompute them, especially since there are a number of preprocessing steps that limit what we pay attention to.

The discussion here about full text search versus semantic search is interesting. In our experience, both are highly relevant. Sometimes it's most useful for our customers to segment their conversation data by exact text matches, and other times semantic clustering is most effective. I think there's plenty of reason to offer both kinds of capabilities.

It's built into the inbox view, so GMail is extracting the action from the message content and placing a button on the row element. Sorry if that wasn't clear from my initial post.

GMail seems to have opened a vector to amplify dark patterns by placing action buttons on messages. My least favorite is the LinkedIn accept invitation button, which I've clicked now several times by accident because I've spent years using GMail without it taking actions like opening GitHub PRs and accepting LinkedIn invites.

I can't find a way to disable this feature. Does anybody know how?

For query plans using a sort node, there can be a major difference in performance depending on the row width.

It's worth mentioning that it's a bad idea to invalidate refresh_token grants ever during the lifetime of an authorization. I've seen APIs do this immediately upon sending the response to the token endpoint, which makes the system unusable due to the frequency of network transmission errors that would result in having to contact the resource owner to grant access again. Even an expiry after days and years is only likely to result in more support requests to the API maintainer without increasing security enough to justify it.

The reason this bad practice is common is that it is allowed by the spec in https://tools.ietf.org/html/rfc6749#section-6 as an optional action to take on refresh grants. Please, do not do this.

Your response highlights a good idea to mitigate the risk I was trying to highlight in mine.

They want to have a rapid response path (little to no delay using staging envs) to respond to emergencies. The old SOP allowed all releases to use the emergency path. By not using it in the SOP anymore, I'd be concerned that it would break silently from some other refactor or change.

Your notion is to maintain the emergency rollout as a relaxation of the new SOP such that the time in staging is reduced to almost nothing. That sounds like a good idea since it avoids maintaining two processes and having greater risk of breakage. So, same logic but using different thresholds versus two independent processes.

Appreciate the detail here. It's a great writeup. Wondering what folks think about one of the changes:

  5. Changing the SOP to do staged rollouts of rules in
     the same manner used for other software at Cloudflare
     while retaining the ability to do emergency global
     deployment for active attacks.

One concern I'd have is whether or not I'm exercising the global rollout procedure often enough to be confident it works when it's needed. Of the hundreds of WAF rule changes rolled out every month, how many are global emergencies?

It's a fact of managing process that branches are liability and the hot path is the thing that will have the highest level of reliability. I wonder if anyone there has concerns about diluting the rapid response path (the one having the highest associated risk) by making this process change.

edit: fix verbatim formatting

The other part of this story I did not see mentioned is that I suspect that password expiration also makes organizations more vulnerable to social engineering hacks because legitimate users (I have done this) become locked out due to poorly managed password expiration, then have to call in to restore access. The use of insecure identity and authentication mechanisms like student IDs and security questions is a recipe for abuse.

Good riddance to password expiration.

Kinesis is not necessarily well-suited fan-out. It is very well suited for fan-in (single consumer, multiple producers).

Each shard allows at most 5 GetRecords operations per second. If you want to fan out to many consumers, you will reach those limits quickly and have to implement a significant latency/throughput tradeoff to make it work.

For API limits, see: https://docs.aws.amazon.com/kinesis/latest/APIReference/API_...

Pick up some data sheets for IC parts that might be useful in a system you'd like to design. These are published by the manufacturer, and they contain a lot of design requirements about how to layout properly a PCB and some about the theory of operation of the parts. You can piece together a lot of practical information this way.

You're correct that a table-level lock is a useful tool to ensure serializability, but it's also a drastic one since it blocks other transactions requesting a higher access level. The concurrency control system is trying to provide a generic way to both guarantee consistency and maintain a high throughout on a variety of workloads, and locking tables always would diminish throughout for many common workloads. For example, one where inserts and reads tend to occur simultaneously at high volume.

To implement the serializable isolation level, the database system must track access to every single row you access, even the ones you read or filter out. (The need to track reads, even for rows filtered out of a select statement is surprising but necessary.)

Consider a common scenario where you SELECT a set of rows and take a SUM over a column. Suppose your query and another query begin reading from the same committed state of the same table. Suppose that the other query uses an UPDATE command on a set of rows in the table, and that the other query commits before yours does. In order to be consistent, the database system must detect the situation where the other query updated a row that affects the filter WHERE you scanned the table, otherwise your sum could be incorrect if the other query's committed state would cause your query to compute a SUM over a different set of rows or over modified values in your SUM. The only way for the database system to guarantee there is no conflict is to keep track of every single row your query accesses, even if it is a row passed over by a WHERE clause!

Serializability is a well studied concept. You can find lots of good resources about algorithms for implementing concurrency control and for detecting whether or not two transactions are serializable. The high-level summary is that it takes a lot of operational bookkeeping to guarantee that two queries have no conflicts, especially when you are using real-world examples having many filters and joins.

Agree this is pretty much inexcusable.

Logging request or response payloads without an explicit whitelist should raise flags for any developer. There are very few cases where you can assert that not only in the present but also for all future use cases of a system, the entirety of a payload will not contain sensitive user data.

Only a whitelist will suffice to maintain good security. It's common for developers to attach sensitive data for debugging and other use cases under arbitrary paths.

Systems can improve further by adding patterns and other heuristics to drop values from the whitelist that look like sensitive data.

Great write-up. Is the long-term vision to go completely to the vectorised query execution model, or are there cases where a row-oriented plan might be better, such as cases when there are complex computations involving multiple columns of a single row?

This is a great tool. We use it to generate an Entity Relationship Diagram from our canonical DDL file checked into our repo.

Here's the basic recipe:

  1. Spin up a fresh Postgres instance on Docker using -P to claim an available ephemeral TCP port
  2. Use `docker inspect` to read the Postgres port
  3. Run DDL script on the fresh instance
  4. Run SchemaCrawler Docker container using --network host option so it can connect to Postgres
     and using -v so it can save a schema image to the host filesystem

This entire process is a `/bin` script checked into our repo, so we can update `/doc/db-schema.png` any time. It takes about 15s total since we have to pause for the Postgres instance to come online.

This "asymmetry" is one of the reasons why having capital makes you powerful, and living paycheck-to-paycheck (maximum need for liquidity) makes you powerless.

Wholeheartedly agree. There are so many powerful development and debugging tools that are easy to configure in a standard IDE outside of a container environment. Docker is an excellent packaging and deployment tool. During more than 3 years of use, I've rarely encountered runtime issues caused by differences between my host and the container environment (they occur periodically when using libs for things like image processing). So long as you have logging and other service monitoring configured so you never have to deal with SSH and attaching to containers, you seldom have to think about Docker and you have a system ready to deploy in a variety of environments.

It's also great to use Docker to run test databases and other tools locally. For example, we have a script that takes a DDL file, spins up a fresh Postgres in an ephemeral Docker container, then runs SchemaCrawler on the database from yet another container to generate a useful entity-relationship diagram. The tool is portable and repeatable and carries no risk of affecting a production system.

Vault has always appeared to me to be a great technology for a larger tech org looking to implement more granular access control and increased auditability.

At a smaller scale, we've been satisfied with encrypting secrets using KMS and then placing the resultant ciphertext into environment variables that we commit with our terraform scripts. Our system does not allow granular access control, but it is relatively simple to implement using a couple of Bash scripts and allows committing all of the config, including secrets, all at once so that deploys are fully reproducible.

To allow a bit more granularity in isolating different environments, we use different encryption keys depending on whether we have a dev/uat/prod deployment. We grant the target application role access to particular secrets based on which KMS keys it can use.

The other trick is that we decrypt all of the secrets in our environment at the entry point of our application code, so that the environment is fully decrypted by the time the service runs. This means there is no trace of secret management in our application code.

There are intermediate steps of this scheme where we could revoke access to KMS keys for some developers, so that we could allow users to deploy services but not necessarily be able to use and access the encrypted secrets.

Regardless, I always appreciate knowing these kinds of technologies exist because they seem to me to solve very challenging problems at scale.

A major problem with this is that non-expert users might interpret it to mean that the "green lock" always means everything is ok. That is dangerous advice since it's possible to publish a phishing site having proper SSL. Users need more context than what is offered here to avoid becoming victims of phishing scams.

I also found the content itself difficult to read in both layout and copy. The character names were confusing, and I don't think the three concepts of privacy, integrity, and identity were conveyed in a clear enough sense so that a non-expert could interpret how those are actually 3 different things.

As a programmer and practitioner, I'm curious about what kinds of skills and training it takes to program quantum computers.

Can you shed some insight into what's really different about the tools and task of programming a quantum computer versus using classical programming languages and tools? Do you think quantum computer programming will rapidly become standard training for CS undergrads, or do you expect it to remain a niche skillset like FPGAs, etc, since it will only supplement and not replace classical computers.

Also, nice to meet you. Your essays have been inspiring over the years.

I think there's a crucial component missing from a lot of the good advice here: pick a couple of languages to learn deeply enough to get into the language specification. To really appreciate the interplay between code and execution, you need to dig into some of the differences between how languages are actually implemented and distributed. This will teach you surprising things about the language's execution semantics and what it means to avoid undefined behavior.

Even interpreted languages like Python and Java have tremendous depth in their runtimes and design, such as JITs, memory management, and more.

Set a goal to learn as much about at least a couple of languages as you can, and you'll be able debug and understand programs much better. It will also ease your learning of new languages, because many of their features will fit into things you know already.

Well put.

Do you still do consulting?

Please reach me at brandon {at} brandonreiss {dot} com to explore an opportunity with a small e-commerce retailer/wholesaler.

There is no such thing as a measure of the health of a society. Inequality is one of many indicators that can help you predict the social stability and quality of life achievable in a particular society.

Inequality is important because of the studied link between peace and prosperity. Individual participants of a civilization give their social system power by buying into a social contract that their (mostly) nonviolent participation secures them the necessities of life.

When we have natural disasters, we sometimes see the rupture of the social contract. Now, imagine that there's a continuum of levels of nonviolent participation depending on individual cost-benefit analyses occurring all of the time around you.

When inequality shifts so dramatically over just a generation or two, there's reason to be concerned that people's buy-in on the social contract might shift, too.

There seems to me to be an optimal, market-based solution to this: form a Rideshare Drivers Alliance, hire brilliant former Uber programmers and data scientists, and build an app intended to run in the background on drivers' phones that optimizes for their interests. If you build something awesome, you should be able to charge millions of drivers a small monthly fee to show them metrics and data aligned to their interests. The real difficulty​ to overcome is the information imbalance between drivers and rideshare services.

Edit: typo