robjs

In some applications, using an ICMP payload and getting a quote of the IP header + 8-bytes of the original packet back in ICMP error messages is part of the application. For example, traceroute utilises the fact that it gets part of the payload back in a ICMP TTL exceeded message to identify _which_ traceroute request was being responded to.

I write mainly about networking topics, either standards updates or open source projects. Infrequently at best.

Kudos on recognising “Fairmount” versus lobbing it in with Glen Park or Noe Valley. (I don’t /really/ have a complex about this - but the neighbourhood does have some interesting history!)

(My background: I’ve inherited RSVP-TE deployments in a number of continent-wide, and global networks — which has involved driving standards to improve its scalability, and subsequently driving segment routing in the industry and production deployments.)

The issue one has at any kind of scale is that it is non-trivial to acquire capacity that can be coherent in terms of different optimisation dimensions for your network. For example, a network I worked on could acquire limited capacity on Europe to India cable systems, alternate routes were significantly different latency - but there was significant EU-India demand in the network. What were the options for placing this traffic on the network? IGP weights - sure - but this means there is no selective placement of that traffic (i.e., everyone has to take the same route), which one might not be able to support commercially. Looking beyond that, there are limited options _other_ than MPLS-TE based on RSVP-TE. Path Computation Engine (PCE) support, even when it emerged, was RSVP-TE-centric. So, commercially, those networks didn’t have a choice to deploy traffic engineering — and it wasn’t for want of trying. Significant cable system deployments have been driven on routes such as the one that I mention above — so there was capital to be deployed to fix the problem, it’s just that building such systems take years to be built. Did their architects want to deploy RSVP-TE? Pre-SDN (and SDN-in-the-WAN like B4 and SWAN) what option did they have to meet that business requirement? I would postulate very little (at least at the time that I was engaged in these discussions there were no clear alternatives). In fact, I would postulate that the existence of TE in B4 and SWAN shows that there is value in traffic engineering practically. Greenfield/ground-up systems still implemented.

RSVP-TE itself though was not well thought through. The systems design discussion that I think is very interesting here is considering the lessons that we can learn from such a technology. Distributed state in the network, that causes large amounts of signalling following failures, and requires midpoints to be aware of all demands that traverse them and admit them is fragile by its very nature. The scaling analysis that was done during the architectural work (RFC5439 for example) did not think of the RSVP-TE distributed system’s different points of dynamism — it concentrated on steady-state cost, but we’ve demonstrated time and again in production (over many, many years) that practically the system’s scaling was to do with the cost and scaling of dynamic resignalling following events rather than steady-state utilisation (I’ve presented effusively on this, see https://research.google/pubs/pub45800/ and https://youtu.be/NtED7CUHLNE).

Rather than raising the question, from a systems design perspective, as to whether MPLS-TE was a religious mistake — let’s raise the one around how complex distributed systems that have multiple vendors of their equipment (i.e., not ecosystems that are controlled by one party) can iterate on solving business problems without religiously filling in the gaps of protocols that don’t work. In my view, this would be a huge step forward for the networking industry.

Finally, let’s not fall into the trap that we over-generalise. Higher scale networks within a limited geography (terrestrial UK, US etc.) may not have the same business considerations — and therefore may not need the same approach to traffic placement. There’s, as always, a set of trade-offs here.

It is almost impossible to describe the feeling of not being able to think in this way. I'm a senior software engineer at a large company, I spent much of my time diving into different code bases, and in meetings where I am often unfamiliar with the specifics of a situation and need to quickly reload context. When I am in a state that I have brain fog, I absolutely cannot do these things - I need to sit and prepare for a meeting, and even then I can't think quickly enough in it to be able to comment on anything in a meaningful way. Creativity is not possible, I can't think around a problem at all. Understanding unfamiliar code becomes extremely taxing (if not impossible). Whilst I'm not as bad as some of these folks that are described in the article, coherently forming sentences how I would normally (I'm someone that "thinks out-loud" often) is just not possible. It's debilitating.

For folks that don't have it, it can be hard to explain. Like someone in this article, I tend to just cancel things when I'm in this state (which can be for days, or weeks -- luckily I get to emerge as my body deals with being able to eat again). I just need to sleep - partially because of the emotional toil of the frustration of being cognitively impaired overnight, but also from the physical toll having a flare-up causes my body. I'm not really making this post (and sharing things that I probably usually wouldn't) for any reason other than to just say to folks, hey - be understanding to your coworkers, these conditions are poorly understood, and difficult to deal with. We all tend to feel guilty for not pulling our weight during the time that we're sick, so sometimes just knowing that our coworkers are cutting us some slack really helps :-) Thanks!

Nuance. These clickbait articles really miss this. I wish we had better editors.

Quite honestly -- this kind of marketing hype and hyperbole is what is wrong with the whole area of SR today (and I say this as someone that was _very_ involved). We've completely lost the ability to say what it is we're solving, and why we're doing it. We're driving disparate architectures into silicon where there's opportunity cost for the functionality. We're having political disagreements within the IETF based on folks trying to keep political control of technologies, not worrying about the efficacy for the industry. It's all pretty broken. YMMV.

The whole "IP 2.0" presumption that appears to be being made here is that suddenly some external traffic source will be able to select a route through someone else's network -- but this just isn't the commercial reality. Some more performant paths are going to have costs associated with them (even if it's just to build more capacity), so there is going to be a cost of choosing that route through the network. That cost is going to need to be covered somewhere - so you are very unlikely to actually be able to get to choosing a path without some commercial contract. Guess what? We've already had those -- they just tend to use the DSCP bits to indicate what the traffic class, and hence associated requested SLO is - not an explicitly chosen path.

Equally, let's think about how this would even work - if you are going to choose a path through the network to get better QoS, you're going to need to know something about what IDs to use, which implies knowledge of the topology. Inter-domain topology exposure is going to /significantly/ increase the complexity and fragility of inter-domain routing -- there are reasons that we don't run a global link-state protocol :-)

In conclusion - I think this is hype with little technical justification, and is unlikely to have any different impact than other intra-domain traffic engineering that the industry has been running for many years.

Even then, you absolutely cannot distill how successful an employee someone will be down to a single document, but equally, you can't interview everyone - so there's some level of pragmatism that you have to exercise to be able determine who to interview. My experience here is that the filter criteria doesn't have to be the typical screening that a bunch of us probably are frustrated by (e.g., "does the resumé mention Python? No. Route it to /dev/null").

I've tried the following.

1. I'll filter by folks that I think have demonstrated some interesting commitment in their journey to where they are. For example, interesting side projects along with their professional experience, or a non-traditional route to their role (e.g., didn't attend college for the "expected" subject). My experience here has been that these folks often have different insights that have correlated to being easy to work with, and fruitful for the teams that I've engaged with.

2. I'll try and determine folks that are able to communicate in their resumé some particular outcome that shows that they had a wider perspective than their immediate role. This tended to me to show that the person was able to think a bit wider than the task that was on their plate, which again has correlated with good teammates to me.

An industry mentor of mine who runs a R&D for a large, and successful software/hardware business unit once explained their approach to building teams in a really interesting manner to me. They explained that they wanted to have experienced folks on the team, so that they didn't re-learn lessons that the team should already have learnt, and instilled the culture and best practices of good developers -- but they wanted junior engineers on the team to drive a healthy disregard for what was asserted to be impossible, or written off as folly based on prior experiences. Their experience was that this balance helped grow great engineers from the junior folks, whilst keeping the senior folks on their toes -- seeing that their bias against an idea wasn't always the right thing.

Something that stuck with me from this conversation: We naturally become more conservative in terms of what we view as possible as we become more experienced -- after all, we tried X before, and it really didn't work out -- let's not waste our time doing X again...!

I interviewed 10s of candidates at a FAANG last year. A significant number of them were older than me. A non-trivial proportion of those were interviewing for positions that were more junior to me. Of those folks we hired a good number of them. If you see age-bias in the interviewing process, it's already a red-flag as to the team not clearly understanding how to make the best of the pool of engineers that they can potentially hire, and I'd avoid them for that reason.

I'd also love to hire experienced folks into my team. A great thing about the FAANG that I'm at, is that we're explicitly empowered in the hiring process - and hiring managers really _have_ to care about what interviewers think. I would encourage not writing this class of company off as being only youth-focused. :-)

To combat the fact that there's no right answer here, the teams that I work in have adopted a couple of strategies.

1. Start out with a reasonable sketch of the high-level design of the system and/or library that is being written. Thinking through this high-level approach lets everyone start with a reasonable shared mental model of how things will look as development proceeds. It also means that some of the complexities of "which of the more complex cases that we're going to come across are we going to solve, and which can be a TODO" be decided up-front.

2. After this initial design exercise, do design is small chunks, iteratively. Take the smallest possible set of functionality, and have a developer in the team (not just the tech lead...) determine the options for design, and propose one as the solution. These small units of design can be informed as the developer wishes -- i.e., they can do prototyping if it makes sense.

3. At all costs, avoid development approaches and team dynamics where refactoring is not considered an option, or discouraged. As you observe in your question, getting the design right might involve understanding more of the problem, or the requirements, and that might need the "oh , we should have solved this like that!" moment that only comes from having implemented something, or watching your users try and use the solution you built entirely differently to how you expected. Ensure that there's a team culture, or an openness to the idea that you definitely won't be right the first time. Perfect is the enemy of "done" or "shipped".

Personally, I prefer to start to think about the problem in an abstract form before coding, because this allows me to separate the consideration of the best way to write maintainable, testable code from the problem of the system, library or application design. Of course, YMMV, but this is what works for me :-)

I've worked for an operator all the time that I've been in the IETF, and its definitely pedantry, not-invented-here, and lack of understanding of real issues that prevents us making significant progress. I personally have had more than one go at trying to improve IETF<->operator communication, and made little to no progress.

A much more successful model has been writing code, co-developing it with other operators and vendors if possible, and then working directly with vendors to push their implementations. This model self-selects on solutions that are actually used (because there's non-standards-focused engineers involved), and rather than worrying about potential edge cases, get to handle the problems that occur in practice. This is a bit harder to do with changes that require global scope -- but all technologies we develop now need to coexist with legacy, so I'm not clear that it's not the best model as we go forward.

- Understand whether a solution that is being proposed actually addresses a problem that a real network or technology system has.

- Reshape a set of proposals that have already reached "we already implemented this" into something different.

Both of these challenges require significant investment from the community. People have to be willing to stand up and critique the drafts (which they do), but also take the subsequent steps of going to work with these folks to help them understand how better they might address real gaps, or even to explain why the ideas aren't going to work in practice. The problem is that for most technical contributors, this work isn't moving anything forward -- it's more "good of the Internet" work. My observation is that there are limited cycles available from the folks in the IETF to do this work, but the number of new drafts coming in has increased at a rate that out-strips it (source: >15y working in the IETF routing area in general). Equally, there is limited support from the folks that employ IETF contributors for doing this work -- would they rather spend time fixing standards that they have customer demand from, or stopping standards that they probably won't need to ever implement (and thus have little to no negative affect on them)? These two challenges for the IETF have really exacerbated the culture clashes there.

Whilst eqvinox's analysis above draws the line at a particular contributing company, in my experience, this isn't solely the case. If we look at the IPv6 data plane for segment routing being progressed in the SPRING working group, it has the same hallmarks. A solution was proposed that it wasn't really clear what the problem it solved was, there was no significant technical debate to say that it wasn't needed or was harmful ahead of time (6man and spring didn't see these contributors), and only later down the line - when there was significant investment of a number of companies in it - was its implementability, and efficacy discussed. At this point there's zero chance that this technology will actually be morphed or deprecated (at best there'll be a competing solution), even if there's no standardisation of it.

Overall - I don't see anything particularly new here, other than another outlet for the frustration of not necessarily being able to push forward standards in the Internet industry. The other outlet has been open source - as we've seen more push towards just running code. Some areas of the IETF have embraced this one with much more ease (SPDY->HTTP/2.0, QUIC adoption etc.), but the routing area - with its implementations relevant to quite a small number of implementing vendors - has been harder to crack. (Source: I work with a team that took this route, and has really struggled to bring ideas back into the IETF and have them openly evaluated.)

The restaurant actually cooked the food twice - and the second time, I asked if I just could go down there and get the food - but apparently this isn't possible. Uber just use the excuse that their drivers are contractors - and they can't do anything.

I'd understand if this were the middle of nowhere, and on a Friday or some other busy time. But at 8PM on a Tuesday in San Francisco... Yeah. Probably they should sort this out.