News for hackers. Stuff that matters.
Re:What they don't teach you about sockets by TYPE_FASTER on Tuesday July 26, 2022 @01:41PM (#32241330)
What really helped me understand and troubleshoot network communications at the socket level was the book by W. Richard Stevens[0]. I think this is because he starts with an introduction to TCP, and builds from there. Knowing TCP states and transitions is important to reading packet dumps, and in general debugging networking.

Also, no matter what platform you're using, there is probably a socket implementation somewhere at the core. You're best off understanding how sockets work, then understanding how the platform you're working with uses sockets[1].

Once I read the W. Richard Stevens book, I was able to read and understand RFCs for protocols like HTTP to know how things should work. Then you're better prepared to figure out if a behavior is due to your code, or an implementation of the protocol in your device, or the device on the other end of the network connection, or some intermediary device.

[0] - http://www.kohala.com/start/unpv12e.html

[1] - https://docs.microsoft.com/en-us/windows/win32/winsock/porti...

Re:What they don't teach you about sockets by gwright on Tuesday July 26, 2022 @06:00PM (#32244626)
I had the privilege of working for Rich as a junior developer and then later with him as a co-author. He was my mentor and friend. Seeing messages like this over 30 years later really makes my day and reminds me how much I miss him.
Re:What they don't teach you about sockets by sn41 on Tuesday July 26, 2022 @10:10PM (#32246213)
I eventually decided to do theoretical CS for my research, but all of these books showed me as a student what attention to detail really meant.

In fact, the typesetting of these books also made me look into groff and play around with it.

I vaguely remember that there was a phrack prophile on W. Richard Stevens just around the time he died. Such a great loss.

Thank you for your inspiring books!!

Re:What they don't teach you about sockets by teleforce on Wednesday July 27, 2022 @06:47AM (#32248881)
Yes indeed, the typesettings for these books are really awesome. The template is also being used by other authors notably by Jon Snader in his VPNs Illustrated book and if it's not obvious enough the name of the book is paying homage to the original TCP Illustrated series [1]. Co-incidentally Jon also wrote 44 Tips for TCP/IP network socket programming before the VPN book [2].

[1] VPNs Illustrated: Tunnels, VPNs, and IPsec: Tunnels, VPNs, and IPsec:

https://www.oreilly.com/library/view/vpns-illustrated-tunnel...

[2] Effective TCP/IP Programming: 44 Tips to Improve Your Network Programs:

https://www.informit.com/store/effective-tcp-ip-programming-...

Re:What they don't teach you about sockets by Quarrel on Tuesday July 26, 2022 @08:37PM (#32245714)
When I had to write a TCP stack in the 90s, you & Rich had a book ready for me! Thanks for that. :)

He wrote lots of the great, practical, 90s references. Shame he left us so young.

Re:What they don't teach you about sockets by swellguy on Tuesday July 26, 2022 @08:13PM (#32245600)
Not to cause offense, but can you indicate his cause of death? This idle question has been in my mind since 2002 or so when I used his books in graduate school. Was surprised to realize he was already dead back then.
Re:What they don't teach you about sockets by cantrevealname on Tuesday July 26, 2022 @10:14PM (#32246235)
According to Wikipedia[1], he died in 1999 at the age of 48, which is so young that I also would like to know what happened. He was apparently healthy and working and just fine a week before his death. But there's a bizarre resistance among some people regarding discussing cause of death. If a President suddenly died, we shouldn't ask about cause of death "out of respect"? I'm a big believer in privacy, but it's silly to not talk about cause of death after someone has passed away. In fact, we should know causes in order to improve all our lives (to be more cautious of heart disease, or suicide, or motorcycle accidents, etc.).

[1] https://en.wikipedia.org/wiki/W._Richard_Stevens

Re:What they don't teach you about sockets by bruce511 on Wednesday July 27, 2022 @12:32AM (#32246865)
Sometimes though (and I'm not saying it was in this case) the cause of death is perceived as stigmatising, and so is withheld to protect the person's name, or protect the family.

This happened, for example, with Isaac Asimov who died in 1992. He contracted Aids from a blood transfusion [1]. In the 80s and 90s this was sufficiently stigmatising to warrent being kept quiet.

Today things like drug overdosing and suicide are sometimes considered shameful,and details are suppressed. Or worse; imagine having to answer that your relative was shot by police while shooting kids in an elementary school...

In truth people die of embarrassing things all the time, so in some societies it is impolite to ask.

Again, to be clear I am NOT suggesting that there was anything shameful about this death, I'm making a more general point about the nature of reporting cause of death.

[1] https://en.wikipedia.org/wiki/Isaac_Asimov

Re:What they don't teach you about sockets by jprd on Wednesday July 27, 2022 @02:58PM (#32255014)
Thank you for the post, and I'd like to highlight that the only important driver here is Shame.

We all experience it (some much much much more than others), and yet we never discuss it, by its very nature. It can be debilitating and will naturally lead oneself to dark places.

Shame kills.

Re:What they don't teach you about sockets by antod on Wednesday July 27, 2022 @12:07AM (#32246747)
His TCP/IP Illustrated book was formative in my career. I wish I had've salvaged that company copy when that business shutdown - it probably just got thrown out :(
Re:What they don't teach you about sockets by hinkley on Tuesday July 26, 2022 @02:05PM (#32241659)
There were many authors in that era who had practically compulsory books. Stevens is one of the few to have two. His Unix Programming book was referred to as the Bible.
Re:What they don't teach you about sockets by sophacles on Tuesday July 26, 2022 @03:36PM (#32242892)
I still refer back to APUE often and push my junior folks to read it.
Re:What they don't teach you about sockets by Agingcoder on Wednesday July 27, 2022 @08:33AM (#32249675)
The Linux programming interface, by Michael Kerrisk, is an excellent modern update.
Re:What they don't teach you about sockets by lanstin on Tuesday July 26, 2022 @02:14PM (#32241785)
We had to debug an issue with ARP during a deploy of docker containers a year or two ago. A nearly complete understanding of the lower layers is not that much knowledge and quite useful at times.
Re:What they don't teach you about sockets by sophacles on Tuesday July 26, 2022 @03:34PM (#32242871)
I agree with your main point - a decent working knowledge down-stack really helps with network stuffs.

"nearly complete" strikes me as a whole lot of hubris though - I've spent almost 20 years at a career that can be summed up as "weird stuff with packets on linux", and the only thing that I'm certain of is: every time I think I'm starting to get a handle on this networking thing, the universe shows me that my assessment of how much there is to know was off by at least an order of magnitude.

Re:What they don't teach you about sockets by macintux on Tuesday July 26, 2022 @05:08PM (#32244036)
I once interviewed someone with “Internet guru” on his résumé. I advised him there were only a handful of people I’d consider for that title.
Re:What they don't teach you about sockets by twic on Wednesday July 27, 2022 @07:16AM (#32249102)
Perhaps it can be "nearly complete" in a pareto sense. If you know the structure and semantics of IP, UDP, and TCP packets, including their most common few options, and DNS and HTTP, then you know <<10% of what there is to know about networks, but >90% of what you will see in a random pcap file, and >90% of what you know to solve a randomly selected networking problem.
Re:What they don't teach you about sockets by arpa on Wednesday July 27, 2022 @01:07AM (#32247031)
i have no idea how networks work, and i have been doing link splitting, bonding, packet scheduling and dabbling in bgp in 2004. The knowledge i've gained since helps immensely when debugging simple networks created by kube-proxy, but i am just petrified by amount of the iceberg that remains underneath. And to this day i'm slightly perplexed when udp socket operation in php throws `connection reset` exception.
Re:What they don't teach you about sockets by 5e92cb50239222b on Tuesday July 26, 2022 @03:11PM (#32242579)
"Nearly complete"? No matter how deep your knowledge is, you're only scratching the surface. To pick a small example, there's something like 20 TCP congestion avoidance algorithms alone (many are available in the mainline Linux kernel, and they can be picked depending on the task and network at hand), and I believe it took something like a decade of research, trial and error to solve the bufferbloat problem.

https://en.wikipedia.org/wiki/TCP_congestion_control

https://lwn.net/Articles/616241/

Re:What they don't teach you about sockets by hn_go_brrrrr on Tuesday July 26, 2022 @03:25PM (#32242765)
Only CUBIC and BBR seem to have any meaningful usage, however: https://www.comp.nus.edu.sg/~ayush/images/sigmetrics2020-gor...
Re:What they don't teach you about sockets by icedchai on Tuesday July 26, 2022 @03:56PM (#32243154)
I tried out BBR on a system that had a ton of packet loss. It made a tremendous difference!
Re:What they don't teach you about sockets by n_kr on Tuesday July 26, 2022 @04:53PM (#32243855)
> No matter how deep your knowledge is, you're only scratching the surface.

I understand this is just emphasis, but no, its not magic, its not innate ability, its just software man! If you have dug deep enough, and understood it, that's it. Key phrase is IMO 'understood', but that's universal.

Re:What they don't teach you about sockets by aflag on Tuesday July 26, 2022 @07:07PM (#32245206)
I think the point is that it may be impossible for a single human to have "nearly complete understanding" of how the networking stack work. But maybe what was meant was nearly complete understanding of the fundamentals. That's certainly achievable. But networking in the kernel is a beast of a thing with specialists in small parts of it. But I don't think there's a single human that know nearly all that those specialists know combined.
Re:What they don't teach you about sockets by Dowwie on Tuesday July 26, 2022 @02:38PM (#32242141)
for those unfamiliar with C, if anyone can recommend sufficient reference material to understand the C examples presented in the book, that would be helpful
Re:What they don't teach you about sockets by TYPE_FASTER on Tuesday July 26, 2022 @03:26PM (#32242772)
Some languages, like Python[0], have a low-level socket library that maps pretty closely to the C API used in the book.

[0] https://docs.python.org/3/library/socket.html

Re:What they don't teach you about sockets by throwaway1777 on Tuesday July 26, 2022 @02:48PM (#32242287)
You’re opening a can of worms but Kernigan and Ritchie wrote the canonical book on C and you cant go wrong with that.
Re:What they don't teach you about sockets by quietbritishjim on Tuesday July 26, 2022 @06:06PM (#32244696)
It's with stressing how different the K&R book is from books in the past that introduce a programming language. Usually those sort of books are fairly long and you end up with a decent but incomplete understanding.

With K&R, it's quite short, it's an easy going read, and yet when you get to the end you've covered the whole of C (even the standard library!).

Re:What they don't teach you about sockets by harry8 on Tuesday July 26, 2022 @09:44PM (#32246101)
Let's be controversial. K&R is lovely, I learned from it, everything good everyone says about it is true and K & R and both beautiful, sexy, clever, kind and wonderous men.

If you are learning programming (and who among us isn't?), the code examples stink to high heaven. No really. Don't program like that! K&R didn't! If you're learning programming and C it will teach you a bunch of really bad habits /because/ it is a book for experienced programmers with fully developed taste and the examples were designed to illustrate a particular facet of the language as pithily as possible. It's great for that.

I realize any criticism of the sacred texts will result in inspiring revulsion and possibly bile among those who think of such things as above criticism and any commentary is about the shortcomings of the commentator not the holy book but there it is... It's 2022 and I'm old enough to take it. ;-)

Alistair Moffat "Programming, Problem Solving and Abstraction" is one I found that seems better to learn C programming. Maybe other people have better recommendations than that..?

Separate to that "The C Puzzle Book" is superb when you're learning to really thoroughly understand pointers, arrays and so on. Koenig's "C Traps and Pitfalls" isn't bad for some of the dark corners.

Re:What they don't teach you about sockets by quietbritishjim on Wednesday July 27, 2022 @06:59AM (#32248960)
I didn't give K&R that praise because it's a sacred text and I'd feel guilty or embarressed to say otherwise. I said all that because that's how it worked for me: I bought the book, read the whole thing (in far less time and with far less effort than I expected), and at the end of it all I understood C.

I'm exaggerating, but only a little. I did read some more about C afterwards and I wouldn't claim to be a C master. But I never came across any new concepts once the book was complete, or had to fundamentally rework my existing understanding of any concepts. (Not like C++... Oh, there are templates that take templates as template parameters? Or, oh, rvalue references aren't any different to normal references once overload resolution has completed? etc.)

Yes I agree it caters for people that already have a fairly good understanding of coding.

Re:What they don't teach you about sockets by wruza on Tuesday July 26, 2022 @03:19PM (#32242705)
The most confusing part should be “inheritance” of sockaddr structs, afair (the rest should read as if it was javascript). If this is it, then be sure that no magic happens there, structure quirks matter much less than the values they are filled with.
Re:What they don't teach you about sockets by waynesonfire on Tuesday July 26, 2022 @03:08PM (#32242541)
curious whether the book help w/ understanding routing and firewalls?
Re:What they don't teach you about sockets by TYPE_FASTER on Tuesday July 26, 2022 @03:51PM (#32243076)
Yes. To really understand routing and firewalls, I'd suggest implementing a firewall for yourself using Linux or OpenBSD. I call out OpenBSD because when I went through that process years ago, OpenBSD was lightweight from resource and complexity perspectives, so I could focus on network configuration. There might be better options with Linux these days, it's been a long time since I looked.
Re:What they don't teach you about sockets by waynesonfire on Tuesday July 26, 2022 @08:46PM (#32245775)
like at the kernel level? even if not, seems like a such a great exercise.

i guess i'm also interested in routing... any tips on creating a network of computers to experiment with? I guess ideally, you'd have real hardware to assemble, . what's the next best option? VMs? jails w/ vnet?

Re:What they don't teach you about sockets by buzer on Wednesday July 27, 2022 @01:11AM (#32247057)
It depends a bit on what you want to learn exactly. E.g. do you want to just learn about basic routing, BGP, MPLS, certain vendor specific stuff (like Cisco) etc. If you mostly just want to try out what you can archive on Linux you can get quite far with doing things via network namespaces (+whatever software you want to use, e.g. BIRD or Quagga) these days. Large pro on doing things via namespaces is that you can set them up & tear them down a lot quicker compared to VMs or physical hardware & scripting those steps is a lot easier. VMs & physical hardware provide more ways to do things. At least in the past there were various limitations on what you could do emulate with Cisco VMs for example (especially on some more advanced Nexus features).
Re:What they don't teach you about sockets by TYPE_FASTER on Tuesday July 26, 2022 @11:11PM (#32246512)
Back then, I put OpenBSD on a spare P133 with two network adapters to do something like this: https://www.openbsd.org/faq/pf/example1.html

VM and/or containers make sense today. I've got an old Mac Pro tower I found on Craigslist in the basement. 64G of RAM for it was less than $150.

Since Cisco offers virtual versions of their routers, you can look at what people are doing to build home labs for practicing for Cisco certifications.

The homelab subreddit has a lot of interesting examples of peoples' setups as well.

Re:What they don't teach you about sockets by tptacek on Tuesday July 26, 2022 @03:33PM (#32242858)
Very yes on both, especially with firewalls.