I recall hearing a theory that having individuals with differing circadian rhythms within the same social group provided an evolutionary advantage as not all members of the group would be asleep – and consequently vulnerable to external dangers – at the same time.
Hypothetically, say local sunrise is 6 AM, and sunset 6 PM, a lark rises at 5 AM and retires at 9 PM, and an owl rises at 7 AM and retires at 11 PM.
If the lark wishes to be immediately productive on waking, they might generate noise, vibration, or odors in the owl's sleeping area, or may simply resent the owl for continuing to sleep while the lark has started working, and thus may wake the owl intentionally. If they have a task that requires cooperation, they could either wait 2 hours for the owl to wake, or go rouse them immediately. At that point, the owl has had 6 hours of sleep, and could probably function on that amount for that day. The owl, on the other hand, can work cooperative tasks immediately on waking, as the lark is already awake, and has no reason to ever awaken the lark prior to the natural conclusion of their sleep cycle.
The converse act to the lark waking the owl 2 hours early is the owl keeping the lark up 2 hours late, after which the lark will likely just wake later. Each has a 2 hour sleep deficit, but the owl would have been re-clocked by daylight all day, and thus it would be more difficult for them to retire early after being awakened early (unless the weather was gloomy and overcast). They would also desire to awaken 2 hours later than usual the next morning. But there the lark is again, waking the owl up early again. The lark can cause the owl to be chronically sleep deprived, and the owl cannot effectively retaliate.
So the owl can, at best, create rules that prohibit larks from disturbing others' sleep before "a reasonable hour". Commuting, artificial lighting, time zones, and daylight savings have all combined to make that "reasonable hour" less reasonable.
Daylight savings is a particularly execrable lark tradition. While the owl is sleeping, the lark changes all the clocks, and then wakes up the owl, waving the clock in their face, so the owl does not murder the lark immediately. And then the larks only relent when the evidence of the sunrise would reveal the ruse. Only the larks have any real incentive to redefine civil time to get the owls out of bed earlier. So when someone says, "your body-clock is messed up," the best response is, "your civil time is messed up," and then roll over for a few more zees.
That is not the case: if I usually wake up at 6am, I will again wake up at 6am if I had at least 4h of sleep. So a "lark" will be similarly sleep deprived in circumstances you describe (eg. in young adult life and lots of nights out).
The reason while early-risers had more success getting their schedule accepted throughout history is because they needed no artificial light source to perform whatever they needed, and artificial light was costly for a long time.
However, the solution in a global world today is simple: move to a timezone where your productive hours will align with whatever the societal norm is, and do remote work.
Or 11 AM and 3 AM?
Or sleeps straight through to 6 PM before they rise?
If you keep going, you'll go through nocturnal and wrap back around to lark.
I assume that sleep-wake cycles plot neatly onto a population standard normal distribution. Everyone on the early-riser half of the bell curve is larkish, and everyone on the late-riser half of the bell curve is owlish. As with most things, if you live more than 2 standard deviations from the median, you may sometimes experience problems arising from your deviation from the norm.
Based on studies of people living in an actual cave with no timepieces or natural light sources, there may also be a component for the natural duration of one's cyclical body-clock. I suspect larks cycle faster than the median, and owls cycle slower, but I'm not aware of any supporting evidence for that.
This also applies to oncall rotations at software companies!
"Why We Sleep" and "The Body Keeps the Score" are two of the most helpful books I've ever read (and both found by reading HN book suggestions).
I try to sleep through weed (indica). I'm not to happy about it but that works more or less.
I've never particularly had a problem going to sleep once I'm in bed though, and have just struggled rolling out of bed within a reasonable timeframe (apparently a very normal symptom of ADHD, which I'm prescribed the dex for), causing me to stay up late to try and get things done, causing my sleep cycle to be pushed back even further. At one point during university, when I had nothing I needed to be awake and on time for, I was doing 20 hours up, then 10 hours in bed.
Now instead of taking 1+ hours to get out of bed and get going, it takes me 20 minutes. My brain just needs a bit of a jumpstart in the morning.
I can't recommend enough getting professional help for chronic issues like poor sleep patterns, if you're in a country where it's affordable. The worst that can happen is they say there's nothing they can do, on the other hand, it might be that it might be as simple as medication.
Fixed my insomnia on accident. Tried the 4HB diet. The supplements advice did it for me. Magnesium, calcium, and potassium. Overnight fix. Turns out, the culprit was my poor diet. Now, I only have trouble sleeping if I try to sleep less than 12 hours after waking. Best of luck.
I have taken magnesium supplements and multivitamins before (I do quite a bit of exercise), and many dietary permutations.
I do notice my insomnia becomes worse on a caloric deficit so it might be accentuated by nutrition, but I doubt it's the only cause.
As someone who was fortunate to fix a self-created sleeping disorder where I managed to completely f up my circadian rhythms, I wish I had known about TRE years ago. As far as I know it's not (yet) part of the usual list of sleep hygiene/CBT advice, but for me at least it's been the most comprehensive change I've made to my lifestyle that has benefited my rhythms and sleep quality.
I'm in the midst of a full diagnostic course including cerebral scans, a sleep study and blood analysis, but my neurologist told me there's a good chance that nothing actionable will be revealed and that the pill will remain the best treatment for the rest of my life.
I just wanted to say to you:
(1) I feel you, it sucks to feel that you're underperforming b/c your clock is messed up, and what further sucks more is that people, especially the internet, will tell you you're doing something wrong, when you're actually a disciplined guy that has tried everything.
(2) Get pro help if you can. Seeing my GP and my neurologist is the first time I've been assured that there's a good chance that this is something that's entirely out of my control, and that _actually feels good_.
(3) "If the mountain will not come to Muhammad, then Muhammad must go to the mountain" - if you can't fix it just roll with it, that's what I've done at times. I've told a few of my bosses I'm a night owl, and they've fully respected it and given great freedom over my schedule.
You've got plenty of options with your skillset... If you wanted to f* off to the beach and code as a freelancer late at night I'm guessing you could also go do that!