I think many folks in eng are used to being told they were smart either in high school or college or an early job and it became part of their self image. They might be smart but they are tossed into a field of people who come from similar backgrounds and now they are average which just doesnt fit with their self image.
First 10 years of my career, I felt bearded Unix Sysadmins were the smartest people I knew :->, whereas I was fully Dilbertian in view of The Evil Pointy Haired Management.
I've been trying to develop "PeopleSmarts" or "Soft Skills" or "Emotional Intelligence" or whatever we call it last 5+ years, and it's... just as much harder and more challenging than technical skillset, as my mentors always told me and I never believed. It's a far fuzzier, squishier, less certain set of problems, and there just isn't O'Reilly-quality literature out there :P.
I have confidence that given time I can learn technical concepts or figure out technical issues.
I think path in front of me to be soft-skill-savvy, emotionally intelligent, is going to be long and hard.
So, how to properly setup LUNs on AIX or optimize a SQL statement or design an ERP architecture is one kind of smart; how to work with your own team to figure out productive way forward is another kind of smart; how to work with client and business and management stakeholders to satisfy their actual needs is another important kind of smart.
In terms of alignment among humans, most (but not all) human governments agree on certain principles, which are encoded in international law and negotiated within the United Nations.
E.g., most (not all) people agree that torture and slavery are unacceptable.
Thus, it is possible and important to get AGI to abide by these high-level human values. E.g. if an AGI system is put in charge of maximizing the GDP of a nation, it should do so in accordance with generally recognized principles of human rights and human dignity.
There's certainly a lack of agreement among nations but at least we mostly agree on some things and getting AI to agree with us on the things we agree on is important.
If you like podcasts, here's a great podcast episode which explores a philosopher's perspective on how to make moral decisions (e.g. should abortion be legal) in a situation of moral uncertainty:
https://futureoflife.org/2018/09/17/moral-uncertainty-and-th...