It's a pretty strong argument to say that Microsoft under Gates was technically obsessed, but that really faltered under Balmer.
Microsoft continued to win profits, but they made major strategic missteps that cost the revenue.
Amazon feels like it's going down the same path: empowering the tree-gazers without remembering that the forest also matters.
{Revenue attributable to previous R&D} (aka coast) vs {Revenue attributable to current R&D} (aka acceleration)
I did several experiments, and noticed that whenever I produced analysis that was in line with what management expected - my analysis was praised and widely disseminated. Nobody would even question data completeness, quality, whatever. They would pick some flashy metric like a percentage and run around with it.
Whenever my analysis contradicted - there was so much scrutiny in numbers, data quality, etc, and even after answering all questions and concerns - analysis would be tossed away as non-actionable/useless/etc.
if you want to succeed as a Data Scientist and be praised by management - you got to provide data analysis that supports managements ideas (however wrong or ineffective they might be).
Data Scientist's job is to launder management's intuition using quantitative methods :)
So, a synonym for 'consultant?' :)
I've always used them as "independent 3rd parties" who were listened to.
This has been going on for mellennia, unfortunately.
I want to make the move from development into consultancy and would appreciate hearing how you did it.
I cannot DM you, but if you have the time to type a small paragraph, I'm all ears.
If you don't want to make that public, you can directly email me (<HN-username>@gmail.com)
It was a LinkedIn recruiter message which I usually ignore. However, my SO did not (she is in IT as well) and convinced me to join a hiring event. I ended up liking it a lot and went through the hiring process. Soon, I started out on the most junior level and joined my first project with 3 very senior colleagues after a few weeks.
The learning curve was very steep both on the technical level and also regarding the consulting aspect - at first there was nothing I could 'consult' on due to lack of experience. This changed with growing experience, with the guidance of senior colleagues and my private efforts to gain skills and expertise.
Let me know if this was helpful!
Thanks a lot!
A few years ago there was a problem with storm-water infiltration into my (elderly at the time) mother's property from her neighbor. I, being a dutiful son and a civil engineer, investigated it and came up with the probable cause, the likely effects of non-action and the most cost-effective solution. I presented it to my mother in the most layman-like terms that I could. She said she'd think about it – meaning she'd refer i.e. defer – to her daughters. In the meantime I had a very layman-like chat with my mother's carer and told her the situation in layman's terms. The carer listened and said that what I said I made total sense to her. Later on, one of my sisters accosted me and stated that it was completely obvious what the problem and the solution was – "even the carer could see it". Human foreheads don't have the real estate for where my eyebrows wanted to ascend.
My advice is to consider whether the message should be separated from the messenger somehow.
Recent tech layoffs is the other side of that growth, when cheap money is gone and company is forced to stick to core competencies and shutdown growth plans
We are accepting a whole lot of assumptions every day.
The definition of rational is being informed enough to know what said chemicals will do in the short term and long term in order to make an informed decision, but then I'm reminded we dont get taught any of the above unless we specialise at a Uni, so most people cant make any sort of informed decision.
Had the same problem when I was an economics researcher -- publication bias for what stakeholders want to hear (often the government) is rampant because that's where funding for the economics department mostly comes from.
This is also known as The Iron Law of Institutions.
it isn't clear to me how the grounds for realizaing the theory are reconciliable with its conclusion
-- Robert A. Heinlein, Tunnel in the Sky, 1955
This is the only reason why a 60+ PowerPoint slide deck can cost several millions dollars
It's easier to poke holes in predictions of the future than in interpretations of the past, especially when the people making those decisions have likely reached their decision-making status through "how things have always been".
It took a lot of analysis and time to clarify to higher ups that we weren’t just P-hacking , but at least they were concerned about that.
> Data Scientist's job is to launder management's intuition using quantitative methods :)
It’s no different than the days when grey bearded wisemen would read the stars and weave a tale about the great glory that awaits the king if he proceeds with whatever he already wants to do.
The beards might be a bit shorter or nonexistent, but the story hasn’t changed.
I was the infra lead on a data lake project and got take part all the way to breaking down the data and turning into PowerBI reports. The result was "sell more" and to clients who marketing already identified, years ago, as whales.
There were some interesting other insights, esp. w/r/t to niche products that sold around weird dates (Easter, Memorial Day, 4th July -- but not obvious gift days like Valentines or X-Mas), but it led to a lot of "you're doing it wrong!" recriminations and follow up projects.
I would say it was a very engineering driven org however, so if you could present compelling data it could go a long way.
Also someone to blame if it doesn't work out
Nonetheless, the CTO went on a multi-year, 10s of millions of dollars, huge data tech stack & staffing reorg shake up... with really zero data points explaining the driver, or what we would measure to determine it was successful.
So it became a self referential decision that we are successful by doing what he decided, and we are doing it because he decided it.
Ouch. This is savage, but sadly correct in many cases.
HOWEVER, to play devil's advocate here, I've also seen corporate data scientists overstate the conclusions / generalizability of their analysis. I've also seen data scientists fall prey to believing that their analysis proves would should be done, rather than what is likely to happen.
The role of an executive or decision maker is to apply a normative lens to problems. The role of the data scientist / economist / whatever is to reduce the uncertainty that an action will have the desired effect.
Positive claims are about what is true. Normative claims are about what should be true, or rather what decisions we should make. Put another way, positive claims deal only with facts while normative claims deal also with values.
GP is saying that it's the data-scientist's job to give the executive the facts and it's the executive's job to decide what to do about the facts.
So often, those models proved to be over-fitted and not generalizable.
But too many decision makers simply can't properly judge such results.
Where do business analysts fit into this dichotomy? Their whole job is to poke around in Tableau in order to surface high-ROI strategies for the business to pursue. (Where, in choosing which proposals to surface to management, they're effectively making 90% of the strategic decisions.)
Or how about corporate buyers in trading and retail companies?
Or quantitative investment managers?
If understanding the data generating processes is the goal, I'd rather see some useful plots than wade through a technical description of some model whose assumptions were flagrantly violated.
The skepticism isn't a problem, the unequal application of it, the potential to harm careers, and the chilling effect as people wisen to how best meet their own personal goals is.
The SNAFU principle: communication is only possible between equals. When an hierarchical divide exists the subordinate will tell the superior what he wants to hear.
Sadly true. As humorously depicted here[0]:
In the beginning was the DEMO Project. And the Project was
without form. And darkness was upon the staff members
thereof. So they spake unto their Division Head, saying, "It
is a crock of shit, and it stinks."
And the Division Head spake unto his Department Head,
saying, "It is a crock of excrement and none may abide the
odor thereof." Now, the Department Head spake unto his
Directorate Head, saying, "It is a container of excrement,
and is very strong, such that none may abide before it." And
it came to pass that the Directorate Head spake unto
the Assistant Technical Director, saying, "It is a vessel of
fertilizer and none may abide by its strength."
And the assistant Technical Director spake thus unto the
Technical Director, saying, "It containeth that which aids
growth and it is very strong." And, Lo, the Technical
Director spake then unto the Captain, saying, "The powerful
new Project will help promote the growth of the
laboratories."
And the Captain looked down upon the Project, and He saw
that it was Good!
And then when I got old, I realized, there is even a reason that some people want status quo... because they have usually been around long enough to see society fall apart into anarchy and mass murder, so in their mind, they are doing the right thing.
- Galileo Galilei
A modern characterisation of "The Galileo Affair" would be that he was SWAT'ed by someone he was really really mean online to.
Thus the whole "Galileo affair" starts as a conflict initiated by a secular Aristotelian philosopher, who, unable to silence Galileo by philosophical arguments, uses religion to achieve his aim. [1]
While delle Colombe was almost alone in arguing publicly against Galileo, there was a group of scholars and churchmen who supported his Aristotelian views. After Galileo referred disparagingly to delle Colombe as 'pippione' ('pigeon'), his close friend the painter Lodovico Cigoli coined the nickname 'Lega del Pippione' ('The Pigeon League') for delle Colombe's group.[2]
The affair was complex since very early on Pope Urban VIII had been a patron to Galileo and had given him permission to publish on the Copernican theory .. this was very much a case of personal vendettas and internal politics rather than a straight up case of "The Church Versus Galileo".
[1] https://en.wikipedia.org/wiki/Galileo_affair#cite_note-Spell...
[2] https://en.wikipedia.org/wiki/Lodovico_delle_Colombe
[3] https://en.wikipedia.org/wiki/Dialogue_Concerning_the_Two_Ch...
>> Whenever my analysis contradicted - there was so much scrutiny in numbers, data quality, etc, and even after answering all questions and concerns - analysis would be tossed away as non-actionable/useless/etc.
It's a good sign at the company that I run, anytime our analysts/data scientists come up with metrics that say we're killing it, or that our ideas should bear a ton of fruit, the kneejerk reaction is to be extremely skeptical of the results. Usually they're still right.
When the data scientists say we're fucking something up, we tend to pay a lot more attention.
Only the paranoid survive, after all.
I found it incredibly stressful to discover and provide analyses (even experimental results) that wasn't expected, or contradicted prior beliefs. The findings were always very harshly scrutinized, and typically lead to tons of pointless extra work to 'understand what is going on'.
My friend runs a successful market research agency and she says she gets called in when management have decided they need to make a change but need evidence to sell it to the shareholders and staff.
That was a plot point in Dirk Gentley's Holistic Detective Agency (1989), though the observation much pre-dates this.
doesn't help that most of this data goes through multiple layers of BS where each person is putting it through filters to make themselves look better. And a good chunk of people don't have enough understanding of stats to understand when they are being tricked
Are you going to take the time / money to set up a warehouse, get all the data into with an ETL product, set up dbt or some other transformation layer, set up a BI tool and build the reports and dashboards, etc.
Regardless the size of your data, you still need to get it in one place and model it in a way it's actually usable.
Getting your data to the cloud is expensive, but then you can’t do anything with it because distributing it to process in multiple stages is too expensive and you’re already paying so much to keep all that useless data.
kthnxbye
> Most corporate decision making is highly political, the needs of/whats best for the business is just one parameter in a complex equation.
100% Individual humans are emotional creatures with their own wants and needs, and it's important to understand how organizational incentives drive decision making.
> Data Scientists are commonly disregarded by VPs in large corporations, despite the claims about being "data driven".
This has not been my experience, though. The more common thing I've seen is that, sometimes data is boring and doesn't really show much actionable insight, but as everyone wants to justify their job, I've seen data scientists come up with really questionable conclusions that fell apart on further inspection (call it "p-hacking the enterprise").
Plus, a lot of this data in these data wearhouses is messy. Often times data scientists are siloed at the end of the process, but then you get "garbage in/garbage out" results, where there is some bug in data tracking that isn't understood until it's too late. Much better in my opinion to have data engineers and data scientists work much more closely with product engineering teams up front so they can help ensure the data they collect is accurate.
Oh, and a bunch of data scientists with zero domain knowledge for whatever data they are analyzing, preferably with PhDs in maths, but some ML background will do. And agile, because of course all those Palantir dashboards can only be developed using agile.
Once all is said and done, zero insight was created but a whole lot of consultants, contractors and project managers have been paid handsomely, while some higher ups can now put "implemented agile and big data at X" on their LinkedIn profiles.
Most of my analyses provide very little value because they are sort of common sense to people with domain knowledge. When I ask people what could be more useful, one of two things usually happen: 1) it's impossible due to data and/or infrastructure limitations, 2) what they ask turns out to be nonsensical in further analysis (like asking for average of something that follows a very fat tailed distribution with a few observations dominating the phenomenon. Of course it's usually impossible to explain this to people).
The more I think about this, the more I think that in truly data powered companies, both the decision making and data analysis have to be carried out by more or less the same people. The organizational hierarchies have to be much flatter. Essentially the employees will have to be some kind of "secret agents" who have both the skills and the mandate to steer the company in the direction they see fit. I sort of see this already happening in the FAANG companies where, or so I hear, it's very difficult to get hired, the staff count is quite small compared to traditional companies and the senior engineers have a lot of power in the company.
Using math PhDs or Palantir or whatever as a sort of modularized black box for "insights" while giving them no real skin-in-the-game does not work.
My favorite example is the WW2 bomber diagram shown to illustrate survivorship bias. Sure, working from data and first principle one could identify the vulnerable spots of the bomber. Or one could asl the designers or have an engineer, heck even a contemporary field mechanic, take a look at the actual drawings of the plane. And reach the same conclusion, faster, with a ton of additional insight and improvement ideas that can actually be implemented...
Which itself both takes time and is wildly unpredictable - neither of play well with today's Taylorist managements schemes.
I've been using LangChain and was really interested in how this could summarize primary sources, so I ran it on Hillary Clinton's emails. This was over unstructured PDF documents uploaded directly from pymupdf to OpenAI LLM through LangChain. ------------------------
A Storyteller Summarizes Hillary Clinton’s Emails
Input: Raw PDF of 2016 Democratic National Committee email leak
------------------------
Prompt: I want you to act as a storyteller. You will come up with entertaining stories that are engaging, imaginative and captivating for the audience based on the summary text provided. It can be fairy tales, educational stories or any other type of stories which has the potential to capture people's attention and imagination. You have been given a section of a large set of emails to summarize and come up with an entertaining story.
------------------------
Output: Once upon a time, there was a group of diplomats working hard to make the world a better place. They had been discussing how to gain leverage over the Pakistanis to move them to more aggressively pursue Al-Qaeda. After much deliberation, they decided to use a combination of incentives and coercive measures to try to get the Pakistanis to cooperate. They offered incentives such as increased control over how Kerry-Lugar money was used and increased support for counterinsurgency capabilities. They also used coercive measures such as targeting military leaders who gave support to Al-Qaeda and its allies, and attempting to reduce Pakistani anxieties about the Indian front. In the end, their efforts paid off and the Pakistanis began to cooperate more with their mission. The diplomats had succeeded in making the world a better place.
The story follows Hillary as she juggles her busy schedule, responding to requests from her colleagues and friends. She is always willing to help, and her dedication to her work is evident. Despite her busy schedule, she is still able to find time to help those around her. A powerful leader was tasked with finding out whether a senator supported the surge in Iraq and obtaining a copy of his recent speech about troops to Afghanistan. The leader reached out to her team for help, and they worked together to draft a memo outlining four strategic interests, three of which overlapped with the leader's goals. With the help of her team, the leader was able to obtain the information she needed and use it to make a positive impact.
President Obama was faced with the challenge of how to win the war in Afghanistan without causing too much destruction and civilian casualties. He decided to increase the size of the Afghan army and police force, and to pay them better wages so they could live comfortably. He also decided to send more US Special Forces and tens of thousands of troops to the region. Despite these efforts, the Taliban continued to gain strength and support from the Pashtun community. Obama realized that he needed to find a better solution. He decided to focus on carefully thought out measures such as providing better pay for the Afghan army and police, and using diplomacy to negotiate with the Taliban. In the end, Obama was able to find a peaceful solution that allowed the US to win the war without causing too much destruction.
A Prime Minister of a country had a fear of losing his sight due to a childhood rugby injury. He had his eyes checked and it was found that he had two minor tears in his retina. He visited the world-renowned Moorfields Hospital for further examinations and the doctors decided against any further operations. The Prime Minister was grateful to the doctors and staff of the NHS, especially Moorfields Hospital. Meanwhile, the US Secretary of State was in contact with the Foreign Minister of Honduras to discuss the situation in the country. She asked her staff to call the US Congressman Payne to update him on the situation. Through her diplomatic efforts, she was able to make a positive impact on the lives of many.
A small island nation called Haiti faced many challenges, but the people of Haiti were determined to build a stable, prosperous and democratic future. The United Nations appointed President Clinton as a special envoy to Haiti, and he appointed Dr. Paul Farmer as the Deputy U.N. Special Envoy. Dr. Farmer had dedicated his life to helping the people of Haiti for the past 20 years. Despite the efforts of the U.S., U.N. and other countries, Haiti still faced many challenges. To help the people of Haiti, the U.S. State Department took a closer look at the challenges and Secretary Clinton's chief of staff briefed Members on the initial findings. The U.S. hoped to better target and coordinate assistance, engage the Haitian Diaspora, and make assistance sustainable. To help the people of Haiti, the U.S. proposed granting temporary protected status to Haitians living in the U.S. and included Haiti in security initiatives such as the Merida Initiative. With the help of the U.S., U.N. and other countries, the people of Haiti could build a brighter future.
There was also a great conflict between two powerful nations, Morocco and Guinea, over a passport that had been taken away from a citizen of Guinea. The leaders of the United States and France stepped in to help and offered ideas to help the two nations come to a resolution, but the leaders of Morocco and Guinea were still unable to come to an agreement. The leaders of the United States and France then decided to meet in Paris to discuss the situation further and were able to broker a deal between Morocco and Guinea, returning the passport to its rightful owner.
There was also a man named Tom who was nominated for a prestigious position, but his nomination was held up due to a disagreement between two powerful forces. A brave negotiator was able to convince the two sides to lift the hold on Tom's nomination, allowing him to achieve his dream.
I told my VP that the engineering foul ups in the current product are easily fixable. Standard tooling and patterns exist to re-architect and solve the bottlenecks. What is much harder is a data architect to make sense of the complex data and make sure there is good value for our customers.
Guess what position I don't have on the team, and won't have due to budget issues.
The expertise we need in the industry is people who understand applications in-and-out and make great decisions on what data is worth keeping for present and future applications. And what data is needed to be kept, but only in aggregates (or anonymized, which reduces costs of maintenance)
But I have thousands of data points, thousands, I tell ya!
Though I think there’s a better way— that is executive data science, what I do at Zapier. The key is that I’ve built up a huge amount of econometric and economic / business skills that I apply to affect good change in collaboration with company leaders. It allows me to work with Execs using sensible analysis. I improve growth and output by helping us catch errors of assumption before they go into production and cost growth / bad surprises. I also help the executives gain alignment around good information. This multiplies their departments’ output by allowing them to work better together, more in concert. That helps avoid issues with data being bent to decisions.
They typically carry a lot of hard-won valuable domain knowledge (that I combine with my economic-statistic knowledge and skills for rigor).
It’s my job to ensure Execs start with good sensible information regarding objectives. They usually ask fantastic questions and share a lot of great analysis of their own.
There are times I learn about what I might call controversial implications. This is typical of innovation using technology. It’s in these moments that I feel I create the most value by highlighting the trade offs I believe we face / potential regret.
My day to day is mostly economic / econometric analysis, conceptualization and formulation of technology, and writing, but also occasional meetings to gather information, feedback, answer questions, teach and collaborate with others.
I try to effectively “sample” the org by interacting with people up and down the org, so I can assist Execs in incorporating directors, managers and ICs knowledge / pool good information / systematize planning, etc.
I worked in a "data-driven" company of 5,000+ employees, with 4 data scientists who were spilt into two separate non-collaborating teams. In effect, they were so under resourced they got nothing done.
There's an agenda by some level of management, and they use "data" to forward their agenda, or disregard it due to "unexplainability" (a legitimate concern for BD/ML) if it disagrees with the agenda.
The real issue is that business people usually ignore what the data says. Wading through data takes a huge amount of thought, which is in short supply. Data Scientists are commonly disregarded by VPs in large corporations, despite the claims about being "data driven". Most corporate decision making is highly political, the needs of/whats best for the business is just one parameter in a complex equation.