For the first decade of my career, I really believed this, and spent a lot of time doing split tests, and studying analytics, and trying to "make things better" by understanding the numbers as they were given to me.
This was a mistake.
I've since learned that these numbers will rarely help me make meaningful improvements to my product, and my business. Sure, they can be useful so that I'm not "running blind," but they simply aren't going to show me how to create an ingenious idea that takes things to the next level.
Analytics will help you optimize things to a "local maximum", but they'll blind you to the real possibilities of creating something new that can completely transform a business. As soon as I understood this distinction, I've been quite a lot more effective.
There's a similar problem with things like "user interviews." A common pitfall is to ask people which features they want. That has limited use. The real work you need to do is the "creative thinking" that others haven't done. Figure out what people don't know they want; learn what the numbers can't tell you. Then go and build it. Yes, understand the numbers, choose a good business model, and optimize based on those numbers, but don't let the numbers create the product. It's a dead end.
Of course not. There's no substitute for straight-up creativity and deep thinking.
But once you have your ingenious idea, you still have to design it, make sure it's clear to users, that they find it and can use it effectively. Your "ingenious idea" may turn out to be largely sabotaged if a button you thought had an intuitive label is misunderstood by 90% of users, or a link you thought was highly visible is being scrolled past by nearly everyone.
Yes, analytics is all about optimizing things to a local maximum. But you might not be anywhere near your local maximum. It's astonishingly easy for the first version of your ingenious idea to only be achieving 5% or 10% of the actual local maximum potential. We shouldn't downplay the difficulty or achievement involved in getting even close to a local maxima.
And you're correct that in user interviews, if you only ask what features they want, you're drastically limiting the value you might uncover. On the other hand, you'd better not ignore the features users are frequently requesting either. A lot of users are pretty smart and know exactly what they need, at least to get to that local maxima.
You could have multiple UIs hitting the same endpoint. Also, why limit yourself with such crude metrics?
> good UX doesn’t so much rely on Google Analytics but on a UX engineer’s depth of knowledge about human psychology
UX in theory, and UX in application are two different things. You could have the best models of how users will interact with your site, but until you deploy and measure, you have no idea what will happen.
https://github.com/onurakpolat/awesome-analytics
https://github.com/awesome-selfhosted/awesome-selfhosted/blo...
A button turns out to be below the fold on common small screen sizes that a new designer forgot to consider. A bad translation results in 90% of users in a particular country misunderstanding something. A JavaScript library doesn't work on a common Android phone you don't test with. Latency issues make something virtually unusable from the other side of the country because of a single badly written function that's easy to rewrite -- but you still have to catch it!
You need ALL forms of feedback. It's not a question of some being more important than others. They're all important and play their own unique roles. Analytics is for catching AND debugging all the things that go wrong at scale in the real world, as opposed to the artificial and limited environments used for user testing.
And even so, while there are a lot of countries and languages, a collection of tests in all of them, which is all it takes for these examples, is not "big data". Again you are crediting analytics for things that people were fully capable of and responsible for doing in the days without traffic data. I realize it is great for the resume and sounds sexier to do "analytics" as opposed to just basic software testing (you should see what people are calling "AI" in other industries these days). And I'm certainly happy to agree that analytics has some value, say in improving wording and stuff that a single instance won't tell you, but again these aren't cases of that.
Similarly with languages, that you cans er from backend metrics.
More data seems always nice, but analyzing more data isn't making analysis simpler and there is a big privacy impact in analytics, especially when outsourcing to data collectors, who can gather data across sites. (Which then also gives Google information which services are interesting to user and can be integrated into search etc.)
How would the users know they can't use a feature they don't know it exists?
Let's say you add a brilliant new feature X, but due to a bug the users can't load the code for that feature so they never see it. How would they know to submit a form feedback for a feature they don't know it's there?
I'm getting the impression that people are just ignoring all advice about communicating with customers in their startups and just throwing stuff out there to see what sticks. Besides being wasteful in doing stuff no one wants anyway, what if that bad feature crippled your product and your paying customers have permanently switched to a competitor the instant your change frustrated them? And now you're bankrupt and can't afford analytics. Relying on analytics as a crutch to catch these things was the mistake in the first place.
Fun story I actually had forgotten about till now: I briefly worked at a tiny startup out of my school in the ending days of the internet bubble, trying to sell a "data mining" software product. Way too soon before it was cool, sadly. It was really hard to make the case that people needed to pay us $100k for the benefits we could get from their data. And companies certainly weren't going to go for a pitch like "if you launch a broken product we will catch that fact". They spend a lot of money to be sure that doesn't happen already. We even had one major customer figure out that they could just have an engineer perform a simple counter over incoming communications that would catch all they needed to know, and hence they didn't need our product anymore. That was kind of the end for us, in fact.
Also, after you've already reached product-market fit, it's important to take your product to its "local maximum".
Quantitative data lets you drill down into different dimensions. Because it is much easier to collect at scale (it's the sum of your users' interaction w/ your product, after all!), it's much easier to make representative decisions.
To be clear, I support qual + quant user research (+ split testing). They are all tools that help influence product direction. I have, however, seen a lot of cases where qualitative research uncovers an insight that doesn't actually match real world behavior when we bake it into the product.
Compare that to a split test, where I don't know why something is happening, but I am able to optimize against goals. It gives less insight but is more foolproof-actionable. With insight, you have to go insight -> improved_action. With testing, you just have the improved action.
To me, qualitative research is crucial to build a model of the environment. These can be used to generate hypotheses, which you then split test. Any time I see someone take direct insight and build a product feature from it, I have a lot of questions about what alternate ideas they tested. And how to know the one that was developed is as close to optimal as you can get, given the fixed resources available.
As another example, consider that in 2007 Apple developers were begging for an iPhone SDK, and Steve Jobs crushed their hopes by telling them to just make web apps. A year later Apple came out not just with an iPhone SDK, but with an entire App Store. (Though I suppose some developers [Epic] and users [HN] wish they had just come out with an SDK, and that the iPhone wasn't locked down.)
That being said, they do a lot of user testing of the next big thing before it is revealed publicly.
And yet, Macs will still send usage and performance data to Apple so they can incorporate that information into future product versions and find out about system software issues.
Your SaaS or webpage probably gathers 80% of what Google Analytics does in your log files. It could without doubt provide deeper insights that GA is capable of by adding your own behaviour tracking code (which can be written with the understanding of your specific problem dom ain, rather than be an "everything to everybody" generalised solution).
Nobody is suggesting we shouldn't collect usage and performance data, the thesis here is that we shouldn't send all that directly to the worlds most profitable advertising agency, just because they'll draw us some pretty graphs for free.
> Nobody is suggesting we shouldn't collect usage and performance data
This thread degenerated into an argument because somebody stated outright:
> Don't use analytics at all but focus on your product.
That’s what people in this thread are arguing against – don’t use analytics at all – not the subject of the article, don’t use Google Analytics.
I guess the point is not that crash reports and slowdown data aren't useful (they are), but that they tend to give you incremental improvements.
That being said, incremental improvements over a decade can make a big difference, as Apple also demonstrates.
Where exactly do we draw the line to what's core for a company to track in order to run their business and what can be opt-out?
Leads are useless without measuring how many people bounced off your shitty online shop, because they couldn't understand your UI.
Staying time is useless without measuring closed tabs, because your website is unreadable with ads.
The irony behind it is that ad metrics are used to buy/sell a website's worth for ads. Sometimes I feel like it's like a bubble that's invented on purpose to not have a measurable outcome of anything.
But I agree they measure the wrong end points or rather they measure too many end points for me to make sense of which one is better. The goal of overall traffic sometimes works against keeping bounce rates low or staying time high.
In the end you would think products sold matters. But an over promising website will over hype and under deliver causing returns or bad reviews which may affect future sales. The message has to describe the product but still meet the market's demand.
Asking what users want is not the right way to do interviews. I advise you read "Just enough research" by Erika Hall. The goal of interviews is to gather enough data points to understand the user needs and struggle, and to know their journey through the product with accuracy, and also understanding why they use or don't use a function. It is not a way to get a "wish list".
Then, designers usually have tons of tools and methods to process this data, take decisions and try creative solutions (usually more than one), and play them back to the users through prototypes to see which ones work better. You can check design thinking as a starter, but there are many more.
I would disagree on user research. It works on certain types of broad questions like messaging and branding. It's probably overused though.
Understanding > Knowledge
Understanding builds better product. Full stop.
Feedback loops. Hypothesis, experiment, compare expected vs actual. Lather, rinse, repeat.
QA, test, analytics are for verifying you're on track, that your predictions are useful. Nothing more.
They're not for charting a course, planning a journey.
That sounds creepy as fuck. Did you have informed consent from the users?
Do you know of any legal document mentioning this, that storing the clicked element is allowed, but not the click position? They can be both used to run the business and improve the product, thus being in the "necessary" category.
Tech startups do definitely have this problem of focusing on website analytics where the product is NOT a website or app. If we're generous we can assume many people here develop for these kinds of companies. Some waste a lot of time looking for up-and-to-the-right arrows for investors or trying to be data-focused when data about the website isn't actually all that important. Many of these companies might actually be better off with no analytics to waste time on. I'd still argue it's better to check in every once in a while to look for problems and ask yourself some questions.
The idea of removing analytics where the product is an app or a website is silly. This would be like arguing a grocery store shouldn't track what people are buying from their stores, and instead just source good products. You need to do both. What are you going to do when I ask what is or is not working? Tell me your feelings? Shake an 8-ball? Aside from detecting problems, analytics can be a jumping off point for innovation if you're smart about it. What can we do that's more like what's working? How can we improve this page type?
There are for sure people who over-focus on analytics (often on the wrong data points) instead of creativity, but these are not mutually exclusive. If I were to list the millions of dollars I've earned and saved via analytics this would be a very long post. Sadly, most of those millions were for other people, but it's a very valuable tool for optimizing and creating if you use it correctly.
How do you find out if your "useful" feature isn't as useful as you thought it was? Or is it really always "the user can't find it"?
Analytics (quantitative data) can help you find areas of bottlenecks to explore further by doing qualitative user research and getting to the ‘why’ behind the people problems standing in the way of your metrics you are tracking (retention, adoption, etc.). This is called ‘triangulation’ using quant and qual research methods to understand your users more deeply than just looking at data can achieve.
Where are you getting your information from?
Related, survey respondents are known to be weird.
Science is not the only form of truth.
i feel like there is a underapreciated difference in wether you are selling a physical product/service or a digital one.
both benefit from focus on the product, but anlytics on the website is vastly more helpful if your product essentially _IS_ the web-ux.
This is provably false. You do not need intrusive analytics to develop fantastic products.
Have people somehow forgotten about good old-fashioned user testing? It is expensive, time consuming, and amazingly effective. Most importantly you can actually talk to your users because they are people instead of data points.
User feedback >>> analytics.
I've recently interacted with a startup where the amount of resources they spend on trying to get them both, is making them blind to the power of one. It's not a pretty sight when all the numbers are tracked, plotted and planned on, yet nothing seems to work.
By default, GA only sends one hit on page load. If there's no second hit, there's no way to tell if someone was looking at the page for a second or a minute or an hour.
Heat maps are great too but Google Analytics is still used as the foundation for figuring out which types of users are clicking and not clicking on what, both in isolation and as part of a pathway between elements/pages/etc.
When we sent users their "data", many of them were in disbelief at how little data they received. Many had come to believe that all tech companies are secretly building inventories of user data to sell to 3rd parties, when really most of us just want to know if our heavy users of Feature A are also heavy users of Feature B, or if Feature C is more popular with new users but not old users.
The strange part is that tech companies are taking the brunt of the bad PR for things like gathering customer feedback and serving relevant ads, while traditional companies like cell phone providers and credit card companies are actually selling customer data. The latter doesn't get enough attention despite being a much more widespread issue.
Facebook doesn't sell your data, but your phone provider and credit card company probably do. But ask the average person who's selling their data, and Facebook will get all the blame.
Very very very few journalists have time to grind axes for any reason. They're too busy feeding the beast.
Could be as simple as occasionally removing mentions of companies that advertise with the paper in peices about customer data creating the effect, not J. J. Jamison telling people to get him pictures of zuckerberg :D
Relevant ads are the kind you get on DuckDuckGo: relevant to the content you're looking at. E.g. if you look at a site about origami, you get ads from arts&crafts supply stores.
What you've probably meant are called predatory ads: they chase you around the Web wherever you go, like a predator chases its prey.
For you, it might be wrong, but when the advertiser is buying millions of ad impressions and is looking for a 0.01% hit rate, the math shows that you're one of the more likely future customers.
I usually buy a new printer when it runs out of ink because it's cheaper than buying a new ink cartridge.
Also look at third party solutions like external feeds.
Even if it is cheaper, the externalities are not fully priced in, please think of the planet, thanks.
Of course, as you note, printer companies realized people were actually doing this (due to the perverse incentives created by their ridiculous razors/blades business model) so now most printers come with pathetic "starter" ink cartridges that run out after about 10 pages.
The irritating bit though is that many cartridges don't last long (intentionally and unintentionally) once you open/install them, regardless of whether you actually print anything. So, if you don't print a lot, it might still make (financial) sense to buy a new printer... ;-[
I am tempted though to get a model with refillable ink tanks, but most of what little printing I do is on an old b/w laser printer which I've had forever. Still looking for the holy grail color laser printer that is cheap, networked, duplexing, compact, and can print photos.
Apple is the only company that still occasionally does this, though even they have driven off into the realm of every application having its own entirely novel interface. Go back in time to MacOS or Windows in the 1990s and you'll find an entirely different paradigm: every application has the same interface... or at least the same interface paradigms. Learn the computer once and you've learned the computer.
Features were remarkably discoverable. They were organized logically in menus. Keyboard shortcuts were always available and usually intuitive. I remember opening a new app I'd never used before on Windows 95 and just unthinkingly hitting a keyboard combo and it doing the general thing I expected, or mousing to where I expected to find a feature I imagined should be there only to find that it actually was there. I never used Mac Classic much but I heard it was similar.
The web is what really broke this. Web UIs overtook desktop UIs due to the difficulty of installing local software and the power of trendiness. Web UIs were never uniform and couldn't be since the web was anarchistic and wild and often driven by designers who wanted to make their product look a specific distinct way.
I remember in the days immediately before the web there being talk of algorithmic generation of UIs from data schema. If the UI is thoroughly standardized then it becomes at least thinkable to examine data structures and generate user interfaces from them, even UIs that aren't horribly ugly or hard to use. This was building on a previous generation of incredible WYSIWYG UI design tools. Then the web came and all that stuff was completely abandoned.
Today's UI design tools in things like Xcode and Android Studio are horrible by comparison to what people were using in 1995. Go back and try Visual Basic on Windows 95. The VB language sucked but the UI designer was aeons ahead of anything we use today.
... which you can entirely do by analyzing web server logs.
You dont need google for this, pushing your users (with violation of GDPR consent) into google monstrosity. Dont do that, I block every freaking google domain from cdns, fonts to analytics.
But I don't block 1st party analytics.
I have no reason to. I have visited your site, I dont have anything against YOU following what I read. It is your site. But you will ask me for consent for giving those data to google. And I will say 'no'. And I am not the only one.
Just to inform you, that the person/entity that allows Google to gain access to PII data is directly responsible for this - if google is fined due to GDPR violation, you can get fined to by providing it the way to get users data. They will survive. You might not.
Have your analytics, but you will not sell my soul (which GDPR explicitly forbids you - you are handing over my PII data to 3rd party that is known for violating it and that makes you accomplice) for you having your graphs.
You can get those data from web server logs. You will have all the data that you need. Actually more data, as no one will block them.
Needing "google analytics" is just a huge, giant, hype driven, lie. You don't need them to analyze what you already have in YOUR logs.
chaos_emergent: please do explain, what data google analytics is offering to you than what is already in your server logs? Without violating GDPR even more? Yes, you can surely track something more, again "on your side". Dont give it to google as it WILL get blocked and you will have a distorted picture of how your site is being used. If you want real data, skip 3rd party analytics. Found a way to require to be unblocked? I will skip your site, you have just lost a user. A paying user if the content is worth the money. And sites with selling my data for a graph or two are not worth it.
Just for an info, I have written a mitming proxy that takes care of those (cname cloaking) and a lot of other things, including fingerprinting, supercookies etc., with support for various blocklist (domain to adblock), js injection where the injected scripts are handled in same manner as blocklist (you can stockpile them and make rules), changing validity of cookies (to session cookies for instance), saves your data trough highly effective caching and even helps spying cdns save some bandwidth as they are mostly no longer visited. And yes, works as transparent proxy too (for router enforced usage). Imagine fully armored firefox for whole network, regardless of browser, freebsd,linux,windows,arm8. Working on cross platform gui, release soon.
So I might know a thing or two. ;)
A good product, that does what it advertises, does not need analytics. But a bad product, that somebody desperately wants to make successful, or at least successful enough to sell to a PE and exit, needs analytics.
While some people have an uncanny sense of vision, and seem hit on the right ingredients more often than seems fair, but most companies aren't led by this kind of person.
You need things that tell you when and how to course-correct, this is what analytics gives you. Now, of course, this needs to be balanced against privacy concerns. I push back on things that track literally everything (the tools that record every click and cursor movement are fascinating, but undeniably creep), and I try to avoid sending any PII to 3rd-parties. The amount of stuff Google Analytics phones home about by default is also pretty troubling.
I'm on board with basically every privacy-based criticism of tracking, but I don't buy this argument that only bad products benefit from it.
That may not be so for a game developer who wants to modify gameplay that is heavy or reliant on things like particular mouse cursor movement and mouse click usage. Especially if a meticulous gameplay goal is the objective.
...
"You don't know these things until you measure them."
If you don't build big, bloated tools using ultra-high-level frameworks and if your product is a simple tool that performs a single, useful task ... then you do know these things and you don't need analytics to tell you.
Huh? Analytics is how you focus on your product.
By instrumenting your product with analytics, you can find out if customers are using your new useful feature or if they can't find it. If they're performing a task quickly because it's easy, or slowly because they're struggling with the UX. And you find out that customers on a certain mobile device are suffering huge performance issues, for example.
You don't know these things until you measure them. That's analytics.
Obviously analytics are only one piece of product improvement -- there's sitting down with users for 30 minutes to watch them use the product, interviews, surveys, etc.
But analytics are a critical piece. You can't focus on the product without analytics.
Analytics isn't just about conversion. Analytics is about the entire product experience.