News for hackers. Stuff that matters.
Re:Tech-savvy audiences block Google Analytics by superasn on Tuesday August 31, 2021 @04:56AM (#28365454)
That's why I prefer goaccess(1) over any other tool that uses client side Javascript.

It's basically a real time website log analyser which gives you enough information to know whats happening on your website but doesn't require any pesky Javascript etc to do it.

Also since it is works by analysing your log files it can never be blocked.

(1) https://rt.goaccess.io/?20210826211303

Re:Tech-savvy audiences block Google Analytics by scrollaway on Tuesday August 31, 2021 @04:59AM (#28365468)
By nature it won't filter out bots, however.

One of my clients' website traffic is composed of over 75% bot traffic. Server-side logs are unusable for anything other than site performance.

Re:Tech-savvy audiences block Google Analytics by Cyberdog on Tuesday August 31, 2021 @06:11AM (#28365973)
> One of my clients' website traffic is composed of over 75% bot traffic. Server-side logs are unusable for anything other than site performance.

I'm unclear how broad you intend that second sentence to be, but there's still a ton of info you can glean just from server-side logs:

- Referrer info, and, by extension, popular search terms being used to find your site;

- Paths on your site causing 5xx errors (so pages which might be triggering an error in a server-side script)

- Paths on your site causing 4xx errors, and associated referrers (might be broken links on your own site; might be stale search engine indexing)

- Mobile vs desktop access statistics

Finding this data among a bunch of bot-induced noise might be annoying, but if they're good bots and sending proper UA headers specifying their botness, it's easy enough to filter out. Even otherwise there might be typical bot-like behavior you can find and account for such as not sending a referrer header or trying to access known exploitable PHP scripts (in which case you should block that IP address for a few hours or days - there are programs which can do this sort of thing automatically but frustratingly I can't recall the name of one off the top of my head right now).

Granted, a lot of this can be spoofed, but I'm pretty sure the number of people sending spoofed referral or UA headers is dwarfed by the number of those (like me) who block Google Analytics and similar cruft entirely.

Re:Tech-savvy audiences block Google Analytics by scrollaway on Tuesday August 31, 2021 @06:24AM (#28366051)
No, you're right, I shouldn't have written something so dismissive. (I do include error tracing as part of "performance" for what it's worth but those have their own system from within the app itself)

Frankly I would love to see some serious low-config solutions to analyzing server-side logs. Oh, especially Fastly. Client in question uses Fastly and it blew my mind to find out that there was nothing in place to answer simple questions such as "what are the slowest paths to respond", "which paths are a cache hit most often", "which paths are most hit overall", etc. And being able to look at various dimensions such as browser, bot traffic, country of origin, etc. If you have suggestions…

Re:Tech-savvy audiences block Google Analytics by Cyberdog on Tuesday August 31, 2021 @05:44PM (#28374084)
Any log analyzer will tell you which path is the most hit. For slowest paths, I think a server daemon could theoretically log how long it took to serve the page from request in to last byte out, but I don't know if any of them do that - you might have to set up a custom format for logging, and then from there you'd need to tell your analyzer how to interpret that field. For cache hits, I guess it'd depend on what sort of cache you have in mind, but that might be something you could only effectively log at the application level.
Re:Tech-savvy audiences block Google Analytics by second--shift on Tuesday August 31, 2021 @11:34AM (#28369346)
| frustratingly I can't recall the name of one off the top of my head right now

fail2ban

Re:Tech-savvy audiences block Google Analytics by dschooh on Tuesday August 31, 2021 @05:11AM (#28365545)
> By nature it won't filter out bots, however.

I have two questions about this:

- Since you know they are bots, why couldn't you filter them?

- On the other hand, couldn't there be bots that run JavaScript which would be tracked client-side?

Re:Tech-savvy audiences block Google Analytics by scrollaway on Tuesday August 31, 2021 @05:27AM (#28365660)
> Since you know they are bots, why couldn't you filter them?

A decent amount can be filtered by UA. This is inconvenient because UA is a very large piece to log and index on, so you need to do UA processing to do anything useful with it and … well by that point it just becomes a chore and I suspect there's good logging services that do this better than you'd spend your time doing yourself.

> On the other hand, couldn't there be bots that run JavaScript which would be tracked client-side?

They exist but they're more rare by nature, because running JS at bot scale is expensive.

Re:Tech-savvy audiences block Google Analytics by dschooh on Tuesday August 31, 2021 @08:04AM (#28366823)
> well by that point it just becomes a chore and I suspect there's good logging services that do this better than you'd spend your time doing yourself.

Which is why the OP suggested using tools like goaccess.

Re:Tech-savvy audiences block Google Analytics by superasn on Tuesday August 31, 2021 @05:50AM (#28365828)
Goaccess does have a separate panel for Crawlers/Bots(1) but I think it's based on the UA.

There is also an option to ignore IPs/Referrers too which can work very well for such cases.

(1) https://goaccess.io/features#:~:text=Spot%20aggressive%20hos...

Re:Tech-savvy audiences block Google Analytics by marvinblum on Tuesday August 31, 2021 @09:55AM (#28368093)
The middle way is using something like Pirsch's API [0] from your backend.

[0] https://pirsch.io/

Re:Tech-savvy audiences block Google Analytics by AlexAndScripts on Tuesday August 31, 2021 @09:39AM (#28367926)
Can't you have a client side system do nothing but tell the server it's not a bot?
Re:Tech-savvy audiences block Google Analytics by qualudeheart on Tuesday August 31, 2021 @05:47AM (#28365806)
Can I evade tracking by disgusing my traffic as bot traffic? Sounds like a startup idea.
Re:Tech-savvy audiences block Google Analytics by scrollaway on Tuesday August 31, 2021 @06:00AM (#28365901)
Oh I suspect you'll have a decent amount of sites not serving you ads or analytics if you put "googlebot" or "chrome lighthouse" in your UA.
Re:Tech-savvy audiences block Google Analytics by faeyanpiraat on Tuesday August 31, 2021 @08:21AM (#28366998)
If you use googlebot you’ll likely just get blocked altogether.

Using Lighthouse might be an interesting experiment though.

Re:Tech-savvy audiences block Google Analytics by ignoramous on Tuesday August 31, 2021 @05:50AM (#28365833)
You're likely going to end up implementing a poor man's Tor.