cullenking
here's someone else with the same problem: https://www.reddit.com/r/Appliances/comments/10ijvmy/why_is_...
As per other comments, my guess is somewhere between 1% and 5% of ipad mini sales are to pilots. It's gotta be significant or they wouldn't drop a line that in their main copy!
It’s really starting to suck. The worst part is these are coming from consumer vpn providers that we can’t just block (lots of real users paying real money on our service are vpn users) or they come from big residential isps. The whole residential relay thing is a hard adversary, since it’s spread across so many real users paying ips that you can’t whackamole it. I’d say 1% of residential ips are participating in a botnet (there’s prob some real numbers out there if I cared to look).
This is just going to push everyone harder into walled gardens. Web input biggest inbound but our content is our mote, we are going to have to turn off non logged in content viewing, which will have real business impacts.
As for the latter comment....not sure what your implication is. Yes, bot/spam mitigation is whackamole, but there are consequences for not playing the game of whackamole. Luckily residential proxies are few and far between so far, but they will grow in popularity. When they do, and I can't get by with the occasional individual residential IP ban, we'll come up with other methods to handle.
Luckily the signal is strong with vulnerability scanning, which makes it pretty easy to automate. The only reason to put up whole ASN mitigation (captcha/turnstile, outright bans) is just efficiency. Nothing stopping individual IP banning. The scrapers are the tricky ones, since they more easily hide in legit traffic. However legit traffic has patterns that scrapers do not emulate (at least for a service like ours with millions of pieces of user generated content that's easily walkable), so you can still pull out the signal. It's just a little trickier.
Definitely a continual arms race though.
We collect signals that help determine good vs bad networks. For example, large amounts of requests to .php endpoints, large amounts of empty accounts from the same /24 subnet, etc etc. All these signals let us automatically determine risk, and then put up a challenge. Authenticated users never see the challenge even if they are on a risky network (VPN 99.9% of the time), unless the network has been identified as 100% malicious, then it gets a full block.
Here's a small snapshot of the dashboard:
https://cos.ridewithgps.com/screenshots/6a7c54d0-12Aug26-358...
This was probably a total of 3-4 days of work, spread out over a couple months of iterative claude led hacking. I didn't know exactly what to build, but had some of the key architectural ideas in my head. Opus+Faable made easy work of it all, and ended up guiding some really slick improvements for performance.
I would say this has dropped about 20% of all traffic to our service, though it turns out turnstile is a massive target for bots, so replacing that with something custom is next on the list.
It helps that we have something closer to a lifestyle business, where I can ask for a brief paragraph about your relationship to the outdoors and cycling, but that just means I had 500 slop cover letters gushing about cycling. The three that made it through were short concise honest and linked to real world activities they did.
Good luck, it’s a hard problem , and very very adversarial. You have true scam level applications from North Korea and India, and you have unqualified people trying to appear qualified. Sprinkled in are unqualified people who would be a good hire because of raw capability, and qualified people who are looking to do bare minimum.
Instead, I hittup ebay, got six used gen3 processors, found a "good deal" on a couple tb of new ram (still insanely expensive), and came out with the same overall horsepower for a total of $20k instead of $110k.
I know this is about consumer desktop, but seeing the comments about upgrading old hardware caused me to chime in. This is happening in the production/enterprise level in some segments.
If you have a logging stack, you can easily find crawler/bot patterns, then flag candidate IP subnets for blocking.
It's definitely whackamole though. We are experimenting with blocking based on risk databases, which run between $2k and $10k a year depending on provider. These map IP ranges to booleans like is_vpn, is_tor, etc, and also contain ASN information. Slightly suspicious crawling behavior or keyword flagging combined with a hit in that DB, and you have a high confidence block.
All this stuff is now easy to homeroll with claude. Before it would have been a major PITA.
Instead of using integers IDs for the hexes, we created an encoded version of the ID that has the property that removing a character gets you the containing parent of the cell. This means we can do basic containment queries by querying with a low resolution hex (short string) as a prefix query. If a gps track goes through this larger parent cell, the track will have hexes with the same prefix. You don’t get perfect control of distances because hexes have varying diameters (or rather the approximation, since they aren’t circles they are hexes), but in practice and at scale for a product that doesn’t require high precision, it’s very effective.
I think at the end of this year we’ll have about 6tb of these hex sets in a four node 8 process ES cluster. Performance is pretty good. Also acts as our full text search. Half the time we want a geo search we also want keyword / filtering / etc on the metadata of these trips.
Pretty fun system to build, and the concept works with a wide variety of data stores. Felt like a total hack job but it has stood the test of time.
Thanks uber, h3 is a great library!
IMHO git bisect is slower, especially depending on the reload/hot-reload/compile/whatever process your actual app is using.
this is a common problem in any complex codebase that has a culture of using third party dependencies to solve small problems. you see this conversation all the time with modern frontend development and the resulting dependency tree you get with npm etc....
We are the world's largest library of bike routes, and we enable cyclists to go on better rides, more often. We have a website and mobile apps that allow people to discover the best riding in their area, and get turn by turn navigation using either our mobile apps or the bike computer of their choosing. Come join us in taking Ride with GPS to the next level! We have two openings right now, and are starting to build out the hiring plan for a third:
Senior Software Engineer - API & Product Development: We are looking for an experienced backend engineer to join our small and effective engineering team with a focus on supporting web and mobile app development using our APIs. The right candidate for this role brings extensive experience supporting modern product development, in collaboration with frontend and mobile developers, product management, and design. This requires excellent communication and collaboration skills, both on the engineering side, and from a product perspective. We use rails, but prior rails experience is not required.
Details, and application process available here: https://ridewithgps.com/careers/2025-senior-software-enginee...
Senior Software Engineer - API Development: We are looking for an experienced backend engineer to join our small and effective team with a focus on our APIs and supporting our platform at scale. This doesn't mean you are isolated from product development &emdash; everything we do serves our users in some way, and being a small team we regularly share responsibilities. However, this role will spend more time on efficiency and system design rather than delivering this quarter's new features. The right candidate should have a depth of experience supporting a large API surface area with efficient, well organized code, and should be excited about maintaining and improving performance over time. Experience with developer tooling, database design, query optimization, and DevOps workflows will serve you well in this role. We use rails, but prior rails experience is not required.
Details, and application process available here: https://ridewithgps.com/careers/2025-senior-software-enginee...
Senior Software Engineer - iOS Development: In mid July, we will officially start the hiring process for an iOS developer, and potentially another Android engineer. We are reviewing applications for qualified candidates at this time, and will officially post the job by July 15th. If you think you are an excellent fit please apply now, however there might be some delays in screening, interviewing, etc while we finalize our hiring plan. We have a technically interesting, battery efficient set of mobile apps that act as a companion to our website, and need another iOS or Android engineer to help us take our apps to the next level.
Draft job position and application process available here: https://ridewithgps.com/careers/job_postings/2025-ios-engine...
We just replaced our top of rack firewall/proxies that were 11 years old and working just fine. We did it for power and reliability concerns, not because there was a problem. App servers get upgraded more often, but that's because of density and performance improvements.
What does cause a service blip fairly regularly is a single upstream ISP. I will have a second ISP into our rack shortly, which means that whole class of short outage will go away. It's really the only weak spot we've observed. That being said, we are in a nice datacenter that is a critical hub in the pacific northwest. I'm sure a budget datacenter will have a different class of reliability problems that I am not familiar with.
But again, an occasional 15m outage is really not a big deal business wise. Unless you are running a banking service or something, no one cares when something happens for 15m. Heck, all my banks regularly have "maintenance" outages that are unpredictable. I promise, no one relaly cares about five nines of reliability in the strong majority of services.
Interesting reading through the related submission comments and seeing other hard to troubleshoot bugs. I don’t think atop devs are to blame, my guess is that what you have to do to make a tool like atop work means you are hooking into lots of places that have potential to have unintended consequences.
I pay for our rack outright every 3-4 months from what I can tell. Still takes the same number of infra/ops/sre people as well. We staff 2, but really just have 1.25 worth of FTE work, you just need more for redundancy.
Pretty nuts! This is also why I am so dismissive of performance optimization. Yeah, I'll just buy a new set of three machines with 2tb of ram each in a few years and call it good, still come out ahead.