News for hackers. Stuff that matters.
Re:ByteDance’s Bytespider is scraping at much higher rates t... by benreesman on Monday October 07, 2024 @07:13AM (#41764852)
Indiscriminate scraping is a dick move.

But if you’re going to do it, do it properly. I would have hung it off the Like button with an ungodly ZooKeeper ensemble and trained a GBDT on which parts of which URLs I could just obliterate with Proxygen.

We’d have it all in about 4 days. Don’t ask me how I know.

The second worse thing about the AI megacorps after being evil is being staffed by people who use Cursor.

Edit: on the back of the valued feedback of a valued commenter I’d like to acknowledge that I made a sloppy mistake and have corrected in haste, making no excuses. It would be super great if the largest private institutions in history of the world took the care with give or take everything that I do with trolling on a forum.

Re:ByteDance’s Bytespider is scraping at much higher rates t... by vasco on Monday October 07, 2024 @07:30AM (#41764947)
> But if you’re going to do it, do it proerly

Top shelf unintentional irony.

Re:ByteDance’s Bytespider is scraping at much higher rates t... by Rinzler89 on Monday October 07, 2024 @09:09AM (#41765631)
Exactly. That line of reasoning just feels like established players kicking the ladder from under them in order to maintain their moat, when competitors start to catch up: "Hey, web scraping and data mining should only be allowed the right way, where the right way = our way."

"Free market" to them = the market where they get to write the rulebook.

Re:ByteDance’s Bytespider is scraping at much higher rates t... by earth-adventure on Monday October 07, 2024 @09:57AM (#41766079)
Pretty sure he meant the typo.
Re:ByteDance’s Bytespider is scraping at much higher rates t... by vasco on Monday October 07, 2024 @10:10AM (#41766185)
Exactly.
Re:ByteDance’s Bytespider is scraping at much higher rates t... by viraptor on Monday October 07, 2024 @07:16AM (#41764866)
> is being staffed by people who use Cursor.

Any specific reason?

Re:ByteDance’s Bytespider is scraping at much higher rates t... by trunch on Monday October 07, 2024 @07:38AM (#41765003)
Not OP but other than what core functionality they can demo to investors, every AI company seems to have extremely lacking:

- web design (basic features take years to implement, and when done break the website on mobile)

- UI/UX patterns (cookie cutter component library elements forced into every interface without any tailoring to suit how the product is actually used, also makes a Series C venture indistinguishable from something setup in a weekend)

- backend design (turns out they've been hemorrhaging money on serverless Vercel function calling instead of using Lambda and spending a minute implementing caching for repeat requests)

- developer docs (even when crucial to business model, often seems AI generated, incomplete, incoherent)

And this usually comes from hiring much less developers than is needed, and those that are hired are 10x Cursor/GPT developers which trust it to have done a comprehensive job at what seems like a functional interface on the surface, and have little frame of reference or training for what constitutes good design in any of these aspects.

Re:ByteDance’s Bytespider is scraping at much higher rates t... by benreesman on Monday October 07, 2024 @07:40AM (#41765017)
dawg I ChatGPT’d that license, busy building rn.
Re:ByteDance’s Bytespider is scraping at much higher rates t... by benreesman on Monday October 07, 2024 @08:08AM (#41765185)
I was the guy trolling, downvote me.

Don’t downvote the person who submitted a substantial comment far more valuable than it’s GP.

Re:ByteDance’s Bytespider is scraping at much higher rates t... by raverbashing on Monday October 07, 2024 @07:54AM (#41765096)
> (turns out they've been hemorrhaging money on serverless Vercel function calling instead of using Lambda and spending a minute implementing caching for repeat requests)

Oh but why can't the AI do basic backend programming anymore? /s

Re:ByteDance’s Bytespider is scraping at much higher rates t... by benreesman on Monday October 07, 2024 @07:18AM (#41764884)
Plenty of smart people use Cursor I shouldn’t have been dismissive.

I meant people who don’t work at Cursor.

Re:ByteDance’s Bytespider is scraping at much higher rates t... by t_6_t on Monday October 07, 2024 @07:52AM (#41765080)
I agree on your first point, but even Lex Fridman endorses Cursor, so not sure why you're so negative on that.
Re:ByteDance’s Bytespider is scraping at much higher rates t... by trunch on Monday October 07, 2024 @07:56AM (#41765106)
Only on HN could Lex Fridman's endorsement mean anything when it comes to an IDE
Re:ByteDance’s Bytespider is scraping at much higher rates t... by benreesman on Monday October 07, 2024 @07:56AM (#41765108)
I remember when people said things like “X is endorsed by John Carmack”.

But Engine John is just a guest on the real show.

I didn’t hang onto the first fortune I made in this game, which in general is a real nuisance, but it has the silver lining that I’m still working and likely will be when Zuckerberg realizes that he hasn’t graduated a class of legitimate E5s since 2017.

Re:ByteDance’s Bytespider is scraping at much higher rates t... by t_6_t on Monday October 07, 2024 @08:10AM (#41765196)
Totally get where you’re coming from with the 'endorsement culture'—it’s easy to overhype a product just because a big name is behind it. But I think tools like Cursor actually help a lot of devs, not just because Lex Fridman likes it, but because they improve workflows. They might not be ‘the show,’ but they’re definitely part of what’s driving the evolution of how we work. Even Carmack’s influence is part of how we build modern tech, no?
Re:ByteDance’s Bytespider is scraping at much higher rates t... by benreesman on Monday October 07, 2024 @08:19AM (#41765259)
^ it’s considered polite to use an “Edit:” annotation when dramatically changing what someone replied to.

Cursor is a great product, done by brilliant people.

I very much doubt that they set out to dramatically amplify the byte-denominated output and performance-cycle clout of the exact group of people who want to generate their code but find emacs macros or protobuf or any of the more sophisticated zero-temperature codegen mechanisms too high a complexity bar to clear before blasting metric tonnage of generated slop into our collective lives.

The target audience of Cursor is the last group of people you want jizzing on your codebase at scale while simultaneously capturing mindshare proportional to NVIDIA’s market capitalization.

It’s somewhere between dorky and cringe when Jensen signs some girl’s breasts like he’s Mick Jagger or something. That vibe in your repository is my job security in ten years.

Re:ByteDance’s Bytespider is scraping at much higher rates t... by sshine on Monday October 07, 2024 @11:09AM (#41766772)
> people who use Cursor

What's Cursor?

Re:ByteDance’s Bytespider is scraping at much higher rates t... by ro_bit on Monday October 07, 2024 @11:11AM (#41766809)
AI based code editing tool (https://www.cursor.com/)