joe_91

Same.... I'll install it in a few months when all the major security bugs have been found and patched!

Tell that to the thousands of apps/sides out there which rely on scraped data ;) (Including all search engines/LLMs/price comparison sites etc)

True, but also need to know the basics well of what constitutes good code and how it should scale vs just working code. Too many people relying on LLMs to produce stuff which just about works but give users a terrible experience as it bearly works.

Nice concept. I've definitely seen this play out in practice.

A lot of sites aren't impossible to scrape, but they're steadily getting more expensive. We're having to lean more on residential proxies, headless browsers etc just to get the same data that used to be straightforward...

Actually as someone who is over 225lbs and only going up in weight monthly I don't think its fair to presume things about others like that....

As another commenter said - its simple, not easy...

Until they find out that there are unintended side effects...

Better off just exercising and eating as clean as you can!

Instead of focusing solely on the "market boom" and technological advancements, we should be having a serious conversation about the societal implications and the long-term sustainability of this scraping free-for-all.

The "higher stakes" mentioned aren't just about cost and effort; they're about the potential erosion of online privacy, the destabilization of websites reliant on ad revenue, and the ethical responsibility of data practitioners.

It would be interesting to explore ways of profit sharing to cover loss of ad revenue, better caching to reduce the cost on websites themselves, and simple ways to opt in or opt out of personal information being used online!

If only the rest of the western world could do this imagine how much better it could be to live in large cities...

Looks great! I've had the same issue as you with many projects - so many people leave the marketing till the end so don't beat yourself up about it now. (Hopefully you will have learned a lesson haha!)

Lets look at what you could do:

- Start in one local area - ideally one near you or one that you know. Make it work there and then expand.

- See if you can engage any local groups which might engage with the idea - I could see local city/town facebook groups liking this. Or maybe on nextdoor.com.

- Check out the book "The cold start problem" by Andrew Chen. He goes into many examples of lots of companies that had this issue and how they over came the issue and got their first customers - http://andrewchen.com/wp-content/uploads/2022/01/ColdStartPr...

Meetup really could be a 10x better platform. As someone who's run a group on it having to pay to then put in the work to run a group on top of that is just crazy.

They could have monetised the platform in so many other different ways which could have helped the community instead of just going for the easy pickings of charging the event organisers.

Why not charge a percentage on top of any payments take? Create/integrate with a room booking system in cities to help organisers book? (and take a fee on this)

The only thing Meetup does well is discovery - the domain name is amazing for the type of site it is and it is easy enough to search for groups in your city and find what you want. Everything else is pretty terrible.

I don't know how you could justify having more than 3 dev's working on this site the way it is at the moment. It just screams lazy...

Travel between countries in Europe is improving every year too. Next week I'll be traveling from Bordeaux to Berlin (over 1,600km) - it's faster than the car (16 hours by car vs 12 hours by train), and cheaper than flying, in the summer at least (150 euro by train, vs 300 euro by plane - booking 6 weeks before).

That will improve next year too with the direct Paris Berlin train that should only take 7 hours.

I doubt many website owners feel they have an ethical responsibility to provide public APIs to their data so people wouldn't scrape their content. It is up to them what they want to do with it.

I just feel like the whole thing is a chicken and egg problem - both sides could be nicer to the other but no one wants to go first!

I do like the idea of identifying web scrapers in the user agent but I'm not sure how many websites would use it...

What does one get for actually being nice and being an "ethical web scraper"?

I don't see many people bothering to be honest, developers who are out there scraping are mostly just doing a job, they want to be done as quick as possible and most don't care about things like this unless there are repercussions

My only thought on their refunding process is that maybe they don't have a simple way to just click one button to refund everyone for that specific campaign?

We should probably not assume bad intent on their part!

HN also has moderators who can step in when things look like they're getting out of control which really helps. Or in some cases just point people to the rules and tell them to stop causing flamewars or get out.

It really makes a difference to the overall discourse. Something that is missing on most other places on the web!

I think it's interesting how they justify giving 1 million of the donation proceeds to the Organisers but then decide that the rest just doesn't make the cut.

It would be nice if they provided some evidence showing the organisers having anything to do with - "promotion of violence and harassment".

With any of these large scale protests there will be some bad eggs who appear to cause havoc...

This is where a decentralised fundraising platform would come in handy :)

Madness that they haven't gotten back to your access request in 6 months!

Why even bother having the API there - so much value can be added by people building on top of YouTube and other large sites, its a shame that most of these large sites do nothing to provide API access and people have to go out of their way to scrape them them...

That looks like a great resource! How often is the data set "updated"?

I'd imagine most people's use cases need data which can change from day to day or week to week but I do think that this is fantastic if I was to have a project which was looking at data across a longer timeframe.

Haha I love that people forget how google/bing are out there scraping everything and anyone who scrapes anything for any other reason is a "bad guy".

You can get around some web scraping blockers by just setting your user agent as Googlebot too which I find funny...

I'm scraping about 30 sites for work at the moment, but have a few that are using Cloudflare which has been a b*tch to deal with. Tried numerous libraries and different proxy providers, but reliability is patchy. Previous fixes like https://github.com/Anorov/cloudflare-scrape don't seem to work anymore after Cloudflare updates, so I've switched to using a pretty optimised headless browser with good proxies instead.

It's unfortunate but most of the time people tend to imitate what the people in their social/peer group are doing. That's partly why parents want their kids going to good schools - they'll pick up good habits from those around them.

If your friends are all talking about investing and saving to buy a house chances are you'll start looking into it too. I know its not an excuse either but people want to be part of a group more than anything and if that group were to mock you for your financial literacy or none of them are bothering to be financially savy I can see why many people are failing to grasp some of these concepts.

In my opinion financial literacy should be promoted via movie stars/ athletes. The very people many poor people look up too. I know Kevin Heart the actor was saying that this is something he really thinks would change a lot of poor peoples lives for the better and he's helped roll out campaigns to try and teach people some financial skills: https://hartofitall.chase.com/index.htm

It's going to get really interesting when the world has 1-2 working aged person for 2-3 retirees...

Not sure the old people will be able to riot on the streets or anything but it will be an interesting problem for humanity to solve!

Maybe the world will end up with all the Mormons doing everything until they eventually take over since they have such large families.

When I worked at Dell they were giving out cash bonuses for people to submit patent ideas with different levels of rewards depending on how far your patent idea got in the process.

The crazy thing is, some people (mostly the Indian cohort) were flat out coming up with idea's and submitting them to the internal patent committee every week/month since the amount offered for them would be the equlivent of 6 months/1 years salary.

At this stage - I don't see wide spread adoption happening outside of Japan or somewhere where a country really pushes for it.

Battery/recharging tech is improving year on year. For their latest model S plaid they've claimed: Tesla says that 15 minutes should be enough to replenish 187 miles (301 km) of range.[1]

In 2-3 years when the rest of the worlds car companies/ fast chargers have caught on I just don't see how Hydrogen can compete with all the downsides it has.

1. https://insideevs.com/news/515641/tesla-models-plaid-chargin...

Another point I think is relevant is that I feel many solutions that you've described are easy to understand, quick fixes. In a world that is growing more and more complicated I think people are more prone to gravitate towards the simple.

Another reason, in my opinion for the leftist narrative being so captivating at the moment is that people are delaying adulthood, kids, family more and more. When you are young, or have little, true life experience you don't see why the simple solutions to help all these people wouldn't work. You haven't been in the trenches of life to realise that compromise is needed and that we don't always understand the 2nd or 3rd order effects that our actions will have.

Again all this is my own personal experience but I think it has some bearing on todays political minefield.