adinosaur123

I don't wish to hijack this thread, but I've been pondering a similar question. I've been working on a product that requires a very large amount of data that, as far as I can tell, can only be gathered by scraping (real estate data - even data vendors like estated.com don't have stuff like sales data).

Many, many websites contain legal language that forbids automatic data collection/scraping. How can a business be built in such a case?

Perhaps OPs tool only scrapes a select few sites that don't prohibit scraping, but that seems like the exception, not the norm.

Are there any court cases that provide precedence regarding the legality of web scraping?

I'm currently looking for ways to get real estate listings in a particular area and apparently the only real solution is the scrape the few big online listing sites.

> Would you mind sharing this tool

It's not online. I suppose I could spin it into a web application if the demand was there.

I use to spend a lot of time on Reddit looking for conversations that were relevant to my business. Promoting our products in the context of those conversations usually resulted in sales leads.

I built a tool earlier this year that streams comments from those forums, analyzes them, and pings me when a relevant conversation is found. I generally keep it running throughout the day in my local Terminal.

> "I also believe these greater restrictions will result in more job loss, more hunger, more despair and desperation (...) and more death from causes other than COVID. And I wonder, are these premature deaths any less worrisome than COVID deaths?”

There's been a total of 164 registered COVID-19 related deaths in San Francisco county, the vast majority of which had other comorbidities, assuming trend[1]. On the other hand, there has been nearly three times (~460) as many deaths in San Francisco related to overdoses between January and August alone[2].

It's hard to disagree with his sentiment.

[1]: https://www.cdc.gov/nchs/nvss/vsrr/covid_weekly/index.htm#Co...

[2]: https://www.sfchronicle.com/projects/2020/sf-overdoses-inter...

I headed to Townsend the other morning to grab breakfast and was surprised to see the sign. That's the fourth(!) "spot" that I used to visit regularly that has now permanently closed in my area.

Does anyone know where the state/city legislature is hosted online? I'm genuinely curious on what the laws are that give officials authority to close businesses like they have over the past 9 months.