Google's robots.txt - Hackdot

Hackdot

News for hackers. Stuff that matters.

Sections

Top
Latest
Ask HN
Show HN
Jobs
Polls
Older Stuff
Search

About

Every story and comment from the Hacker News API, in a layout borrowed from 2001.

A comment on Google's robots.txt. Parent comment

Re:Google's robots.txt by sfall on Sunday February 23, 2014 @05:32AM (#7285501)

my guess lawyers

[ Parent | on HN ]

There are no comments on this comment yet.

Ask HN

What use are the non-major cryptocurrencies?
OpenAI Astra post-launch slump?
What is the state of self-hosting email in 2026?
Advice for those who want to build stuff but don't know where to start
Has AI coding changed who builds your internal tools?
What customer acquisition channel is working best for you?

Show HN

Free Dictionary API – Wiktionary data, no API key required
VPN Manager
EDNA – 100% on-device human sensing so AI can perceive the user
Podcast News
Back endskills – agent skills for safer auth, data and migration code
Browserr – An MCP-first browser for AI agents (one Rust binary)

Jobs

Autonomous (YC F25) is hiring MTS to build AI agents that manage your wealth
Tinfoil (YC P25) is hiring an applied AI security engineer in San Francisco
Y/N (YC S23) Is Hiring a Growth Lead to Distribute Superhuman A.I. Storytelling
UpCodes (YC S17) is hiring SDR's to sell AI compliance software

Stories and comments come from the Hacker News API and belong to their authors. Not affiliated with Y Combinator or Slashdot.