Robots.txt Tester
Fetches robots.txt for a host and lists user-agents, disallow rules and sitemaps.
Reading the result
A bare Disallow: / blocks crawling of the whole host for that user-agent.
Limitations
Robots.txt controls crawling rather than indexing.