Glossary
What is robots.txt?
Robots.txt gives crawler instructions about which areas to request. It is not access protection.
In practice
Check that production content is not accidentally blocked after a staging migration.
Important limitations
Known blocked URLs may still appear in results. Use authentication for private content and appropriate indexing controls for public pages.
Further reading: official background information
A suitable next step
Want to apply this?
Explore the related service or discuss your situation with our team. First an introduction, then a clear proposal.