What one may find in robots.txt
During the reconnaissance stage of a web application testing, the tester
(or attacker) usually uses a list of known subdirectories to brute force
the server and find hidden resources. For that purpose, a list of known
subdirectories is used such as the one provided with Skipfish or wfuzz.
If any extra context information is available, it may be added to the list.
Most importantly, once a test is concluded (or command execution has been
achieved) and a listing or server configuration is disclosed, the list is
updated with any missing entries.