.Google.com's Gary Illyes shows Googlebot gathers web links rather than "complying with" them, opposing official documents.Googlebot collects links instead of observing all of them in real-time.Gary Illyes discussed this relevant information in Google.com's 'Search Off The Report' podcast.Google.com's documentation negates the true spider habits.