What Web Crawler Does Google Use? Googlebot and Its Family
Article Summary
Google’s main web crawler is Googlebot, and for most sites the version doing the heavy lifting is Googlebot Smartphone. Around it sits a family of specialized crawlers for Images, Video, News and Shopping, plus GoogleOther and a few special-case fetchers. Each has its own user agent token and its own relationship with robots.txt.
- For Google Search, think Googlebot Smartphone first. Googlebot Desktop still visits, just less often for most sites.
- Both Googlebot variants answer to the same robots.txt token, so you can’t block one and keep the other.
- Blocking a URL in robots.txt stops crawling, not indexing. A noindex tag only works if Googlebot is allowed to fetch the page and see it.
- A user agent string proves nothing on its own. Verify with a reverse and forward DNS lookup, or against Google’s published IP ranges.
- If crawl blockers keep reappearing after site changes, TAMA ties website audits, internal linking and performance tracking into one workflow, which makes those regressions easier to catch early.
Someone tightens the firewall, spots an unfamiliar bot called Storebot-Google working through the product pages, and blocks it. It wasn’t on the short list of crawlers anyone recognised. Nothing breaks visibly. The site still ranks. But the crawler Google uses for its Shopping surfaces can no longer reach the catalogue, and nobody connects the two events for a long while.
That mistake is common because the honest answer to “what web crawler does Google use” is slightly bigger than one name. Google’s primary web crawler is Googlebot, the program that fetches pages so they can be processed for the Google Search index. Googlebot itself comes in two versions (Smartphone and Desktop), and Google runs several other named crawlers for specific products.
If you manage a site, the distinction matters in practical ways. It decides which version of your pages gets judged, which robots.txt rules actually apply, and whether that “Googlebot” in your logs is even Google.
What web crawler does Google use? Googlebot, explained
Googlebot is the generic name for Google’s two main Search crawlers: Googlebot Smartphone and Googlebot Desktop. According to Google’s own documentation on Googlebot, both exist to fetch web pages for Google Search, and both identify themselves with the same product token in robots.txt.
Here’s how it works. Googlebot starts from URLs Google already knows about, follows links it finds on those pages, and picks up new URLs from sitemaps you submit. It requests each page, receives the HTML, and can render JavaScript using a recent version of Chromium so it sees roughly what a browser would see. Then it hands what it fetched over for processing.
Think of it like a mail carrier working a route. The carrier only reaches addresses that exist on the route; a house with no street connecting to it simply doesn’t get visits. Pages with no internal links and no sitemap entry are in the same position (they exist, technically, but Googlebot has no reason to knock).
Discovery is the part people underrate.
Most crawl conversations jump straight to robots.txt and speed. Our honest view is that the bigger, quieter problem on most sites is pages Googlebot never finds, not pages it finds and refuses to fetch. If you’re auditing a site with a flat, well-linked structure, you can skip this worry. On a sprawling catalogue or an old blog with years of unlinked posts, start here.
Googlebot Smartphone vs Googlebot Desktop: which one shows up
For most sites, Googlebot Smartphone does the bulk of the crawling. That follows from Google mobile-first indexing: Google primarily uses the mobile version of a page’s content for indexing and ranking, so it makes sense for the mobile crawler to do most of the fetching.
Googlebot Desktop hasn’t disappeared. It still visits, and you’ll see it in your logs. It’s just the minority caller.
The consequence is blunt. If something exists on your desktop layout but not on your mobile layout (a block of product specs, a set of internal links, structured data someone only added to the desktop template), Google may effectively not see it. The mobile version is the version that counts.
You can tell the two apart in server logs by the full user agent string: the smartphone variant identifies itself as a mobile browser on Android, while the desktop variant presents as a desktop browser. Both carry the Googlebot identifier. That’s useful for log analysis, but it changes nothing in robots.txt, because both obey the single Googlebot token. There’s no way to allow one and block the other there.
One practical check we’d recommend: open the mobile version of your most important template on an actual phone and compare it line by line against desktop. Tedious? Yes. It also catches the missing-content problem faster than any report.
The specialized crawlers behind Images, Shopping and News
Googlebot handles Google Search, but other Google products send their own crawlers. Google groups these in its overview of Google crawlers and fetchers, splitting them into common crawlers, special-case crawlers and user-triggered fetchers. The table below covers the ones site owners run into most.
| Crawler | robots.txt token | What it fetches for | Follows robots.txt? |
|---|---|---|---|
| Googlebot Smartphone | Googlebot | Google Search (main crawler for most sites) | Yes |
| Googlebot Desktop | Googlebot | Google Search | Yes |
| Googlebot Image | Googlebot-Image | Google Images and image features in Search | Yes |
| Googlebot Video | Googlebot-Video | Video features in Search | Yes |
| Googlebot News | Googlebot-News | Google News (uses Googlebot’s user agent strings) | Yes |
| Google StoreBot | Storebot-Google | Google Shopping surfaces, such as product and checkout pages | Yes |
| Google-InspectionTool | Google-InspectionTool | Search testing tools like URL Inspection and the Rich Results Test | Yes |
| GoogleOther | GoogleOther | One-off crawls by Google product teams, including research and development | Yes |
| AdsBot | AdsBot-Google | Ad landing page quality checks | Only rules that name it; it ignores the * wildcard group |
Notice the pattern here: a rule aimed at Googlebot generally applies to the Search crawler and its media siblings, because Google’s specialized Search crawlers fall back to Googlebot rules when no more specific group exists for them. StoreBot is a separate token with its own job. AdsBot is the odd one out, and that catches people who assume a blanket User-agent: * disallow covers every Google visitor.
It doesn’t.
Crawling and indexing are different jobs
Crawling is fetching. Indexing is deciding what the fetched content means and whether to store it. Google’s guide to how Google Search works describes three stages: crawling, indexing and serving results. Googlebot only does the first.
A library is a decent comparison. Acquiring a book and cataloguing it are separate steps, done by different people, and plenty of donated books never make it onto a shelf. Google treats crawled pages the same way: after Googlebot fetches a page, Google analyses the text, images and other signals, groups duplicates, picks a canonical version, and then decides whether the page belongs in the Google Search index at all.
Here’s the example that makes the difference real. Picture a staging folder, /staging-v3/, blocked in robots.txt. A partner links to one of those staging URLs from their site. Googlebot can’t fetch the page, so Google never reads the content… but it knows the URL exists, and it can still appear in results as a bare URL with little or no description. Blocked from crawling, not from indexing.
The opposite case trips people up too. A page is crawled happily, every day, and never gets indexed, because Google judged it a near-duplicate of another page or simply not useful enough. Crawl activity in your logs is not a promise of rankings.
GoogleOther, Google-Extended and AI fetching
This is where most current confusion lives.
AI features inside Search, such as AI Overviews, are part of Google Search, so the content they draw on gets fetched by Googlebot, the same as a normal blue-link result. There isn’t a separate “AI Overviews crawler” to allow or block. If you block Googlebot to keep a page out of AI answers, you’re also pulling it out of regular Search (which is rarely the trade anyone actually wanted).
Google-Extended is the piece people misread. It’s a robots.txt product token, not a crawler with its own user agent. Disallowing it tells Google you don’t want your content used for training future Gemini models or for grounding in certain Gemini features. Google states it doesn’t affect a site’s inclusion or ranking in Google Search. So it’s a usage control layered on top of crawling done by existing Google crawlers, not a separate bot you’ll spot in your logs.
GoogleOther is different again. It’s a generic crawler that Google product teams use for fetches that aren’t tied to Search indexing, such as research and development crawls. It follows robots.txt under its own token. Blocking it doesn’t touch your Search visibility, though we’d only bother if it’s genuinely straining your server; there’s no strong reason to block it by default.
Our advice, stated plainly: for a site that depends on organic traffic, blocking Googlebot to manage AI exposure is almost always the wrong move. The exception is content you’d rather not have in Google Search at all, in which case noindex and access controls are the honest tools. If AI answer visibility is the thing you’re actually trying to influence, measuring it is a separate discipline (we wrote about that in our guide to LLM visibility tracking and measurement), and optimizing for it has its own playbook, covered in our look at AI Overviews optimization tools.
How to check that “Googlebot” is really Google
A user agent string is a claim, not proof. Anyone can send a request labelled Googlebot, and scrapers do it constantly because they expect sites to wave Googlebot through.
Google gives you two ways to check. The first is a DNS round trip:
- Take the IP address from your log line and run a reverse DNS lookup on it.
- Confirm the hostname ends in googlebot.com, google.com or googleusercontent.com.
- Run a forward DNS lookup on that hostname and make sure it resolves back to the original IP. Skip this step and a spoofer with control over their own reverse DNS can fool you.
The second way is easier to automate. Google publishes its crawler IP ranges as JSON files, split by crawler category, and you can match incoming IPs against them. If you’re writing a firewall rule, this is the approach we’d pick (DNS lookups on every request are slow, and nobody wants that in a hot path).
The limitation nobody mentions: verification tells you the request is from Google, not which product it serves. A verified Google IP fetching your checkout page might be StoreBot, AdsBot or a user-triggered fetcher acting on someone’s request. Read the user agent token for that, after you’ve confirmed the IP is genuine.
Googlebot and robots.txt: what a block actually does
Yes, Googlebot obeys robots.txt. Google’s standard crawlers respect the rules you publish there, as Google explains in its overview of Google’s web crawling. The catch is that robots.txt controls fetching and nothing else.
That one fact explains most robots.txt disasters. A few practical rules follow from it:
- Want a page out of Search? Use a noindex meta tag or header, and leave the page crawlable so Googlebot can read that instruction.
- Never block CSS or JavaScript your pages need to render.
- Google ignores crawl-delay, so it won’t slow Googlebot down. If your server is struggling, fix capacity or return temporary 503 or 429 responses briefly.
- Name AdsBot explicitly if you run ads and want to restrict it. The wildcard won’t do it.
After any robots.txt change, run your key URLs through Search Console’s URL Inspection tool and confirm they’re still fetchable. A stray slash in a Disallow line can wipe out a whole directory, and you’d rather find that in a test than in a traffic report.
User-triggered fetchers are the other exception. When a person asks a Google tool to fetch something (site verification, for example), those fetchers generally ignore robots.txt because a human initiated the request. Don’t treat their appearance in logs as Googlebot misbehaving.
Keeping crawl problems from coming back
Fixing a crawl problem once is easy. Keeping it fixed is the hard part, because sites change constantly. A template update drops internal links from mobile, a plugin rewrites robots.txt, a migration leaves a few hundred pages with no links pointing at them. Each one quietly shrinks what Googlebot can reach.
This is where a recurring audit earns its keep. TAMA connects website audits, internal linking and SEO performance tracking in one workflow, so a page that loses its links or a section that drops in organic performance shows up alongside the content work rather than in a separate spreadsheet someone forgets to open. Orphaned pages matter here specifically: an AI internal linking tool addresses the discovery gap from the first section, since Googlebot follows links to find things.
To be fair about scope: for the raw record of what Googlebot actually requested, your server logs and Google Search Console stay the source of truth. And a small site with a handful of pages honestly doesn’t need an automated platform for this; a manual check after each change is fine, and nobody should talk you out of that. If you want to compare approaches before committing, our automated website audit guide walks through what a repeatable audit should check.
Frequently asked questions about Google’s web crawler
Does Googlebot crawl every page on my site?
No. Googlebot crawls pages it can discover and is allowed to fetch, and Google decides how much to crawl based on factors like how your server responds and how much your content seems to change. Pages with no links and no sitemap entry may never be visited at all.
Is blocking Google-Extended going to hurt my Google rankings?
Google says no. Google-Extended controls whether your content is used for Gemini model training and certain grounding uses, and Google states it doesn’t affect inclusion or ranking in Google Search.
What is Google-InspectionTool, and why does it look like Googlebot?
It’s the crawler behind Google’s Search testing tools, including URL Inspection in Search Console and the Rich Results Test. It mimics Googlebot so that your test results reflect what Search would see. You’ll usually find it in your logs right after someone on your team runs a test, which makes those entries easy to explain.
Can I make Googlebot crawl my site faster?
Not directly, and there’s no setting that forces it. What helps is removing friction: keep your sitemap accurate, fix server errors and slow responses, link important pages from places Googlebot already visits, and use URL Inspection to request indexing for an individual URL that changed. Those steps make crawling easier; they don’t guarantee timing.
Should I block Googlebot Desktop now that mobile-first indexing is standard?
You can’t, at least not separately. Both variants share the Googlebot token in robots.txt, so any rule hits both.
Google uses Googlebot, mostly the smartphone variant, and a handful of named siblings for everything else. Most damage comes from treating them as one bot, or as no bot worth checking. Pick your three most important page templates, confirm the mobile versions carry the same content and links as desktop, and run them through URL Inspection. That single pass catches more crawl trouble than a stack of reports.
FREE WEEKLY GROWTH TIPS
Increase your traffic, leads and revenue
One short email a week with an SEO or GEO tactic you can apply the same day, plus the checklists we use for clients. See what is working right now in Google and in ChatGPT.
Free. One email a week. No spam, unsubscribe anytime.
Want more content like this in your Google results?
Add us as a preferred source and Google will show you our articles more often.