Ask Google to Crawl Your Page or Site

Tweet about this on TwitterShare on FacebookShare on Google+Share on LinkedIn

GoogleDo you want Google to come and crawl your page or site faster? Why not just ask, or tell it to anyway.

One of the things that doesn’t get a lot of attention, or mentioned that often is how to tell Googlebot that you posted a new page that you would like crawled and hopefully indexed.

I know a lot of people have difficulties getting crawled and indexed. People post something new and expect Google to find and index it almost immediately, which doesn’t always happen. This can depend on a lot of factors such as: How new your website is, how many backlinks you have, even what kind of script your site is based on, just to name a few.

People talk about using pinging services and sites to help speed up the process. The expectation is you go to a service like Pingomatic add your website link and details, hit the ping button, and somehow all the bots and crawlers are going to come rushing. Maybe you have better luck with things like these than I do. Every time I have ever tried a ping service the results were so poor I have doubts about whether they do anything at all.

Other people use them to ping a blog after they have left a comment hoping that Google will come running and find the link back (backlink) to their website. Seeing how I never had any luck with pinging my own posts, I have doubts about doing this as well. To me it seems like it’s more work than what it’s worth.

When I post something new with a WordPress based website Google always crawls and indexes it within 30 minutes or less, and usually more like 5 to 10 minutes. Part of that could be attributed to the built-in update services (ping) feature that WordPress comes with. Other scripts I use that don’t include a pinging feature can take several days, and sometimes even up to a week on occasion.

Food for thought:
This reminds me. Remember when Dominos Pizza had a guaranteed delivery time of 30 minutes or less? Maybe Google can do something like: Google: We’ll crawl and index your site in 30 minutes or less…The guarantee? I don’t know about that part yet!

If it seems like Googlebot is taking a long time to find and index your new posts you can simply submit your new posts (URLs) in Google Webmaster Tools using Fetch as Googlebot. Obviously you will need to have an account with a verified site setup to do this.

How to Submit URLs to Google with Fetch as Googlebot

1. Login to Google Webmaster Tools

2. Find Diagnostics, and then “Fetch as Googlebot”

3. Enter the address (URL) to your new post or page you want crawled, and click “Fetch”

Fetch as Googlebot

 

4. It will say something like “Your request was completed successfully.” near the top of the page. Your link should also appear in the URL status area along with the time. If it doesn’t show up right away be patient. Sometimes it takes a few minutes. You may need to come back or refresh this page.

5. Now you can click on the “Submit to index” link.

WMT Submit to Index

 

6. A little window will open or appear and you can choose “URL” or “URL and all linked pages”. Make your selection and click submit.

WMT Submit URL

 

At the time of this post you are allowed:
You can submit 500 urls per week and 10 urls with all linked pages per month.

7. Now you can see it says: “URL submitted to index”. You will also notice I have 499 fetches remaining for the month.

WMT URL Submitted

 

There is no guarantee that your post will be added to Google, but at least you told it where to find it and to consider it. Supposedly after doing a Fetch as Googlebot it will be crawled within 24 hours.

It makes more sense to me if you tell Google where to find your new post or page, and to have it crawled within 24 hours than to rely on a 3rd party pinging service that you’re not even sure about.

Obviously if Google crawls and indexes your site, posts, and pages frequently then you shouldn’t have to do this.

More info:
Originally (or a while ago anyway)
You could submit 50 URLs per week and 10 URLs with all linked pages per month.

Currently (at the time of this post anyway)
You can submit 500 URLs per week and 10 URLs with all linked pages per month.

So you can submit more URLs now, but you still might want to choose your URLs with linked pages wisely since you are only allowed 10 per month.

Resources and info at Google:
Ask Google to crawl a page or site
Submit URLs to Google with Fetch as Googlebot

About: Jeremy LeSarge - AKA: Ray (209 Posts)

I am the site owner and administrator of DialMe.com. I provide help and tips for Boonex Dolphin on the main part of this website where you will also find an assortment of other resources. Here, on the blog I write about a variety of topics surrounding WordPress, technology, social media/networking, SEO, and webmaster resources.




71 Comments

  1. Reply

    I’m using this everytime whenever I post a new article on my blog and that article will appear in Google search results within 5 -10 minutes. The only thing I’m not too sure/ don’t quite understand is what’s the difference between both options of : ‘URL’ and ‘URL and all linked pages’. I used ‘URL’ everytime and never tried the ‘URL and all linked pages’ before.
    Peter Lee recently posted..Why an External Hard Drive is Important to YouMy Profile

    • Reply

      I don’t use fetch as Googlebot for a WordPress post. Google always indexes mine in just a few minutes. I use it with other non-Wordpress sites that seem to take much longer to index.

      From what I gather the difference between URL and URL and all linked pages is:
      URL – Just the page content is crawled by Google.
      URL and all linked pages – Your page is crawled and Google will also follow, crawl, and index any links that you have included in your post.

  2. Tan

    Reply

    I am not sure, but maybe install WordPress plugin like “Google XML Sitemaps” may help. According to setting options, whenever you have new post it will notify search engine.

  3. Worli

    Reply

    Using feedburner can ease everything. It automatically ping all major pining services when the post is being published.

    • Reply

      I do have a couple of feeds set up in Feedburner for other sites, but they tend to take much longer than WordPress. Go figure. I have no complaints about crawling and indexing with WordPress. Other site scrips are a much different story though.

  4. Reply

    I use the same plugin as Tam, but I also have submittet my rss feeds to google webmastertools.
    Then it does all the work for me. 😉

    I also twitterfeed to get my new posts posted on facebook and of course twitter, thereby giving me more time to write posts, guest posts and commenting. 😀

  5. Reply

    I’m so glad you also inserted images, otherwise I wouldn’t have understood anything. I’ll give a try these days and I’ll analyse the results. Thanks!

  6. Pete Goumas

    Reply

    I already submitted URL’s to Google via Fetch as Googlebot and its result is great.

    • Reply

      For me it depends on the site. Sometimes I use it, and other times I don’t bother. It’s worth checking out and testing to see how it works for your sites though.

  7. Syeda Mehwish

    Reply

    Thanks for sharing your information on how to submit URL’s to Google via Fetch as Googlebot.The screen shots make it very easy to understand.

  8. Reply

    Thanks for sharing Fetch as Googlebot, I was not aware of it. I used it for my site and many pages which were not indexed earlier got index in few hours.

  9. Harriet

    Reply

    This is a really good idea. I’ll look into this for my site, hopefully it may be able to give it a little boost!

    • Reply

      It can’t hurt any, and it is very fast and easy. So your not out anything other than a few short minutes of your time.

  10. Rob R

    Reply

    I have mixed success using Fetch in Webmaster tools. Often times it gets stuck on Pending and then ultimately fails. When I test my sitemaps they work fine. Some days my posts get indexed within 30 minutes, and other times, it takes several days. It’s a bit frustrating – especially on posts that I’m really proud of and want indexed ASAP. It’s almost like Google knows this and purposely teases me.

    • Reply

      I never had it it fail or get stuck with a pending status. I do have mixed luck with how long it takes to actually appear in the index. It seems to vary from one to the next sometimes.

  11. Kathy

    Reply

    Interesting. I didn’t know about that feature in Webmaster Tools before. As you say, WordPress sites seem to get their pages indexed in less than an hour anyway, but for other platforms this is certainly a good option. How long did you find indexing took doing it this way? Less than the 24 hours they mentioned?

    • Reply

      It seems to vary. It usually gets crawled within 24 hours, but then a little longer to actually show up in a Google search. I think if it happens to get crawled in a few hours, it tends to show up within 24 hours in Google. If it takes nearly 24 hours before it is actually crawled then it seems to be more like 36-48 hours before it appears in Google. Sometimes it seems to be the luck of the draw, or how prompt Google is about crawling the page.

  12. Amy

    Reply

    Such a useful article! This method seems to be a really nice tool to get the new posts indexed as soon as possible. Thanks for sharing this helpful information!

    • Reply

      No problem. My WordPress posts I usually don’t bother because Google indexes them super fast, but if you have other sites where Google seems to be very slow this could help them. It certainly would be worth the few minutes it takes.

  13. Reply

    Since the update on the 24th my site has went to the second page. i have done some changes to it but the changes are taking forever to change on google. yahoo and bing has already made the changes. What do i have to do to get google to notice.. please help thanks

  14. Leo

    Reply

    I guess I’ve never had a problem with Google crawling my site. Every site I’ve put up generally appears in the search results within a day or two. But this is useful advice if it doesn’t.

    • Reply

      It’s definitely worth a try it seems like it is taking a while for Google to index a new post.

  15. Reply

    It might have something to do with the changes that’s what i’m thinking. I’m just trying to get my rankings back up. I need some good high pr back links for my local site here in Nashville TN. Anyone want to help?

  16. Reply

    I’ve been doing a lot of stuff mostly social bookmarking. I still see no changes I think I’m just going to wait a couple of weeks until this update cleans up a little bit. I do know that i wasn’t penalized and i wasn’t over doing the onpage seo (keyword stuffing). I’m thinking and from what i been seeing it was the backlinks that i had.. Thanks Yall

  17. Reply

    Good suggestions given here. Seems, it’s better to ask Google bot to crawl a new post, instead of waiting for it to do it naturally. Thanks for sharing. This was very insightful.

  18. Reply

    My new post are not crawled immediately. I update the latest Exam results in india. But many websites similar to mine is getting crawled and indexed with in few minutes. I tried all the way in google search,but still helpless in getting my new post to be crawled. Please help and suggest any ideas. Thanks in advance

    • Reply

      My WordPress sites are almost always crawled and indexed within minutes. My other non-WordPress sites take much longer. If I use the fetch as Googlebot feature in Google Webmaster Tools they are usually indexed in 24 hours give or take a little.

  19. larry

    Reply

    I have been seeing in webmaster tools that google has not crawled my blog in over 2 weeks. I have posted daily and nothing.

  20. Reply

    Hi sir, I have build a site at http://heavengb.com. Earlier, I chose culture and tourism topics for my site and added content. Later I decided to build the site only on Biodiversity of my area (Gilgit Baltistan, Pakistan). Then I deleted my all data from my server. I also deleted all indexed pages including homepage from Google Webmaster Tools. Later, now I have somehow build my site but Google is not indexing it. Firstly, I created and uploaded the sitemap to Google, but site not indexed. Then I used “Fetch as Google Bot” option and still my site is not being indexed. Please help.

  21. Amanda Gates

    Reply

    I also follow this process to get my website indexed quickly and generally, it gets indexed within 24 hours. I tried pinging, submitting rss feeds, etc. earlier & didn’t help anyway. You are right to say that when Google itself is giving you a platform to update your updates, why not use it?

  22. Reply

    even Im not too sure abt the services of pingomatic but ppl still beleive in submitting the URLs there.

    Anyways, your other method can be followed but again, there is no guarentee that Google would come & index your page in a shortest possible time.

  23. Aliuddin Faisal Faruqui

    Reply

    My website was last indexed on september 8 and couple of minutes I tried all the steps hope this solves my problem

  24. Reply

    I tried using both pinging services and Fetch as Google that you mentioned in the article, but, nothing is working. New articles are not being indexed, I am also updating my site regularly with fresh content.
    Any idea what could be wrong? Also, if possible tell me how can I check when google bot last visited my site?

    Regards,

  25. Reply

    I have uploaded my side with new content but in google search engine it shows description and title of my old content site..

    can you pls let me know the solution to indexed site ASAP.

    Regards

    • Reply

      Google blogspot does a pretty decent job at indexing fast. You probably don’t need to do so with blogspot, but it can be helpful for other sites, cms, scripts, etc.

  26. Reply

    i have tried absolutly everything and can not get my website to be indexed at all??
    Please help?
    its been 3 weeks since i added it to webmaster and nothing….

  27. Adela Deeks

    Reply

    Need I repeat what they already mentioned up there? I’m just here to attest to this. It truly helps. You get the results you want to see faster than before.

  28. Damien Town

    Reply

    Great article thk u. I submitted my pages and got an error message, “There was an error submitting your request”. It didn’t provide any further information. Any suggestions?

    • Reply

      When that happens instead of selecting URL try URL and all linked pages. Sometimes the URL only option errors out for some reason. Google might be having some problems or issues at the moment. You can either try again in a few minutes or use the URL and all linked pages option instead.

  29. Reply

    Great post. Thanks a lot for sharing this useful info. I was much frustrated as my site was not being re-indexed after its removal from Webmasters by me. Now everything is fixed. Thanks again for sharing.

  30. Reply

    good post,i have faced many problems in google,i am getting 90% of trafic from bing and Google is ranking my my keywords for high compet but not low.i think google is only collecting money from ads.
    thanks

  31. Holly

    Reply

    This is one of the dilemma of most new sites to be indexed in Google and to be on top later on the target keywords.

  32. Reply

    It’s not that easy to just ask Google crawl your site because it has its own standard and you really have to work on it.

  33. Reply

    I use Linklicious Basic, but I don’t even know whether it works (it does show the exact time that the page was crawled).

    Whether a page gets noticed, crawled and indexed seems totally arbitrary. I wonder if the $14.50 I’m paying a month is really worth it.

    Google Webmaster is a great tool to crawl and index your own site, but how do you get Google to crawl a blog that you just left a comment and link?

    • Reply

      I am not familiar with Linklicious. I know there are some online tools you use to check the last Google Crawl. I don’t know how accurate they are though. Some people will go and ping a blog post that they just left a comment on hoping that Google will re-crawl it and find their link. Of course your comment would need to be approved and visible first. I never messed with all of that myself to see what the results are. I just know some people do.

  34. Reply

    So glad that I’ve found your post about my problem! I made some post in squidoo and until now I can’t find it on the Google’s search engine. I also made articles that I want Google to crawl, hope your tool that shared to us will help. 😀

  35. Reply

    Thank you so much for this information. I couldn’t find a good explanation of how this works and what ‘fetches remaining’ etc meant. Excellent article and now i understand, i wish i knew about it years ago!

  36. Reply

    I admit, It’s not that easy to just ask Google crawl your site because it has its own standard and you really have to work on it

  37. Reply

    Such a very fundamental info for webmasters and bloggers. But still useful reminder. Sometimes most of those webmasters forget.

Leave Comment

Your email address will not be published. Required fields are marked *

CommentLuv badge