Do you want Google to come and crawl your page or site faster? Why not just ask, or tell it to anyway.
One of the things that doesn’t get a lot of attention, or mentioned that often is how to tell Googlebot that you posted a new page that you would like crawled and hopefully indexed.
I know a lot of people have difficulties getting crawled and indexed. People post something new and expect Google to find and index it almost immediately, which doesn’t always happen. This can depend on a lot of factors such as: How new your website is, how many backlinks you have, even what kind of script your site is based on, just to name a few.
People talk about using pinging services and sites to help speed up the process. The expectation is you go to a service like Pingomatic add your website link and details, hit the ping button, and somehow all the bots and crawlers are going to come rushing. Maybe you have better luck with things like these than I do. Every time I have ever tried a ping service the results were so poor I have doubts about whether they do anything at all.
Other people use them to ping a blog after they have left a comment hoping that Google will come running and find the link back (backlink) to their website. Seeing how I never had any luck with pinging my own posts, I have doubts about doing this as well. To me it seems like it’s more work than what it’s worth.
When I post something new with a WordPress based website Google always crawls and indexes it within 30 minutes or less, and usually more like 5 to 10 minutes. Part of that could be attributed to the built-in update services (ping) feature that WordPress comes with. Other scripts I use that don’t include a pinging feature can take several days, and sometimes even up to a week on occasion.
Food for thought:
This reminds me. Remember when Dominos Pizza had a guaranteed delivery time of 30 minutes or less? Maybe Google can do something like: Google: We’ll crawl and index your site in 30 minutes or less…The guarantee? I don’t know about that part yet!
If it seems like Googlebot is taking a long time to find and index your new posts you can simply submit your new posts (URLs) in Google Webmaster Tools using Fetch as Googlebot. Obviously you will need to have an account with a verified site setup to do this.
How to Submit URLs to Google with Fetch as Googlebot
1. Login to Google Webmaster Tools
2. Find Diagnostics, and then “Fetch as Googlebot”
3. Enter the address (URL) to your new post or page you want crawled, and click “Fetch”
4. It will say something like “Your request was completed successfully.” near the top of the page. Your link should also appear in the URL status area along with the time. If it doesn’t show up right away be patient. Sometimes it takes a few minutes. You may need to come back or refresh this page.
5. Now you can click on the “Submit to index” link.
6. A little window will open or appear and you can choose “URL” or “URL and all linked pages”. Make your selection and click submit.
At the time of this post you are allowed:
You can submit 500 urls per week and 10 urls with all linked pages per month.
7. Now you can see it says: “URL submitted to index”. You will also notice I have 499 fetches remaining for the month.
There is no guarantee that your post will be added to Google, but at least you told it where to find it and to consider it. Supposedly after doing a Fetch as Googlebot it will be crawled within 24 hours.
It makes more sense to me if you tell Google where to find your new post or page, and to have it crawled within 24 hours than to rely on a 3rd party pinging service that you’re not even sure about.
Obviously if Google crawls and indexes your site, posts, and pages frequently then you shouldn’t have to do this.
More info:
Originally (or a while ago anyway)
You could submit 50 URLs per week and 10 URLs with all linked pages per month.
Currently (at the time of this post anyway)
You can submit 500 URLs per week and 10 URLs with all linked pages per month.
So you can submit more URLs now, but you still might want to choose your URLs with linked pages wisely since you are only allowed 10 per month.
Resources and info at Google:
Ask Google to crawl a page or site
Submit URLs to Google with Fetch as Googlebot
I am the site owner and administrator of DialMe.com. I write Boonex Dolphin Tutorials on the main part of this website where you will also find tips, webmaster resources and more. Here, on the blog I write about a variety of topics surrounding WordPress, technology, social media/networking, SEO, and webmaster resources.
Boonex Dolphin, Wordpress, Seo, Technology, Web Server and Hosting Specialist.
I’m using this everytime whenever I post a new article on my blog and that article will appear in Google search results within 5 -10 minutes. The only thing I’m not too sure/ don’t quite understand is what’s the difference between both options of : ‘URL’ and ‘URL and all linked pages’. I used ‘URL’ everytime and never tried the ‘URL and all linked pages’ before.
Peter Lee recently posted..Why an External Hard Drive is Important to You
I don’t use fetch as Googlebot for a WordPress post. Google always indexes mine in just a few minutes. I use it with other non-Wordpress sites that seem to take much longer to index.
From what I gather the difference between URL and URL and all linked pages is:
URL – Just the page content is crawled by Google.
URL and all linked pages – Your page is crawled and Google will also follow, crawl, and index any links that you have included in your post.
I am not sure, but maybe install WordPress plugin like “Google XML Sitemaps” may help. According to setting options, whenever you have new post it will notify search engine.
Tan recently posted..Yahoo Pipes Tutorial Wrap Up
Google XML Sitemaps is a great plugin. It sure can’t hurt any to use it.
Using feedburner can ease everything. It automatically ping all major pining services when the post is being published.
I do have a couple of feeds set up in Feedburner for other sites, but they tend to take much longer than WordPress. Go figure. I have no complaints about crawling and indexing with WordPress. Other site scrips are a much different story though.
Thanks for sharing so clearly how to submit URL’s to Google via Fetch as Googlebot. Nice post. Very useful.
Catherine Lockey recently posted..Stand Out with Social, Search, and Content!
Glad you liked it.
I use the same plugin as Tam, but I also have submittet my rss feeds to google webmastertools.
Then it does all the work for me.
I also twitterfeed to get my new posts posted on facebook and of course twitter, thereby giving me more time to write posts, guest posts and commenting.
Sounds like you have a good game plan. Always nice to have more time to get some serious work done.
I’m so glad you also inserted images, otherwise I wouldn’t have understood anything. I’ll give a try these days and I’ll analyse the results. Thanks!
Elena recently posted..Licensed Practical Nurse Duties
Images certainly can help making following along easier.
I already submitted URL’s to Google via Fetch as Googlebot and its result is great.
Pete Goumas recently posted..Men’s Gold Rings Throughout History
For me it depends on the site. Sometimes I use it, and other times I don’t bother. It’s worth checking out and testing to see how it works for your sites though.
Thanks for sharing your information on how to submit URL’s to Google via Fetch as Googlebot.The screen shots make it very easy to understand.
Syeda Mehwish recently posted..About NutriSystem Coupon & Review
Glad they were helpful. I hope it works well for you.
Thanks for sharing Fetch as Googlebot, I was not aware of it. I used it for my site and many pages which were not indexed earlier got index in few hours.
Glad it worked out well for you too.
This is a really good idea. I’ll look into this for my site, hopefully it may be able to give it a little boost!
Harriet recently posted..Sony Ericsson Xperia X8
It can’t hurt any, and it is very fast and easy. So your not out anything other than a few short minutes of your time.
I have mixed success using Fetch in Webmaster tools. Often times it gets stuck on Pending and then ultimately fails. When I test my sitemaps they work fine. Some days my posts get indexed within 30 minutes, and other times, it takes several days. It’s a bit frustrating – especially on posts that I’m really proud of and want indexed ASAP. It’s almost like Google knows this and purposely teases me.
Rob R recently posted..I May Have Just Freaked Out Tawny Kitaen
I never had it it fail or get stuck with a pending status. I do have mixed luck with how long it takes to actually appear in the index. It seems to vary from one to the next sometimes.
Interesting. I didn’t know about that feature in Webmaster Tools before. As you say, WordPress sites seem to get their pages indexed in less than an hour anyway, but for other platforms this is certainly a good option. How long did you find indexing took doing it this way? Less than the 24 hours they mentioned?
It seems to vary. It usually gets crawled within 24 hours, but then a little longer to actually show up in a Google search. I think if it happens to get crawled in a few hours, it tends to show up within 24 hours in Google. If it takes nearly 24 hours before it is actually crawled then it seems to be more like 36-48 hours before it appears in Google. Sometimes it seems to be the luck of the draw, or how prompt Google is about crawling the page.
Such a useful article! This method seems to be a really nice tool to get the new posts indexed as soon as possible. Thanks for sharing this helpful information!
Amy recently posted..A Closer Look To The Device Used In Dental Implants
No problem. I hope you get some use out of Fetch As Googlebot.
Wow, this is cool. I had no idea this could be done. Thanks for sharing this tip Ray.
david recently posted..The THRIVE Movie – A Story About Personal Transformation
No problem. My WordPress posts I usually don’t bother because Google indexes them super fast, but if you have other sites where Google seems to be very slow this could help them. It certainly would be worth the few minutes it takes.
Since the update on the 24th my site has went to the second page. i have done some changes to it but the changes are taking forever to change on google. yahoo and bing has already made the changes. What do i have to do to get google to notice.. please help thanks
I guess I’ve never had a problem with Google crawling my site. Every site I’ve put up generally appears in the search results within a day or two. But this is useful advice if it doesn’t.
Leo recently posted..CPA Salary
It’s definitely worth a try it seems like it is taking a while for Google to index a new post.
It might have something to do with the changes that’s what i’m thinking. I’m just trying to get my rankings back up. I need some good high pr back links for my local site here in Nashville TN. Anyone want to help?
Just don’t overdo it. There has been a lot of talk about being to aggressive lately.
I’ve been doing a lot of stuff mostly social bookmarking. I still see no changes I think I’m just going to wait a couple of weeks until this update cleans up a little bit. I do know that i wasn’t penalized and i wasn’t over doing the onpage seo (keyword stuffing). I’m thinking and from what i been seeing it was the backlinks that i had.. Thanks Yall
Good suggestions given here. Seems, it’s better to ask Google bot to crawl a new post, instead of waiting for it to do it naturally. Thanks for sharing. This was very insightful.
My new post are not crawled immediately. I update the latest Exam results in india. But many websites similar to mine is getting crawled and indexed with in few minutes. I tried all the way in google search,but still helpless in getting my new post to be crawled. Please help and suggest any ideas. Thanks in advance
My WordPress sites are almost always crawled and indexed within minutes. My other non-WordPress sites take much longer. If I use the fetch as Googlebot feature in Google Webmaster Tools they are usually indexed in 24 hours give or take a little.
I have been seeing in webmaster tools that google has not crawled my blog in over 2 weeks. I have posted daily and nothing.
I use joomla with a special ping plugin for new articles and a xml sitemap..works great…
Frank recently posted..MP3´s kostenlos – Wie komme Ich zu meinen Lieblingssongs ?
Hi sir, I have build a site at http://heavengb.com. Earlier, I chose culture and tourism topics for my site and added content. Later I decided to build the site only on Biodiversity of my area (Gilgit Baltistan, Pakistan). Then I deleted my all data from my server. I also deleted all indexed pages including homepage from Google Webmaster Tools. Later, now I have somehow build my site but Google is not indexing it. Firstly, I created and uploaded the sitemap to Google, but site not indexed. Then I used “Fetch as Google Bot” option and still my site is not being indexed. Please help.
Hurra ! Yesterday I become the first pagerank( 1 )
for my site…Thaks for your helpfull tips!
Stuttgart recently posted..Zum 20.Todestag von Sonja – Memorial
I also follow this process to get my website indexed quickly and generally, it gets indexed within 24 hours. I tried pinging, submitting rss feeds, etc. earlier & didn’t help anyway. You are right to say that when Google itself is giving you a platform to update your updates, why not use it?
even Im not too sure abt the services of pingomatic but ppl still beleive in submitting the URLs there.
Anyways, your other method can be followed but again, there is no guarentee that Google would come & index your page in a shortest possible time.
I just Want to ask, Is it worth to say that, More google crawl rate it means More traffic and More Higher visibility In google search?
And if it’s not, then why people want higher Google crawl rate?
Saliya Nikhil recently posted..The list of dofollow forums for bloggers to increase backlinks
I have been seeing in webmaster tools that google has not crawled my blog in over 2 weeks.
I have followed the information you have given hope it will help my site to crawl faster by google.
My website was last indexed on september 8 and couple of minutes I tried all the steps hope this solves my problem
I tried using both pinging services and Fetch as Google that you mentioned in the article, but, nothing is working. New articles are not being indexed, I am also updating my site regularly with fresh content.
Any idea what could be wrong? Also, if possible tell me how can I check when google bot last visited my site?
Regards,
I have uploaded my side with new content but in google search engine it shows description and title of my old content site..
can you pls let me know the solution to indexed site ASAP.
Regards
I am still new to this and am trying to figure it all out. Thanks for the tips!
I do not use fetch as Google because Google crawls my blog’s post within a minute. Your post is helpful, thanks…
Google blogspot does a pretty decent job at indexing fast. You probably don’t need to do so with blogspot, but it can be helpful for other sites, cms, scripts, etc.
i have tried absolutly everything and can not get my website to be indexed at all??
Please help?
its been 3 weeks since i added it to webmaster and nothing….
Need I repeat what they already mentioned up there? I’m just here to attest to this. It truly helps. You get the results you want to see faster than before.
Great article thk u. I submitted my pages and got an error message, “There was an error submitting your request”. It didn’t provide any further information. Any suggestions?
When that happens instead of selecting URL try URL and all linked pages. Sometimes the URL only option errors out for some reason. Google might be having some problems or issues at the moment. You can either try again in a few minutes or use the URL and all linked pages option instead.
Great post. Thanks a lot for sharing this useful info. I was much frustrated as my site was not being re-indexed after its removal from Webmasters by me. Now everything is fixed. Thanks again for sharing.
Rehmat recently posted..Link Building-Best Techniques to Build Links
Google can be frustrating that is for sure. Good to hear you got your issues sorted out.
I use Google Webmaster Tools to submit a sitemap.xml file which gets Google to be aware of my new pages.
Google Webmaster Tools is definitely nice to have and use.
good post,i have faced many problems in google,i am getting 90% of trafic from bing and Google is ranking my my keywords for high compet but not low.i think google is only collecting money from ads.
thanks
This is one of the dilemma of most new sites to be indexed in Google and to be on top later on the target keywords.
It’s not that easy to just ask Google crawl your site because it has its own standard and you really have to work on it.
I use Linklicious Basic, but I don’t even know whether it works (it does show the exact time that the page was crawled).
Whether a page gets noticed, crawled and indexed seems totally arbitrary. I wonder if the $14.50 I’m paying a month is really worth it.
Google Webmaster is a great tool to crawl and index your own site, but how do you get Google to crawl a blog that you just left a comment and link?
I am not familiar with Linklicious. I know there are some online tools you use to check the last Google Crawl. I don’t know how accurate they are though. Some people will go and ping a blog post that they just left a comment on hoping that Google will re-crawl it and find their link. Of course your comment would need to be approved and visible first. I never messed with all of that myself to see what the results are. I just know some people do.
So glad that I’ve found your post about my problem! I made some post in squidoo and until now I can’t find it on the Google’s search engine. I also made articles that I want Google to crawl, hope your tool that shared to us will help.
hello admin !
thanks for posting it for us,
tell me anything else we can do for fast crawling.
Akhilesh recently posted..Munna Bhai MBBs-3 Movie
Thank you so much for this information. I couldn’t find a good explanation of how this works and what ‘fetches remaining’ etc meant. Excellent article and now i understand, i wish i knew about it years ago!
I admit, It’s not that easy to just ask Google crawl your site because it has its own standard and you really have to work on it
Yeah funny i found this page looking for a pinger service even though my past experience is similar to yous (no luck) thank you for the useful webmasters tip.
Curtis recently posted..Does The iPhone Have Swype?