How to use Robots meta tag
How do you explain a spider of a search engine you only want him to index just the first page of your website or he is allowed to index the whole website. You use the so called ROBOTS tag.
The robots meta tag is not the same as the file called robots.txt You should use these two together. Both are used by the seach engines like Yahoo and Google. If you use this meta tag the wrong way you might shut the search engines out. So the influence of this meta tag is significant.
Meta robots tag controls spiders from crawling and indexing your webpage or post. In this post, we will see all types of tags in detail. We can use more than one meta tags in a webpage but its not a good practice because it can cause conflicts. Hence it should occur only once in a webpage.
The robots meta tag lets you utilize a granular, page-specific approach to controlling how an individual page should be indexed and served to users in search results. Place the robots meta tag in the <head> section of a given page, like this: section of a given page,
The following table shows all the directives that Google honors and their meaning. Note: it is possible that these directives may not be treated the same by all other search engine crawlers. Multiple directives may be combined in a comma-separated list (see below for the handling of combined directives). These directives are not case-sensitive.
<meta name="robots" content="noindex, nofollow" />
<meta name="robots" content="index, nofollow">
<meta name="robots" content="noindex, follow">
<meta name="robots" content="noarchive" />
<meta content='all' name='robots'/>
<meta content='noodp, noydir' name='robots'/>
Noodp= No Open Directory Project
Noydir= No Yahoo Directory
These are used if your website is listed in one of these directories with information you do not want used in the results pages. This might be the case if you have old, outdated listings that no longer apply. They tell robots not to use information from these sources, and they are optional.
The noydir and noodp are Meta Robots commands that instruct search engine bots on how to handle the usage of data for your site in the Yahoo! Directory and the Open Directory Project/DMOZ Directory.
The noydir command corresponds to the Yahoo! Directory and the noodp command correlates to DMOZ.
noydir Explained
When a site is listed in the Yahoo! Directory there is a title and description associated with that listing. So, Yahoo! started showing that title and description as the linking text and snippet text in the search engine results page (SERP) whenver that website would show up (most commonly only the homepage). The noydir command overrides this functionality and gives control back to the website owner to control the linking and snippet text in the SERPs via the Title tag and Meta Description tag on their website. By regaining this control you can adjust those fields whenever desired.
noodp Explained
The same functionality goes for the noodp command. However, this is directed at Google. Google partnered with ODP to power the Google Directory so if you website was listed in DMOZ then Google would pull that listing information as the linking and snippet text in the Google SERPs. The noodp overrides that functionality and give you control back.
Implementation
The most common implementation is to do both commands simultaneously as in your question like such:
<meta name=”robots” content=”noydir,noodp” /> (content field order does not matter)
They can also be listed separately as such:
<meta name=”robots” content=”noydir” />
<meta name=”robots” content=”noodp” />
You can also direct the commands to the specific bots as such:
<meta name=”googlebot” content=”noodp” />
<meta name=”slurp” content=”noydir” />
It is hoped that these examples illustrate several of the methods at your disposal that will help you to tailor the way search engines index a given web page.
For further reading:
Google and NOODP
Yahoo! and NOYDIR
MSN and NOODP
By specifying meta name as “robots”,you are actually blocking all search engine robots. However if you want to block specific search engine bot then you can specify bot name instead of “robots”
BOT Name:
Google: GOOGLEBOT
Yahoo: SLURP
MSN: MSNBOT
ASK: TEOMA
So, if you want to make your page nofollow for Google bot only then you can use below tag.
<meta name="Googlebot" content="noindex" />
Note: In meta name I have given “Googlebot” instead of “robots” to make it applicable for Google search engine bot only.
The robots meta tag is not the same as the file called robots.txt You should use these two together. Both are used by the seach engines like Yahoo and Google. If you use this meta tag the wrong way you might shut the search engines out. So the influence of this meta tag is significant.
Meta robots tag controls spiders from crawling and indexing your webpage or post. In this post, we will see all types of tags in detail. We can use more than one meta tags in a webpage but its not a good practice because it can cause conflicts. Hence it should occur only once in a webpage.
Using the robots meta tag
The robots meta tag lets you utilize a granular, page-specific approach to controlling how an individual page should be indexed and served to users in search results. Place the robots meta tag in the <head> section of a given page, like this: section of a given page,
1234567<!DOCTYPE html>
<html><head>
<meta name="robots" content="noindex" />
(…)
</head>
<body>(…)</body>
</html>Valid indexing & serving directives
The following table shows all the directives that Google honors and their meaning. Note: it is possible that these directives may not be treated the same by all other search engine crawlers. Multiple directives may be combined in a comma-separated list (see below for the handling of combined directives). These directives are not case-sensitive.
| Directive | Meaning |
|---|---|
all | There are no restrictions for indexing or serving. Note: this directive is the default value and has no effect if explicitly listed. |
noindex | Do not show this page in search results and do not show a "Cached" link in search results. |
nofollow | Do not follow the links on this page |
none | Equivalent to noindex, nofollow |
noarchive | Do not show a "Cached" link in search results. |
nosnippet | Do not show a snippet in the search results for this page |
noodp | Do not use metadata from the Open Directory project for titles or snippets shown for this page. |
notranslate | Do not offer translation of this page in search results. |
noimageindex | Do not index images on this page. |
unavailable_after: [RFC-850 date/time] | Do not show this page in search results after the specified date/time. The date/time must be specified in the RFC 850 format. |
Example meta tag robots
<meta name="robots" content="index, follow"><meta name="robots" content="noindex, nofollow" />
<meta name="robots" content="index, nofollow">
<meta name="robots" content="noindex, follow">
<meta name="robots" content="noarchive" />
<meta content='all' name='robots'/>
<meta content='noodp, noydir' name='robots'/>
Noodp= No Open Directory Project
Noydir= No Yahoo Directory
These are used if your website is listed in one of these directories with information you do not want used in the results pages. This might be the case if you have old, outdated listings that no longer apply. They tell robots not to use information from these sources, and they are optional.
What does “noydir,noodp” mean?
The noydir and noodp are Meta Robots commands that instruct search engine bots on how to handle the usage of data for your site in the Yahoo! Directory and the Open Directory Project/DMOZ Directory.
The noydir command corresponds to the Yahoo! Directory and the noodp command correlates to DMOZ.
noydir Explained
When a site is listed in the Yahoo! Directory there is a title and description associated with that listing. So, Yahoo! started showing that title and description as the linking text and snippet text in the search engine results page (SERP) whenver that website would show up (most commonly only the homepage). The noydir command overrides this functionality and gives control back to the website owner to control the linking and snippet text in the SERPs via the Title tag and Meta Description tag on their website. By regaining this control you can adjust those fields whenever desired.
noodp Explained
The same functionality goes for the noodp command. However, this is directed at Google. Google partnered with ODP to power the Google Directory so if you website was listed in DMOZ then Google would pull that listing information as the linking and snippet text in the Google SERPs. The noodp overrides that functionality and give you control back.
Implementation
The most common implementation is to do both commands simultaneously as in your question like such:
<meta name=”robots” content=”noydir,noodp” /> (content field order does not matter)
They can also be listed separately as such:
<meta name=”robots” content=”noydir” />
<meta name=”robots” content=”noodp” />
You can also direct the commands to the specific bots as such:
<meta name=”googlebot” content=”noodp” />
<meta name=”slurp” content=”noydir” />
It is hoped that these examples illustrate several of the methods at your disposal that will help you to tailor the way search engines index a given web page.
For further reading:
Google and NOODP
Yahoo! and NOYDIR
MSN and NOODP
How to block specific search engine bot
By specifying meta name as “robots”,you are actually blocking all search engine robots. However if you want to block specific search engine bot then you can specify bot name instead of “robots”
BOT Name:
Google: GOOGLEBOT
Yahoo: SLURP
MSN: MSNBOT
ASK: TEOMA
So, if you want to make your page nofollow for Google bot only then you can use below tag.
<meta name="Googlebot" content="noindex" />
Note: In meta name I have given “Googlebot” instead of “robots” to make it applicable for Google search engine bot only.
Post A Comment:
0 comments: