You may well have certain pages that you do not want Google, et al, to index in the SERPS. Examples include log in pages, thank you pages, and perhaps content of little search value such as legal blurb. This can let you control a little more, albeit very slightly, the pages you want Google to show and those that you don’t.
You can use the Custom Layout Update control (in the Custom Design tab) to set the robots control on a per-page basis. Simply add the following code into the Custom Layout Update textarea in the Magento admin:
You can vary the value by using:
NOINDEX (not to index (list in SERPs) a specific page
NOFOLLOW (not to follow links on a specific page)
NOARCHIVE (not to store an old cached copy of the page)
NOSNIPPET (not to show a description under your google listing – will also not show a cached link).
Whilst the last two entries might possibly be useful if you are fixing/trying to hide a content co-up, and want to try and hide historic evidence, the noarchive is sensible if the page content can change frequently over time.
There are some other values you can put in. See the noarchive initiative for more information,