Make your web page more easily indexed by search engines

At Indiana University, the Pages server has a robots.txt file that denies access to all robots, including search engine robots. Faculty and staff may change their robots.txt search engine crawl settings at Search Engine Crawl Settings for Pages. However, all Pages service web pages can be included in the IU Search Engine's index of pages; see Publish a web page at IU using Pages. Web pages on the Webserve server (e.g.,, have no such restrictions on indexing.

If the server that hosts your web page permits search engines to index it, you can easily include indexing information using <meta> tags. Information in <meta> tags can be searched by title, author, subject, description, or any combination of those categories, for example:

 <meta name="subject" content="Rebel Alliance FAQ"> <meta name="author" content="Luke Skywalker"> <meta name="keywords" content="Rebel alliance seeks to overthrow the evil empire and restore freedom to the galaxy"> <meta name="description" content="Read this if you want to know the truth about the Empire.">

You can use these four <meta> fields to customize descriptions used in databases or to force a match to certain keywords. For this reason, it can be useful to make the subject attribute of a document contain a standard document identifier or combination of identifiers, such as "FAQ", "Resume", etc.

This is document ahfd in the Knowledge Base.
Last modified on 2018-10-04 14:34:43.

Contact us

For help or to comment, email the UITS Support Center.