Why Can't I Get Indexed by the Search Engines? Unfortunately, this is an all too common question. If it makes you feel any better, you're not the only one frustrated about the length of time it takes to be indexed, or the many pitfalls involved. It often takes anywhere from two days to as much as six months to be listed on a search engine. A web position submitter report will give you current time estimates for each engine so you'll know what to expect. However, an engine at any time could choose to delay their indexing beyond the "norm" for maintenance or other reasons. On the flip side, you could get lucky and submit just a couple days before an engine does a complete refresh of their database. Therefore, submission times can never be an exact science since we're all ultimately at the mercy of the engine. If you've submitted your site and have waited the estimated time to be indexed and there's still no listing, what do you do?
Here are some tips that may help:
— First, be sure you're not already indexed but just don't know it. Unfortunately, none of the major engines will e-mail or notify you as to if and when you've been indexed. The method to determine if a page or domain has been indexed varies from one engine to another, and in many cases, it's difficult to tell for sure. Never assume that you're not indexed just because you searched for a bunch of keywords and you never came up in the first few pages of results. You could be in there but buried near the bottom. In addition, it's not very practical to review the status of a number of pages on each major engine each week.
— Make sure you have uploaded the pages to your site before submitting them. This one seems obvious, but submitting a page that does not exist or submitting with a subtle typo in the URL is a goof we might all make at one time or another.
— If you have information inside frames, that can cause problems with submissions. It's best if you can create non-framed versions of your pages. You should then submit the non-frames versions of your pages which can of course point to your framed Web site. Alternatively, you can enter your relevant text within the noframes area of a framed page which most search engine spiders will read.
— Search engine spiders cannot index sites that require any kind of registration or password. A spider cannot fill out a form of any kind. The same rule applies regarding indexing of content from a searchable database, because the spider cannot fill out a form to query that database. The solution is to create static pages that the engines will be able to find.
— Dynamic pages often block spiders. In fact, any URL containing special symbols like a question mark (?) or an ampersand (&) will be ignored by many engines.
— Most engines cannot index text that is embedded in graphics. Text that appears in multimedia files (audio and video) cannot be indexed by most engines. Information that is generated by Java applets or in XML coding cannot be indexed by most engines.
— If your site has a slow connection or the pages are very complex and take a long time to load, it might time out before the spider can index all the text. For the benefit of your visitors and the search engines, limit your page size to less than 60K. In fact, most Webmasters recommend that your page size plus the size of all your graphics should not exceed 50K-70K. If it does, many people on dial up connections will leave before the page fully loads.
— If you submit just your home page, don't expect a search engine to travel more than one or two links away from the home page or the page that you submitted. Over time they may venture deeper into your site, but don't count on it. You'll often need to submit pages individually that appear further down into your site or have no link from the home page.
— If your Web site fails to respond when the search engine spider pays a visit, you will not be indexed. Even worse, if you are indexed and they pay a visit when your site is down, you'll often be removed from their database! Therefore, it pays to have a reliable hosting service that is up 99.5% of the time. However, at some point a spider is going to hit that other 0.5% and end up yanking your pages by mistake. Thus, it pays to keep a close eye on your listings. To check on whether you are listed in Yahoo and Google go to this site: http://www.top25web.com/cgi-bin/report.cgi
— If you have ever used any questionable techniques that might be considered an overt attempt at spamming (i.e., excessive repetition of keywords, same color text as background, or other things that are warned about), an engine may ignore or reject your submissions. If you're having trouble getting indexed in the expected amount of time, make sure your site is spam-free.
— If your site contains redirects or meta refresh tags these things can sometimes cause the engines to have trouble indexing your site. Generally they will index the page that it is redirecting to, but if it thinks you are trying to "trick" the engine by using "cloaking" or IP redirection technology, there's a chance that it may not index the site at all.
— If you're submitting to a directory site like Yahoo, Open Directory, NBCi, Looksmart, or others, then a human being will review your site. They must decide the site is of sufficient "quality" before they will list it. I recommend you read the submission guide of the site.
— A number of engines no longer index pages residing on many common free web hosting services. The common complaint from the engines is that they get too many "junk" or low-quality submissions from free web site domains. Therefore, they often choose not to index anyone from those domains or they limit submissions from them. It's always best to buy your own domain name (very important) and place it on a respected, paid hosting service to avoid being discriminated against.
— Some engines have been known to drop pages that cannot be traveled to from the home page. HotBot has been rumored to do this. You may want to consider submitting your home page that links either directly or indirectly to your doorway pages.
— Make sure you're submitting within the recommended limits. Some engines do not like more than a certain number of submissions per day for the same domain. If you exceed the limit, you may find that all your submissions are ignored. Some submission consultants feel it is dangerous to submit more than ONE page a day to a engine for a given Web site. For those who wish to be ultra-conservative in their approach, the WebPosition Submitter includes a checkbox to limit submissions to one URL per day per engine.
— Lately I have learnt that a site with just a home page and external links, or multiple url's with the same site content may very likely get your site banned. I know, it seems only logical that one should buy up all the url extensions associated with a name but apparently the new spiders feel as if this practice constitutes spamming.
— Last but not least, sometimes the engines just drop submissions at random through technical errors and bugs. Therefore, some people like to resubmit once or twice a month for good merit in case they do miss a submission. Certainly if you've followed all the "rules" and are still not listed, re-submit! Sometimes a little persistence is all that's needed.
If any of the above scenarios apply to your submission, you should make the necessary adjustments and re-submit. If that still does not work, you should consider e-mailing or calling the search engine and asking them politely why you have not been indexed yet. Sometimes they will reply back with "Sorry, there was a problem with our system and I've made sure you'll be indexed within the next couple days." Or, sometimes they'll tell you why you were not indexed. In other cases, they will ignore your e-mail and you'll have to keep e-mailing or calling them until they respond. Still, it's definitely worth the effort to get your site listed with the major engines assuming you also take the time to optimize your pages so you'll achieve top rankings. |
A Closer Look at Search Engines and Directories
Search Engines uses spiders, whereas Directories typically use people and require one to select a category for inclusion. Many search engines are free, while others require you to pay for inclusion. Usually, a search engine’s spider will include the pages on your site in its database once you have submitted the request to be added, but sometimes they can’t for a number of reasons. They might have problems with frames or image maps on a Web site, they might simply miss a page, and so on. Even though a number of spiders constantly crawl the Web looking for sites, I suggest you take a proactive approach and submit all appropriate pages on your site to the search engines to guarantee that all your important pages are properly listed. But before you submit, check the search engine’s submission document to be sure submitting more than one page is permitted, because you don’t want your site to be rejected. A search engine might also have restrictions on the number of pages you can submit in a single day—perhaps only 5 or 10 pages are allowed to be submitted.
Some of the search providers share technology. Many search engines and directories either partner with or license the use of another search engine or directory’s search technology. Being indexed by these engines means your Web site is likely to be found in other major search services. For example, Google’s results can be found on AOL, Netscape, and even sites such as CNN.
The ranking criteria can differ to determine who gets top placement so even though two search engines might use the same database they can provide different search results. For example, some search engines determine how often a keyword appears on the Web page. It is assumed that if a keyword is used more frequently on a page, then that page is more relevant than other pages with a lower usage of that keyword. Some search engines look for the keyword in the title of the Web page and assume that if the keyword is in the title, then that page must be more relevant than those that don’t have the keyword in their title. Some search engines determine where keywords are used and assume that pages with keywords in the headings and in the first couple of paragraphs
are more relevant. Some search engines use the number of links pointing to a particular page as part of their ranking criteria. Some search engines use information contained in meta-tags; others don’t look at the meta-tags at all.
Develop an SEO plan
First off, make certain that your pages are search engine optimized. This includes the page URL, title, keyword and description meta-tags, alt tags on images, keyword density on pages, and title tags on linkable text.
Next you need to keep of log of the search engines submitted to, as well as when you submitted. In most cases you don't want to submit a site that already has a high ranking (considered spamming), and you don't want to submit more than once per month.
The following are recommended:
• Convert graphics to HTML text.
• Edit elements of the HTML code on every page of the site.
• Remove or reduce the use of Flash (flash cannot be indexed by the engines).
• Create a specialized text file called robots.txt and have it placed in the root directory of the site.
• Rewrite page text to reflect more commonly searched terms.
Keyword Selection Is Key
Careful keyword selection is the heart of the SEO plan. Site owners who are on top of their SEO game have a list of top-priority keywords that they use on their site, with reasonable repetition, in strategic places. You should never let a site go for six months without checking the keywords to make sure they’re still appropriate. If a site’s focus or positioning changes, new keywords are needed.
With your SEO plan, you need to track the progress of your site and assess the competition.
Google Analytics (free from Goggle) is one of the best way to track which pages are visited, find out your visitor's internet browser, and where they have come from. Why the browser? Well, sometimes it is a good idea to optimize a site, or make another version available for a particular browser if the page does not display well (Netscape and IE have always shown pages differently). There are many other free sites that offer tracking resources as well, and typically all one does is apply a snippet of code on a page which is tracked to provide site statistics.
Evaluate your competition, and identify your top 5 competitors. If they have a higher ranking than yours look at how their meta-tags vary from your own (view source), and then consider the other elements such as the visual look of the site, services they offer (such as free ones), the navigation structure (ease of use, A.D.A. accessibility, and text links), and the quality of their page text content. After you have appraised their site, look at your own again and compare them.
Indicate yes or no for the following statements about each of your landing pages:
• This page has a unique HTML page title.
• The HTML page title contains my target keywords.
• This page contains 200 or more words of HTML text.
• HTML text on this page contains my exact target keywords.
• This page can be reached from the home page of the site by following HTML text links (not pull-downs, login screens, or pop-up windows).
• The HTML text links from other pages on my site to this page contain my target keywords.
The advantages of surfing:
• Surfing helps you find and assess the quality of sites linking to you and locate new sites that may want to link to you.
• It helps you find new search products and opportunities that may be useful for promoting your organization.
• It helps you to think like a searcher, using a variety of techniques to find important information.
• And it helps you get familiar with the wide range of available search engine and directory listings.
Here are a couple tools/techniques I would recommend.
The Google Toolbar
The Google Toolbar, which can be downloaded from http://toolbar.google.com/, is a free add-on to your browser (Internet Explorer or Firefox) that contains several features such as a toolbar that displays the Google PageRank value for the web page being viewed.
Keep current on the latest SEO news:
• www.forums.searchenginewatch.com
• www.highrankings.com/forum/
• www.webmasterworld.com
• www.searchengineforums.com
And lastly, the major players in the search engine industry are:
• Google (http://www.google.com/)
• Teoma (http://www.teoma.com/)
• Yahoo! Search (http://www.yahoo.com/)
• MSN Search (http://search.msn.com)—Soon to be powered by MSNBot.
Popular directories include:
• Yahoo! (http://www.yahoo.com/)
• LookSmart/Zeal (http://www.looksmart.com/)
• Open Directory (http://www.dmoz.org/)
• About.com (http://www.about.com/)
• Business.com (http://www.business.com/) |