Market Via Search Engine

Why Can't I Get Indexed by the Search Engines?

Unfortunately, this is an all too common question. If it makes you feel any better, you're not the only one frustrated about the length of time it takes to be indexed, or the many pitfalls involved. It often takes anywhere from two days to as much as six months to be listed on a search engine. For example, last month Excite finally updated its index for the first time since last August! Luckily, Excite is the most extreme case lately, but waiting several weeks to a month can also be extremely frustrating.

The WebPosition Submitter report will give you current time estimates for each engine so you'll know what to expect. However, an engine at any time could choose to delay their indexing beyond the "norm" for maintenance or other reasons. On the flip side, you could get lucky and submit just a couple days before an engine does a complete refresh of their database. Therefore, submission times can never be an exact science since we're all ultimately at the mercy of the engine.

If you've submitted your site and have waited the estimated time to be indexed and there's still no listing, what do you do now?

1. First, be sure you're not already indexed but just don't know it. Unfortunately, none of the major engines are kind enough to e-mail or notify you as to if and when you've been indexed.

The method to determine if a page or domain has been indexed varies from one engine to another, and in many cases, it's difficult to tell for sure. Never assume that you're not indexed just because you searched for a bunch of keywords and you never came up in the first few pages of results. You could be in there but buried near the bottom.

In addition, it's not very practical to check the status of a number of pages on each major engine each week. Fortunately, WebPosition has a URL verification feature in the Reporter that makes this process much easier. Each time you run a mission, it will report which URLs exist and do not exist in each engine. If you're using WebPosition and are not finding your URLs after submitting, be sure to see this page for common pitfalls to watch out for:

2. Make sure you have uploaded the pages to your site before submitting them. This one seems obvious, but submitting a page that does not exist or submitting with a subtle typo in the URL is a goof we might all make at one time or another. If you're using WebPosition's Submitter, there's a checkbox on tab 2 that forces WebPosition to verify that all your URLs are valid before submitting them.

3. If you have information inside frames, that can cause problems with submissions. It's best if you can create non-framed versions of your pages. You should then submit the non-frames versions of your pages which can of course link to your framed Web site. Alternatively, you can enter your relevant text within the NOFRAMES area of a framed page which most search engine spiders will read.

4. Search engine spiders cannot index sites that require any kind of registration or password. A spider cannot fill out a form of any kind. The same rule applies regarding indexing of content from a searchable database, because the spider cannot fill out a form to query that database. The solution is to create static pages that the engines will be able to find.

5. Dynamic pages often block spiders. In fact, any URL containing special symbols like a question mark (?) or an ampersand (&) will be ignored by many engines.

6. Most engines cannot index text that is embedded in graphics. Text that appears in multimedia files (audio and video) cannot be indexed by most engines. Information that is generated by Java applets or in XML coding cannot be indexed by most engines.

7. If your site has a slow connection or the pages are very complex and take a long time to load, it might time out before the spider can index all the text. For the benefit of your visitors and the search engines, limit your page size to less than 60K. In fact, most Webmasters recommend that your page size plus the size of all your graphics should not exceed 50K-70K. If it does, many people on dial up connections will leave before the page fully loads.

8. If you submit just your home page, don't expect a search engine to travel more than one or two links away from the home page or the page that you submitted. Over time they may venture deeper into your site, but don't count on it. You'll often need to submit pages individually that appear further down into your site or have no link from the home page.

9. If your Web site fails to respond when the search engine spider pays a visit, you will not be indexed. Even worse, if you are indexed and they pay a visit when your site is down, you'll often be removed from their database! Therefore, it pays to have a reliable hosting service that is up 99.5% of the time. However, at some point a spider is going to hit that other 0.5% and end up yanking your pages by mistake. Therefore, it pays to keep a close eye on your listings.

10. If you have ever used any questionable techniques that might be considered an overt attempt at spamming (i.e., excessive repetition of keywords, same color text as background, or other things that the WebPosition Page Critic warns you about), an engine may ignore or reject your submissions. If you're having trouble getting indexed in the expected amount of time, make sure your site is spam-free.

11. If your site contains redirects or meta refresh tags these things can sometimes cause the engines to have trouble indexing your site. Generally they will index the page that it is redirecting TO, but if it thinks you are trying to "trick" the engine by using "cloaking" or IP redirection technology, there's a chance that it may not index the site at all.

12. If you're submitting to a directory site like Yahoo, Open Directory, NBCi, Looksmart, or others, then a human being will review your site. They must decide if the site is of sufficient "quality" before they will list it. I recommend you read the submission guide on the directory tab of the WebPosition Submitter. It contains tips to improve your chances of obtaining a good listing on these directories.

13. A number of engines no longer index pages residing on many common free web hosting services. The common complaint from the engines is that they get too many "junk" or low-quality submissions from free web site domains. Therefore, they often choose not to index anyone from those domains or they limit submissions from them. It's always best to buy your own domain name (very important) and place it on a respected, paid hosting service to avoid being discriminated against.

14. Some engines have been known to drop pages that cannot be traveled to from the home page. HotBot has been rumored to do this. You may want to consider submitting your home page that links either directly or indirectly to your doorway pages.

15. Make sure you're submitting within the recommended limits. Some engines do not like more than a certain number of submissions per day for the same domain. If you exceed the limit, you may find that all your submissions are ignored. Fortunately, WebPosition's submitter will warn you regarding current limits and recommend you stay within them. Some submission consultants feel it is dangerous to submit more than ONE page a day to a engine for a given Web site. For those who wish to be ultra-conservative in their approach, the WebPosition Submitter includes a checkbox to limit submissions to one URL per day per engine.

16. Last but not least, sometimes the engines just lose submissions at random through technical errors and bugs. Therefore, some people like to resubmit once or twice a month for good merit in case they do lose a submission. Certainly if you've followed all the "rules" and are still not listed, re-submit! Sometimes a little persistence is all that's needed.

If any of the above scenarios apply to your submission, you should make the necessary adjustments and re-submit. If that still does not work, you should consider e-mailing or calling the search engine and asking them politely why you have not been indexed yet. Sometimes they will reply back with "Sorry, there was a problem with our system and I've now made sure you'll be indexed within the next couple days." Or, sometimes they'll tell you why you were not indexed. In other cases, they will ignore your e-mail and you'll have to keep e-mailing or calling them until they respond. Still, it's definitely worth the effort to get your site listed with the major engines assuming you also take the time to optimize your pages so you'll achieve top rankings.

Is Submitting Manually Better?

If you have a lot of extra time on your hands, you could submit your pages manually to the search engines each time you change your content. However, this can be time consuming if you are trying to optimize for many search engines on a regular basis. It is also prone to error since you must re-type the URLs each time correctly, or at least paste them in one by one.

Some people are wary of automated submission tools because they've heard they are also prone to errors. In the case of some submission products or services on the market, this is a valid concern. One problem is that many submission tools fail to look for a "success" message after the submission is made. They just assume that all will be "cool" and they move on to the next submission. WebPosition Gold, however, will not report a submission as successful unless it receives back a specific string from the search engine indicating the submission was accepted.

A few people shy away from automated submissions for fear that the submissions will be sent to the search engine so quickly that it will "red-flag" them as using an automated tool. However, with WebPosition, you may choose to submit one page per day per engine if you like so that the speed of the submission becomes a non-issue. In addition, WebPosition emulates a common browser so that your submissions look no different to the search engine than if you'd used Internet Explorer to submit.

As a rule you will only want to resubmit your Web site each time you update or change it and only if you are not ranking well. The search engines will not be instantly aware of the changes you made to your Web site. Therefore, you will generally want to tell the engine about your changed pages by re-submitting them. You can go to each search engine and individually submit each of your pages, or you can use WebPosition Gold's Submitter to create Missions to do it automatically.

Unfortunately, there are pitfalls to submitting since each engine has their own set of rules and submission limits. Unlike many other submission products and services, WebPosition will try to warn you about known limits so you can avoid being penalized or having your submission ignored. If you try to submit too many pages at once, WebPosition will prompt you and suggest submitting only some of the URLs for that day. Surprisingly, the engines penalize for exceeding these limits but rarely ever warn the Webmaster if they exceed them.

Also, resubmitting before the engine has had time to index your page is not necessary. After submitting, it will take anywhere from a couple of days to months for your submission to be integrated into their database (i.e., indexed), and for your ranking to change. The amount of time varies for each search engine. The WebPosition submission report generated after you submit will give you estimated times for each engine so you know what to expect.

Regarding directories such as Yahoo, you will never need to resubmit unless the content of your Web site changes significantly or the description they chose for your listing is not accurate. The method to update or change your listing on a directory will vary. See the Directory Submission Guide of WebPosition's Submitter for details.

A common myth on the Web is that submitting to thousands of "engines" will gain you all kinds of new traffic. In reality, there are not even thousands of "engines" to submit to. The bulk of those submissions go to what are called FFA sites that are not search engines at all. Instead, they are simply a list of links to the last fifty or so URLs that were submitted to the service. Nobody ever uses these lists to find anything because they are just a random list of Web sites, generally with no organization to the list. The primary purpose of these sites is to gather e-mail addresses from Web marketers so they can be sent offers for other products and services. By submitting, you are unknowingly giving your consent for them to e-mail you.

So why do all those submitters on the market submit to these FFA and other "junk" sites? Because the general public perceives that "more is better." Therefore, many of the competitors to WebPosition spend the bulk of their resources trying to add more junk sites to their list so they can say they submit to more than the next guy. However, 95% of the traffic on the Web is going to come to you from the top 15 to 20 engines and that's only if you rank near the top. In addition, if you don't take the time to be positioned near the top of the search results for the sites you do submit to, then the odds of anyone finding you're site is extremely remote. Helping you achieve top rankings on the major engines is exactly what WebPosition Gold is designed to do, and does it very well.

We've tested bulk submitters in the past against two different "fresh" Web sites each getting one or two random visitors a day. Both sites continued to get one or two visitors a day after bulk submitting every month for three months to about two thousand sites. This, along with feedback we've received from other Web marketers proves that submitting to thousands of sites is not a traffic generator.

The other argument for bulk submission tools is that it will increase your link popularity. The more sites that link to you, the higher rankings you can achieve. Link popularity does influence rankings. However, the problem is that the search engines do not normally spider the databases of these obscure search engines, so being in those indexes does not improve your link popularity.

In addition, having your site listed on an FFA page does not normally improve your link popularity. That's because most FFA pages only maintain the last 50 to 100 submissions. Older submissions get bumped off the list as new ones come in. Therefore, you have to hope that a major search engine will index that FFA page before you get bumped off the list.

Since thousands of people normally use the same bulk submitter you do, you'll often get bumped off an FFA page within hours of your submission. This doesn't give the engine much of a chance to find your FFA link, particularly when they often take weeks just to index a page after its been submitted. This assumes that someone takes the time to submit each FFA page to the major search engines and the engine doesn't decide the page is spam when it arrives.

Admittedly, you can get lucky sometimes and temporarily increase your popularity by a handful of links if you submit to enough FFA sites often enough. However, based on our tests, the number of links achieved often does not exceed a dozen links at best. The bigger problem is that those links will not influence your rankings nearly as heavily as having a link to your site from a highly popular site like Yahoo or Open Directory, or from Web sites with related content to yours.

In conclusion, your position on the major engines is what matters, not how many submissions you make. Focusing your efforts on optimizing your pages for the major engines will reap far more rewards that bulk submission.

TIP: If you do choose to try a bulk submitter to see the results for yourself, do NOT use your real e-mail address! Otherwise you'll receive dozens, if not hundreds of e-mails from the sites you submitted to within 24-48 hours. Someone in our office made that mistake once and is still paying the price from being on every spam list under the sun.

Go.com Re-organized / Infoseek Bites the Bullet

Well, it's official. The Infoseek search engine owned by Disney and hosted at Go.com is no more. After continued financial losses and according to many analysts, some mismanagement by Disney, Go.com has decided to reorganize and down-size. The new purpose of Go.com is to provide a doorway to other Disney properties such as ESPN, ABC, and others, along with their affiliates under one umbrella. They wish to leverage the assets they have while spending as little money as possible on new technology for Go.com.

The Go.com Web site continues to exist, but not as a distinct search engine. Instead, the Infoseek search engine has been replaced with mirrored results from Goto.com, a completely separate company. Unlike most other Goto affiliates who display only the top three Goto listings as "sponsored sites," Go redirects all their search traffic to Goto.com. I find this new relationship rather ironic since it was only last year when Goto was suing Go for trademark infringement regarding its name and traffic light logo.

Contact Us at IQBS

Job Listings | Benefits | Submit Resume