This essay has been submitted by a student. This is not an example of the work written by our professional essay writers.
A web crawler is a simple program that scans through Internet pages to look for data. There are lots of different uses for a web crawler. The most common is associated with web crawlers are related to search engines. Search engines use web crawlers to gather information about what is available on the web pages. Their main purpose is to get data so that when Internet users enter a keyword or term in the search engine, they can easily provide the user with a long list of web sites.
A web crawler needs a starting point which would be a URL. The crawler then browses suing the URL and then seeks for hyperlinks and websites. Then the same crawler then browses those links.
When a search engine's web crawler visits a web site, it reads the text and the hyperlinks. Using the information gathered by the crawler, a search engine will then choose which each site is about using the information. The website is then included in the search engine's database and its page ranking process.
Search engines are not the only ways of using a web crawler. Researchers use web crawlers to assess different parts of a market. There are many uses of web crawlers as well. A web crawler can be used by anyone looking to get information using the internet.
Web crawling is an important way of collecting data and keeping up with the rapidly changing website Internet. A large number of web pages are always being added all the time and information about different topics are always changing. Web crawlers are a way for search engines to keep up to date.
A directory is a directory on the internet it links to other web sites and sorts them into categories
A web directory is not a search engine and doesn't display lists of web pages by keywords like a normal search engine like Google. It lists web sites by category. Web directory entries not known by web crawlers but by humans. The category is based on the web site instead of keywords.
Web directories also allow websites to directly submit their site to be in the search.
On the World Wide Web, a directory is a subject guide, typically organized by major topics and subtopics. The best-known directory is the one at Yahoo. Many other sites now use a Yahoo-like directory including major portal sites.
One major advantage of directory submission is that it is an easy and rapid way to create link dispersion on search engines without needing to spend too much. Other advantages of directory submissions are screenshots and deep links.
There are few disadvantages of directory; one major disadvantage is that directories are time consuming. When you add your site to directory, it is important for you to select the most appropriate category for your web site.
There are several advantages of directory submissions but it is a time consuming task. However, software may help to quicken the process, but you still need to supervise the process.
Overall, there are more advantages than disadvantages to directory submissions, and your web site will hugely gain from a well-studied plan of submission.
3. Meta search engines
A Meta search engine searches other search engines and then combines the results that are received. The user is not using just one search engine but a combination of many search engines at once to get the best results from web searching.
Meta search engines use several other web search engines and then combine the results in one list. Because they do not need to create their own search engine, there are lots Meta search engines.
No two Meta search engines are alike. A Meta search engine doesn't rank web sites. It searches lots of web sites altogether and will deliver a page of all the results of the best sites from each search engine. The better your rank on each search engine, the higher your site will be on a Meta search engine.
Searches many different search engines
Able to eliminate duplicate records, or pages on similar topics at the same address
They will sort the pages into topics, or will delete duplicate pages
Can sometimes return too much material.
Detailed searching techniques may not be possible.
Some don't sort the results very well, or will only show some of the search results.
The problems outweigh their benefits. All have restrictions as a search engine. They are subject too much information, when searching takes too long. Most only get the top 10 to 50 web sites from each search engine; the number of websites may be a lot less than found by doing a normal search on one of the search engines. Complex searches on individual search engines are not usually available. Often they exclude one of the major search engines such as Google or Ask.
4. Children's Search Engines
These are search engines designed for kids. They aim is to filter out sites that parents and teachers may find inappropriate for children and make searching on the internet a safer process.
For homework, music and messages, children seem to use the Internet more every day. But some sites with sexual content, pornographic pictures and videos are on the web there are huge numbers of pages that are inappropriate for a child. You can avoid these sites by using children's search engines. These search engines filter out inappropriate websites and only offer results that are deemed acceptable for children to view.
Unlike the main search engines, which have automated software to analyse websites, search engines for kids like Yahooligans, Ask for Kids, MSN for Kids or KidsClick. These all filter our inappropriate sites for children.
Some sites offer games and other entertainment, and many are designed to be visually appealing for children. Each one is slightly different in its approach.
Children are growing up with this technology are often much more familiar with it than their parents ever will be. The main goal is to stop them from accessing inappropriate websites for children, the major search engines all have filters you can turn on. This allows your child to learn how to use a search engine properly.
Another approach would be to have filtering software's that will not only stop search engines from taking children to inappropriate websites, but also stop them from using it by directly typing in a web address. This type of software has its downsides though; it may block sites your child needs to access for homework for example. No search engine or software can guarantee that your child will never see anything you disapprove of.