Search engines (PS) are already decent time is a mandatory part of the Internet. Today they are huge and most complicated mechanisms that are not only a tool for finding any necessary information, but also quite exciting areas for business.

Many search users have never thought about the principles of their work, about how to handle user requests, how the system data is built and functioning. This material will help people who are engaged in optimization and, understand the device and the basic functions of search engines.

Functions and concept PS

Search system - This is a hardware and software package that is intended to implement the Internet search function, and responding to a user query that is usually specified as a text phrase (or more accurate search query), issuing a reference list to information sources carried out by relevance. The most common and large search systems: Google, Bing, Yahoo, Baidu. In RuNet - Yandex, Mail.Ru, Rambler.

Consider in more detail the value of the search query itself, taking the Yandex system for example.

The request must be formulated by the user in full compliance with the subject of his search, as simple as possible and briefly. For example, we wish to find information in this search engine: "How to choose a car for yourself." To do it, open main page And enter a request to search "how to choose a car." Then our functions are reduced to enter the links to information sources provided on the network.

But even acting in this way, you can and not get the information you need. If we got similar negative result, you just need to reform your request, or in the search database there is really no useful information by this kind The query (such is quite possible at the specified "narrow" query parameters, like, for example, "how to choose a car in Anadyri").

The most basic task of each search engine is to deliver to people exactly the type of information they need. And to commit users to create "correct" type of requests for search engines, that is, phrases that will correspond to their principles of work, practically impossible.

That is why search engine development specialists make such principles and algorithms of their work that would give users to find their information. This means that the system should "think" just as a person thinks when finding the necessary information on the Internet.

When he enters his request to the search engine, he wishes to find what he needs, as easy as possible and faster. Having received the result, the user accounts for its assessment of the system, guided by several criteria. Did he get to find the necessary information? If not, how many times did it have to reformate the text of the query to find it? How relevant information was they obtained? How fast search system Processed its request? How comfortable were search results? Was the necessary result first, or was located on a 30th place? How many "garbage" (unnecessary information) was found along with useful information? Will there be a relevant information for it, when using PS, in a week, or in a month?

In order to get the right answers to such questions, the search developers are constantly improving the principles of ranking and its algorithms, add them new opportunities and functions and any means are trying to make the system work faster.

The main characteristics of search engines

Denote the main search characteristics:

Fullness.

Fullness is one of the most important search characteristics, it is the ratio of the numbers found on request of information documents to their total number on the Internet relating to this request. For example, there are 100 pages with the phrase "how to choose a car", and in the same query there were only 60 out of total, then in this case the search full of 0.6 is 0.6. It is clear that the larger the search itself, the greater the likelihood that the user will find exactly the document that he needs, of course, if he exists at all.

Accuracy.

Another basic function of the search engine is accuracy. It determines the degree of compliance with the user's request for the found pages on the network. For example, if the key phrase "how to choose a car" there is a hundred documents, in half of them contain this phrase, and in the rest you simply have such words (how to choose a car radio, and install it in the car "), then search accuracy equal to 50/100 \u003d 0.5.

Than the search is more accurate, the sooner the user will find the information you need, the less diverse "garbage" will meet among the results, the less documents found not to correspond to the meaning of the request.

Relevance.

This is a significant component of the search, which characterizes the time passing from the moment of publishing information on the Internet until it is published in the search engine index base.

For example, the next day after the information about the release of the new iPad, many users turned to the search with the corresponding types of requests. In most cases, information about this news is already available in the search, although very little has passed since its appearance. This is due to the presence of "fast base" from large search engines, which is updated several times per day.

Search speed.

This function as the search speed is closely related to the so-called "stability of loads". Every second to search great amount People, such a workload require a significant reduction in time to process one request. Here, the interests of both the search engine and the user coincide: the visitor wants to get results as quickly as possible, and the search engine must work it up as quickly as possible, so as not to slow down the processing of subsequent requests.

Visuality.

A visual representation of the results is an essential element of search convenience. According to a multitude of requests, the search engine finds thousands, and in some cases millions of different documents. Due to the fuzziness of the compilation of key phrases for searching or not accuracy, even the very first results of the query do not always have only need information.

This means that a person often has its own search among the results provided. A variety of components of PS issuing pages help to navigate in search results.

The history of the development of search engines

When the Internet began to develop, the number of its regular users was small, and the amount of information for access was relatively small. Mostly access to this network had only specialists of research areas. At that time, the task of finding information was not so relevant as now.

One of the very first methods of organizing wide access to information resources was the creation of site catalogs, and the references to them began to be grouping on topics. So the first project was the resource Yahoo.com, which opened in the spring of 1994. Subsequently, when the number of sites in the Yahoo directory increased significantly, the option of finding the necessary information on the catalog was added. It has not yet been fully a search engine, since the area of \u200b\u200bsuch a search was limited only by sites included in this directory, and not absolutely all resources on the Internet. Link catalogs were quite widely used earlier, but at present, almost fully lost their popularity.

After all, even today, huge catalogs have information about slightly part of the websites on the Internet. The most famous and large catalog in the world has information about five million sites when Google base contains information about more than 25 billion pages.

The very first real search engine was Webcrawler, which arose in the 1994th year.

Next year, Altavista and Lycos appeared. Moreover, the first was the leader in the search for information a long time.

In 1997, Sergey Brin, together with Larry Page created the search Google car as research project In Standford University. Today, Google, the most popular and popular search engine in the world.

In September 1997, the Yandex PS was announced (officially), which currently He is the most popular search system in RuNet.

According to N. September 2015.The share of search engines in the world is distributed as follows:

• Google - 69.24%;
• Bing - 12.26%;
• Yahoo! - 9.19%;
• Baidu - 6.48%;
• AOL - 1.11%;
• ASK - 0.23%;
• Excite - 0.00%

According to N. december 2016., Signs of search engines in RuNet:

• Yandex - 48.40%
• Google - 45.10%
• Search.mail.ru - 5.70%
• Rambler - 0.40%
• Bing - 0.30%
• Yahoo - 0.10%

Principles of search engine

In Russia the main system The search is Yandex, then Google, and then [email protected]. All large search systems have their own structure, which is very different from others. But still you can select the main elements common to all search engines.

Indexing module.

This component consists of three robot programs:

Spider (in English. Spider) - a program that is intended to download web pages. "Spider" downloads a certain page, at the same time removing all links from it. HTML code is downloaded with almost every page. For this, robots use HTTP protocols.

"Spider" functions as follows. The robot transmits a request to the "Get / Path / Document" server and other HTTP query commands. In response, the robot program receives a flow of text that contains the information of the service type and, of course, the document itself.

• URL of the downloaded page;
• date when the page is downloaded;
• hTTP response server header;
• hTML code, "body" pages.

Crawler. ("Traveling" Spider). This program automatically enters all links that are found on the page, and also highlights them. His task is to decide where in the future the spider should be found, based on these links or based on the specified list of addresses.

Indexer. (Robot indexer) is a program that analyzes the pages that spiders downloaded.

The indexer fully disassembles the page to composite elements and analyzes their analysis, applying its morphological and lexical types of algorithms.

The analysis is carried out over various parts of the page, such as headlines, text, links, stylistic and structural features, HTML tags, etc.

Thus, the indexing module makes it possible to undergo the references of the specified number of resources, download pages, extract the reference mass to new pages from the received documents and make their detailed analysis.

Database

Database (or the search engine index) is a set of data storage, an array of information in which the adjusted parameters of each processed indexing and downloaded document are stored.

Search server.

It is very important element The entire system, because from the algorithms underlying its functionality, the speed is directly dependent and, of course, the search quality.

The search server works as follows:

• The request that comes from the user is subjected to morphological analysis. The information environment of any document available in the database is generated (it will continue to display as a snippet, i.e. the text information field of the corresponding request).
• The data obtained transmits as the input parameters with a specialized ranking module. They are processed in all documents, and in the end, for each such document, its rating is calculated, which characterizes the relevance of such a document as a user's request, and other components.
• Depending on the conditions of the user specified by the user, this rating may well be adjusted by additional.
• It is then generated by SNIPPET himself, i.e. For any document found from the corresponding table, the title, annotation that most meets the query is retrieved, and the link to this document, while the found wordforms and words are highlighted.
• The results of the search received are transmitted to the person in the form of a page that search results (SERP) are issued.

All these elements are closely related to each other and function, interacting, forming a distinct, but rather difficult mechanism for the functioning of the PS, which requires enormous costs of resources.

Search engines are considered the most popular sites on the Internet. It is not surprising, because to find something on the network, you first have to use the search services.


There are quite a lot of them and not even experienced Internet users, they know at least 2-3 search engines.

What are the search engines? Someone knows about one search engine, some know about 10 search services, but in fact they are much more.

Of course, their popularity is different, as well as the interface, and the quality of the results. We will present you a list of different search engines so that you can compare them.

Popular search engines

Developed search services for various categories of people. The most famous covers countries or even the whole world, and less well-known are designed for regions or user interests.

Below is 10 search engines that you must know:

1. - Everything is known about this search engine, it is the most popular in Russia and provides a wide functionality to its users (starting with electronic payments, ending with a webmaster panel).
2. - Leader among all sites in the world. They enjoy people from all countries and consider the highest quality search engine. As with Yandex, clients offer many additional tools.
3. - Search and information system, taking into account the Russian-speaking morphology. The site has several separate versions, for example, Xrambler, through which you can search immediately in several services.
4. - With abroad, this system has tremendous popularity. She has long been translated into Russian, but he never received great fame in RuNet. The users are available to different search (pictures, videos, etc.).
5. - a company known in RuNet, is developing in several directions at once. In addition to its own search engine and the most popular mail service, they are owners of such projects as classmates and VKontakte.
6. - created as a state resource in Russia, but did not receive mass distribution. You can carry out different search formats through it, as well as get useful information.
7. - Designed to search for information in the Chinese Internet segment. Who knows, perhaps and it will be useful for you. You can receive news, search for pictures, music, get cards and much more.
8. - Project all famous company Microsoft. By volume of traffic, this site ranks second in the world search engines ranking. He has been working since 1998 and during this time it was upgraded many times.
9. - Only foreigners enjoy this service. Project American and in addition to the search engine, the company has a mass of other sites and services. It is worth noting that they have the largest directory of references to the websites of the world cobweb.

If one of the search engines fail to find information, then why not look for it through another site? Now you have several quality resources, and you will definitely find what you need.

total

Introduction

Few people can now present the Internet without searching, searching for issuing and organizing all these information search engines (IPS). But even recently, all information of the Internet has fun in several directories, the name is still on the hearing (Dmoz, Yahoo).

Today, the amount of information on the Internet is so huge that it is not possible to fit it into any directories. For processing, storing information, search organizations are created and continue to be created, powerful software products that we call search engines (PS). Each search engine (search engine) has its own databases, their own processing, search, ranking, and information issuing algorithms.

Internet search systems is

You can give the following academic definition to search engines. Search engine is a set of programs and technical means for organizing on the Internet of a user search, in which on the text request, the user receives a list of relevant (relevant query) results.

The issuance is made in the form of a list of references to the source of information with a brief description (preview) Sometimes with the photo.

For the first example, let us remember the world leader of the search "Google" and the leader of Runet Search Engine "Yandex". In addition to these search engines, you can call another dozen existing search engines, about which we will talk to a little lower.

Opinion: Google Search Systems, Yandex and others are not generators (manufacturers) of content, but are aggregators (batteries) of content and in the main mass, someone else's content. It is worth remembering that the use of someone else's content to create their own traffic and its monetization can be characterized as "piracy", which, in practice, of course, does not occur.

Rating

• And Google share the first two places of leaders: about 49% and 45%.
• Third place: search Mail.ru is about 3%;
• The remaining search engines are floating below 1%.

I watch statistics on Google Analytics:

• yandex / Organic 40.26%
• google / Organic 38.93%
• mail.Ru / ORGANIC 0.60%
• rambler / Organic 0.52%
• bing / Organic 0.12%

Statistics inexorab: to the search for Yandex and are also addressed, and if you consider that 3% is a good result compared to 45%, then you can call the third in popularity.

In this regard, the reasoning about the popularity of search engines besides Yandex and Google, can be attributed to the drying, and special promotion of sites in other search engines (not Yandex and Google), which do not deserve attention.

How search engines work

The question is how search engines work, the same common as the question of the sky. If the sky is blue, then search engines collect information on the Internet, processed it, rank and go to the search query.

The theory of the Internet search is much more important and in the article should not be present. However, the highlights will be useful to us:

Internet search engines do not store documents, that is, they do not make downloading and unloading documents completely into their repository;

IPS use the Internet as a decentralized document repository. Search engines occasionally bypass the Internet, choose the necessary, according to their algorithms, information and partially placing it (information) into its database (Database). From here a few problems:

• IPS is used to issue not all Internet information, but only part;
• Internet information is often changing. About 1500 thousand pages per day are added, from here a possible "empty issuance";
• There is a large number of doubles (duplicate content). Unfortunately, I have no accurate duplicate data, and a 5% duplicated digit seems to be overestimated;
• A lot of advertising, which also bypass search engines;
• "Wandering" of search robots in the network repeatedly increases the load not resources (not refers to search engines);
• Most commercial sites (about 83%), and have a small informative value.

For these and some other reasons, the overwhelming majority of IPS Internet use keyword search scheme (Search Engines), and not a classic search for information classification scheme.

Search features by keywords

Despite the changing algorithms of search engines, the advertisement of which is trying to convince us that the cars become smarter and dirty, the search for search engines is a search by keywords.

I like this search scheme by keywords.

As you can see, the operation of the Internet search engine is based on the search for new documents (SPIDER + CRAWLER search robot), indexing detected documents (indexer) and executing a user query (Search Engine Results Engine). The names of search robots used for these purposes are listed in brackets.

As I said, most search engines do not copy the full texts of documents into their database. To search for a document indexing, its search image is created. To organize the software search, the indexing robot creates an image of the software document, the so-called derived method (DERIVED). That is, the title and set keywords.

However, it is fairly precisely possible that all IPS pay attention to the following:

• The presence of a key word in