A quick way to check the indexing of pages in Yandex and Google. How to find out how many pages are in the Yandex or Google index? How to check whether a page is indexed or not

We've released a new book, Social Media Content Marketing: How to Get Inside Your Followers' Heads and Make Them Fall in Love with Your Brand.

Site indexing is the process of searching, collecting, processing and adding information about a site to a search engine database.

More videos on our channel - learn internet marketing with SEMANTICA

Indexing a site means that a search engine robot visits the resource and its pages, studies the content and enters it into the database. Subsequently, this information is returned based on key queries. That is, network users enter a query into the search bar and receive a response in the form of a list of indexed pages.

In simple terms, it goes something like this: the entire Internet is a huge library. Any self-respecting library has a catalog that makes it easier to find the information you need. In the mid-90s of the last century, all indexing was reduced to such cataloging. found keywords on websites and formed a database from them.

Today, bots collect and analyze information based on several parameters (errors, uniqueness, usefulness, availability, etc.) before entering it into the search engine.

Search robot algorithms are constantly being updated and becoming more complex. Databases contain a huge amount of information, despite this, searching for the necessary information does not take much time. This is an example of high-quality indexing.

If the site has not been indexed, then the information may not reach users.

How Google and Yandex index sites

Yandex and Google are perhaps the most popular search engines in Russia. In order for search engines to index your site, you need to report it. You can do this in two ways:

Add a site for indexing using links on other resources on the Internet - this method is considered optimal, since the pages found this way are considered useful by the robot and are indexed faster, from 12 hours to two weeks.
Submit your site for indexing by filling out a special search engine form manually using the services of Yandex.Webmaster, Google Webmaster Tools, Bing Webmaster Tools, etc.

The second method is slower; the site is queued and indexed for two weeks or more.

On average, new sites and pages are indexed in 1–2 weeks.

It is believed that Google indexes sites faster. This happens because the Google search engine indexes all pages - both useful and unhelpful. However, only high-quality content gets ranked.

Yandex works slower, but indexes useful materials and immediately excludes all junk pages from the search.

Indexing a site works like this:

the search robot finds the portal and examines its contents;
the information received is entered into the database;
in about two weeks, material that has successfully passed indexing will appear in the search results upon request.

There are 3 ways to check the indexing of a site and its pages in Google and Yandex:

using tools for webmasters - google.com/webmasters or webmaster.yandex.ru;
by entering special commands into the search bar, the command for Yandex will look like this: host: site name + first-level domain; and for Google - site: site name + domain;
using special automatic services.

Checking indexing

This can be done using:

search engine operators - look in the help or;
special services, for example rds bar;

How to speed up site indexing

The speed at which new material appears in search results depends on how quickly the robots perform indexing, and the faster the target audience will come to the site.

To speed up indexing by search engines, you need to follow several recommendations.

Add a site to a search engine.
Regularly fill the project with unique and useful content.
Navigation around the site should be convenient, access to pages should be no longer than 3 clicks from the main page.
Place the resource on fast and reliable hosting.
Configure robots.txt correctly: eliminate unnecessary restrictions, block service pages from indexing.
Check for errors, number of keywords.
Make internal linking (links to other pages).
Post links to articles on social networks and social bookmarks.
Create a sitemap, or even two, one for visitors and one for robots.

How to block a site from indexing

Block a site from indexing - deny search robots access to the site, some of its pages, part of the text or image. This is usually done to hide sensitive information, technical pages, development-level sites, duplicate pages, etc. from public access.

You can do this in several ways:

Using robots.txt, you can prevent the indexing of a site or page. To do this, a text document is created at the root of the website, which sets out the rules for search engine robots. These rules consist of two parts: the first part (User-agent) indicates the recipient, and the second (Disallow) prohibits indexing of any object.
For example, prohibiting indexing of the entire site for all search bots looks like this:

User-agent: *

Disallow: /

Using the robots meta tag, which is considered the most correct way to block one page from indexing. Using the noindex and nofollow tags, you can prevent robots of any search engines from indexing a site, page or part of the text.

An entry to disable indexing of an entire document would look like this:

You can create a ban for a specific robot:

What does indexing affect during promotion?

Thanks to indexing, sites are included in the search engine. The more often the content is updated, the faster it happens, since bots come to the site more often. This results in a higher search ranking.

Indexing the site in search engines gives an influx of visitors and contributes to the development of the project.

In addition to content, robots evaluate traffic and visitor behavior. Based on these factors, they draw conclusions about the usefulness of the resource, visit the site more often, which raises it to a higher position in search results. Consequently, traffic increases again.

Indexation is an important process for promoting projects. For indexing to be successful, search robots must ensure that the information is useful.

The algorithms that search engines use are constantly changing and becoming more complex. The purpose of indexing is to enter information into the search engine database.

A search engine index is a database that stores information about web documents. The purpose of this “storage” is obvious - using the database, the search engine can quickly and accurately calculate the values of ranking factors, the numerical value of the URL relevance to the user’s request and generate search results.

For SEO specialists and website owners, this means one thing - how many pages are indexed, how many documents can be found in the search results for targeted queries, and how many URLs can attract traffic.

How to quickly check indexing in Yandex and Google?

There are a lot of methods and services for checking page indexing, and we have three tools that show the number of documents in the index by domain:

Just specify the domain in any of them. The tool will help you check your site for indexing and tell you how many documents are in the Google and Yandex database, taking into account subdomains or without them:

Search operators for checking indexing

You can find out the indexing of a site using search operators. For Yandex, these are the operators “site:”, “host:” and “url:”.

For Google - “site:”, “inurl:” and “info:”.

Both search engines provide approximate results, but you can use additional parameters to see how many pages were indexed in a week, for example:

In order to check the indexing of a specific page on a site, use the “url:” operator for Yandex and “info:” for Google.

Webmaster and Search Console

In Webmaster, go to the “indexing” tab - “pages in search”. Here you can find statistics on documents added and removed from the search. You can also view the history of changes for the entire site for a certain date interval:

You can check your site's indexing in Google using Search Console. In the new version of the console, go to the “status” tab - “indexing of sent URLs” and get a picture of changes, errors, excluded pages and a graph of impressions:

Possible indexing control

It is important to understand that crawling or crawling a site page does not mean it is included in the index. Sometimes it is necessary to prohibit indexing of pages that are not needed in the search, for example: URLs with technical GET parameters, service or user documents (shopping cart, personal account, etc.), documents with UTM tags. Therefore, it is advisable for every site to have:

Sitemap.xml is a file that helps search robots crawl the site, taking into account the specified priorities.

Robots.txt is a file that specifies crawl rules and parameters for all search engine robots or individually for each:

Using the Disallow command, you can prevent indexing of individual URLs or sections of the site.

Crawl delay — sets the minimum time for the robot to contact the server to avoid loads on the site.

Clean param - allows you to specify CGI parameters in URLs that are not important for indexing, that is, various user identifiers, sessions, and so on. Used only in Yandex.

Open pages should be in search, only then can you think about ranking. Don’t forget to check the site for indexing, track changes and successful positions in the search results!

Ask a question or leave a comment

It is very important that all pages of your site are indexed in search engines (Yandex, Google, etc.).

Firstly, if the page is not in the index, then people will not be able to find it and you wasted time (and possibly money) on its creation, content and design. Each page in the index is a source of visitors.
Secondly, if a page is not in the index, this may indicate technical problems on the site, such as duplicate content, site or hosting glitches.
Thirdly, this page may play a technical role, for example, participate in a linking scheme (or contain paid links for which you will not get paid if the page is not in the index).

Working with clients, I have repeatedly encountered the fact that due to problems with indexing there were bad positions. This is a technical problem that I usually correct in the first month of cooperation, due to which there is a noticeable increase in visitors and positions already from the 2nd month.

Below I will consider manual and automated ways to check the indexing of pages in Yandex and Google. I'll show you how to check site indexing in general and each page separately.

How to find out the number of pages on a site

This can be done in several ways:

Now that we know the actual number of pages, we need to check how many of them are indexed in Yandex and Google

We look at the indexing of the site as a whole

In this case we find out How many pages of the site are indexed in the search engine?. What does this give us? Knowing the actual number of pages on the site, we can compare whether it corresponds to the number of indexed pages. And if it matches, then everything is in order, and if not, then you need to figure out the problem and find out which pages are missing (or which pages have duplicates).

Site indexing in Yandex

Several ways.

As you can see, the data is slightly different. This is due to the fact that the url:your-site design shows not only pages, but other types of files (doc, xls, jpg, etc.). The webmaster shows exactly the number of pages.

Website indexing in Google

Here, similarly to Yandex, there are 2 ways:

Manually using the site:your site construct. The effect will be approximately the same as with Yandex.
Using Google Webmaster Tools https://www.google.com/webmasters/(analogous to Yandex.Webmaster)

automatic methods

What's next

Now that we know how many of the actual pages are indexed, there can be 3 situations:

The number of pages in search engines and on the website is the same. This is an ideal option, it means everything is fine with the site.
The number of indexed pages is less. This means there are problems with the site (the most popular problem is uninformative or non-unique content)
The number of indexed pages is greater. Most likely you have a problem with duplicating pages, i.e. one page can be accessed at several addresses. This is bad for promotion, because... The static weight of the page is blurred and in addition there are many pages with repeated content.

To further diagnose the site, we will need to find out which pages are accurately indexed and which are not included in the index.

How to check the indexing of one page

We may need this when we want to check a specific page on our site (for example, recently published) or a page on someone else's site (for example, where we bought a link and are waiting for it to be indexed)

How to check the indexing of all pages individually

In this case, we will check all pages of the site for indexing at once and as a result we will find out Which specific pages are not indexed in the search engine?.

Here we need to not only know the number of actual pages on the site, but also the list of addresses of these pages (their urls). This is probably the most difficult thing in this article. We seem to have received a list of pages when we generated the site map, but the addresses there are not contained in pure form and you need to be able to work with some kind of data processing program to extract them. Therefore, we will use another program.

How to get a list of all pages on a site

Before collecting links, you need to configure the Exclude Patterns parameter. This is done to exclude unnecessary links when collecting, for example, in my case, when collecting a lot of addresses like: https://site/prodvizhenie/kak-prodvigayut-sajjty.html? replytocom=324#respond, which point to a comment on the page. And I only need the page address. Therefore, I configured exclusion of addresses using the *replytocom* mask:

Next, we start collecting urls and when the program finishes collecting them, go to the Yahoo Map / Text tab and copy the addresses from there (The save button does not work, because we are using the free version of the program)

Now we have the addresses of all pages.

How to check page indexing automatically

Everything is simple here. After launching the program, add the list of urls of your site collected in the last step and add them to the list of source urls. The program allows you to check indexing in Yandex, Google and Rambler, select the search engine you need and run the check:

After you have received a list of pages that were not included in the index, you need to understand why this happened. If everything is in order with the page, then in order for it to be included in the index, you can purchase links to it or several retweets from upgraded accounts.

Conclusion

The ability to check the indexing of your website pages will allow you to work more productively with search engines, as well as identify existing problems with the site.

Pages indexed by search engines are very important, because essentially only by indexing the pages of a website or blog can Google understand what queries to display them for in search results. The more pages indexed by Google, the correspondingly more search traffic can be obtained for more queries.

How the number of indexed pages affects the site’s position in search results

Has anyone heard of the concept “Google dance”? This is the state to which your website or blog reaches if you develop it according to the rules, sharply increasing the amount of traffic from . All indexed pages rise in the rankings, since the weight of the site or blog has increased significantly. Therefore, the more pages a site or blog has in the Google index at that time, the more search traffic you can potentially receive. By the way, the number of indexed pages is one of the parameters that Google takes into account when determining.

How to check the number of pages indexed by Google

Go to the Google search engine, or if you use Google Chrome, then simply write in the address bar of your browser:

I entered the name of my blog, you enter the name of the site or blog that you want to check. Be sure to enter the address without “www.” at first.
Do not put a space after the colon!

Click “Search” and Google will display a list of pages in its index. Simple enough!

Now you can check cached pages of your blog or website in Google.

Cache is Google's temporary storage for website pages. The Google spider visits any website, depending on the update frequency from a minute to a month. When you visit a website, Google saves the site’s contents in its temporary storage until your next visit.

How to check the latest page cache in Google

Enter the following command back into the Google search bar:

I entered the address of the main page of my blog, you enter the name of the page that you need to check. Let me remind you again, do not put spaces after the colon. Click "Search". And you get the latest cached page in Google with the exact date and time when the Google spider visited the site.

That's all, I hope I have made it clear how to determine the number of indexed google pages. These are important points for novice bloggers and webmasters))) Remember that the number of indexed pages can affect the results. The more high-quality pages in the index, the better. If not, ask questions in the comments!

For a number of reasons, search engines do not index all pages of a site or, conversely, add unwanted ones to the index. As a result, it is almost impossible to find a site that has the same number of pages in Yandex and Google.

If the discrepancy does not exceed 10%, then not everyone pays attention to it. But this position is valid for media and information sites, when the loss of a small part of pages does not affect overall traffic. But for online stores and other commercial sites, the absence of product pages in the search (even one out of ten) is a loss of income.

Therefore, it is important to check the indexing of pages in Yandex and Google at least once a month, compare the results, identify which pages are missing in the search, and take action.

Problem with indexing monitoring

Viewing indexed pages is not difficult. This can be done by uploading reports in panels for webmasters:

(“Indexing” / “Pages in search” / “All pages” / “Download XLS / CSV table”);

Tool capabilities:

simultaneous checking of indexed pages in Yandex and Google (or in one PS);
the ability to check all site URLs at once by;
There is no limit on the number of URLs.

Peculiarities:

work “in the cloud” - no need to download and install software or plugins;
uploading reports in XLSX format;
notification by email about the end of data collection;
storage of reports for an unlimited time on the PromoPult server.