Get Google To Index URLs & Discover Penalties In The New Search Console

[et_pb_section fb_built=”1″ admin_label=”section” _builder_version=”3.0.47″][et_pb_row admin_label=”row” _builder_version=”3.0.48″ background_size=”initial” background_position=”top_left” background_repeat=”repeat”][et_pb_column type=”4_4″ _builder_version=”3.0.47″][et_pb_text admin_label=”Text” _builder_version=”3.21.4″ background_size=”initial” background_position=”top_left” background_repeat=”repeat” hover_enabled=”0″ header_font=”||||||||” header_2_font=”||||||||” header_2_font_size_last_edited=”on|phone” header_2_font_size=”28px” header_2_line_height=”1.1em”]

Get Google To Index URLs & Discover Penalties In The New Search Console

Google recently sunset the old version of their Search Console tools and migrated all webmasters over to the “new version” of Search Console. One of the features missing from the new version of Google’s webmaster tools is the “Fetch As Google” function underneath the Crawl tab. This was how webmasters could request that Google crawl and index new URLs on their site. However, websites can still do this in Search Console. It just works differently now.

Below, I’ll walk through how any webmaster can setup Search Console and request that Google crawl and index URLs on their website. I’ll also show you how to find penalties that Google might have levied against your website.

This can potentially help websites ensure their content ranks faster, but more importantly, it can help them ensure all their URLs are indexed in Google Search.

Watch a video of me briefly walking through this process

Setting up Search Console to crawl and index website URLs

Hopefully, you’ve already set up Google Search Console on your website and visit it regularly. It’s a great hub to check for helpful information about the way Google sees your website.

We wrote a good article on some of the helpful modules inside of Search Console here. The article talks about how to use this data to improve search results.

If you don’t have Search Console set up yet, you can click this link and follow the easy-to-use wizard.

It is very easy to verify your site with Search Console. The easiest way is by connecting it to your Google Analytics account, but if it’s not connected to your Gmail account, you can simply use the HTML tag option and throw a small line of code into the head of your website. Once the code is added, the site can be verified immediately.

Doing “Fetch as Google”  in New Search Console

One of the most important and commonly used features inside of the old Google Search Console was the ability to have Google test, render, crawl, and index any URL that you submitted through the tool.

Why this was great: You could have Google simply crawl a single URL or crawl a URL and all linked pages (often the majority of your site). If the URL wasn’t blocked (the tool would tell you if it was), Google would allow you to index the URL(s) quickly and they would typically appear in Search in approximately 24 hours. This allowed publishers to have new content indexed in Search immediately; while ensuring that their URLs were able to be indexed in the first place.

How you can still do this: You can see me walking through it in the video at the top; however, it’s actually really simple. At the top of the page inside of the new Google Search Console, there is a bar that says “Inspect any URL in https://yourwebsite.com”. Simply put any URL from your website into that tool and press “enter” (see GIF below).

If the URL is brand new, Google will crawl it and let you know if it can be indexed or not.

Since is it new, it will show up as not in Google’s index and you can then click on “Request Indexing”.

Google will then run a test to ensure that the URL can be indexed properly. Afterward, it will say “URL submitted to index”.

How long until Search Console tells Google to index the URL? While even Google won’t commit to a specific timeframe, in most cases URLs that are submitted to Search Console begin appearing in Google Search within 24 hours. Sometimes it can take slightly longer, but most of the time it takes less than 24 hours.

How long until Google crawls the URL if it isn’t submitted in Search Console? The fastest way to have new content rank — or even appear at all — inside of Google Search is to request Google to index it via Search Console. Without your request, it is possible that it takes days or even months before Google indexes a new URL. It depends on a lot of factors that Google does not readily share. That being said, many websites will notice that Google has indexed their new URLs within hours of publishing despite the fact that they haven’t submitted the URL in Search Console.

Will having Google crawl and index my page through Search Console help with rankings? No. Simply put, it will not help the content rank any higher for specific keywords simply because you submitted a URL to Search Console. However, this practice can help you ensure that the URL is not blocked and is able to be indexed by Google. It can also help it appear faster in Google Search than by waiting for Google to crawl it naturally.

Should I have Google crawl pages or articles when I update them?

Yes!

You should definitely have Google re-crawl or re-index a page or post after you update it.

It is really hard to say how often Google may be crawling different parts of your site. Adding your site-map to Search Console can actually help Google do this better automatically. This is actually something that Google recommends as a best practice for SEO.

If you actually look at your server weblog files, you could see this on your own; however, almost no one does this anymore. Unless you’re a total geek, it’s not really all that necessary either.

Outside, of giving Google the best understanding possible of how your site architecture and hierarchy are outlined in your site-map, anytime you make updates to part of your site you should have Google re-crawl the page and index it again.

Throwing in an updated page’s or article’s URL into the new Search Console Inspect URL tool will allow you to see all Google’s indexing stats on the page. Then, you can request that Google index the page again just like how you would with a new URL.

This will ensure that Google Search is indexing your pages or articles based on the most recent updates. This is important if you have recently added or improved the content on a page in an attempt to make it rank higher.

Is Google penalizing my site for any reason?

A common phrase I often hear from webmasters and publishers is … “Google has penalized our site because we have noticed a loss in rankings.”

First, it is always important to double-check an organic traffic drop for all the important signals and factors to better understand what you’re dealing with.

Second, if Google has actually penalized your site (ad-heavy, web spam, the site was hacked, etc.) they will let you know most of this inside of Search Console or Webmaster Tools.

One of the most common things I’ll hear from publishers is that they believe that too many ads are affecting their SEO. While ads can affect the visitor’s experience, and this DOES impact SEO if it is not managed properly, Google is actually really clear when they would penalize or rank a site lower because of being ad-heavy.

You can actually check this report on your site inside of Search Console by clicking this link.

Furthermore, manual penalties are easily shared with webmasters inside of Search Console. These could be the result of a manual reviewer finding something on your site that clearly violates webmaster guidelines. They are usually pretty clear about what it is they’d like to have you fix.

Lastly, if your website has been hacked or compromised, Search Console is often the first to warn you. If it is set up properly, you will get emails about potentially hazardous behavior occurring on your site.

Often, Hackers will proxy the website so that the website owner, and even the vast majority of the visitors, won’t see the hacked version of the site. This allows them to infiltrate the site and run malware to a small sub-section of visitors unnoticed. Google Search Console will show this to you inside the interface often before anyone else is able to notice.

But, I lost ranking and don’t see any penalties. What happened?

Rankings and organic traffic are definitely correlated, but they aren’t completely inter-connected. You can lose organic traffic without losing any rankings.

It might seem hard to believe but these things aren’t as easy as you might imagine. If you’re traffic drops, it can be a lot of different things causing the drop. What’s more, losing traffic could be just as much about the way Google Search is changing as it is about another site’s content ranking above yours.

Many times, we are seeing more and more sites that have ranked well for evergreen keywords (i.e. “King Henry The IV) see a drop in organic traffic without any loss in keyword rankings. With Google’s Knowledge Graph and other rich snippets being more prominently featured in more result, typical organic results are seeing a varied performance.

That’s why looking at things like impressions, average ranking position, and click-through-rate inside of Search Console can be helpful.

Ultimately, if you have in fact lost true keyword rankings, clicks from Google search (organic traffic), and are concerned about why Google is ranking your content lower than it used to it’s worth looking at three things.

What to do to improve rankings if you don’t have a penalty:

1. Schema mark-up so that you can appear in more rich results inside Google Search
2. Improve “search relevance” for your declining content
3. Explore existing and new keywords that are good bets for delivering more organic traffic

What should I do next inside the New Search Console?

For now, it’s good to check it regularly. Google has added a lot of helpful performance stats and coverage reports to help you understand how Google is viewing your site, how it is performing in Google Search, and if there are any errors or penalties that are worth fixing on your site.

By monitoring Search Console and regularly updating your site-map, submitting new URLs, and having Google re-index updated pages or articles you can ensure that your website is primed for success in Google Search.

While the world of SEO can be daunting and complex, Search Console is a source of truth directly from Google regarding your website. It is a tool that should be the first place you come to examine issues with your organic traffic.

Thoughts, questions, or comments? Leave them below.

[/et_pb_text][/et_pb_column][/et_pb_row][/et_pb_section]