Fixing “Duplicate Without User-Selected Canonical” in Google Search Console

Q: How do I fix mobile usability errors in Google Search Console?

Add the viewport meta tag <meta name="viewport" content="width=device-width, initial-scale=1"> to your HTML <head> . This stops mobile browsers from auto-scaling the page and resolves errors caused by oversized fonts and links spaced too closely. Confirm the fix with Google’s Mobile-Friendly Test.

Q: What is the difference between robots.txt and sitemap.xml?

robots.txt tells crawlers which paths they should not crawl (e.g. Disallow: /private/ ), while sitemap.xml lists the URLs you want crawled along with metadata such as lastmod , changefreq , and priority . The robots.txt file can reference the sitemap with a Sitemap: directive.

Today I received an email from Google Search Console flagging an issue with my blog pages: a “duplicate without user-selected canonical.” You know what? I did not have a duplicate webpage. My webpage was simply available over two protocols, HTTP and HTTPS, so Google saw it as a duplicate of itself. The problem was that I had not declared a canonical URL, so Google’s crawler had no way of knowing which version was the one to index.

A duplicate without user-selected canonical

As a result, my blog failed to index correctly because of that missing canonical definition. So what is canonical, and how do you optimise webpages so Google actually “sees” them? Getting noticed was never really my priority — my blog is still a work in progress, more scrapbook than portfolio — but I was intrigued to see what happened once I made it more search-engine friendly. Here is every step I took to optimise my blog, and the results after Google’s crawler came back through. Let’s go!

What is SEO (Search Engine Optimization)?

Search Engine Optimization (SEO) is the practice of formatting and structuring web pages so search engines index them correctly and rank them higher in search results. When you publish online, you generally want to be found through search, and Google remains the most prominent engine as of 2022, with its pros and cons discussed by Paul Gil in “The Best Search Engines of 2022.” In simple terms, we define SEO as:

Search Engine Optimization helps to index web pages with search engines and potentially rank higher in search results.

Essential SEO Steps to Index a New Blog

Why did I put “potentially” in my SEO definition? I think that not all SEO activities might lead to better ranking and I want to focus mainly on the essential SEO steps.

1. Fixing Indexing Issues With Google Search Console

To check whether your website is correctly indexed by Google, and to fix issues like small, hard-to-read text on mobile, head to Google Search Console and look at which pages are not indexed, and why.

Whoosh! I have 65 pages not indexed due to 6 reasons!

Page indexing in Google Search Console

Fixing Mobile Usability Errors With the Viewport Meta Tag

My website also had mobile usability problems. These typically come from font sizes that are too small, links spaced too close together, or a missing viewport declaration.

My CSS had oversized fonts, so I dug into it. Adding the viewport tag solved the issue — that problem comes from the automatic scaling some mobile browsers apply when no viewport is set, as pointed out in Tips for passing Google’s “Mobile Friendly” tests.

The viewport meta tag tells the mobile browser how to control the page’s dimensions and scaling. The fix is to add the following tag to your HTML header.

<meta name="viewport" content="width=device-width, initial-scale=1">

Mobile usability is passed

To confirm that your website is mobile-friendly, you can use a free tool provided by Google Search Console.

Canonical Tags: Resolving Duplicate Content

A canonical tag is an HTML <link rel="canonical"> element that tells search engines which URL is the authoritative version of a page when several URLs serve the same content. Canonicalisation issues happen when the crawler gets confused and treats some of your webpages as duplicates. As Jean Abela explains [2], you fix the issue by explicitly marking the page you want indexed — in WordPress, the Yoast SEO plugin handles this for you. Alternatively, you can edit the HTML headers directly to add a canonical tag. Simply, you add the canonical tag to the webpage header like this:

<link rel="canonical" href="https://example.com/page" />

After updating your HTML with canonical tags, tell Google’s crawler you have fixed the issue using Search Console [3], and request the page be re-indexed.

Pending validation of my fixed canonical issue

Pages With Redirect: Excluding Non-Indexable URLs

Since not all pages need to be indexed, I hand-pick only the required pages. For instance, I might not index some pages with redirects, such as my contact page.

2. Optimising Title, Description, Headers, and Links

Some say keyword optimisation matters less now that search engines have got smarter. I am not convinced either way — nobody has proved it conclusively. Paid tools like Ahrefs are still in demand, but free alternatives such as Ubersuggest are available too.

Ubersuggest is a fantastic starting point for SEO of small blogs like mine. What do I really like the most? It automatically checks missing tags, links, and pages with duplicate title tags. Site speed is not critical for my blog since it is mostly text in Markdown, but for more visual websites, Ubersuggest’s site-speed checks would matter more.

Ubersuggest tested my website

Another useful tool to analyse website performance is the PageSpeed Insights, which also provides information for desktop and mobile devices about accessibility, SEO and render-blocking resources.

PageSpeed Insights

3. Choosing the Best Keywords for SEO

To optimise webpages for SEO, we define the keywords tag in HTML. The keywords can be defined within the meta tag:

<meta name= "keywords" content= "Python code examples">

A properly optimised page has well-chosen keywords that tell search engines what your website is about and bring in visitors through related searches. According to Stephen Hockman [9], keywords should ideally be included in:

Surely, into the keywords and title meta tags of the page;
Ideally, into content headers, preferably H1 and H2;
Your page introduction;
Page body content;
The concluding section of your webpage;
Images, into their filenames, ALT tags, titles;
Classes for <a> and <div> HTML elements;
Titles defined for <p> HTML elements.

You can research keywords related to your webpage content with the help of tools like Google Keyword Planner, ahrefs.com, semrush.com (read their fantastic guide in [8]), Moz, etc., to find more popular keywords for your page. These tools can also help you understand the competition for the keywords and the search volume and trend. There is also a comparison of keyword research tools by Nathan Gotch in “We Tested 11 Keyword Research Services (Here’s the Top 3).”

In short, you need to know which keywords fit your website’s content and purpose — what is the topic, what are you trying to achieve, and who is your audience? The goal of keyword research is to find terms that bring good-quality traffic matching your target readers.

Should you need an automated tool for keyword research optimisation and other SEO tasks, outranking.io is an excellent SEO optimisation tool powered by AI to rank better in search results, improve organic reach, and avoid duplicate content penalties. As I have mentioned above, I am affiliated with and recommend outranking.io for “AI-assisted Workflow” that helps in SEO content writing.

You can also use Surfer SEO for comprehensive keyword research within your niche, providing high-potential keywords and content suggestions to enhance your Google ranking efforts. With Surfer AI, you can easily create SEO-optimised articles for Google ranking and revenue generation.

4. Building Quality Backlinks

The number of good-quality backlinks (links referring to your website) influences your ranking — the more, the better. According to [11], you should aim to get links that have:

Relevant content;
High authority such as well-known media or with .edu/.gov domains;
Good traffic;
Good backlinks as well;
Good outgoing links, which can be checked with Ahrefs Site Explorer;
Been indexed on Google, which can be checked with: “site:example.com.”

5. Creating Sitemap.xml and Robots.txt for Crawler Control

To control which web pages search engine crawlers index and which they do not, you create two simple files in the root directory of your website. Robots.txt is a plain-text file that tells search engine crawlers which pages or sections of a website should not be crawled. Sitemap.xml is an XML file that lists the URLs of a website, along with additional metadata about each URL (such as when it was last updated), to help search engines crawl the site more intelligently.

For instance, the contents of robots.txt may look something as follows:

User-agent: *
Crawl-delay: 10
Disallow: /private/
Sitemap: https://daehnhardt.com/sitemap.xml

The “Disallow: /private/” directive tells crawlers not to access any pages within the “private” directory. Crawl-delay is respected by some crawlers such as Bing and Yandex, but Googlebot ignores it entirely — Google has never supported this directive, and recommends setting your preferred crawl rate in Search Console instead, as documented in its robots.txt specification.

As we see, robots.txt can contain a reference to the sitemap.xml file.

<urlset xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" xsi:schemaLocation="http://www.sitemaps.org/schemas/sitemap/0.9 http://www.sitemaps.org/schemas/sitemap/0.9/sitemap.xsd">
<url>
<loc>https://daehnhardt.com/tag/ai/</loc>
<changefreq>monthly</changefreq>
<priority>1.0</priority>
</url>
<url>
<loc>https://daehnhardt.com/tag/blogging/</loc>
<changefreq>monthly</changefreq>
<priority>1.0</priority>
</url>
</urlset>

The sitemap.xml lists all the pages on a website, along with metadata about each page. The purpose of a sitemap is to help search engines and other crawlers understand the structure of the website and the relationships between its pages. Sitemaps can include information such as the date a page was last updated, the frequency with which it changes, and its relative importance within the website. Sitemaps can also include information about images, videos and other multimedia content on the website. This information helps search engines crawl a site more intelligently and index its content more effectively. Sitemaps are beneficial for websites with a large number of pages or for websites that are difficult for search engines to crawl.

Sitemaps can be submitted to search engines through Webmaster Tools/Search Console for faster discovery and indexing. If you are interested in how I have created my sitemap, check my XML file for this blog.

Here are some best practices for creating a sitemap:

Keep your sitemap up-to-date: Make sure to update your sitemap every time you make changes to your website, such as adding new pages or removing old ones.
Use the correct XML format: Follow the guidelines set by the Sitemap protocol, which is the standard format for sitemaps.
Use correct URL structure: Ensure all URLs in your sitemap are correct and follow a consistent structure. Avoid using URLs that redirect or have a “?” in them.
Limit the number of URLs: keep each sitemap to 50,000 URLs or fewer, and under 50MB uncompressed — that is the hard limit set by the sitemap protocol. If you have more URLs than that, create multiple sitemaps and submit them all.
Prioritize your pages: Use the tag to indicate the relative importance of different pages on your website.
Include the last modified date: Use the tag to indicate the last time a page was updated. This helps search engines determine which pages are more important to crawl.
Indicate the change frequency: Use the tag to indicate how often a page is likely to change. This helps search engines to determine how often they should crawl a page.

6. Publishing High-Quality Content

Indeed, these activities will only work when our website has interesting, valuable, engaging content.

Final thoughts

SEO is a maintenance job, not a one-time fix. Correct canonical tags, a current sitemap, and mobile-friendly pages are the baseline that keeps a site indexable — get those wrong and little else you do will matter much. I have laid out the steps that fixed my indexing issues, and I will update this post once I see how the changes affect my rankings. Thanks for reading — let me know in the comments if you hit an issue I have not covered.

Tree Tree Brothers, Autumn

I update this article periodically with new ideas, so click here and save this blog post to your favourite Pinterest board. Pinning it will ensure you can refer to this detailed article later.

SEO and Indexing FAQ

What does “Duplicate without user-selected canonical” mean in Google Search Console?

It means Google found two or more URLs serving the same content and could not determine which one to index because no canonical URL was declared. A common cause is the same page being reachable over both HTTP and HTTPS. Fix it by adding <link rel="canonical" href="https://example.com/page" /> to the page <head>, then request re-indexing in Search Console.

How do I fix mobile usability errors in Google Search Console?

Add the viewport meta tag <meta name="viewport" content="width=device-width, initial-scale=1"> to your HTML <head>. This stops mobile browsers from auto-scaling the page and resolves errors caused by oversized fonts and links spaced too closely. Confirm the fix with Google’s Mobile-Friendly Test.

What is the difference between robots.txt and sitemap.xml?

robots.txt tells crawlers which paths they should not crawl (e.g. Disallow: /private/), while sitemap.xml lists the URLs you want crawled along with metadata such as lastmod, changefreq, and priority. The robots.txt file can reference the sitemap with a Sitemap: directive.

Do meta keywords still help SEO?

Google ignores the meta keywords tag for ranking, so it has little direct SEO value. Keyword research still matters, but the keywords belong in the title tag, headings (H1/H2), introduction, body, and image ALT text rather than a <meta name="keywords"> tag.