How to Detect (and Fix) Duplicate Content on Your Site

When a website’s keyword rankings drop, oftentimes duplicate content is to blame. All too often, websites unintentionally have duplicated pages appearing in the Google search index, and some estimate that up to 29% of the web is actually duplicate content. Duplicate content can cause a website to be filtered in the results, but can be easily identified.

While detecting duplicate content is very easy, fixing it is sometimes even easier. In this post, we’ll answer a few of the most common duplicate content questions:

• How to check and fix duplicate title tags
• How to check and fix duplicate meta descriptions
• How to check and fix duplicate content on your website
• How to check and fix duplicate content on the web 

Step 1: Check for and Fix Duplicate Meta Tags

Every URL of your website should have a unique title and a unique meta description. Your page titles shouldn’t be over 60 characters and your meta descriptions shouldn’t be over 140 characters. Figuring out how to fix duplicate meta descriptions and tags on your site may seem daunting but luckily there are tools in place to help you locate them.

There’s a handy tool called Screaming Frog SEO Spider that will help you detect duplicate page titles and duplicate meta descriptions. It’s free for up to 500 URLs and if your website has more than 500 pages, you can get the paid version. Run Screaming Frog on your site and then go to Page Titles and select Duplicate from the drop-down menu.

Once you know which pages have duplicate titles, all you have to do is fix them. Once you’re done with the page titles, do the same for the meta descriptions.

Step 2: Check for and Fix URL Canonicalization Issues

Google sees YourSite.com and www.YourSite.com as two different URLs. If you can access your website through both instead of having one redirect to the other, Google will see two URLs with the exact same content. 

The best practice is to have YourSite.com (without the “www”) 301 redirect to www.YourSite.com (with the “www”). If you want to see an example, type madfishdigital.com into your browser and you’ll see how you get redirected to www.madfishdigital.com.

You can set up this redirect from your .htaccess file. A word of caution, if you don’t know what you’re doing and mess up your .htaccess, your whole website could go down. Have an expert take care of this for you if you don’t feel confident you can do it yourself. If you want to give it a shot, here’s a great tutorial.

Step 3: Check for Internal Duplicate Content

There are various reasons websites have internal duplicate content. Some websites have print-friendly versions of their pages, or the same type of content in both HTML and PDF formats. Blogs are often notorious for duplicate content – this happens when identical content shows up on both the direct post URL and home page.

Many e-commerce sites have a problem with duplicate content as well. This is usually caused by multiple products using the same description and features on their pages.  Bad e-commerce site architecture can also cause users and web crawlers to find the exact same content through searches, category pages, tags, and the product pages themselves.

To detect internal duplicate content on your site, go to one of your pages and copy some of its content. 

Next you’ll need to do a Google search using advanced search operators. It’ll look like this: site:yoursite.com “text you copied”. Make sure to put the text you took from your website between quotes. 

If you have several pages on your website with the exact same content,there are three ways to fix this issue:

• If you have two versions of the same piece of content (such as web and PDF versions), just tell Google not to index the PDF version. Put all your PDFs in one folder and use the robots.txt file to tell Google not to index that folder. This is a good tutorial on how to create a robots.txt file.
• If there’s absolutely no reason to have two or more URLs with the same content, redirect all the duplicate URLs to the main one using 301 redirects. Here’s a tutorial on 301 redirects.
• If you have a legitimate reason to have multiple URLs with the same (or very similar) content, designate a canonical URL. This is a way of telling Google “I know I have multiple URLs with the same content. I have a reason for it. This is the URL I want you to count.” This is perfect for when pages A, B, and C are similar but you want to keep all three. Just tell Google that page A is your preferred choice and even though B and C won’t redirect to A, Google won’t think they’re duplicates. Here’s a tutorial from Google on setting up “rel=canonicals”.

Step 4: Check for External Duplicate Content

External duplicate content occurs when two or more domains have the same page copy. Sometimes tricky “scrapper” sites troll the Internet searching sites for fresh content. They swipe YOUR content and pass it off as their original content and your site gets banned because your site is recognized in search engines as duplicated content.

Another way that external duplicate content can occur is when e-commerce websites have products being sold across multiple sites and use the same page copy and descriptions. 

To find all the duplicate content that is not on your site, take that same search query you did in step 3 and add a dash at the beginning. It’ll look like this: -site:yoursite.com “text you copied”. 

If you run an e-commerce site, other companies may be using  the exact same description to list your products. How do you fix that? Ideally, you want every page of your site to have unique content, but if you have an e-commerce site with tens of thousands of products, this might not be possible. 

In cases like this you need to have one goal: to make your website the best place for people to buy the products you sell. There are a lot of ways to do this, but think about it: if 10 companies sell the exact same product, what reason can you give people to buy it from you? Here are some ideas:

• You can have a better-looking website that’s easy to use and provides a better overall shopping experience
• You can have more product reviews than your competitors, increasing the value of your site
• You can be the least expensive  option
• You can have a lot of great content, such as videos and infographics
• You can use social media to position your company as the go-to resource in your space —establishing thought leadership and a strong brand goes a long way

Step 5: Search for Almost-Duplicate Content

Let’s say you have an e-commerce website with 100 different products. And, let’s say you’ve written 20-word descriptions for each of your products. Here’s the problem: each of your product pages has 20 unique words and a lot of content that doesn’t change across your whole website (like navigation menus, headers, footers, etc.).

There are two things you can do about this:

• Reduce the amount of content that is common to all pages
• Add some unique content to each one of your pages. Write longer descriptions, explain how the products can be used, ask customers to leave reviews, tell stories, share testimonials and leave comments. Get creative. You’ll get a better response if you offer a free giveaway to people who add content to your site.

Step 6: Check for Duplicate Content Caused by CMS Issues

Most content management systems (CMS), such as WordPress, Drupal, or Joomla, have duplicate content issues out of the box. The good news is that it’s easy to fix them.

1. Make a list of the types of pages on your site, such as home page, categories, products, member profiles, etc.
2. Take one page of each type and copy some of the text on the page
3. Search for that text on Google (put it between quotes) and see if the text appears on more than one page. If it does, you have a duplicate content issue you need to fix.

In most cases, you want to fix the issues blocking folders using Robots.txt (tutorial here) or using the rel=canonical technique mentioned above.

Take Proactive Steps to Ensure Your Content Isn’t Duplicated

• Avoid empty web pages (placeholders) whenever possible
• Minimize similar content – merge similar pages or create unique content for each page
• Keep internal linking consistent
• Use only top-notch domains that are country-specific
• Carefully review webmaster guidelines (under Google Search Central)
• Understand how content is displayed (especially sites that show the same content on different pages such as forums and blogs)

Work With an SEO Agency

Fixing duplicate content issues will considerably improve your Google rankings and organic traffic. When you’re ready to have an SEO agency examine your online presence and improve your organic search traffic, we’ll be here.