Duplicate Content for index.html

[superTallDan]

In the Crawl Diagnostics Summary, it says that I have two pages with duplicate content which are:

www.mywebsite.com/

www.mywebsite.com/index.html

I read in a Dream Weaver tutorial that you should name your home page "index.html" and then you can let www.mywebsite.com automatically direct the user to index.html.  Is this a bug in SEOMoz's crawler or is it a real problem with my site?

Thank you,

Dan

[SEMCLIX]

The code should definitely go into the websites root directory's .htaccess, however .htaccess can be weird, a few days ago I ran into a similar issue with a client's website, and I was able to remedy the issue with a variation of the code.

index Redirect RewriteCond %{THE_REQUEST} ^[A-Z]{3,9}\ /([^/]+/)index.(php|html|htm|asp)\ HTTP/ RewriteRule ^(([^/]+/))index.(php|html|htm|asp)$ http://yoursite.com/$1 [R=301,L]

If you give me the URL for the site I will take a look at it and let you know what would be feasible.

[KeriMorgret]

Hi Daniel, can you share with us the URL of your site? We can take a look at it and give you a more precise answer that way. Thanks!

[superTallDan]

I eventually figured out that your method was a 301 redirect and I definitely broke my site trying to use the code you posted. .. haha.  Its ok though.  I just removed the code and it went back to normal.  At first, I was editing the .htaccess file in the public_html folder which wasnt working.  Then I tried the root folder for the site (I created the .htaccess file since it did not exist.)  Neither of those worked.  (I am using Bluehost so I do not think that I have root access and I am not sure if it is a Linux server or not.)

If there is an easy way to explain what I am doing wrong, please do so.  Otherwise, I will use canonical.

Thanks for everything!

[SEMCLIX]

@Dan

Thanks for your reply.  It seems like there are lots of different ways to solve this problem.  I just watched this video on Matt Cutt's blog where he discusses his preference for 301 redirects over rel canonical tag.

Where would you say your solution fits in?

sorry about the delay of this response, i didn't realize the that you were asking me a question right away. When placing the code I provided in my previous answer this will cause a 301 perminant redirect to the original URL. That's actually what the

[R=301,L]

portion of the code is stating (R) redirect (301) status is referring to. After reviewing the Matt Cutts video, I realize that I should have asked you if you were operating on a Linux server  that you had root access to. We actually utilize both redirects and canonical tags since it was recommended by the on-page optimization reports. Heck Google uses them, I would assume because it's easier for the user to be referred to a single page URL. Obviously though if you don't have server header access, and are not familiar with .htaccess (you can accidentally break your site) then the canonical solution is appropriate

[superTallDan]

Josh,

Thanks for your reply.  It seems like there are lots of different ways to solve this problem.  I just watched this video on Matt Cutt's blog where he discusses his preference for 301 redirects over rel canonical tag.

Where would you say your solution fits in?

Thanks,
Dan

watch?v=zW5UL3lzBOA

[usaccess608]

use the link rel tag for all my homepages for the http://www.yoursite.com

[SEMCLIX]

Odd enough I just recently answered this question. The SEOmoz crawler is correct, because without a redirect you will be able to access both versions of the page in your browser.

www.yoursite.com/

www.yoursite.com/index.html

To resolve this issue simply rewrite the index.html to the root url by placing the following code into your .htaccess file into your root directory.

Options +FollowSymlinks                                                                                                            RewriteEngine on

Index Rewrite                                                                                                                                RewriteRule ^index.(htm|html|php) http://www.yoursite.com/ [R=301,L]                                      RewriteRule ^(.*)/index.(htm|html|php) http://www.yoursite.com/$1/ [R=301,L]

You can also do the same with the index file in any subdirectories that you might create, by simply placing a .htaccess into those sub directories and using variations of the above code. This is how you create nice tight URLs without the duplicate content issue that look like - http://www.semclix.com/design/business/

[RyanKent]

It is a problem which you need to fix. You need to canonicalize your pages.

www.mywebsite.com

www.mywebsite.com/

www.mywebsite.com/index.html

mywebsite.com/

mywebsite.com/index.html

Those are all various URLs which most likely lead to the same web page. I say "most likely" because these URLs can actually lead to different pages.

You need to tell crawlers and search engines how you organize your site. There are several ways to achieve canonicalization. The method I prefer is to add the following line of code to each page:

The URL provided should be the preferred URL for your page.