Moz Q&A is closed.
After more than 13 years, and tens of thousands of questions, Moz Q&A closed on 12th December 2024. Whilst we’re not completely removing the content - many posts will still be possible to view - we have locked both new posts and new replies. More details here.
Disallow: /404/ - Best Practice?
-
Hello Moz Community,
My developer has added this to my robots.txt file: Disallow: /404/
Is this considered good practice in the world of SEO? Would you do it with your clients?
I feel he has great development knowledge but isn't too well versed in SEO.
Thank you in advanced,
Nico.
-
Thank you Lesley.
This really helps a lot. I appreciate it very much. This is my site by the way: http://devilswink.com/
Thanks.
Nico.
-
This comes down to personal preference in my opinion. I think honestly it is neither here nor there. The instances that your 404 page could come up in the SERP's is more than likely pretty low and at the same time it really does not offer any useful content. So disallowing it would not really be any loss. One reason why it might be disallowed is that you have an e-commerce site that rotates products. When a product is deleted, the developer has a 301 to the 404 page, then with the robots.txt saying not to index the 404 page, the other page will drop out of search engines. If this is the case I would rethink that strategy. If you notice a lot of sites like amazon and other big sites leave the page in the index even if the product is no longer for sale. The thought is traffic is traffic, the hardest part in the whole equation is getting someone to your site, if the page is ranking, why delete it.
The only time I can think that I would specifically allow it and optimize it is if you have a cool 404 page. Some companies actually spend a bit of time on their pages and it gets them a little pop of viral traffic from social sharing sites like reddit. If you do have one that is funny or unique I would allow it and actually optimize it for a term like "funny 404 page" or something like that.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
What is the best way to deal with an event calendar
I have an event calendar that has multiple repeating items into the future. They are classes that typically all have the same titles but will occasionally have different information. I don't know what is the best way to deal with them and am open to suggestions. Currently Moz anayltics is showing multiple errors (duplicate page titles, descriptions and overly dynamic urls). I'm assuming that it's showing duplicate elements way into the future. I thought of having the calendar no followed at all but the content for the classes seems valuable. Thanks,
Technical SEO | | categorycode0 -
Best practices for controlling link juice with site structure
I'm trying to do my best to control the link juice from my home page to the most important category landing pages on my client's e-commerce site. I have a couple questions regarding how to NOT pass link juice to insignificant pages and how best to pass juice to my most important pages. INSIGNIFICANT PAGES: How do you tag links to not pass juice to unimportant pages. For example, my client has a "Contact" page off of there home page. Now we aren't trying to drive traffic to the contact page, so I'm worried about the link juice from the home page being passed to it. Would you tag the Contact link with a "no follow" tag, so it doesn't pass the juice, but then include it in a sitemap so it gets indexed? Are there best practices for this sort of stuff?
Technical SEO | | Santaur0 -
How to find and fix 404 and broken links?
Hi, My campaign is showing me many 404 problems and other tools are also showing me broken links, but the links they show me dose work and I cant seem to find the broken links or the cause of the 404. Can you help?
Technical SEO | | Joseph-Green-SEO0 -
Vanity / Short URLs 301?
Hi everyone, I'm working on a website that uses a lot of short urls eg http://www.forest.com/oaktrees. A quick check reveals these are currently 302 status. My question is should these be made 301s - a lot of them are in off-page content and looking at GA attract a lot of clicks. I've not managed to see a definitive answer to this after several Google searches. All help and advice greatly appreciated. Bw Jon
Technical SEO | | CoL-PR0 -
Cloaking? Best Practices Crawling Content Behind Login Box
Hi- I'm helping out a client, who publishes sale information (fashion sales etc.) In order for the client to view the sale details (date, percentage off etc.) they need to register for the site. If I allow google bot to crawl the content, (identify the user agent) but serve up a registration light box to anyone who isn't google would this be considered cloaking? Does anyone know what the best practice for this is? Any help would be greatly appreciated. Thank you, Nopadon
Technical SEO | | nopadon0 -
Video Sitemaps <video:content_loc>and<video:player_loc></video:player_loc></video:content_loc>
Hi guys, If I'm creating a video sitemap do I need to use both: video:content_locandvideo:player_loc</video:player_loc></video:content_loc> Or could I just use video:content_loc?</video:content_loc> Thanks
Technical SEO | | Tug-Agency0 -
How do you disallow HTTPS?
I currently have a site (startuploans.org) that runs everything as http, recently we decided to start an online application to process loan apps. Now, for one certain section we configured ssl to work (https://www.startuploans.org/secure/). If I go to the HTTPS url for any of my other pages they show up...I was going to just 301 everything from https but because it is in a subdirectiory I can't... Also, canonical URL's won't work either because it's a totally different system and the pages are generated in an odd manor. It's really just 1 page that needs to be disallowed.. Is there any way to disallow all HTTPS requests from robots.txt while keeping all the HTTP requests working as normal?
Technical SEO | | WebsiteConsultants0 -
How do I redirect index.html to the root / ?
The site I've inherited had operated on index.html at one point, and now uses index.php for the home page, which goes to the / page. The index.html was lost in migrating server hosts. How do I redirect the index.html to the / page? I've tried different options that keep giving ending up with the same 404 error. I tried a redirect from index.html to index.php which ended in an infinite loop. Because the index.html no longer exists in the root, should I created it and then add a redirect to it? Can I avoid this by editing the .htaccess? Any help is appreciated, thanks in advance!
Technical SEO | | NetPicks0