Moz Q&A is closed.
After more than 13 years, and tens of thousands of questions, Moz Q&A closed on 12th December 2024. Whilst we’re not completely removing the content - many posts will still be possible to view - we have locked both new posts and new replies. More details here.
How to prevent development website subdomain from being indexed?
-
Hello awesome MOZ Community!
Our development team uses a sub-domain "dev.example.com" for our SEO clients' websites. This allows changes to be made to the dev site (U/X changes, forms testing, etc.) for client approval and testing.
An embarrassing discovery was made. Naturally, when you run a "site:example.com" the "dev.example.com" is being indexed. We don't want our clients websites to get penalized or lose killer SERPs because of duplicate content.
The solution that is being implemented is to edit the robots.txt file and block the dev site from being indexed by search engines.
My questions is, does anyone in the MOZ Community disagree with this solution? Can you recommend another solution? Would you advise against using the sub-domain "dev." for live and ongoing development websites?
Thanks!
-
Hey there, in addition to Oleg's comment you can add an htpasswd file to your server to require a username and password to be entered before any users or robots are allowed to access your website.
You can find information on setting this up here.
-
Thanks for your prompt feedback Oleg!
Your proposed action is exactly what we're implementing.
Randy Holland, Sprout Digital
-
So....
- If the dev site has not been indexed yet, you can block crawlers via robots.txt
- If the dev site is already indexed and you want it removed, add meta NOINDEX tag to all pages allow the site to be crawled via robots.txt (reason: you want google to crawl and noticed the noindex tag on the pages so that they remove it from search results. if the site is indexed and you block crawler via robots.txt, google will keep the pages indexed but won't crawl them again). Once deindexed, you can block via robots.txt again
As long as its blocked (and you build that into your process), having the dev site on the same domain shouldn't be an issue. We have our own dev domain + server that autoblocks all pages from being indexed.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Does changing content and design of the website gonna affect my all the backlinks i have made till now
i have been working on my link profile for a month now, after learning about 5 step moz methodology i have decided that i would like to change all of the content of my site and taylor it to what my customers need, am i gonna loose all the domain authority if make changes? if it gonna affect, hows that gonna come out
Web Design | | calvinkj0 -
Multiple websites for different service areas/business functions?
I'm wondering what the implications are for having multiple domains for different service areas of a company? I realize having multiple domains for one company can be troublesome because of the possibility of duplicate content, keyword cannibalization, and linkbuilding to multiple domains. But when the domains are for very different service offerings/unique business functions that each serve their own purpose (and have different positionings), is there a downside to having more than one domain? Any thoughts would be appreciated!
Web Design | | KevinBloom0 -
Website organic traffic unchanged, impressions took a 98% drop in the last week.
Hi all, I have a very curious predicament and I'd be grateful if someone could shed some light on the situation. As mentioned in the title, organic traffic to our website has remained unchanged, but organic impressions have taken a 98% drop in the last week. This happened suddenly over one day; on October 22, impressions were 700, on October 23, they were 500, and on October 24 they drastically dropped to 50. The next two days they were at 22 and then up to 35. Organic traffic, however, showed the normal "weekend drop" as of October 24, and is still showing normal level (even increased a bit) continuing into this week. These are organic impressions according to Google Analytics and Google Webmaster tools. We did perform a complete site redesign a month ago. Could this be an effect from the redesign? We also noticed drop in Domain Authority, but our competitors suffered a similar (if not greater) drop as well, so we wondered if it could be due in part to the algorithm update. If anyone could shed some light on the situation I would be so appreciative! Thanks!
Web Design | | Joanne_Pendon0 -
Learning Center on Subdomain or New Domain
Hello, With so much emphasis for SEO to develop unique, information content I am working to build out a "Learning Center' for one of my clients in the finance industry. As you can guess, this is quite a competitive space and their rankings have become somewhat stagnant so we are looking for new ways to develop original, share-worthy content for the site. So my question is, in order for them to retain the most SEO value should we develop this on a subdomain such as learning.website.com or a new domain altogether. Note: We do not want to develop this under the current site domain as it we want o only post informational content & courses and keep these pieces outside of the "sales" side of their website. Therefore, I thought the subdomain would be the best bet so that we would retain the most value for SEO... however, some of my colleagues disagree. Some vote it should be within a directory under website.com. Some say a new site & domain altogether. My vote is to place this under a subdomain such as http://learning.website.com with the idea that the site's authority will influence the root http://website.com. The argument I was provided against the subdomain was: "essentially adding a subdomain would mean you're sort of starting over again in terms of building authority. adding a folder will ride the coattails off the already established authority." Lastly, for clarification, the current domain is set up as http://website.com so the subdomain would be http://learning.website.com, the directory would be set up as http://website.com/learning/ and a whole new domain would be http://newwebsite.com. This is for a Wordpress site. Thoughts? All feedback is much appreciated!
Web Design | | TinaMumm0 -
Website title next to a post title-how to remove it?
I just have checked on some of the keyword I am ranking for and found in the serp that next to the post I have also the site name. But I thought that I have remove it. Does somebody know how to remove it? perhaps I did not do it correctly. I am also using yoast seo plugin but I do not have it set there to show the site name after posts name. Can somebody help me to fix this please? I have also attached an image from the serp where is behind the post title also Villas Diani-the site name Thank you very much! Iris O1oj4W0.jpg
Web Design | | Rebeca10 -
Indexing Dynamic Pages
Hi, I am having an issues among others, regarding indexing dynamic pages. Our website, www.me-by-melia, was just put live and I am concerned the bottom naviagtion pages (http://www.me-by-melia.com/#store, http://www.me-by-melia.com/#facebook, etc) will not be indexed and create duplicate pages. Also, when you open these pages in a new tab, it takes you to homepage. The website was created in HTML5. Please advise.
Web Design | | Melia0 -
Recommended Website Monitoring Tools
Hi, I was wondering what people would recommend for website monitoring (IE is my website working as it should!). I need something that will:
Web Design | | James77
1/. Allow multiple page monitoring not just homepage
2/. Do header status checking
3/. Do page content checking (ie if the page changes massively, or include the word "error") then we have an issue!
4/. Multiple alert possibilities. We currently use www.websitepulse.com and it is a good service that does all the above, however it just seems so overly complex that its hard to understand what is going on, and its complex functionality and features are really a negative in our case. Thanks0 -
How do you visualize website structure
How do you visualize a website structure in terms of (categories of) pages and interlinking. I use such visuals for discussing what you are actually doing now and what can be improved. I have made visuals I few times myself (basically making boxes representing categories of pages and lines representing internal links), but I found that I soon ran into a scheme of huge proportions and needed more paper and more time. Appreciate your thoughts!
Web Design | | NewBuilder2