Crawl solutions for landing pages that don't contain a robots.txt file?

[Nomader]

My site (www.nomader.com) is currently built on Instapage, which does not offer the ability to add a robots.txt file. I plan to migrate to a Shopify site in the coming months, but for now the Instapage site is my primary website. In the interim, would you suggest that I manually request a Google crawl through the search console tool? If so, how often? Any other suggestions for countering this Meta Noindex issue?

[Nomader]

No problem Tom. Thanks for the additional info — that is helpful to know.

[BlueprintMarketing]

Bryan,

I’m glad that you found what you where looking for.

I must have missed the part about it being 100% Instapage when you said CMS I thought  meant something on else with instapage I think of it as landing pages not a CMS

I want to help so you asked about Google search console how often you need to request  google index  your site.

First make sure

You should have 5 urls in Google search console

your domain, http://www. , http:// , https://www. & https://

• nomader.com
• https://www.nomader.com
• https://nomader.com
• http;//www.nomader.com
• http://nomader.com

you should not have to  requests google index once you’re pages are in googles index. There is no time line to make you need to requests  google index.

Use search consoles  index  system to see if you need to make a request  and look for notifications

Times you should request google crawl when adding new unlinked pages , when making big changes to your site , whatever adding pages  with out a xml sitemap or fixing problems  / testing.

I want to help so as you said you’re going to be using Shopify.

Just before you go live  running on Shopify in the future you should make a xml sitemap of the Instapage site

You can do it for free using https://www.screamingfrog.co.uk/seo-spider/

you’re running now name it  /sitemap_ip.xml or /sitemap2.xml  upload it to Shopify

& make sure it’s not the same name so it will work with your Shopify xml sitemap  /sitemap.xml

submit the /sitemap._ip.xml to search console then add the Shopify /sitemap.xml

You can run multiple xml sitemaps as long as they are not  overlapping

just remember never add  non-200 page, 404s, 300sno flow , no index or redirects  to a xml sitemap  ScreamingFrog  will ask if you want to when you’re making the sitemap.

Shopify will make its own xml sitemaps and and having the current site as a second xml sitemap will help to make sure your change to the site will not hurt the intipage par of  the Shopify site

https://support.google.com/webmasters/answer/34592?hl=en

know  adding a XML  Sitemap is a smart move

I hope that was of help I’m so about miss what you meant.

respectfully,

Tom

https://builtwith.com/relationships/nomader.com

https://builtwith.com/redirects/nomader.com

[Nomader]

Thanks so much for your thoughtful, detailed response. That answers my question.

[seoelevated]

Bryan,

If I understand your intent, you want your pages indexed. I see that your site has 5 pages indexed (/, /help, /influencers, /wholesale, /co-brand). And that you have some other pages (e.g. /donations), which are not indexed, but these have "noindex" tags explicitly in their HEAD sections.

Not having a robots.txt file is equal to having a robots.txt file with a directive to allow crawling of all pages. This is per http://www.robotstxt.org/orig.html, where they say "The presence of an empty "/robots.txt" file has no explicit associated semantics, it will be treated as if it was not present, i.e. all robots will consider themselves welcome."

So, if you have no robots.txt file, the search engine will feel free to crawl everything it discovers, and then whether or not it indexes those pages will be guided by presence or absence of NOINDEX tags in your HEAD sections. From a quick browse of your site and its indexed pages, this seems to be working properly.

Note that I'm referencing a distinction between "crawling" and "indexing".  The robots.txt file provides directives for crawling (i.e. access discovered pages, and discovering pages linked to those). Whereas the meta robots tags in the head provide directives for indexing (i.e. including the discovered pages in search index and displaying those as results to searchers). And in this context, absence of a robots.txt file simply allows the search engine to crawl all of your content, discover all linked pages, and then rely on meta robots directives in those pages for any guidance on whether or not to index those pages it finds.

As for a sitemap, while they are helpful for monitoring indexation, and also provide help to search engines to discover all desired pages, in your case it doesn't look especially necessary. Again, I only took a quick look, but it seems you have your key pages all linked from your home page, and you have meta directives in pages you wish to keep out of the index. And you have a very small number of pages. So, it looks like you are meeting your crawl and indexation desires.

[Nomader]

Hi Tom,

Unfortunately, Instapage is a proprietary CMS that does not currently support robots.txt or site maps. Instapage is primarily built for landing pages, and not actual websites so that's their reasoning for not adding SEO support for basics like robots.txt and site maps.

Thanks anyway for your help.

Best,

-Bryan

[BlueprintMarketing]

hi

so I see the problem now

https://www.nomader.com/robots.txt

Does not have a robots.txt file upload it to the root of your server or  specific place where Developer and/or CMS / Hosting company recommends  I could not figure out what to type of CMS you’re useing  if you’re using one

make a robots.txt file using

http://tools.seobook.com/robots-txt/generator/

https://www.internetmarketingninjas.com/seo-tools/robots-txt-generator/exportrobots.php

https://outdoorsrank.com/learn/seo/robotstxt

It will look like this below.

User-Agent: *
Disallow:

Sitemap: https://www.nomader.com/sitemap.xml

it looks like you’re using Java  for your website?

https://builtwith.com/detailed/nomader.com

I am guessing  you’re not using a subdomain to host the Landing Pages?

If you are using a subdomain you would have to create a robots.txt file for that but from everything I can see you’re using your regular domain. So you would simply create these files ( i’m in a car on a cell phone so I did quick to see  check if you have a  XML site map file but I do think you do

https://www.nomader.com/sitemap.xml

You can purchase a tool called Screaming Frog SEO spider  if your site is over 500 pages  you will need to pay for it it’s approximately $200 however you will be able to create a wonderful site map you can also create a XML site map by googling   xml sitemap  generators. However I would recommend Screaming Prod because you can separate the images and it’s a very good tool to have.

Because you will need to generate a new site map whenever you update your site or add Landing Pages it will be done using screaming frog and uploaded to the same place in the server. Unless you can create a dynamic sitemap  using whatever website of the  infrastructure structure using.

Here are the directions to add your site  Google Search Console /  Google Webmaster Tools

https://support.google.com/webmasters/answer/34592?hl=en

If you need any help with any of this please do not hesitate to ask I am more than happy to help  you can also generate a site map in the old version of Google Webmaster Tools / Google Search Console.

Hope this helps,

Tom

[Nomader]

Thanks for the reply Thomas. Where do you see that my site has the robots.txt file? As far as I can tell, it is missing. Instapage does not offer robots.txt as I mentioned in my post. Here's a community help page of theirs where this question was asked and answered: https://help.instapage.com/hc/en-us/community/posts/213622968-Sitemap-and-Robotx-txt

So in the absence of having a robots.txt file, I guess the only way to counter this is to manually request a fetch/index from Google console? How often do you recommend I do this?

[BlueprintMarketing]

You don’t need to worry about instapage & robot.txt your site has the robots.txt & instapage is not set to no index.

so yes use google search console to fetch / index the pages it’s very easy if you read the help information I posted below

https://help.instapage.com/hc/en-us#

hope that helps,

Tom

[BlueprintMarketing]

If you cannot turn off  “Meta Noindex“ you cannot fix it with robots.txt I  suggest you contact the developer of the  Instapage  landing pages app. If it’s locked to no index as you said that is the only of for countering a pre coded  by the company Meta Noindex issue?

I will look into this for you I bet that you can change it but not via robots.txt. I

will update it in the morning for you.

All the best,

Tom