Robot.txt and canonical URL

🙋Get Help
,
1

Hello,
my client has just put the site live here https://www.javierlorente.es/, and he says that the robot.txt are in noindex, nofollow (which is the case on the wp instance)
IDK how to fix this, or where to check if it’s right or not.
and also that the cannonical url is pointing to the vercel url, so I have changed the url in the settings, but IDK if this will change the cannonical.
…I’m really not an seo person, so I’m a bit lost.
or do I need to do something with the head tag plugins ?

2

If you go there https://www.javierlorente.es/holaluz-opiniones/ you will see it’s on noindex, nofollow
<meta data-rh="true" name="robots" content="noindex,nofollow"/>

I have unchecked the rule on the wp side, but this still like that and I have no idea why
this is the repo https://github.com/javierlorente/jl

4

Hi @alexaspalato

I don’t see any custom robots.txt in your repo
You can add your custom robots.txt to specify the pages you want the search engines crawlers to index

Here you have more info about it: https://docs.frontity.org/guides/seo#robots-txt

I’m not a SEO expert but I think having a custom robots.txt should fix most of these SEO problems
Can you check this?

6

I will ask the client to prepare the robot.txt file, as he is SEO
but how can I don’t understand how I can have this file to the settings…
it’s not explained, and it’s not in the example repo neither
robots: ‘url-of-the-file’ , or how ?

7

Hi @alexaspalato

In this SEO guide you can read

You can add the robots.txt file at the root of your theme (next to the frontity.settings.js file) and when you build and deploy your app for production, it will be automatically picked up by the Frontity server and served at https://your-site-url.xyx/robots.txt.

With the deploy to Vercel should be as easy as that (placing that document at the root of the project and deploy)

Here you have an example of a robots.txt in a Frontity project

Hope this helps

PS: We’ll take your feedback to improve the docs regarding this. Thanks for it

8

Thank you!
I have seen that, but doesn’t show how to put in the settings…

9

You don’t need to add anything to the settings, just place the file there and deploy.

10

Ah ok, I didn’t well, I understood that it was IN the settings, when it was NEXT to the settings !!!
Thanks!

11

I asked the client to send me what he wants in the robots.txt, but he says there is already a robots here https://www.javierlorente.es/robots.txt
but that there is this in the site

   <meta data-rh="true" name="robots" content="noindex,nofollow"/>

and that we need to put this

<meta data-rh="true" name="robots" content="index"/>

and IDK how or what is wrong
My clients says that it’s not the robots, so I need to know how to change this code

12

Let me check this with the Development team and I’ll get back to you ASAP

14

If you visit this call for example:
https://frontity.javierlorente.es/wp-json/wp/v2/posts?_embed=true&categories=6&page=1&per_page=50
You will see that the meta tag for robots is arriving in head_tags:

Screen Shot 2020-10-22 at 16.03.19
Screen Shot 2020-10-22 at 16.03.191076×368 19.9 KB

This means that the WP <head> is being populated with that tag and you should check the settings of WP or maybe some SEO plugin that is doing that.

1 Like
15

Thanks,
first in wp I checked the nofollow as it is what I do generally when I work headless
is it different in frontity ?
Now we have unchecked this.
will that need some time to be taken ?
we are not using any seo plugin

16

Once you remove these tags, the content from https://frontity.javierlorente.es/holaluz-opiniones/ might be indexed by google, so you need another solution different than the robots tag.

I read somebody mentioning this theme that might sort out that: https://wordpress.org/themes/nude/ but I never tried it myself, so not sure.

EDIT: this is the thread I couldn’t find yesterday talking about this issue: What Wordpress PHP Theme should I use with Frontity?

17

It’s finally fixed by unchecking the nofollow in the wordpress back office.
But I’m surprised it works like that, this setting should not be reflected in frontity, as the wp instance should hidden from google…
Perhaps it comes from the head tags plugin and should be fixed!

18

Get rid of the robots.txt , it’s essentially defunct in regards to Googles SEO nonsense.

As for the noindex, nofollow metadata, I’ll check it out with our test site. Interesting.

19

I just checked the website, maybe you already got a fix. Your robots.txt files is :
User-agent: *
Allow: /

After an SEO test your website seems open to google.