Use a robots.txt file in the root

luisherranz · 15 January 2020 12:54

Description

Right now we are returning a default robots.txt file:

frontity/frontity/blob/dev/packages/core/src/server/index.tsx#L31-L36


// Serve static files.
app.use(async (ctx, next) => {
  const moduleStats = await getStats({ target: "module" });
  const es5Stats = await getStats({ target: "es5" });
  const stats = moduleStats || es5Stats;

Even though we want to provide a Frontity package to configure the robots.txt file using the Frontity settings in the future, it’d be great to be able to configure it using a file in your root directory as well, similar to what we are doing for the favicon.ico.

User Stories

As a Frontity project developer
I want to use a robots.txt file
so that I can get control of my robots easily

Possible solution

I guess we have to check if the file exists first. If it does return it and if it doesn’t, return the default string like we are doing right now.

christian.nascimento · 22 January 2020 21:00

Is not possible to use now.json routes to get this file? Right now I need to put some redirects in robots.txt

luisherranz · 7 May 2020 06:38

@koli14 has just made a pull request to implement this feature

Moving this to Currently in Development.

luisherranz · 7 May 2020 06:39

I’m sorry we didn’t answer you @christian.nascimento!

SantosGuillamot · 5 June 2020 09:18

This has been included in the latest release! Thanks @koli14 for implementing it

You can now add the robots.txt file at the root of your theme (next to the frontity.settings.js file) and when you build and deploy your app for production, it will be automatically picked up by the Frontity server and served at https://your-site-url.xyx/robots.txt.