301 redirects stored in WordPress database

Description

There are some WordPress plugins that store the redirects in the database. As Frontity is accessing the REST API, it’s not aware of these redirects, so they could stop working if users are not using the Embedded mode. We should find a way to support it for all those users migrating to Frontity who want to keep the redirects.

Possible solution

We could add some logic to tiny-router to fetch the urls in parallel to getting the content from the REST API, and modify the Frontity side if the fetch return a 301 in the Header. This would mean that is doing two different requests for the same url. It could include a setting to specify this behaviour:

  • none: the default behaviour, where Frontity doesn’t do the new fetch.
  • 404: it just does the fetch if it’s a 404 error.
  • all: it does the fetch for all the urls.
  • regex: allow users to filter using regex.

Implementation proposal

After a lot of deliberation and trial and error here we are :slight_smile:

You can refer to the pull request which contains all the details.

1. router state

I propose the following state and the naming:

/**
 * How the router should handle 301 redirections that can be stored in the
 * WordPress database e.g. via a 
 * Redirection plugin: https://wordpress.org/plugins/redirection/.
 *
 * - "none" - Does not handle them at all.
 *
 * - "all"  - Always make an additional request to the WordPress instance
 * to check if there exists a redirection.
 *
 * - "error" - Only send the additional request to the WordPress instance
 * if the original request returned a 404. This is the default.
 *
 * - Regexp  - Make the additional request for links that match the given regex.
 *
 * @defaultValue "none"
 */
redirections: "none" | "all" | "error" | RegExp;

My only doubt here is if we should even provide the option to follow "all" redirections. I think most of the time the users should just use the "error" default and the "all" setting would not make sense for them.

I feel that the risk here is that users might assume that they should use the "all" setting when in reality, they will only need the "error" setting and their performance will suffer without them even noticing because they will make a useless fetch(link, { method: 'HEAD' }) for every link.

2. Modification to source.actions.fetch()

We’ll need something very similar to the following inside of the error handler:

if (e instanceof ServerError) {
  if (state.router.redirections === "error") {
    // TODO: handle errors
    const head = await fetch(state.source.url + link, {
      method: "HEAD",
    });

    if (head.redirected) {
      // TODO: We'll have to preserve the query string as well.
      // TODO: I guess we should also normalize the link with libraries.source.normalize(link);
      const newLink = new URL(head.url).pathname;

      Object.assign(source.data[link], {
        location: newLink,
        is301: true,
        isFetching: true,
        isReady: true,
        isRedirection: true,
      });
      return;
    }
  }
}

3. Should we implement state.source.url together with this feature ?

This feature would be quite a lot cleaner if we could already use the state.source.url.. Otherwise, we would have to parse the link ourselves to get the full backend url.

4. Router actions.

We ll need three modifications in the router actions:

4.1 actions.router.init():

This is needed because we want to “react” whenever the data for the redirection has been fetched on the client and call actions.router.set() with the correct URL again.

observe(() => {
    const data = state.source.get(state.router.link) as RedirectionData;
    if (data.isRedirection && state.frontity.platform === "client") {
      actions.router.set(data.location, { method: "replace" });
    }
  });

4.2 actions.router.beforeSSR()

Handle the redirection on the server side if the user is loading the redirected page directly.

const data = state.source.get(state.router.link) as RedirectionData & ErrorData;
if (data.isRedirection && state.router.redirections === "error") {
  // TODO: Handle all of the the Frontity Options, not just `frontity_name`
  ctx.redirect(
    data.location + "?" + `frontity_name=` + state.frontity.options.name
  );
} else if (data.isError) {

4.3 Change actions.router.set() to use the link from the redirection, if the data for the redirection has already been fetched:

const data = state.source.get(link) as RedirectionData;
if (data.isRedirection) {
  link = data.location;
}

Finally, it’s worth mentioning that the redirections for both RegeEx and all should work analogically, but are omitted here for clarity.

2 Likes

Great work Michal!

This is my feedback:

I think we should! :slightly_smiling_face:

1. Use "no" instead of "none"

To be consistent with the "no" we use for state.theme.autoPrefetch.

export type AutoPrefetch = "no" | "in-view" | "hover" | "all";

2. Default to "error" or "no"?

I’m not sure if the default should be error or no. My current thought is that the default should be no.

3. Use "404" for 404’s and "error" for errors

In Frontity 404’s are only a subset of errors. In order to be consistent, I would call this option "404" instead of "error". If we also think that a setting for all errors is worth (which I’m not sure right now) then I would call it "error".

  • "error": data.isError === true
  • "404": data.is404 === true

4. Make state.router.redirections an array of settings

I agree with you that RegExp and 404’s need to be used at the same time so we should allow for an array of settings:

  • state.router.redirections: "404"
  • state.router.redirections: "\\/some\\/url"
  • state.router.redirections: ["404", "\\/some\\/url"]

It would also be useful if people need to add more than one RegExp:

  • state.router.redirections: [ "\\/some\\/url", "\\/some\\/other\\/url"]

5. Use the same pattern/regexp strings as in handlers

We could also allow the same format of patterns and RegExps that we are using in handlers, because patterns are much simpler for certain URLs, although not as powerful as regular expressions.

  • "/some/custom/:urls" (pattern).
  • "RegExp:\\/some\\/custom\//(?<urls>[^/$?#]*)" (RegExp).

6. Don’t follow redirects

In order to avoid unnecessary bandwidth consumption, the fetch we do to check for a 301 should not follow redirections:

const head = await fetch(state.source.url + link, {
  method: "HEAD",
  redirect: "manual",
});

EDIT: Michal already tried this and it doesn’t work.

7. Support both 301 and 302

I would try to be consistent with how we define 400-500 errors:

const error = {
  isError: true,
  is500: true,
  errorStatus: 500,
  errorStatusText: "Some string describing the error",
};

And also add support for 302 redirections:

const redirection = {
  isRedirection: true,
  is302: true,
  location: newLink,
  redirectionStatus: 302,
};

Also location is the name of the 301 header, but maybe it would make more semantic sense to use a different field in Frontity, like newLink or redirectionLink?

There a couple of extra redirections that use 307 and 308, but to be honest I’ve never seen them used.

8. Bypass Server Side Rendering if there’s a redirection

For what I read in the Koa docs, it seems like ctx.redirect() only populates the status and location headers, but the app keeps executing:

ctx.status = 301;
ctx.redirect("/cart");
ctx.body = "Redirecting to shopping cart";

https://koajs.com/#response-redirect-url-alt

So I think the framework should be smart enough to bypass the React server-side render if the ctx.status is either 301 or 302 to avoid unnecessary work.


Finally, I’d like to propose another related feature that may be worth thinking about at this point: Frontity redirections (as opposed to “Backend Redirections”). I’ll do so in another post later in the day.

I don’t have time today. I’ll try to do it tomorrow.

Basically, I was thinking that maybe we should let room for configuring 301/302 redirections directly in Frontity, instead of in WordPress, and support both.

const state = {
  router: {
    redirections: {
      frontity: {
        "/some-post": "/other-post",
        "/some/custom/:url": "/other/custom/${params.url}",
        "/more/custom/:url": {
          destination: "/other/custom/${params.url}",
          status: 302,
        },
        "RegExp:\\/some-post?photo=(?<id>[^&$]+)>":
          "/some-post/photo-${params.id}",
      },
      source: ["404", "/redirections/stored/in/wordpress"],
    },
  },
};

Maybe not for now, but if we want to add this in the future, we have to take that into account so the API names make sense in combination of WordPress/backend redirections.

yup makes total sense :+1:

I’m generally in favour of this but maybe we could add that feature later on so as not to complicate the implementation too much initially? Since the RegExp is more powerful, there is only a little bit of ergonomics that is gained by allowing the user to specify the “pattern paths”. The downside is that it complicates the implementation because we’ll have to distinguish between strings like "no", "all" and "error" and strings that are patterns that should be matched.

I have already tried this and unfortunately redirect: manual does not work on the client because it will return an “opaque redirect” that does not contain the Location header: https://github.com/whatwg/fetch/issues/763

:+1:

:+1:

Oh, that’s a shame. Now that you say it, I think @david bumped into this one in the wp-comments research as well. Well, we are doing a HEAD request so at least we are not retrieving the body of the second request :slightly_smiling_face:

Not up to me to decide, but to @santosguillamot. But I agree with you that we can add support for both patterns and regexps at a later point.

yup, exactly it’s the same issue that David had :slight_smile:

great :slight_smile: @SantosGuillamot Let me know what your decision is. For reference, this is what is said about adding support for both string patterns like /path/:id and RegExps:

Sure, we can add it later. It seems everything will be doable with RegExps so I think it’s enough with this. We’ll see later if it’s something users are really demanding.

Good morning, practically all the websites in the world (especially the old ones) have redirects so in my opinion this is something very important. In my personal case, if I could configure them in Frontity it would not matter to me, but perhaps the most used plugin on this topic can be made compatible, https://es.wordpress.org/plugins/redirection/ in this way users like me who They are not developers, they would be “autonomous” when configuring new redirects. Theoretically, this plugin has a compatibility with REST API as we can see in the attached image of the plugin configuration. Thanks a lot

[image]

1 Like

Thanks for the feedback :slightly_smiling_face: As you mention there are two ways of adding redirects, from Frontity and WordPress. In order to do redirects from Frontity, we would like to a simpler solution but we will need the Server Extensibility first. If you control your Node server you should be able to do it as explained in this topic.

To create new redirects, users should always add them directly in the frontend (Frontity). But sometimes, in old websites as you say, there are a lot of redirects stored in the DB, and supporting that ones is the purpose of this feature.

I will take a look at the setting you mention to understand better how it works and check if it could be useful. Thanks!

What do you guys think of what I proposed in this message (301 redirects stored in WordPress database) to have both Frontity and WordPress redirections configured under the state.router.redirections setting?

I have I feel that it’s perhaps a bit too opinionated to have Frontity provide that option out of the box? I think that this makes the API more “ugly” and less intutitive and only to accomodate a feature that has not been requested yet by any user.

Handling the 301/302 redirections that are in stored in WordPress database is necessary, but adding redirections in the application is another thing :slight_smile:

  1. I think that for majority of applications, adding a 301 in the application would not be the right thing to do. Redirections in most cases should be done in your hosting (netlify.yaml, vercel.json, etc.) or maybe in nginx in your WP hosting or lastly in your WP database and I feel like putting redirections in your state is a weird pattern that I think we should not encourage.

  2. Once we ship server extensibility, if a user really needs redirections, they should be able to create their own solution because they will have access to the Koa ctx so could just call ctx.redirect(newUrl); return;

Of course, I’m happy to be proven wrong here :slight_smile:

Ok, I get your point Michal, thanks for sharing it :slightly_smiling_face:

Let’s create a draft of the situation to be able to make a decision.

The basic user story would be:

As a Frontity user
I need to redirect some old URLs to new ones
so that links pointing to an old URL are still valid

Old URLs can be present in:

  • 1. External sites
    This is usually the main problem, URLs that the user cannot change because they belong to external sites.
  • 2. Database content
    I guess this could also be categorized as solvable by the developers, although sometimes is easier to set up a redirection than to do a search-and-replace of your entire database. There are WordPress databases with +100.000 posts.
  • 3. Theme code (like in a menu, for example)
    This is usually not a problem because the developer can change that URL.

I am going to assume that 3 is not a problem we should solve, so let’s keep going for 1 and 2. The requirements for those would be:

  • 1. External sites
    Because these old URLs are only present on external sites, only the server needs to be aware of them. These are not needed in the client.
  • 2. Database content
    This is different because the old URLs present in the content can appear inside the client-side rendered app. So, if we want to provide redirections from old URLs present in the content, the client needs to be aware of those redirections as well.

Now let’s see the different options to set up those redirections in a Frontity+WordPress app:

  • Server Redirections
    As Michal mentioned, these can be set up in different places:
    • Hosting redirections
      Like Vercel or Netlify configuration files.
    • Proxy redirections
      Like in Apache or NGINX.
    • WordPress server
      Like the Yoast or Redirection plugins.
    • Frontity Koa server
      Using the upcoming Server Extensibility to manage the redirections from Koa’s server.
    • Frontity server does a request to the backend server
      We can do a request to the backend server to see if it returns a redirection, which would be set up in any of the above methods: Hosting, Proxy or WordPress.
  • Client Redirections
    This is where things get trickier, as we have seen. Let’s see what options we have:
    • Do a request to the server
      As we have seen, we can do a request to the server to see if it returns a redirection. This approach works with any of the methods mentioned in the previous section.
    • Retrieve a list of redirections from the server
      Another way would be to retrieve a list of redirections from the server. That would avoid the need to do an additional request for each client-side navigation.
    • Store the list of redirections in the client state
      The final option would be to store the list of redirections in the client state.

Now, let’s see the pros and cons of these methods:

  • Do a request to the server
    • Pros
      No extra configuration is required. It just works with whatever you have configured in your server.
    • Cons
      Doing an additional request for each client-side navigation just to see if there is a redirection is quite expensive. For that reason, I think this method works better if it is used only under certain conditions, like 404’s or other server errors.
  • Retrieve a list of redirections from the server
    • Pros
      Instead of doing a request for each client-side navigation, this method would do only one.
    • Cons
      We need extra code in the server, something that knows about all the redirections and exposes in an endpoint. That is not easy as there are many methods to create redirections in the server. For example, how do you access Apache or NGINX redirections?
      If the list is big, this download could be pretty expensive in the client.
  • Store the list of redirections in the client state
    • Pros
      No request is needed.
      As the information is already there in the state, it is cheap to consume. Therefore, there is no drawback in checking for all URLs, as opposed to restricting only for 404’s.
      This information, as it is store in Frontity, can be shared for both the client and the Frontity server.
    • Cons
      It doesn’t work with the redirections that are already set up in the server.
      If the list is big, we can increase the size of the bundle.

Ok, I hope that this makes things clearer to think about what cases we want to cover now, what cases we want to cover in the future (to think now about those APIs), and what cases we don’t want to cover.

@david, @mmczaplinski: I am probably missing things so feel free to add your own feedback and I will update the tables to reflect it :slightly_smiling_face:

@santosguillamot, please take a look. We need your decision here.

Great summary guys :slightly_smiling_face: I am writing down some ideas, although I am still not 100% convinced.

The way I see it (I don’t have too much information in this case), the common use cases of redirects would be:

  • Old slugs: I think it’s pretty common in WordPress to change the slug of your posts, and it is redirected to the new url. This is something WordPress does by default and it store the old slugs in the database.
  • Permalink changes: In the WordPress settings, it is also common to change the permalinks of your posts or categories. For example from categories to categoria.
  • Custom redirects: Other custom redirects that the users add with code (or plugins).

With the proposed solution, do a request to the server, these use cases could be easily covered. The old slugs would be covered with the 404 settings and the permalink changes and custom redirects should be fairly simple to replicate with regexp.

I feel this is great, but I assume users might be concerned by server load. They will probably want to migrate most of the redirects to the Frontity side to reduce this. And not only to migrate old redirects, but to create new ones. This is something users have already asked for.

Right now, I feel that is not simple to create redirects that work both in the server and the client in a Frontity app, so I think, in the future, we will need to work on a package, @frontity/redirects for example, that deals with both things. A place where you define your redirects, and they work in Client Side and Server Side Rendering. If possible, it should deal with the Koa’s server for the SSR, and with the routing for the CSR. For this to work I think we would need the Server Extensibility and the Frontity hooks.

So, the way I see it, the best possible workflow to migrate an old WordPress to Frontity would be:

  1. Use the “do a request to the server” method to ensure that everything works, from the beginning.
  2. Start migrating the redirects from WordPress to Frontity.
  3. Create new redirects directly from Frontity. Here my main concerns is the old slugs, as this is something WordPress does by default.
  4. Keep the do a request to the server for the edge cases we didn’t covered in the migration and probably return a 404 or other server errors.

For steps 2 and 3, a redirects package would be really useful.

Would it be useful to redirect the old slugs directly in the REST API?

Not sure if this could be useful somehow: As the old slugs are stored in WordPress database, it isn’t difficult to get them. I have found a PHP snippet that redirects the old-slug to the new-slug in the REST API. So, if I go to mysite.com/wp-json/wp/v2/posts?slug=old-slug it is redirected to mysite.com/wp-json/wp/v2/posts?slug=new-slug. I guess that, to supports this, some changes would be needed in Frontity. The PHP snippet I used was this:

add_action( 'rest_api_init', function() {
    add_filter( 'the_posts', function( $posts, $query ) {
        global $wp_query;
        // Only try to redirect if there were no posts found.
        if ( empty( $posts ) ) {
            // Set up the global wp_query so wp_old_slug_redirect() will think we're in a template & redirect accordingly.
            $wp_query = $query;
            $wp_query->set_404();
            $wp_query->set( 'name', $wp_query->get( 'post_name__in' )[0] );
            // Add filter to old_slug_redirect_post_id in order to prevent the default frontend redirect in favor of a REST API URL.
            add_filter( 'old_slug_redirect_post_id', function( $id ) use ( $wp_query ) {
                $post = get_post( $id );
                $redirect_url = $_SERVER['REQUEST_URI'];
                $redirect_url = str_replace( $wp_query->get( 'name' ), $post->post_name, $redirect_url );
                wp_redirect( $redirect_url, 301 );
                exit;
            } );
            // Finally, call wp_old_slug_redirect() and let our filter interrupt it to handle the final wp_redirect().
            wp_old_slug_redirect();
        }
        return $posts;
    }, 10, 2);
});

If this is possible, and the old slugs are redirected in the REST API, the list of redirects users have to define in the Frontity side would be reduced significantly, as the permalink changes and custom redirects should be easily covered by regexp. And we would still have the “do a request to the server” for the edge cases that are not covered with regexp or the old slugs.


I’ve been taking a quick look at some WordPress redirects solutions, to see if they can be migrated to the Frontity side easily:

I assume with most of the plugins it’s going to be the same. If not, users can also use tools like ScreamingFrog to crawl or the redirects -> https://www.screamingfrog.co.uk/redirect-checker/

Nice. Thanks Mario!

I have kept thinking about this as well and I realized that the current solution to know when there is a 301 redirection in the server is actually storing the information in Frontity as well, in state.source.data.

I think we should use the same system for all the 301 redirections, which means that the redirections created in Frontity also should end up being stored in state.source.data.

So if people want to add redirections stored in Frontity now, they can do it this way:

  • Single redirection populating state.source.data directly:

    const state = {
      source: {
        data: {
          "/old-url": {
            isReady: true,
            isRedirection: true,
            is301: true,
            redirectionStatus: 301,
            location: "/new-url",
          },
        },
      },
    };
    

    This manual population is something I have started recommending recently to create custom pages that are not present in WordPress: How to create Custom Pages? because it’s simpler than creating a code handler.

  • Pattern/RegExp redirection using a handler:

    const categoryRedirection = {
      pattern: "/category/:slug",
      priority: 5,
      func: ({ link, params }) => {
        state.source.data[link].isReady = true;
        state.source.data[link].isRedirection = true;
        state.source.data[link].is301 = true;
        state.source.data[link].redirectionStatus = 301;
        state.source.data[link].location = `/categoria/${params.slug}`;
      },
    };
    

That way all the redirections will be stored in the same place and when packages want to know if there is a redirection in a URL, they can simply check state.source.data.

The hypothetical @frontity/redirections package can have simpler settings, like these:

const settings = {
  packages: [
    {
      name: "@frontity/redirections",
      state: {
        redirections: {
          "/some-post": "/other-post",
          "/some/custom/:url": "/other/custom/${params.url}",
          "/more/custom/:url": {
            destination: "/other/custom/${params.url}",
            status: 302,
          },
          "RegExp:\\/some-post?photo=(?<id>[^&$]+)>":
            "/some-post/photo-${params.id}",
        },
      },
    },
  ],
};

But internally, it will add that info to state.source.data either manually (for single redirects) or with handlers (for patterns/RegExps) to be consistent with the server redirections.

So with the current implementation, we have added:

  • Support in tiny-router for:

    • Checking if the data object is a redirection in the server
      If it is, changing the ctx.status in the same way it is changing it for errors.

    • Checking if the data object is a redirection in the client
      If it is, it does an additional actions.router.set to move to the correct URL.

  • Support in wp-source for “server-stored” or “source-stored” redirections:

    • Checking if the request matches a regexp or is a 404
      If it is, doing an additional request against the state.source.url to see if there is a redirection and storing it in the data object for other packages to do what they need (like the router).

Also, @santosguillamot has talked about the possibility of creating other redirection packages in the future. That package would also add redirections to the data objects, although they won’t be related to the source, but to other origins (Frontity settings, CSV, REST API requests…).

So that can be translated to:

  • tiny-router: Support for changing the routes in both client and server when there are redirections stored in data objects.
  • wp-source: Support for checking redirections in the source and storing them in data objects.
  • other redirection packages: Support for checking redirections in other origins and storing them in data objects.

Due to that, it doesn’t make sense to have the settings for the “source-stored” redirections in state.router.redirections anymore. I think we should move them to state.source.redirections. Other packages will have their settings stored in state.thePackage.redirections or whatever they feel like.

1 Like

Another use case that just came to our mind and I don’t know how it would work with the planned solution. When you create a redirect, you might want to keep the utm campaigns parameters.

Let’s imagine I create a page with the slug /gutenberg-and-frontity/ and I start promoting it. It is posted in an external site with a utm_campaign to track it -> /gutenberg-and-frontity?utm_campaign=external-site.

Then, for SEO reasons for example, I decide to change the slug to /gutenberg-and-react/ and redirect the old slug to this new one. I cannot ask all the external sites to change the link, but I still want to keep the utm_campaign in my Analytics.

This means that the link
/gutenberg-and-frontity?utm_campaign=external-site
should be redirected to
/gutenberg-and-react?utm_campaign=external-site

The default WordPress redirects don’t keep the parameters, which could be a problem for this use case. But for example, the Redirection plugin allows you to keep them, so it doesn’t affect the utm_campaign. I assume other plugins like Yoast also allow you to do the same.

Is this something that would keep working as expected with the planned solution?

Yes, I think so. We won’t do anything special here. That means that if you have configured your redirections to pass the queries, they will. If not, they will not.

We had a conversation about this in the PR: https://github.com/frontity/frontity/pull/601#discussion_r526045152