301 redirects stored in WordPress database

Thanks for the feedback :slightly_smiling_face: As you mention there are two ways of adding redirects, from Frontity and WordPress. In order to do redirects from Frontity, we would like to a simpler solution but we will need the Server Extensibility first. If you control your Node server you should be able to do it as explained in this topic.

To create new redirects, users should always add them directly in the frontend (Frontity). But sometimes, in old websites as you say, there are a lot of redirects stored in the DB, and supporting that ones is the purpose of this feature.

I will take a look at the setting you mention to understand better how it works and check if it could be useful. Thanks!

What do you guys think of what I proposed in this message (301 redirects stored in WordPress database) to have both Frontity and WordPress redirections configured under the state.router.redirections setting?

I have I feel that it’s perhaps a bit too opinionated to have Frontity provide that option out of the box? I think that this makes the API more “ugly” and less intutitive and only to accomodate a feature that has not been requested yet by any user.

Handling the 301/302 redirections that are in stored in WordPress database is necessary, but adding redirections in the application is another thing :slight_smile:

  1. I think that for majority of applications, adding a 301 in the application would not be the right thing to do. Redirections in most cases should be done in your hosting (netlify.yaml, vercel.json, etc.) or maybe in nginx in your WP hosting or lastly in your WP database and I feel like putting redirections in your state is a weird pattern that I think we should not encourage.

  2. Once we ship server extensibility, if a user really needs redirections, they should be able to create their own solution because they will have access to the Koa ctx so could just call ctx.redirect(newUrl); return;

Of course, I’m happy to be proven wrong here :slight_smile:

Ok, I get your point Michal, thanks for sharing it :slightly_smiling_face:

Let’s create a draft of the situation to be able to make a decision.

The basic user story would be:

As a Frontity user
I need to redirect some old URLs to new ones
so that links pointing to an old URL are still valid

Old URLs can be present in:

  • 1. External sites
    This is usually the main problem, URLs that the user cannot change because they belong to external sites.
  • 2. Database content
    I guess this could also be categorized as solvable by the developers, although sometimes is easier to set up a redirection than to do a search-and-replace of your entire database. There are WordPress databases with +100.000 posts.
  • 3. Theme code (like in a menu, for example)
    This is usually not a problem because the developer can change that URL.

I am going to assume that 3 is not a problem we should solve, so let’s keep going for 1 and 2. The requirements for those would be:

  • 1. External sites
    Because these old URLs are only present on external sites, only the server needs to be aware of them. These are not needed in the client.
  • 2. Database content
    This is different because the old URLs present in the content can appear inside the client-side rendered app. So, if we want to provide redirections from old URLs present in the content, the client needs to be aware of those redirections as well.

Now let’s see the different options to set up those redirections in a Frontity+WordPress app:

  • Server Redirections
    As Michal mentioned, these can be set up in different places:
    • Hosting redirections
      Like Vercel or Netlify configuration files.
    • Proxy redirections
      Like in Apache or NGINX.
    • WordPress server
      Like the Yoast or Redirection plugins.
    • Frontity Koa server
      Using the upcoming Server Extensibility to manage the redirections from Koa’s server.
    • Frontity server does a request to the backend server
      We can do a request to the backend server to see if it returns a redirection, which would be set up in any of the above methods: Hosting, Proxy or WordPress.
  • Client Redirections
    This is where things get trickier, as we have seen. Let’s see what options we have:
    • Do a request to the server
      As we have seen, we can do a request to the server to see if it returns a redirection. This approach works with any of the methods mentioned in the previous section.
    • Retrieve a list of redirections from the server
      Another way would be to retrieve a list of redirections from the server. That would avoid the need to do an additional request for each client-side navigation.
    • Store the list of redirections in the client state
      The final option would be to store the list of redirections in the client state.

Now, let’s see the pros and cons of these methods:

  • Do a request to the server
    • Pros
      No extra configuration is required. It just works with whatever you have configured in your server.
    • Cons
      Doing an additional request for each client-side navigation just to see if there is a redirection is quite expensive. For that reason, I think this method works better if it is used only under certain conditions, like 404’s or other server errors.
  • Retrieve a list of redirections from the server
    • Pros
      Instead of doing a request for each client-side navigation, this method would do only one.
    • Cons
      We need extra code in the server, something that knows about all the redirections and exposes in an endpoint. That is not easy as there are many methods to create redirections in the server. For example, how do you access Apache or NGINX redirections?
      If the list is big, this download could be pretty expensive in the client.
  • Store the list of redirections in the client state
    • Pros
      No request is needed.
      As the information is already there in the state, it is cheap to consume. Therefore, there is no drawback in checking for all URLs, as opposed to restricting only for 404’s.
      This information, as it is store in Frontity, can be shared for both the client and the Frontity server.
    • Cons
      It doesn’t work with the redirections that are already set up in the server.
      If the list is big, we can increase the size of the bundle.

Ok, I hope that this makes things clearer to think about what cases we want to cover now, what cases we want to cover in the future (to think now about those APIs), and what cases we don’t want to cover.

@david, @mmczaplinski: I am probably missing things so feel free to add your own feedback and I will update the tables to reflect it :slightly_smiling_face:

@santosguillamot, please take a look. We need your decision here.

Great summary guys :slightly_smiling_face: I am writing down some ideas, although I am still not 100% convinced.

The way I see it (I don’t have too much information in this case), the common use cases of redirects would be:

  • Old slugs: I think it’s pretty common in WordPress to change the slug of your posts, and it is redirected to the new url. This is something WordPress does by default and it store the old slugs in the database.
  • Permalink changes: In the WordPress settings, it is also common to change the permalinks of your posts or categories. For example from categories to categoria.
  • Custom redirects: Other custom redirects that the users add with code (or plugins).

With the proposed solution, do a request to the server, these use cases could be easily covered. The old slugs would be covered with the 404 settings and the permalink changes and custom redirects should be fairly simple to replicate with regexp.

I feel this is great, but I assume users might be concerned by server load. They will probably want to migrate most of the redirects to the Frontity side to reduce this. And not only to migrate old redirects, but to create new ones. This is something users have already asked for.

Right now, I feel that is not simple to create redirects that work both in the server and the client in a Frontity app, so I think, in the future, we will need to work on a package, @frontity/redirects for example, that deals with both things. A place where you define your redirects, and they work in Client Side and Server Side Rendering. If possible, it should deal with the Koa’s server for the SSR, and with the routing for the CSR. For this to work I think we would need the Server Extensibility and the Frontity hooks.

So, the way I see it, the best possible workflow to migrate an old WordPress to Frontity would be:

  1. Use the “do a request to the server” method to ensure that everything works, from the beginning.
  2. Start migrating the redirects from WordPress to Frontity.
  3. Create new redirects directly from Frontity. Here my main concerns is the old slugs, as this is something WordPress does by default.
  4. Keep the do a request to the server for the edge cases we didn’t covered in the migration and probably return a 404 or other server errors.

For steps 2 and 3, a redirects package would be really useful.

Would it be useful to redirect the old slugs directly in the REST API?

Not sure if this could be useful somehow: As the old slugs are stored in WordPress database, it isn’t difficult to get them. I have found a PHP snippet that redirects the old-slug to the new-slug in the REST API. So, if I go to mysite.com/wp-json/wp/v2/posts?slug=old-slug it is redirected to mysite.com/wp-json/wp/v2/posts?slug=new-slug. I guess that, to supports this, some changes would be needed in Frontity. The PHP snippet I used was this:

add_action( 'rest_api_init', function() {
    add_filter( 'the_posts', function( $posts, $query ) {
        global $wp_query;
        // Only try to redirect if there were no posts found.
        if ( empty( $posts ) ) {
            // Set up the global wp_query so wp_old_slug_redirect() will think we're in a template & redirect accordingly.
            $wp_query = $query;
            $wp_query->set_404();
            $wp_query->set( 'name', $wp_query->get( 'post_name__in' )[0] );
            // Add filter to old_slug_redirect_post_id in order to prevent the default frontend redirect in favor of a REST API URL.
            add_filter( 'old_slug_redirect_post_id', function( $id ) use ( $wp_query ) {
                $post = get_post( $id );
                $redirect_url = $_SERVER['REQUEST_URI'];
                $redirect_url = str_replace( $wp_query->get( 'name' ), $post->post_name, $redirect_url );
                wp_redirect( $redirect_url, 301 );
                exit;
            } );
            // Finally, call wp_old_slug_redirect() and let our filter interrupt it to handle the final wp_redirect().
            wp_old_slug_redirect();
        }
        return $posts;
    }, 10, 2);
});

If this is possible, and the old slugs are redirected in the REST API, the list of redirects users have to define in the Frontity side would be reduced significantly, as the permalink changes and custom redirects should be easily covered by regexp. And we would still have the “do a request to the server” for the edge cases that are not covered with regexp or the old slugs.


I’ve been taking a quick look at some WordPress redirects solutions, to see if they can be migrated to the Frontity side easily:

I assume with most of the plugins it’s going to be the same. If not, users can also use tools like ScreamingFrog to crawl or the redirects -> https://www.screamingfrog.co.uk/redirect-checker/

Nice. Thanks Mario!

I have kept thinking about this as well and I realized that the current solution to know when there is a 301 redirection in the server is actually storing the information in Frontity as well, in state.source.data.

I think we should use the same system for all the 301 redirections, which means that the redirections created in Frontity also should end up being stored in state.source.data.

So if people want to add redirections stored in Frontity now, they can do it this way:

  • Single redirection populating state.source.data directly:

    const state = {
      source: {
        data: {
          "/old-url": {
            isReady: true,
            isRedirection: true,
            is301: true,
            redirectionStatus: 301,
            location: "/new-url",
          },
        },
      },
    };
    

    This manual population is something I have started recommending recently to create custom pages that are not present in WordPress: How to create Custom Pages? because it’s simpler than creating a code handler.

  • Pattern/RegExp redirection using a handler:

    const categoryRedirection = {
      pattern: "/category/:slug",
      priority: 5,
      func: ({ link, params }) => {
        state.source.data[link].isReady = true;
        state.source.data[link].isRedirection = true;
        state.source.data[link].is301 = true;
        state.source.data[link].redirectionStatus = 301;
        state.source.data[link].location = `/categoria/${params.slug}`;
      },
    };
    

That way all the redirections will be stored in the same place and when packages want to know if there is a redirection in a URL, they can simply check state.source.data.

The hypothetical @frontity/redirections package can have simpler settings, like these:

const settings = {
  packages: [
    {
      name: "@frontity/redirections",
      state: {
        redirections: {
          "/some-post": "/other-post",
          "/some/custom/:url": "/other/custom/${params.url}",
          "/more/custom/:url": {
            destination: "/other/custom/${params.url}",
            status: 302,
          },
          "RegExp:\\/some-post?photo=(?<id>[^&$]+)>":
            "/some-post/photo-${params.id}",
        },
      },
    },
  ],
};

But internally, it will add that info to state.source.data either manually (for single redirects) or with handlers (for patterns/RegExps) to be consistent with the server redirections.

So with the current implementation, we have added:

  • Support in tiny-router for:

    • Checking if the data object is a redirection in the server
      If it is, changing the ctx.status in the same way it is changing it for errors.

    • Checking if the data object is a redirection in the client
      If it is, it does an additional actions.router.set to move to the correct URL.

  • Support in wp-source for “server-stored” or “source-stored” redirections:

    • Checking if the request matches a regexp or is a 404
      If it is, doing an additional request against the state.source.url to see if there is a redirection and storing it in the data object for other packages to do what they need (like the router).

Also, @santosguillamot has talked about the possibility of creating other redirection packages in the future. That package would also add redirections to the data objects, although they won’t be related to the source, but to other origins (Frontity settings, CSV, REST API requests…).

So that can be translated to:

  • tiny-router: Support for changing the routes in both client and server when there are redirections stored in data objects.
  • wp-source: Support for checking redirections in the source and storing them in data objects.
  • other redirection packages: Support for checking redirections in other origins and storing them in data objects.

Due to that, it doesn’t make sense to have the settings for the “source-stored” redirections in state.router.redirections anymore. I think we should move them to state.source.redirections. Other packages will have their settings stored in state.thePackage.redirections or whatever they feel like.

1 Like

Another use case that just came to our mind and I don’t know how it would work with the planned solution. When you create a redirect, you might want to keep the utm campaigns parameters.

Let’s imagine I create a page with the slug /gutenberg-and-frontity/ and I start promoting it. It is posted in an external site with a utm_campaign to track it -> /gutenberg-and-frontity?utm_campaign=external-site.

Then, for SEO reasons for example, I decide to change the slug to /gutenberg-and-react/ and redirect the old slug to this new one. I cannot ask all the external sites to change the link, but I still want to keep the utm_campaign in my Analytics.

This means that the link
/gutenberg-and-frontity?utm_campaign=external-site
should be redirected to
/gutenberg-and-react?utm_campaign=external-site

The default WordPress redirects don’t keep the parameters, which could be a problem for this use case. But for example, the Redirection plugin allows you to keep them, so it doesn’t affect the utm_campaign. I assume other plugins like Yoast also allow you to do the same.

Is this something that would keep working as expected with the planned solution?

Yes, I think so. We won’t do anything special here. That means that if you have configured your redirections to pass the queries, they will. If not, they will not.

We had a conversation about this in the PR: https://github.com/frontity/frontity/pull/601#discussion_r526045152

1 Like

The PR (https://github.com/frontity/frontity/pull/601) has finally been merged.

I will write the final implementation and I will make a video explaining how they work in the next days :slightly_smiling_face:

6 Likes

This feature has been released :tada: : Read the full Release Announcement.

I have created a demo to explain briefly how it works and how to configure it:

5 Likes

I will write the Final Implementation as soon as possible :slightly_smiling_face:

1 Like

Great! Let us know (@documentation-team) once you have written the Final Implementation so we can check it for documentation purposes

Hi guys, I’m having some trouble making this work, and I think it is because I’m still using a beta version of the wp-source package because of the infinite scroll. Is there a release of the beta packages that includes the redirects?

The infinite scroll PR is merged in dev. We didn’t do a release today because the unit tests started failing for an unrelated issue that is already fixed in this PR.

So if you can wait until tomorrow… :sweat_smile:

Sure thing. Please let me know when it’s released! :slight_smile:

We did the release yesterday, although we haven’t prepared the announcement yet. It’s ready :slightly_smiling_face:

1 Like

We have seen that our hosting provider, in this case Pantheon, change the HEAD methods and use GET instead. This way, if you are using the Embedded mode and the setting redirections: "404", it causes and Infinite loop that breaks the site as explained in this diagram:

If your hosting isn’t changing the HEAD method, it just works fine. We aren’t sure why Pantheon is doing this, but we think we should solve it anyway. We have made this video explaining the issue and the possible solutions:

And this is the diagram we have used. Let us know your thoughts :slightly_smiling_face: .

We should also add another safety measure: Do not return a 301 which redirects again to the same link.

That could happen if the domains are different and the WordPress domain is redirected to the Frontity domain.

The logic should be something like this: if the redirection is internal and the pathname is the same, don’t do the redirection.

const linkParams = new URL(link);
const locationParams = new URL(location);

const linkURI = linkParams.path + linkParams.search;
const locationURI = locationURI.path + locationURI.search;

// Only return the redirection if the location is different.
if (!isExternal && linkURI !== locationURI) {
  return redirection;
}

I haven’t tested this code, it’s just an example.

@luisherranz I can confirm that the infinite loop happens when visiting a non existent url. Not sure what other cases I should test, but if you go to https://frontity.es.aleteia.org/2021/02/23/fake/ for us it enters in a loop.