Issue with having 2 duplicate requests to posts and pages

Hi all! I’m wondering, why I have 2 same requests when looking in the network. See example

Contact - is a page. So the request https://test.silverrailtech.com/home/wp-json/wp/v2/posts?_embed=true&slug=contact returns just an empty array [] and request
https://test.silverrailtech.com/home/wp-json/wp/v2/pages?_embed=true&slug=contact returns json with all the info.
So I don’t understand why the posts request is there at all. Also I have these behaviour on all the pages.

Is it something wrong with my set up or is it a normal thing?

Thanks!

Hi @dev_person

Welcome to the Frontity community.

Interesting, I’ve never spotted that before - but then I don’t really spend much time looking at the network tab in dev-tools.

I’ve tried it with a couple of different themes and they both do this. I think it could be because Frontity only has the slug to work with, and doesn’t know whether it’s a post or a page, so it requests a post with that slug and if that fails it requests a page with that slug.

TBH that’s speculation on my part, so perhaps @mmczaplinski can either confirm or refute that? If that’s the case I wonder what happens with CPT’s - does Frontity try post first, then page, then CPT? :man_shrugging:

1 Like

Yes, your hunch is correct! @mburridge :slightly_smiling_face:

In some circumstances, Frontity cannot know just based on the slug if the entity that you are requesting is a post or a page. If you go to https://my-site/about-us, it’s both a valid page URL and a valid post URL.

For this reason we have to make a request to both endpoints in order to know which one it is.

You check out the relevant source code where we match the link with a pattern and then invoke the postType handler for that link (which could be a post, page or attachement).

The postType handler is created via a “generator function” here

That “generator function” will iterate over post, page and media endpoints fetching the data from each.


@David :

There’s one thing that I’m not certain about. In the postType handler we are fetching the data serially from each endpoint (post, then page and then media). Is the order of the endpoints in the postType handler intentional or not? In other words, do we intentionally check post before page and page before media?

If it is not intentional, we could optimize that part and fetch the data in parallel. I’ve just done a little modification which fetches in parallel and it seems to work just fine with the mars-theme:

libraries/handlers/postType.ts

// Turn an array of promises into a generator. 
// This allows us to iterate over them with `for await...of`
async function* getData(promises: Promise<any>[]) {
  const map = new Map(
    promises.map((p, i) => [i, p.then((res) => [i, res])])
  );
  while (map.size) {
    const [key, result] = await Promise.race(map.values());
    yield result;
    map.delete(key);
  }
}

const promises = finalEndpoints.map((endpoint) =>
  libraries.source.api.get({
    endpoint,
    params: { slug, _embed: true, ...state.source.params },
  })
);

const responses = getData(promises);

// Here we iterate asynchronously. This means that we 
// pull the new response from `responses` as soon as it is received.
for await (const response of responses) {
  const populated = await libraries.source.populate({
    response,
    state,
    force,
  });

  // exit loop if this endpoint returns an entity!
  if (populated.length > 0) {
    // We have to check if the link property in the data that we
    // populated is the same as the current route.

    if (populated[0].link === route) {
      isHandled = true;
      isMismatched = false;
      matchedEndpoint = populated[0].type;
      break;
    } else {
      isMismatched = true;
    }
  }
}
1 Like

Well, it was intentional in order to avoid as many unnecesary requests to the WP REST API as possible and because we thought we would have to wait for all the responses to check in which one the requested entity appears.

But looking at your proposed solution I guess we were wrong, haha. :raised_hands:

OK, I have to admit I’ve never thought of using Promise.race() in that way. So, you are returning the fastest promise and, if the requested entity is not included there, removed it from the list and repeat the same process, waiting for the next fastest response. That would make Frontity to fetch the entity in the shortest possible time, which is awesome. :100:

My only concern would be that WordPress would have to serve more responses in this case (expending more CPU time) but that shouldn’t be a problem once those responses are cached.

EDIT: @mmczaplinski would you mind if you open a PR with your proposal?

EDIT 2: Nevermind, I just saw that you already did that :point_right: [test] Fetch data asynchronously in the postType handler by michalczaplinski · Pull Request #758 · frontity/frontity · GitHub

yup, exactly :smiley:

I’ve finished the PR now (there were some more details than originally in the first draft plus fixing some tests): Fetch data asynchronously in the postType handler by michalczaplinski · Pull Request #758 · frontity/frontity · GitHub

Just so you know, yes this was intentional.

I’ve shared my thoughts in the PR: Fetch data asynchronously in the postType handler by michalczaplinski · Pull Request #758 · frontity/frontity · GitHub

Hey Michal, could you please add a setting to opt-in to this?

Just so you know, we didn’t do parallel requests in the first place because it’s faster, but it is also less efficient.

The number of pages vs posts on big sites is usually very low. Pages are only for static content, like contact pages, landing pages, privacy, about us and so on. When sites grow, they usually do it with posts.

A big site can have 100.000-500.000 posts, but it will rarely have more than 1000 pages.

The same happens with media/attachment pages. They exist, but they are rarely used and rarely linked.

Now, imagine we push this change to Frontity. A blog with 100.000 post requests per day will get 200.000 additional daily requests to their REST API to check for pages and media. That’s not good. If they have 100 pages, only ~0.05% of those 200.000 extra requests will be useful. That’s a lot of wasted bandwidth.

1 Like