The Source API

It seems that it is possible to implement a single plugin for both kinds of WordPress as WordPress.com exposes WordPress.org REST API since 2016. They also encourage its use. :muscle:

They work exactly the same but with some differences:

Retrieving media items from the REST API can be a problem, but it can be avoided by using directly the URLs of media items that appear in the content, as they can be requested with a query to restrict the size.

Apart from that, I was looking at Overmind and I’ll be playing around with it to see what proposal would make the most sense.

I’ll extend a little more tomorrow. :grin:

1 Like

I’m changing the topic title from WordPress Source API to The Source API. The goal here is to decide on a general source API, not only for WordPress. If some developers want to create a source package for Ghost, for example, they would have to stick to this Source API or their package won’t work with the rest of the themes.

When we thought that WP.com and WP.org APIs were different, it was clear that we needed to create some kind of abstraction on top of them, similar to what we had in our old framework.

But now that we know the schema of both WP.com and WP.org APIs can be equal, I have a doubt:

  • Should we leave that schema as it is?
  • Should we create an abstraction on top of it?

If we use the WP schema, people won’t have to learn a new schema. That’s good. They can look at the output of their APIs and use that schema in Frontity. But, if devs start creating other sources (Ghost, Contentful, Medium…) they have to convert them to the WP schema instead of a “more general one”.

I think my vote goes for “stick to the WP schema”.


Some constrains to get you started. I’m sure there are many more:

Constraints:

  • Frontity should be able to retrieve data from the API using:
    • A URL (permalink).
      This is needed to get the data in SSR when using the “direct to node” integration.
    • A URL that points to an endpoint, and some params.
      This is needed to make it “hackable”.
    • A type and an id.
      This is needed to get more content from the client once it has loaded.
      We could get rid of it if getting data using a permalink is as performant as this.
  • Now Frontity doesn’t know if the data it is retrieving is a “list” or an “single”. This is a very important constraint because it means we may have to get rid of the “list” and “single” concepts altogether.
  • We need APIs to make custom lists when:
    • Devs want to exclude certain list form other lists
      This is needed to create carrousels that don’t repeat the same items than the previous carrousel.
    • Devs want to exclude “read” items.
  • We probably need a place to “store” state of the items. Things like “read” for example.

One final suggestion: @David, why don’t you take a look at saturn? There are some really complex things there we had to do because of our old connection API and I’m sure we can come up with much better APIs for those patterns. Things like getting the next page of a list when infinite scrolling, dealing with swipe, the context router and so on.

I’ve been thinking about this and it’d be great to use only urls for the source and router APIs.

@David Could you please investigate how urls are made in both WP.org and WP.com?

For example:

  • Taxonomies:
    • /taxonomy-type/taxonomy-slug
  • Pages:
    • /page-slug
    • /parent-page-slug/page-slug

and so on… including of course custom taxonomies, custom post types, archives, everything…

Once we have that information we can decide how to get the data from the APIs:

  • With guesses:
    Try one endpoint, if it fails try another… and so on.
  • With some settings in the wp-source package:
    The developer can enter the schema of its site. For example, with this config, wp-source can know that /my-custom-taxonomy/something goes to the wp/v2/my-custom-taxonomy/?slug=something endpoint:
settings: {
  taxonomies: ['category', 'tag', 'my-custom-taxonomy']
}
  • With a discovery endpoint:
    A custom endpoint made by us where you enter a slug and it gives you the data: frontity/v1/discovery?slug=/my-custom-taxonomy/something. I’m now less in favor of this solution because it does not solve the WP.com problem.

By the way, we don’t need to support the query urls of WP anymore (?p=id).

@development-team :point_up_2::question:

My vote goes for no schema.

Source API

One of the key ideas of this API is to use only URLs to identify entities, so that it is not necessary to work with ids or types. For this way of identifying entities to work, these URLs must be translated into calls to a REST API (WordPress REST API in this case) in order to get the entities you need. This, in some cases, is not trivial, but it can be easily resolved using the following scheme in the source package configuration:

Settings Schema

type Routes = {
  [pattern: string]: Call
};

type Call = {
  "endpoint": string;
  "params": { [key: string]: any } | string;
  "props": { [key: string]: any } | string;
  "type": "single" | "list";
  "success"?: Call;
  "error"?: Call;
}

Properties of Routes type:

pattern

It is a pattern in string format that allows us to differentiate some URLs from others. Some examples could be:

{
  "/category/:slug": {},
  "/tag/:slug": {},
  "/:slug": {},
  "/cars/:model": {},
  "/products/:name": {},
  // ...
}

Properties of Call type:

endpoint

It can be a string or a function in string format. To differentiate this last case, we should put the identifier "eval: ..." at the beginning of the string. The params object that receives the function as an argument contains the variables obtained after matching the URL with pattern.

{
  "/category/:slug": {
    "endpoint": "/wp/v2/categories",
    // ...
  }
}

{
  "/category/:slug": {
    "endpoint": "eval: ({ params }) => `/wp/v2/posts/${params.slug}`",
    // ...
  }
}

params

Is an object with values that will be passed as the query string. The values of params can be strings or functions in string format, as in the previous case:

{
 "/category/:slug": {
   "endpoint": "/wp/v2/categories",
   "params": {
     "slug": "eval: ({ params }) => params.slug"
     // ...
   },
 }
}

Or it can also be a function in string format that returns an object with values:

{
 "/category/:slug": {
   "endpoint": "/wp/v2/categories",
   "params": "eval: ({ params }) => ({ slug: params.slug })"
     // ...
 }
}

props

An object to store attributes of interest in relation to the entities returned by that URL. It works the same as params:

{
  "props": {
    "isCategory": true,
    // ...
  },
}

In addition to receiving params, it also receives response as an argument, which is the object with the call response. This allows you to assign properties after receiving the entity’s data.

{
  "props": "({ params, response }) => ({ slug: params.slug, type: response.type, id: response.id })"
}

type

It is the type of response we expect to receive from the REST API.
It can only take two values:

  • "single" if we expect to obtain a single entity.

  • "list" if we expect to obtain a list of entities.

success

It is an optional object of type Call that defines another call to the REST API. That call will be requested when the previous one has finished successfully. Useful when you want to chain calls to the REST API. For example, if you want to obtain a list of posts
that belong to a category:

{
  "/category/:slug": {
    "type": "single",
    "endpoint": "/wp/v2/categories",
    "params": {
      "slug": "eval: ({ params }) => params.slug"
    },
    "success": {
      "type": "list",
      "endpoint": "/wp/v2/post",
      "params": {
        "categories": "eval: ({ response }) => response.id"
      },
    }
  }
}

In this case, the functions in string format also receive a response argument, where you can access the response data of the previous call.

error

It works exactly the same as success but when the previous call has failed.

API calls

This part is less defined, but essentially it should be something like the following:

State

source.data

It is the way to access the data that has been requested from the REST API using the fetch action (we will explain it later).

state.source.data["/category/nature"].items.map(url => state.source.data[url])
state.source.data["/category/nature"].pages.map()

state.source.data["/category/nature"].props.isList // true
state.source.data["/category/nature"].props.isSingle // false
state.source.data["/category/nature"].props.isReady

state.source.data["/category/nature"].items // => [...page1, ...page2]
state.source.data["/category/nature"].pages // => [[...page1], [...page2]]
state.source.data["/category/nature"].endpoint // => /wp/v2/posts
state.source.data["/category/nature"].params // => { categories: 17 }
state.source.data["/category/nature"].total // => 24
state.source.data["/category/nature"].totalPages // => 3
// state.router.path = "/category/nature"
state.source.data[state.router.path].props.isTag && <Tag />
state.source.data[state.router.path].props.isHome && <Home />
state.source.data[state.router.path].props.isCategory && <Category />
state.source.data[state.router.path].props.isPost && <Post />
state.source.data[state.router.path].props.isPage && <Page />

Actions

source.create

The create action allows you to define custom lists with a name as the identifier.

actions.source.create({
  name: "carrousel-/category/nature",
  url: "/category/nature",
  exclude: [...read, list1...], /* array of URLs or custom list's names */
  perPage: 5
})

source.fetch

It makes calls to the API and populates the state. Receive two arguments, the URL and a number that indicates the requested page.

actions.source.fetch("/category/nature")
actions.source.fetch("/category/nature", 2)

Another idea is to leave the “hacked” urls to JavaScript.

For example, categories would be:

souce.add("/categories/:slug", async ({ name, params, page }) => {
    let category = data.category.find(category => category.slug === params.slug);
    if (!category) {
        const categories = await effects.api.get({
            endpont: "/wp/v2/categories",
            params: {
                slug: params.slug
            }
        })
        category = { id: categories[0].id };
    }
    const posts = await effects.api.get({
        endpoint: "/wp/v2/posts",
        params: {
            categories: category.id,
        }
    }, page)
    source.populate(name, posts, page);
    source.data[name].isCategory = true;
})

More things we were talking about yesterday:

It seems that we cannot get rid of types and ids completely when accessing entities. If you fetch data from the WordPress API, related entities are specified with an id, like:

{
  "id": 60,
  "type": "post",
  ...
  "author": 4,
  "featured_media": 62,
  "categories": [57, 59, 56, 3, 8, 58],
  "tags": [10, 9, 13, 11],
}

It would be necessary to store the data to access it like showed below. This work could be done by the populate action.

data["/categories/nature"].page[1] 
data["/categories/nature"].items 
data.posts[1] 
data["/post-1"] 
data.posts[2] 
data["/post-2"] 
data.posts[3] 
data["/post-3"] 
data.author[4] 
data["/author/luis"] 
data.media[5] 
data.media[6] 

Also, the populate action could normalize the data and change ids by { type, id, permalink }.

Another “what-if”…

What if we use source.data["name"] to specify only the type and id but the real data is always stored under source[type][id]?

// No entity, just type and id.
source.data["/category/nature"] = {
  type: "category",
  id: 17,
  items: [
    { type: "post", id: 24 },
    { type: "post", id: 28 },
    { type: "post", id: 29 },
  ],
  isCategory: true,
  isList: true
}

// Get the real data from state.source.type.id. 
const { type, id, isList } = source.data["/category/nature"];
const entity = source[type][id];

source.data["/category/nature"].items.map(({ type, id }) => {
  const entity = source[type][id];
  if (type === "post") return <Post entity={entity} />
  if (type === "page") return <Page entity={entity} />
});

Maybe source.data is not the best name though…

More stuff:

  • We need a standard way to work with 404’s
source.fetch(router.link) // <- this url doesn't exist

source.data[router.link] = {
  is404: true
}
  • Developers need to access the real url for things like analytics
// For internal use in source.data() and source.fetch()
router.link = "/category/nature"
router.page = 2

// For analytics...
router.path  = "/category/nature/page/2"
router.url = "https://my-blog/category/nature/page/2"
...
  • Developers may need a way to fetch stuff without having to create names
const entity = source.post[id];
const media = entity.content_media; // <- [{ type: media, id: 1 }], [1]

// option 1, create name.
source.add("my-media", async () => {
  const media = effects.api.get({ endpoint: "/wp/v2/media", params: { include: media.join(',') } });
  source.populate("my-media", media, 1);
})
source.fetch("my-media", 1)

// option 2, fetch directly from endpoint.
source.get({ endpoint: "/wp/v2/media", params: { include: media.join(',') } });
// This is how it is right now:
effects.api.get({ endpoint: "/wp/v2/posts" });

// Maybe we can use this for /wp/v2.
effects.api.get({ endpoint: "posts" });

// And this for other endpoints: 
effects.api.get({ endpoint: "/frontity/v1/discovery" });

Source API

State

Let’s start by explaining how the state data is used and then how that data is requested and stored. The state works with two main concepts: paths and entities.

The state is designed so that you can know which entities correspond to which path, and then access the data of these entities in a simple way.

NOTE: for the data to exist, it will be necessary to request them previously using the fetch action.

example

// Accessing path's data
const data = source.data["/categories/nature"]

// `data` value:
({
  
  // Type & id of the entity
  type: "category",
  id: 2,
  
  // Type & id of the listed entities
  // (when the path is an archive or similar)
  items: [{ type: 'post', id: 60, path }, ...],   // all fetched entities
  pages: [[{ type: 'post', id: 60, path }, ...]], // fetched entities per page
  totalItems: // total number of entities
  totalPages: // total number of pages

  // Booleans to easily identify the type of path
  // inside the WP hierarchy
  isArchive: true,
  isTaxonomy: true,
  isCategory: true,

})

// Accessing the entity's properties
const entity = source[d.type][d.id]

// `entity` value:
({
  id: 2,
  count: 23,
  description: "Beautiful nature locations",
  link: "https://myblog.com/category/nature/",
  name: "Nature",
  slug: "nature",
  taxonomy: "category",
})

The information to distinguish each type of path is based on the WP Template Hierarchy and is as follows:

  • archives: isArchive
    • taxonomy: isTaxonomy (with type, id properties)
      • category: isCategory
      • tag: isTag
      • deal: isDeal
    • author: isAuthor (with id property)
    • postTypeArchive: isPostTypeArchive (with type property)
      • post: isHome, isPostArchive (isFrontPage optional)
      • product: isProductArchive
    • date: isDate (with date)
  • postTypes: isPostType (with type, id properties)
    • post: isPost
    • page: isPage (isFrontPage optional)
    • product: isProduct
    • media: isMedia, isAttachment
  • 404: is404

State API

source.data[path]

Access data related to a path. The path parameter could be also a name that identifies custom lists. See source.add for more info.

source.data["/the-beauties-of-gullfoss"]
source.data["/category/nature"]
source.data["gallery-post-60"]

source.taxonomy[taxonomy]

Access taxonomy entities. taxonomy can be "category", "tag" or any custom taxonomy slug. That value is stored in the type attribute of the source.data[path] object.

source.taxonomy["category"]
source.taxonomy["tag"]
source.taxonomy["deal"] // custom taxonomy

source[taxonomy][id]

Access category, tag, or custom taxonomy’s entities.

source.category[2]
source.tag[13]
source.deal[3]

source.type[type]

Access post types. type can be "post", "page", "attachment" or any custom post type.

source.type["post"]
source.type["page"]
source.type["product"] // custom post type

source[type][id]

Access posts, pages, attachments or custom post type’s entities.

source.post[60]
source.page[7]
source.product[36]

source.author[id]

Access author entities.

source.author[4]

Actions

source.fetch(path, page)

Action that retrieves entities using a path or a name. fetch works using the corresponding fetch function defined with the add action. This action doesn’t return anything but ensures that the received data are correctly populated in the state.

arguments

  • path: string that contains a path or a custom list’s name
  • page: page number (default is 1)

example

source.fetch("/category/nature")
source.fetch("/category/nature", 2)
source.fetch("/the-beauties-of-gullfoss")
source.fetch("custom-list")

There are some URL patterns handled by default. Those are:

postArchive: '/(?s=:search)'
category:    '/category/:slug(/?s=:search)'
tag:         '/tag/:slug(/?s=:search)'
author:      '/author/:name(?=:search)'
date:        '/:year/:month?/:day?(?s=:search)'
postOrPage:  '/:slug'
attachment:  '/:year/:postSlug/:attachmentSlug'

source.add(pattern, callback)

Action that attaches a function to handle a specified name or route pattern, using a string or a regexp. That function should request data from the WP REST API and populate the store with it. For that purpose, you may use the populate action.

arguments

  • pattern: string or regexp

  • callback: async function that will be executed if path matches pattern when calling source.fetch(path). Its arguments are:

    • config: Overmind config object ({ state, actions, effects }).
    • payload: object with the following attributes:
      • name: the path or name that has matched the pattern
      • params: params extracted from name after the match
      • page: page number passed when calling source.fetch(path, page)

example

source.add(
  "/post/:slug",
  async ({ state, actions, effects }, { name, params }) => {
    // Get a post
    const { body: post } = await effects.api.get(
      {
        endpoint: "/wp/v2/posts",
        params: {
          slug: params.slug
        }
      }
    );

    // Populate data response in the store
    actions.source.populate(name, post);
  }
)

source.populate(path, entities, page)

Action that stores entities in the state so you can find them using the API specified in the State API section. Also, it sets all info related to paths for each entity (based on the entity’s link). This action is intended to be used after request entities from the REST API (using api.get effect).

arguments

  • path: path or name
  • entities: array or a single entity, from a WP REST API request
  • page: the requested page number

example

source.populate('/category/nature', posts, 1)

// That action should populate all the state below
source.data['/category/nature']
source.data["/categories/nature"].page[1]
source.data["/categories/nature"].items
source.category[2]
source.data['/the-beauties-of-gullfoss']
source.post[60]
source.data["/author/luis"]
source.author[4]
source.attachment[15]
source.attachment[16]

NOTE: when using a name instead of a path, any property to identify that name should be added explicitly:

source.populate("gallery-1", attachments, 1)

// That action should populate all the state below
source.data["gallery-1"]
source.data["gallery-1"].page[1]
source.data["gallery-1"].items
source.attachment[15]
source.attachment[16]
// ...

// This should be added explicitly
source.data["gallery-1"].isGallery = true

Effects

api.get({ endpoint, params })

Request entity to the WordPress REST API.

arguments

  • endpoint: name of the endpoint if is a /wp/v2 endpoint (e.g. posts), or the full path of other REST endpoints (e.g. /frontity/v1/discovery)
  • params: any parameter that will be included in the query params.

For more info, visit the WP REST API reference.

example

// Get posts from categories 2, 3 and 4
api.get({ endpoint: "posts", params: { _embed: true, categories: '2,3,4' } });

// Get the page 14
api.get({ endpoint: "pages", params: { _embed: true, include: '14' } });

// Other endpoints: 
api.get({ 
  endpoint: "/frontity/v1/discovery",
  params: { path: "/the-beauties-of-gullfoss" }
});

Settings

All the options that can be configured in the package settings, like if it is WordPress.org or WordPress.com, the REST base route, etc.

1 Like

Two doubts I have:


If we want to populate the attributes

totalItems
totalPages

for a list, the action source.populate is ignoring that attributes, as the entities argument doesn’t contain them.

source.populate(path, entities, page)

How would it be better to pass those attributes?

// response = { headers, body: entities }
source.populate(path, response, page)

// other options
const entities = response.body
const totalItems = response.headers["X-WP-Total"]
const totalPages = response.headers["X-WP-TotalPages"]

source.populate(path, entities, page)

source.data[path].totalItems = totalItems;
source.data[path].totalPages = totalPages;


Perhaps, since path and entity are two clearly differentiated concepts, we could leave clearer the separation with

const { type, id } = source.path["/category/nature"]
source.entity[type][id]

even changing path to name so that both things are included

const { type, id } = source.name["/category/nature"]
source.entity[type][id]

const { items, isGallery } = source.name["gallery-1"]
const entities = items.map(({ type, id }) => source.entity[type][id])

would this change worth it?

I hope that with all this I can start programming. :smile::rocket:

@luisherranz, @orballo, let me know what you think!

1 Like

Amazing work @David!!

I think I’d leave things like totalItems, totalPages, isCategory and so on to the source.fetch instead of the populate. We can include fetchedItems and fetchedPages, even isFetching like we had in the old one.

We also need to decide what is an item, what is an entity and if we need both.

Oh and we need to decide if we call it name or path.

I don’t have a strong opinion about this right now. Are entities: taxonomies, authors and post types?

Sure!! :clap::clap:

First of all, great work! :muscle:

I have some doubts:

  • In the case of custom-list, do we define what entities belong to it in a source.add() function?
  • Does the developer need to define all source.add() functions or are some already part of wp-source code? If yes, can a developer override those paths with another add()?
  • I find source.add() name a bit confusing… maybe is better to use something like register or handle?

I like source.path["/category/nature"], source.name["gallery-one"] and source.entity[type][id].

If I remember correctly, in the previous Frontity items was used only in the router, any data stored would be entities. Maybe we should use only entities in wp-source.

I like source.register :+1:

That’s a good point: how do we order all the source.registers? Maybe we need priorities or some clever way to sort them.

Hmm… the thing is that path is just a name. It just happens to start with "/" but otherwise, there’s no difference at all.

We are going to change type for taxonomy to match the API.

What about if we allow either a name or an object in the functions that accept a name and a page? Like source.fetch, source.data and router.set?

source.fetch("/category/nature");
source.fetch({ name: "/category/nature", page: 2 });

We also need to decide if we call it name or path.

@David, what do you think?