This is only a draft of the Implementation Proposal. It is still open for comments and suggestions.
Description
Frontity uses a Koa server to handle the HTTP requests.
Right now, the server only handles the server-side rendering of React and a few other cases:
- Do React SSR.
- Serve static assets generated by Webpack in the
/static folder.
- Serve some special files like
favicon.ico or robots.txt.
Koa uses middleware functions for configuration. To hook into the Koa server and the SSR process of Frontity, we need to add server extensibility, which means packages are allowed to create middleware functions for Koa and expose the objects/functions used by Frontity in the SSR process so they can be modified.
Glossary
Middleware Functions
The functions of the Koa server, like this one:
app.use(async (ctx) => {
ctx.body = "Hello World";
});
Koa Context or ctx
The context that middleware functions receive. It is an abstraction on top of the req/res params of Node servers.
Full documentation can be found at https://koajs.com/.
Node-only modules
For Node-only modules I mean:
- Built-in modules, like
fs, path and so on…
- Packages that are not meant to be sent to the client like koa packages, image optimizers, and so on.
Context / Background
Why do we use Koa for the Frontity server?
We chose Koa instead of other frameworks, like Express, because:
- It is lighter (half the size of Express) so it is faster for serverless environments.
- It doesn’t have dynamic imports, so it is perfect for serverless environments.
- Its middleware functions use a
ctx abstraction that it is easier to understand than the req/res objects of Node.
- Its middleware functions use async/await for
next().
The combination of async/await for the next(), which means that the middleware functions have the opportunity to execute twice (from first to last and then from last to first) and the fact that the middleware can never send or “close” the request with something like req.send() but instead rely on mutations of the ctx properties, make Koa middleware especially good for extensibility.
Why do we need to extend the Frontity server?
Frontity packages need to be able to hook into the Frontity server for different reasons:
- Modify the SSR cycle, to do things like:
- Change the template used.
- Modify the root
<App> to add a new <Provider> for the whole app.
- Modify the HTTP response to do things like:
- Return 301 redirections.
- Modify the
Cache-Control headers.
- Add new URLs to handle things like:
- Return static assets, like
ads.txt.
- Proxy requests to other servers, like
/wp-content to WordPress.
Right now users are forced to add their own NodeJS server on top of Frontity if they want to modify/control the requests, but that approach is not ideal as I explain in the Existing Solution section.
For the modification/control of the SSR process, we are already adding the different parts to the libraries.frontity namespace as I explain in the Existing Solution section below.
Goals
The main goal is that packages can expose middleware functions that:
- Can hook and modify the Frontity SSR process.
- Can handle other types of requests, apart from the SSR.
- Can modify server paramters, like headers.
- These packages/features can be used by other Frontity users just by installing the packages.
Out of Scope
Even though this is called “Server Extensibility”, these middleware functions run only on the HTTP requests. It is out of the scope the hooking and modification of the build process:
- Modifying the settings with code.
- Modifying/extending Webpack.
- Modifying/extending Babel.
Those will be managed with code and/or configuration in the frontity.config.js files, which are planned to be run during the build process.
Existing Solution / Workaround
Controlling the Node server
Right now people that need to hook into the NodeJS server is using the function exposed in build/server.js as a middleware for a NodeJS server which runs on top.
const frontity = require("./build/server.js").default;
const http = require("http");
const server = http.createServer(frontity);
server.listen(3000);
console.log("Listening on port 3000");
That way, they can add more behavior to the NodeJS server as they please. For example, change the headers of the static assets.
const addStaticHeaders = (req, res) => {
const matched = /^\/static\//.url.match(regex);
if (matched) res.setHeader("Cache-Control", "max-age=31536000");
};
But, this is not extensible. Other users of Frontity can’t add a package that adds this functionality and configure it in their frontity.settings.js file.
Also, if people use their own framework to control these middleware functions on top of the Frontity server, they are adding unnecessary layers because Frontity is already running a server powered by Koa.
Modifying the SSR
There is an ongoing process of exposing the parts involved in the SSR in libraries.frontity as part of the AMP package, which means the SSR will be able to be modified from the init and beforeSSR actions.
For example, to modify the template you could do:
import myTemplate from "./my-template";
export default {
actions: {
myPackage: {
beforeSSR: ({ libraries }) => {
libraries.frontity.template = myTemplate;
},
},
},
};
The alternative using middleware functions would be something like:
import myTemplate from "./my-template";
export default {
server: {
myPackage: {
changeTemplate: ({ ctx }) => {
ctx.libraries.frontity.template = myTemplate;
},
},
},
};
Right now, I cannot think of anything regarding SSR that would be better managed in a middleware function as opposed to the init/beforeSSR methods or vice-versa.
Also, I don’t think we should restrict access to libraries.frontity in middleware functions, so I guess we will see over time which approach works better and/or if some things can only be done with one approach or the other.
Implementation Proposal Draft
There are different options for each part.
I am exposing all of them because I don’t have strong feelings for any of the approaches. I do have a current opinion, however, which I express later in this post.
A. Entry Points
- A1. Reuse the index/client/server.js entry points
This means we reuse the same entry points that we already have for the React part and we simply add a new property to the store. We could call it server.
// packages/my-package/src/index.js
export default {
state: {
myPackage: {
//...
},
},
actions: {
myPackage: {
//...
},
},
server: {
myPackage: {
// Add static headers for the static assets (served at publicPath).
addStaticHeaders: ({ ctx }) => {
if (ctx.href.startsWith(ctx.frontity.publicPath))
ctx.set("Cache-Control", "max-age=31536000");
},
},
},
};
Pros:
- We don’t introduce a new entry point and reuse what we already have.
- We can use the
server name, which is more intuitive than app for this use.
Cons:
- When Node-only libraries are needed,
index.js cannot be used anymore and the user needs to divide it to client.js and server.js.
The previous index.js example would need to be divided in client.js and server.js, like this:
// packages/my-package/src/client.js
export default {
state: {
myPackage: {
//...
},
},
actions: {
myPackage: {
//...
},
},
};
// packages/my-package/src/server.js
import myPackage from "./client";
import serve from "koa-static"; // Node-only package.
import { get } from "koa-route"; // Node-only package.
export default {
...myPackage,
server: {
myPackage: {
// Serve the ads.txt using the /public/ads.txt file.
adsTxt: get("/ads.txt", serve("./public"))),
},
},
};
By the way, the possibility of using the client/server.js entry points is already working in Frontity, we would not need to add it as a new feature.
If we go with the same entry points option (A1), we could use the same export than the rest of the properties (default) or we could use a separate one, like server for example.
– A1-1 Reuse default export
export default {
state: {
// ...
},
actions: {
// ...
},
server: {
myPackage: {
// ...
},
},
};
Pros:
- It feels more natural because it works just like
state, actions and libraries.
- We can keep using a single type for the whole package.
– A1-2 Different export
export default {
state: {
// ...
},
actions: {
// ...
},
};
export const server = {
myPackage: {
// ...
},
};
Pros:
-
It is isolated, which may be cleaner when refactoring to client.js and server.js is required, doing this:
// client.js
export default {
state: {
// ...
},
actions: {
// ...
},
};
// server.js
export { default } from "./client";
export const server = {
myPackage: {
// ...
},
};
Cons:
-
So far we have managed to export a single type for a whole package. If we use a separate export, we would need to do two type exports.
This would not be possible:
import SomePackage from "some-package/types";
import OtherPackage from "other-package/types";
export type Packages = MergePackages<MyPackage, SomePackage, OtherPackage>;
And we would have to do something like:
import SomePackage, {
Server as SomePackageServer,
} from "some-packages/types";
import OtherPackage, {
Server as OtherPackageServer,
} from "some-packages/types";
export type Packages = MergePackages<
MyPackage,
SomePackage,
SomePackageServer,
OtherPackage,
OtherPackageServer
>;
- A2. Create a new app.js entry point
The other option would be to use a new entry point, exclusive for the middleware functions, like app.js.
// packages/my-package/src/app.js
export default {
myPackage: {
// Add static headers for the static assets (served at publicPath).
addStaticHeaders: ({ ctx }) => {
if (ctx.href.startsWith(ctx.frontity.publicPath))
ctx.set("Cache-Control", "max-age=31536000");
},
},
};
Pros:
- People don’t need to decide when to divide between
client/server.js, they will just learn to use app.js.
Cons:
- It cannot be named
server.js because that name is already been used when you need to divide index.js in both client and server code, for the case when beforeSSR contains Node-only code, so this would be confusing:
– “Is not server.js supposed to be the file for the server??”
– “Nope…” 
- If we use a separate entry point, we would need to do two type exports.
B. Namespaces
- B1. Use namespaces
We can promote the use of namespaces as we do for other store properties.
export default {
state: {
myPackage: {
//...
},
},
actions: {
myPackage: {
//...
},
},
server: {
myPackage: {
// ...
},
},
};
Pros:
- Packages will be able to reuse or overwrite the middleware functions exposed by other packages.
Cons:
- B2. Don’t use namespaces
Or we can just expose app and let packages add middleware as they please.
export const server = (app) => {
app.use(({ ctx }) => {
// ...
});
app.use(({ ctx }) => {
// ...
});
};
Pros:
Cons:
- Less extensible/hackable because middleware functions will not be accessible.
C. Expose app or abstract app.use()
Koa works with an app instance that has a small API:
-
app.use(middlewareFunction): Add middleware functions to the server.
-
app.callback(): Return a callback function suitable for the http.createServer() method to handle a request. You may also use this callback function to mount your Koa app in a Connect/Express app.
-
app.listen(): Starts the server.
-
app.env: The same value than process.env.NODE_ENV.
-
app.proxy: Setting to trust proxy headers.
-
app.subdomainOffset: Setting to ignore subdomains.
-
app.keys: Set signed cookie keys.
-
app.context: The contect ctx prototype. Useful to add things that can be accessed later in the middleware functions, like ctx.libraries.
-
app.silent: Turn off error output to stderr.
-
app.on(): Listen to events. For example listen to "error".
From those, the main one packages will need to use 99% of the time is app.use().
These are not required at all:
app.callback()app.listen()app.env-
app.proxy (this on will be turned on by default)
And these may be useful at some point, but it is going to be quite rare.
app.subdomainOffsetapp.keysapp.contextapp.silentapp.on()
- C1 Expose app
We could expose the app itself and let packages use app.use() explicitly.
This could be the way to do so if we use the same entry points (A1) and namespaces (B1):
export default {
// ...
server: {
myPackage: {
myMiddlewares: (app) => {
// Configure app here:
app.silent = true;
app.context.db = new Database();
app.use((ctx) => {
// ...
});
},
},
},
};
And this could be the way to do so if we use a separate entry point (A2) and no namespaces (B2):
// packages/my-package/src/app.js
export default (app) => {
// Configure app here:
app.silent = true;
app.context.db = new Database();
app.use((ctx) => {
// ...
});
app.use((ctx) => {
// ...
});
};
Pros:
- Only one API for managing
app and app.use
Cons:
-
We cannot pass ctx as part of an object, like this:
app.use(({ ctx }) => {
// ...
});
So if want to add more things, like the store, we would need to do so inside ctx.
app.use((ctx) => {
const { state, libraries } = ctx;
// ...
});
Destructuring is not straightforward when you need to mutate a prop of the context, like ctx.status or ctx.body, so this could be a bit confusing.
app.use(({ state, libraries, ...ctx }) => {
ctx.status = 200;
ctx.body = "Hi mum!";
});
As opposed to:
app.use(({ ctx, state, libraries }) => {
ctx.status = 200;
ctx.body = "Hi mum!";
});
- C2 Abstract app.use
We can accept funtions where we abstract app.use() away and find an alternative way to expose the app for the uses cases where packages want to hook in the other APIs/settings.
export default {
// ...
server: {
myPackage: {
myMiddleware: ({ ctx, next }) => {
// This is `app.use(ctx, next) => ...`
// but abstracted within the store.
},
},
},
};
Pros:
- It is less verbose.
- It will cover 99% of the server extensibility use cases.
- We can run our own logic before or after the middleware function, so it is possible to pass an object with more elements than only
ctx in the first parameter, and maybe will simplify things if we ever add DevTools for this part of Frontity.
Cons:
- We have to find an alternative way to expose the
app.
To expose the app, I think we have two options.
– C2-1 Expose app in the ctx
We could simply expose app inside the middleware functions:
export default {
state: {},
actions: {},
server: {
myPackage:
myMiddleware: ({ ctx, app }) => {
// Configure app here:
app.silent = true;
app.on("error", () => {
// Do something...
});
},
},
},
};
I am not sure that app settings can be configured inside middleware functions, so if we choose this option we would have to check that out.
– C2-2 Expose app in the package creation
Package modules are usually an object, but they can also be a function. That means we have the opportunity to pass app in that call:
export default ({ app }) => {
// Configure app here:
app.silent = true;
app.on("error", () => {
// Do something...
});
return {
state: {},
actions: {},
server: {},
};
};
D. Keep app.use(ctx, next) or pass an object like ({ ctx, next })
- D1 Keep the app.use(ctx, next) API
We can keep the exact same API than app.use() is using:
app.use((ctx, next) => {
const { state, libraries } = ctx;
// ...
});
This is required if we don’t abstract app.use (C1).
This would mean the store would be part of the ctx object.
export default {
// ...
server: {
myPackage: {
myMiddleware: (ctx) => {
// Access state, actions, libraries or even server.
const { state, server } = ctx;
if (state.frontity.name === "my-site") {
// Delete the middleware from other package...
delete server.otherPackage.someMiddleware;
}
},
},
},
};
Cons:
- People could use destructuring, but it is not straightforward when they need to mutate a prop of the context, like
ctx.status or ctx.body, so this could be a bit confusing. This was explained previously in the C1 section.
- D2 Pass an object like ({ ctx, next })
Or we can pass an object that contains ctx, next and maybe other things, like the store.
app.use(({ ctx, next, state, libraries }) => {
// ...
});
If we use namespaces, this would be:
export default {
// ...
server: {
myPackage: {
myMiddleware: ({ ctx, state, server }) => {
// Access state, actions, libraries and now even server.
if (state.frontity.name === "one-of-my-sites") {
delete server.otherPackage.someMiddleware;
ctx.status = 200;
ctx.body = "Hi mum!";
}
},
},
},
};
Cons:
-
There are a lot of packages for Koa packages and sometimes they are passed directly to the app.use() function, like this:
const ratelimit = require("koa-ratelimit");
export default {
// ...
server: {
myPackage: {
rateLimiter: ratelimit(options),
},
},
};
These packages expect to receive (ctx, next), so they won’t work with ({ ctx, next }) directly.
The way to use them would be this:
export default {
// ...
server: {
myPackage: {
rateLimiter: ({ ctx, next }) => ratelimit(options)(ctx, next),
},
},
};
- D3 Expose both APIs
Finally, we could expose both APIs because they are compatible with each other.
The trick is to do add a reference to the ctx in the ctx itself. Something like this:
const middlewareWrapper = (middlewareFunction) => {
app.use((ctx, next) =>
middlewareFunction(
{
ctx, // Pass ctx.
next, // Pass next.
...store, // Pass state, actions, libraries...
...ctx, // And also spread the ctx itself to copy the original API.
},
next // Pass next as second argument to copy the original API.
)
);
};
I don’t know if genearting a new ctx object is necessary, so this is just an example. I have assumed it is because the reference to the next object needs to be kept in the function’s closure. It will depend on how Koa builds the ctx object that passes to the middleware functions.
Then, these two options are possible:
app.use(({ ctx, next, state, libraries }) => {
// ...
});
app.use(ratelimit(options));
If we use namespaces, this would be:
export default {
// ...
server: {
myPackage: {
myMiddleware: ({ ctx, state }) => {
// Do stuff with ctx, state and so on...
},
rateLimiter: ratelimit({
// Add koa packages directly.
}),
},
},
};
Pros:
- We retain the same API we use in actions and derived state.
- We allow people to use koa packages directly.
Cons:
- People could get confused when debugging the app because they will see
ctx repeated: ctx.ctx, ctx.body, ctx.ctx.body and so on…
- Some of the packages are not so straightforward. For example, when combining
get with serve, this will work:
export default {
// ...
server: {
myPackage: {
serveSomeStatics: get("/public/:file", serve("./public")),
},
},
};
But if you want to use get for your own logic, you may end up back to the app.use(ctx, next) API, because that is the standard in Koa and therefore what get accepts in its callback:
export default {
// ...
server: {
robots: {
serveRobotsTxT: get("/robots.txt", async (ctx, next) => {
// Serve the robots.txt found in the root.
if (await promisify(exists)("./robots.txt")) {
await serve("./")(ctx, next);
} else {
// Output a default robots.txt.
ctx.type = "text/plain";
ctx.body = "User-agent: *\nAllow: /";
}
}),
},
},
};
So if you need access to the store, you need to access it through ctx.
export default {
// ...
server: {
robots: {
serveRobotsTxT: get("/robots.txt", async (ctx, next) => {
ctx.type = "text/plain";
if (await promisify(exists)("./robots.txt")) {
// Serve the robots.txt found in the root.
await serve("./")(ctx, next);
} else if (ctx.state.robots.txt) {
// Output the robots.txt that is stored in the state.
ctx.body = ctx.state.robots.txt;
} else {
// Output the default robots.txt.
ctx.body = "User-agent: *\nAllow: /";
}
}),
},
},
};
E. Share utils using libraries or ctx
We have two options when it comes to enabling middleware functions to share utils with each other.
- E1 Promote sharing utils through libraries
Expose utils in libraries, like the way we do so in the React side.
// Other Package.
export default {
// ...
libraries: {
otherPackage: {
processHeaders: (headers) => {
// Do stuff...
},
},
},
};
And consume them in the middleware functions.
// My Package.
export default {
// ...
server: {
myPackage: {
myMiddleware: ({ ctx, libraries }) => {
// Use utilities from other packages.
libraries.otherPackage.processHeaders(ctx.headers);
},
},
},
};
Pros:
- It is the standard way in Frontity.
- E2 Promote sharing utils through ctx
Expose utils in ctx. This requires one of the methods to expose app to be able to access app.conext.
// Other Package.
export default ({ app }) => {
app.context.processHeaders: (headers) => {
// Do stuff...
};
};
Consume them in the middleware functions.
// My Package.
export default {
// ...
server: {
myPackage: {
myMiddleware: ({ ctx }) => {
// Use utilities from other packages.
ctx.processHeaders(ctx.headers);
},
},
},
};
Pros:
- It is the standard way in Koa.
Cons:
- I guess middleware functions will end up consuming things from
libraries which are also exposed to the React app anyway, so that would mean we have two ways to do the same thing.
Personal preference
Before writing the whole Implementation Proposal Draft I was more inclined towards using a separate entry point, like app.js, and no namespaces.
Now, after seeing the whole picture together, I think it makes more sense to keep this as close to the React app as possible, and fulfill these design principles:
- Frontity should have as few concepts as possible.
- Avoid introducing new concepts. Instead, reuse the same concepts.
- Prioritize using the same concept in different areas over having nicer APIs.
- Frontity should be as extensible by default as possible.
- Frontity should have a simple surface API but a hackable low-level API.
So I would go with:
- A1: Use the same entry points (because of 1.)
- A1-1: Reuse the
default export (because of 1.)
- B1: Use namespaces (because of 1. and 2.)
- C2: Abstract
app.use() (because of 3.)
- C2-2: Expose
app in the package creation (because of 3.)
- D3: Use both
(ctx, next) and ({ ctx, state }) APIs (because of 1. and 3.)
- E1: Share utils using
libraries (because of 1.)
This approach still has an inconsistency with another API: the beforeSSR action when it needs to use ctx.
Right now that API is:
export default {
actions: {
myPackage: {
beforeSSR: ({ state }) => ({ ctx }) => {
// ...
},
},
},
server: {
myPackage: {
someMiddleware: ({ state, ctx }) => {
// ...
},
},
},
};
To make it consistent with the middleware functions, we would have to migrate it to:
export default {
actions: {
myPackage: {
beforeSSR: ({ state, ctx }) => {
// ...
},
},
server: {
myPackage: {
someMiddleware: ({ state, ctx }) => {
// ...
},
},
},
},
};
Thankfully, this can be done with backward compatibility.
Another option would be to deprecate this beforeSSR option altogether and move the logic that needs to deal with ctx to middleware functions. After all, we passed ctx to beforeSSR as an easy fix because we didn’t have server extensibility yet.
Feedback and remaining work
All feedback is welcomed so please go ahead and share yours 
Once we make a decision about the final Implementation Proposal, I will finish it adding:
- The final requirements.
- The test plan draft.
- The individual tasks for the final implementation.
- The suggestion for the documentation.
Please, bear in mind that this section wasn’t part of the opening post. It has been added afterwards and its purpose is to keep a quick summary with the most relevant information about this FD at the top of the thread.
This will make Frontity amazingly extensible.