Run Webpack Configuration Per Site

Description

Right now we are running Webpack once for all sites.

Since we added the possibility of adding custom Webpack/Babel configurations, this means that packages that add a configuration for one site, leak that configuration to the rest.

It would be great to isolate those package configurations from each, and to do that we need to run Webpack once for each site.

When I refer to “run Webpack once” it is not exactly once, because each site needs 3 runs:

  • The client “module” bundle.
  • The client “es5” bundle.
  • The server bundle.

So right now we run it 3 times, each one for all sites. We need to run it 3 times per site.

This switch is also beneficial for other use cases:

  • PWA/Offline packages
    Right now these are not possible because the Webpack packages that create the service worker include all the assets of the Webpack run. Right now, that means that an offline plugin will download all the assets for all the sites, instead for just a single one. This is also solved with this change.

  • Embedded mode assets download

    – We need to have a way to easily download all the assets of a single site at once –

    – If we end up with a folder/manifest/zip that contains all the assets of a single site and it has a way to reference it, like a hash, we can embed that identifier/hash in the HTML response, and WordPress can download all the assets of that site the first time it receives an HTML file with that hash and then serve those assets from the WordPress file system. –

  • Client-only and Server-only sites

    const settings = [
      {
        name: "main-site",
        bundles: ["client", "server"]
        // ...
      },
      {
        name: "amp-site",
        bundles: ["server"]
        // ...
      }
    ];
    

To get a better overview Luis and I had a meeting about it and recorded a video. https://www.youtube.com/watch?v=Qy9y6cSEdPE

Functionalities

  • Isolate Webpack builds of one site from another to avoid leakage of Webpack/Babel configurations.

Requirements

  • End up with a single server.js file.

  • Accept the site name parameter when the dev CLI command is run.

  • Prompt for the site name if it’s not defined when the dev CLI command is run.

Future functionalities

  • Accept the site name parameter when the build CLI command is run.

Dependencies

None.

Possible solution

1. Pass --name or prompt for the name in multisite projects

  • Get rid of dev, build and serve commands in the frontity package because those commands are in the @frontity/core package. (Optional, just in case there’s nothing in the frontity command that requires to be separated from the CLI).

npx frontity dev

  • Throw in dev script (command) in @frontity/core if site is not specified and this is a multisite project, including the list of sites.
  • Capture error in dev CLI and prompt for a site.
const err = new Error("siteName missing");
err.siteNames = { ... }
throw err;
  • Then capture this error in the CLI and prompt for the name.

npx frontity build

  • Never prompts (so it doesn’t throw).

Future functionalities:

  • If it has a name as an argument, it only builds that site.

2. Generate individual server entry points

  • There needs to be a single server entry point for each site.
  • The server entry points need to include only the packages of a single site.
  • We can use site-name/server.ts.

3. Generating a Webpack/Babel/Frontity config for each site

  • In build we can iterate over the sites

    await Promise.all(
      sites.map(async (site) => {
        // ...
      })
    );
    
  • In dev we filter for the specific site

    const sites = [await getAllSites()].filter((config) => config.name === site);
    
  • We must not bundle the external dependencies of the server bundles in this step. They will be bundled in the final single server.js file.

  • We must not bundle the frontity.settings on the server bundles in this step. They will be bundled in the final single server.js file.

4. Generate a top-level Koa server that routes between sites

import { getSettings } from "@frontity/file-settings";

export default (req, res) => {
  const app = new Koa();
  app.use(async (ctx, next) => {
    const servers = await import(`./build/${name}/server.js`);
    // Get settings.
    const settings = await getSettings({
      url: ctx.href,
      name: ctx.query.frontity_name
    });
    // Logic to route between sites.
    const server = servers[settings.name];
    // Run the server...
  });
  return app.callback();
};

Webpack "Dynamic expressions in import()" docs: Module Methods | webpack

5. Bundle server.js

Finally, we generate the final server.js file, bundling the top-level Koa server with the individual server files, the external dependencies and the frontity settings.

  • The entry point of Webpack is the top-level Koa server.
  • In dev, we will do it as well to make sure that if something doesn’t work in the gateway, the developer knows as soon as possible and is able to easily debug it.
// npx frontity dev
const siteServerConfigWebpack = config.webpack.server;
const serverConfigWebpack; /* gateway server webpack config */

await webpackAsync(clientWebpack);
await webpackAsync(siteServerConfigWebpack);

// Start a custom webpack-dev-server.
const compiler = webpack([clientWebpack, serverConfigWebpack]);
// npx frontity build
const gatewayServerWebpackConfig = {};

// [...]

await Promise.all(
  sites.map(async (site) => {
    console.log("Building server bundle");
    await webpackAsync(site.webpack.server);
  })
);

console.log("Building gateway server bundle");
await webpackAsync(gatewayServerWebpackConfig);
console.log();

While working on this, was trying to figure out the best way to run our e2e suites. :grimacing: since the assumption was that we start the server in dev mode with all the sites that is not possible anymore. Should I remove the dev mode from e2e? I don’t see any possible solution so far, besides that.

Thoughts?

Me neither so you can go ahead and remove it. We have a couple of e2e tests that only run in development, but I don’t think that is a problem (frontity/emotion.spec.js at dev · frontity/frontity · GitHub), just ignore them :slightly_smiling_face: