How to resolve dynamic imports in Webpack

This is something we need to solve before we can go on.

The problems

We have packages (Settings and Frontity packages) defined dynamically, not statically:

  • Dynamically: defined in variables
    settingsPackage = "my-pkg"
  • Statically: written in code
    require("my-pkg")

For example, the Settings Package is defined in frontity.config.js like this:

// frontity.config.js
module.exports = {
  settingsPackage: '@frontity/file-settings' 
}

That means that Frontity has to do something like this to import them:

// sever.js
const config = require("./frontity.config.js")

 // ⬇️ this works in Node but not in Webpack
const getSettings = require(config.settingsPackage)

const { settings, packages } = await getSettings()
  • Node runs the code and requires on runtime, so that works.
  • Webpack analyzes the static code to generate the bundle. When it gets to that dynamic import, it doesn’t know what to import and throws an error.

Webpack has a way to overcome this problem, which is what we used in the past. It accepts a regexp or a string that it later converts to a regexp:

const config = require("./frontity.config.js")
const getSettings = require(`./settings/${config.settingsPackage}/index.js`)

const { settings, packages } = await getSettings()
package.forEach(pkg => {
  require(`./packages/${pkg.name}/index.js`)
})

If you do that, Webpack is able to analyze ./settings/${config.settingsPackage}/index.js and import all the possibilities of that regexp, no matter if they are used or not. For example:

  • ./settings/file-settings/index.js
  • ./settings/wp-org-settings/index.js

There’s a couple of problems with the way we want to do things now:

1. Webpack imports all the packages it finds in the regexp

If users forget to npm uninstall a package they are not using anymore, it will be included anyway. This is really bad and counterintuitive. In my opinion, unacceptable.

=> In the past we solved this because each package created its own chunk in the client and we didn’t care about the bundle size in the server.

2. It doesn’t work with node_modules

There’s no way to create a regexp to distinguish between Settings and Frontity packages from the rest in node_modules.

=> In the past we solved this because all packages were created in the ./packages/ folder.


Some possible solutions

1. Force a custom name for Settings or Frontity packages

For example:

  • @frontity/file-frontity-settings (official)
  • wp-org-frontity-settings
  • @frontity/saturn-frontity-theme (official)
  • mars-frontity-theme
  • @frontity/wp-org-frontity-extension (official)
  • graphql-wp-org-frontity-extension

And use a regexp like:

const getSettings = require(`./node_modules/${config.settingsPackage}-frontity-settings/index.js`)
package.forEach(pkg => {
  require(`./packages/${pkg.name}-frontity-(theme|extension)/index.js`)
})

But it’s not ideal because Webpack will include all packages, no matter if they are actually used or not :-1:

2. Analyze what packages are needed and write a file to disk

This must be done in frontity dev or frontity build before Webpack is run:

// build.js
const config = require("./frontity.config.js")
const settingsContent = `export default from ${config.settingsPackage}`
writeFile("./dynamic-settings.js", fileContent)

Then, this is what Webpack sees:

// server.js
const getSettings = require("./dynamic-settings.js")

I don’t like having to write a file to disk to solve this, but I guess it’s ok for settings. But for packages, this approach has a new problem.

The problem of solution 2

A user can have multiple packages for different sites or environments.

For example, it’s frontity.settings.js could be:

 module.exports = {
    site: "my-blog",
    matches: {
      amp: "/amp/$",
      main: "*"
    },
    settings: {
      url: "https://my-blog.com"
    },
    packages: {
      "@my-org/amp-theme": {
        matches: "amp",
        settings: {
          color: "#AAA"
        }
      },
      "@my-org/main-theme": {
        matches: "main",
        settings: {
          color: "#BBB"
        }
      },
      "@frontity/wp-org-source": {
        settings: {
          endpoint: "https://my-blog.com/wp-json/",
          perPage: 13
        }
      }
    }
  }

If we extract the packages used here and write them to disk…

  • @my-org/amp-theme
  • @my-org/main-theme
  • @frontity/wp-org-source

…the client bundle will include both the amp and the main theme. That is bad. It’ll be even worse if the user has different themes for different sites.

…the server bundle will include both but that is ok because it is responsible for rendering both the amp and the main version.

The possible solutions

1. Code split each package

If we go back to code splitting each package, the client bundles can import only the bundles they need on runtime.

There’s a couple of problems with this approach (yes! third generation problems):

  • We need to hack loadable-components to load each store before the SSR or go back to react-universal-component. It seems like react-universal-component won’t work with React Concurrent.
  • Some Frontity packages are really small (a few kbs) and we will be adding an additional HTTP call for each one, which seems unnecessary.

2. Create one entry point for each site/match

If we create a bundle for each match (amp vs main) and for each site, we could have different client bundles for each case and those will only contain the packages needed in each case. It’d be like creating a different app for each case.

It seems like an ideal solution but I don’t know if it has any other drawback.


Ok, this was longer and more complex than expected :sweat_smile:

@development-team If you want to chime in and have more ideas or crazy solutions you’re very welcome!

Ok, one problem with that approach are the Webpack’s service-workers/offline plugins. They don’t work with multiple entry points. So maybe we need to create one Webpack config per site/match instead of only an entry point.

Another one is code-splitting which is based on Webpack stats analysis. Quite probably they won’t work with multiple entry-points or Webpack configs.

Ok, this is a real stopper. We need to figure out a way to make this work.

Ok, I’ve been thinking a lot about this since yesterday. It’s clear that we need to find a compromise, the question is where.

In my opinion the compromise that leads to simpler code and maintenance of the framework and to an easier to understand concept for the developer is: all packages are included in the bundles.

It’s kind of a pity because it won’t be recommendable to have one project for several sites if those sites are very different, but it simplifies everything else and it’s easy to explain.

Main benefits

  • Code splitting with SSR just works, and we won’t create unnecessary chunks for each package.
  • Service workers & offline just works without extra effort.

Main drawbacks

  • All the packages of the app will be included in both the client and server bundles. For example, if users want to use two different themes for two different sites they need to create two apps.

Implementation

  • We have a separate frontity.packages.js file where all the packages are imported.
    => We could create this file automatically and hide it from the developer as I described in the OP, but then it’s not clear for the developer what packages are included in the bundles.
    => We can update it automatically when users install packages from the Frontity Admin, but I think it’s good that it is visible, editable and goes to the control version.
// frontity.packages.js
import saturn from "saturn-theme"
import wpOrgSource from "wp-org-source"

export default [
  saturn,
  wpOrgSource
]
  • The default frontity.config.js contains the file-settings package import.
    => This means the file-settings package will be imported always, no matter if it is used or not. It’s a small price for an easier configuration.
// default frontity.config.js
import getFileSettings from "file-settings"

export default {
  getSettings: getFileSettings 
}

// user custom frontity.config.js
import getWpSettings from "wp-org-settings"

export default config => {
  config.getSettings = getWpSettings 
}

What this means for Frontity hostings

Frontity won’t be prepared to be a monolithic app serving many different sites with different themes/packages (like our old version) but hostings can still be viable if they use a serverless approach: one set of theme/packages, one app.

What this means for our settings

Maybe, we should take this opportunity to simplify our settings because one app is going to related directly to only one site.

I’m ok with the idea of one site » one service. However I don’t understand how everything is connected.

  • When are frontity.packages.js and frontity.config.js used?
  • Does this mean that developers will have to declare their packages in parallel to having them listed in package.json?
  • We will have 3 different files, right? config, packages and settings.

I think I’m missing something else, maybe I already forgot how this would work, but aren’t we installing all of the packages with npm, but those on development that will be in packages? I don’t know if this is something that already is posted somewhere and I missed or something yet to be done.

If you install a package with npm it does nothing until you import them in your code.

In the past we used a Webpack’s dynamic import using a regexp of packages/ but we can’t do that anymore because they come now from node_modules.

We can use frontity.packages.js to list the used packages. We could create it automatically but I think it’s not a good idea because it will obscure what packages are in the bundles. With a frontity.packages.js it’s more clear what’s in it and what’s not. In a hosting this file can be created automatically. You have to create a package.json anyway. And it can be edited automatically by our Frontity Admin as well.

frontity.config.js is similar to next.config.js, for settings in general, webpack overrides, babel overrides…

Yes.

And yes :+1:

One last thing I’ve just realized: apart from the imports, we need each package name to look for their frontity.config.js files. Only the name is ok because these are needed outside of Webpack, in the build phase.

frontity.packages.js could be something like this:

import tendencyTheme from "tendency-theme";
import wpOrgSource from "@frontity/wp-org-source";
import disqusComments from "@frontity/disqus-comments";

export default {
  "tendency-theme": tendencyTheme,
  "@frontity/wp-org-source": wpOrgSource,
  "@frontity/disqus-comments": disqusComments,
};

I think @orballo is right and frontity.package.js is not a good idea (comment here).

I’ll try to make it work automagically in the bundling step.

Right now I’m thinking about these bundles:

  • One server bundle (one file), not matter how many modes.
  • One client bundle (and its chunks) per mode.
  • One inline bundle (one file) per mode.

If there’s no client.js for a mode in any of the packages, that bundle is skipped.
If there’s no inline.js for a mode in any of the packages, that bundle is skipped.

It looks like we are going to need different automatically generated files for the different entry points:

  • All the client.js of all packages for each mode.
  • All the inline.js of all packages for each mode.
  • All the server.js files of all packages. One file is fine for all modes, but it needs the mode info inside the file:
import myThemeReact from "my-theme/src/react/server.js"
import myThemeAmp from "my-theme/src/amp/server.js"
import wpSourceReact from "@frontity/wp-source/src/react/server.js"
import wpSourceAmp from "@frontity/wp-source/src/amp/server.js"

export default {
  react: {
    "my-theme": myThemeReact,
    "@frontity/wp-source": wpSourceReact,
  },
  amp: {
    "my-theme": myThemeAmp
    "@frontity/wp-source": wpSourceAmp,
  }
}

Uh, with this configuration maybe we can create a client bundle per site and per mode instead of just one per mode. I’m not sure if it’s feasible yet, I’ll think about it!

Ok, I think I got it!

I was able to generate two client bundles and switch between them at runtime.

This means we can create one bundle per site and per mode :muscle:

2 Likes

Working perfectly now :slight_smile: