Frontity Connect - First thoughts and implementation

I think it’d be amazing if we could be the creators of our own state manager solution because that would mean we would have absolute control over its implementation and we could craft its API to match our extensibility pattern perfectly.

These are the things we need:

  • Full Typescript support - Overmind & MobxStateTree

  • Zero boilerplate - Overmind & MobxStateTree

  • Easy to learn - Overmind

  • State as a plain javascript object - Redux & Overmind

  • State mutations (devs don’t have to deal with immutability) - Overmind & MobxStateTree

  • Async actions using async/await - Overmind

  • Serializable snapshots - MobxStateTree & Redux

  • Serializable actions - MobxStateTree & Redux

  • Serializable mutations (patches) - MobxStateTree

  • Minimum rerender guarantee - MobxStateTree

  • Avoid breaking object/array references on mutations - MobxStateTree

  • Deterministic mutation (patches) - MobxStateTree

  • Middleware support for actions - Redux & MobxStateTree

  • Middleware support for mutations - none

  • Filter support - none

  • Devtools for actions and state - Redux & Overmind

  • Devtools for components - Overmind

  • Listen to state mutations - MobxStateTree & Overmind

  • Listen to actions from different namespaces - Redux & MobxStateTree

  • Nested derived state - MobxStateTree

  • Small bundle size - Redux & Overmind

I’ve been giving this a lot of thought and we don’t need to maintain immutability, so immer is not needed.

The only way I can imagine to manage the patches with async/await with es5 support is to use an fx function:

const myAction = async ({ state, effects }) => {
  state.user = { name: "Jon" };
  const surname = await effects.getSurname();
  state.user.surname = surname // <- this does not trigger a patch
}

There’s no way to know that the async function has resumed and mutated state.user.surname in es5.

Besides, even in es6 we don’t know when the function is awaiting, so we’d have to do the tracking asynchronously with a setTimeout.

If we use a function to await, we can make synchronous blocks and deep-equals in es5:

const myAction = async ({ state, effects, fx }) => {
  // -> sync block 1 starts
  state.user = { name: "Jon" };
  const surname = await fx(
     // -> sync block 1 ends here, emits patch for 'state.user.name'
    effects.getSurname()
    // -> sync block 2 starts when effect.getSurname returns
);
  state.user.surname = surname;
  // -> sync block 2 ends on return, emits patch for 'state.user.surname'
}

We can do deep-equals on block finish if we are in es5 mode to get the lost mutations.

The only drawback of this is that the middleware for patches run after the sync block ends, instead of at the moment the patch is created. That can lead to problems in filters, like this:

// Filter for myPost.title
const onPatch({ path, value }) => {
  if (path === "myPost.title")
    value = `My blog - ${value}`;
}

const myAction = ({ state, effects }) => {
  // sync block starts
  state.myPost.title = "My post"; // <- this doesn't trigger onPatch until the block ends
  effects.saveTitle(state.myPost.title) // <- wrong value
  // sync block ends, now state.myPost.title path is emitted and value is updated
}

Blocks are useful to batch mutations in React and trigger only one render per block, but patch middleware should be synchronous. The fx function doesn’t solve the problem on es5.

const myAction = async ({ state, effects }) => {
  state.user = { name: "Jon" };
  const surname = await effects.getSurname();
  state.user.surname = surname // <- this does not start a new block
}

We could use an optional babel plugin to support es5 and transpile the code above to:

const myAction = async ({ state, effects, patch }) => {
  state.user = { name: "Jon" };
  patch(); // <- deep-equal, triggers 'state.user' patch
  const surname = await effects.getSurname();
  patch(); // <- deep-equal, no changes
  state.user.surname = surname
  patch(); // <- deep-equal, triggers 'state.user.name' patch
}

One deep-equal per line basically. It’d be slow. Introducing immutability would make it faster but it’s a bit overkill to maintain it only for this.

We can use effects and the babel plugin to finish blocks:

// We use effects.getSurname call to finish block 1 and start block 2
const myAction = async ({ state, effects }) => {
  state.user = { name: "Jon" };
  const surname = await effects.getSurname(); // <-
  state.user.surname = surname
}
// We use the babel plugin to insert fx
const myAction = async ({ state, effects }) => {
  state.user = { name: "Jon" };
  const surname = await fx(effects.getSurname()); // <- babel adds fx
  state.user.surname = surname
}

We fallback to setTimeout(finishBlock, 1) in case block is not finished.

// Neither babel plugin or effects are used
const myAction = async ({ state, effects }) => {
  // Block 1 starts.
  state.user = { name: "Jon" };
  const surname = await fetch("/surname");
  // setTimeout finishes block 1
  state.user.surname = surname // <- patch starts block 2 (only es6)
  // return finishes block 2
}

Another consideration is that if we don’t use a fx function we don’t know that a sync block has started until a new mutation is made. In that case, the only info we have is inside the proxy, so we need a different proxy per action.

Another idea: we maintain two states, one mutable and one immutable. We use immer for that because it minimizes the number of patches. This is great because immer does the heavy lifting and it’s a battle tested library.

Let’s call them state (mutable) and snapshot (immutable).

  • Each action get a deep proxified object created using the mutable state. We only use set.
  • Each time the state is mutated, we use immer to start a draft, mutate it in the same path and close it. We get both the patches and the next immutable snapshot. We also get an immutable snapshot of the current state.
  • We use the immer patches to trigger the middleware.
  • If the middleware doesn’t abort, we store the snapshot as the current snapshot and finish the mutation of state. Both are now in sync.
  • When that block finishes, we minimize the patches (trick here) and trigger the flush middleware with the patches and the last snapshot. One of such middleware is React.

Immutable snapshots can be used to update React. It’s what Redux does with connect. immer minimizes the patches and therefore avoids unnecessary re-renders. It still not perfect as users can do things like:

connect(({ state }) => ({
   // generating a new array each time breaks immutability.
  activeTodos: state.todos.filter(todo => todo.active),
});

So I still think something like memoize-state to solve these cases would be great.

Thanks to the mutable state, we never lose internal references because the internal objects never change, only mutate. Things like this won’t work with immutable objects because the references changes:

const { user } = state;
const result = await fetch("/user");
// If we use immutables here this reference may break when user
// is updated while the fetch is going on.
user.name = result.name 

We get the best of both worlds and the tricky parts are managed by immer.

Another cool thing is that we can add the derived state to the mutable state and it won’t interfere with the immutable state, which is kept serializable. I have an idea on how to solve that part although I’m not sure how it would work with typescript yet.

I also think it’s a good idea to use the fx function for awaits. If later on we discover that setTimeout is good enough, we can easily deprecate it. The other way around would be a mess for the users.

Maybe instead of fx it could be wait or waitFor.

EDIT: I’ve changed my mind :sweat_smile: There’s no way to make ie11 work with instant patches, so I’ve been thinking that we should go back to the idea of the AMP fallback for ie11, avoid the fx function and try to enqueue actions to simulate a synchronous behaviour (using setTimeout in the background) to make it work as clean for the developer as possible.

If we go that route, I think the best approach is to create separate libraries for each part. The first one would be something like immer-await to make immer support to async/await.

The API can be something like:

import { create } from "immer-await";

// This the mutable one.
const state = { user: { name: "John", surname: "Snow" } }; 

const { dispatch, onMutation, onBlock } = create(state);

onMutation((snapshot, patch) => { /* do something */ })
onBlock((snapshot, patches) => { /* do something */ })

dispatch(async state => {
  const { user } = state;
  user.name = "Jon"; // 1. Emits mutation
  await delay(1000);  // 2. setTimeout finishes the block.

  // Starts a new block when first mutation is detected.
  // Finishes the current block if setTimeout hasn't been triggered yet.
  user.surname = "Taergarean"; // 5. Emits mutation.
  effect.save(user.id) // 6. References are never lost thanks to mutable object.
  // 7. Returning a function finishes its block.
})

// Waits until the current block has finished
dispatch(async state => {
  state.user.id = 1; // <- 3. Emits mutation.
  // 4. Returning a function finishes its block.
})

Ok, let’s talk about the derived state. There’s no problem with the code, it can be done. The challenge is to find an API that works with Typescript.

With the Overmind API:

type State = {
  user: {
    name: string;
    surname: string;
    fullname: string;
  }
}

const state = {
  user: {
    name: "Jon",
    surname: "Snow",
    get fullname() {
     return `${this.name} ${this.surname}`;
    }
}

we don’t have access to the snapshot type, which is only:

type State = {
  user: {
    name: string;
    surname: string;
  }
}

There no way to know that fullname is derived state from the whole typing.

So we need a separate object, something we can merge later, like:

type State = {
  user: {
    name: string;
    surname: string;
  }
}

type Derived = {
  user: {
    fullname: string;
  }
}

const state = {
  user: {
    name: "Jon",
    surname: "Snow",
  }
}

const state = {
  user: {
    get fullname() {
      return `${this.name} ${this.surname}`;
    }
  }
}

With separate types we can create the type of the final state we pass in actions with State & Derived and use only State for the snapshots we return in the middleware.

When we apply a snapshot with applySnapshot, we can simply apply it to the current snapshot, get the patches, mutate the state and merge the derived object again. So the not-nested derived state would be fine this way.

The challenge comes with the nested derived state, like objects in arrays:

const state = {
  users: [
    {
      id: 1,
      name: "Jon",
      surname: "Snow",
      get fullname() {
        return `${this.name} ${this.surname}`;
      }
    }
  ] 
}

If you add a new user, it won’t have fullname:

const addUser = ({ state }) => {
  state.users.push({ id: 2, name: "Tyrion", surname: "Lanniester" });
}

Instead of using the derived object we could use the patch middleware:

onPatch(({ state, patch }) => {
  if (patch.path[1] === "users") { // <- for example state.users[1]
    Object.defineProperty(state.users[path[2]], 'fullname', {
      get: function() { return `${this.name} ${this.surname}`; }
    });
  }
})

It’s pretty ugly but the important thing is that it can be done. We can create a nicer API on top.

The problem is Typescript.

If the type is defined like this:

type User = {
  id: number;
  name: string;
  surname: string;
  fullname: string;
};

type State = {
  users: User[];
};

then this fails:

state.users.push({ id: 2, name: "Tyrion", surname: "Lanniester" });

because it doesn’t contain a fullname. So we need the derived types to be optional:

type User = {
  id: number;
  name: string;
  surname: string;
  fullname?: string;
};

type State = {
  users: User[];
};

which are not.

So, that is the point where I am stuck right now :slight_smile:

We also have to find an API for the derived object that has the exact same typing than the final typing that would be merged with state.

For this derived typing:

// State
type User = {
  id: number;
  name: string;
  surname: string;
};
type State = {
  users: User[];
};

// Derived
type DerivedUser = {
  fullname: string;
};
type Derived = {
  users: DerivedUser[];
  numberOfUsers: number;
};

a possible API could be something like:

const state: State = {
  users: []
};

const derived: Derived = {
  users: [{
    get fullname() { return `${this.name} ${this.surname}`; } // <- TS error
  }],
  get numberOfUsers() { return this.users.length; } // <- This is fine
}

This API is simple and elegant and almost works with TS.

We can then analyze the derived object and when we find an array, extract the props of that first element and merge them whenever there is a push to that array.

Another problem is that we need to find a way to make it work with objects:

type User = {
  id: number;
  name: string;
  surname: string;
};
type State = {
  users: {
    [key: string]: User
  };
};

// Derived
type DerivedUser = {
  fullname: string;
};
type Derived = {
  users: {
    [key: string]: DerivedUser
  };
  numberOfUsers: number;
};

const state: State = {
  users: {}
};

const derived: Derived = {
  users: {
    get fullname() { return `${this.name} ${this.surname}`; } // <- TS error
  },
  get numberOfUsers() { return this.users.length; } // <- This is fine
}

Because you don’t know how to analyze them vs other objects.

More things that need some thought: we need derived state functions and access to root state for getters.

MobxStateTree solves it with generating functions that receive self:

const Model = types.model({
  title: 'My awesome title'
}.views(self => ({
  get upperTitle() { return self.title.toUpperCase(); }
}));

And Overmind with

type State = {
  title: string
  upperTitle: Derive<State, string>
}

const state: State = {
  title: 'My awesome title',
  upperTitle: state => state.title.toUpperCase()
}

The thing is… Overmind inherits the string type for the upperTitle when the state is passed to the user, so I should probably take a look at that.

I’ve been also thinking that memoize-state can’t be used as it is because proxifing a big object hundreds of times (once per React component and once per derived state) is going to be expensive for sure.

Instead of that, we could use proxyEqual to reuse the proxified state, store the tracked data and reset the proxy to be reused for the next component or derived state. Then, update the object on each block finish.

The requirement for that is that it must be synchronous but I don’t think components, even in concurrent mode are going to await, right? Obviously, they can trigger async functions but those won’t be tracked.

By the way, we could even have an onTracked or onObserved middleware! :sweat_smile:

By the way, I forgot to update here that this statement is wrong:

Immutable snapshots can be used to update React. It’s what Redux does with connect . immer minimizes the patches and therefore avoids unnecessary re-renders.

Immutables are an efficient way to update React but they still make unnecessary re-renders. Take for example:

const Users = ({ users }) => (
   users.map(user => <User user={user} />)
);

const mapStateToProps = state => ({
  users: state.users // <- this is an array
});

export default connect(mapStateToProps)(Users);

Here, each time some user changes inside users, the whole array gets replaced and therefore this parent component is re-rendered unnecessary.

I have an interesting proposal for the derived state that is flexible and simple for the user both in JS and TS and supports nested derived state.

Derived State

First, the derived state is like this, inspired by Overmind:

const state = {
    users: [],
    numberOfUsers: ({ state }) => state.users.length
}

// later on...
state.users.push({ name: "Jon", surname: "Snow" });
state.numberOfUsers; // === 1

They get the state and return the value.

Types are like this:

type State = {
  users: User[],
  numberOfUsers: DerivedState<State, number>
}
  1. First argument: the state.
  2. Third argument: the returned type.

I agree with the Overmind designer (Christian Alfoni) that MobxStateTree references are not really needed because you can store references in the state and derive the real value in other property like this:

const state = {
    users: {},
    selectedUserId: 1,
    selectedUser: ({ state }) => state.users[state.selectedUserId]
}

// later on...
state.users[1] = { name: "Jon", surname: "Snow" };
state.selectedUserId = 1;
state.selectedUser.name // === "Jon"

Derived Functions

But Overmind doesn’t support derived functions, and those are quite useful. My proposal is that if your derived state returns a function, then it is a derived function:

const state = {
    users: [],
    userById: ({ state }) => id => state.users.find(user => user.id === id)
}

// later on...
state.users.push({ id: 1, name: "Jon", surname: "Snow" });
state.userById(1).name // === "Jon"

The types are:

type State = {
  users: User[],
  userById: DerivedFunction<State, number, User>
}
  1. First argument: the state.
  2. Second argument: the input of the function.
  3. Third argument: the returned type.

Nested Derived State

Now the tricky part, derived state nested in arrays or objects that are populated dynamically.

My idea is, instead of users: [], use a function like users: array(...). Inside the array() or object() goes a generator function that receives the non-derived state and return the combination of non-derived and derived state. Like this:

const user = ({ name, surname }) => ({
    name,
    surname,
    fullname: ({ self }) => `${self.name} ${self.surname}`
})

const state: State = {
    users: array(user)
}

// later on...
state.users.push({ name: "Jon", surname: "Snow" });
state.users[0].fullname // === "Jon Snow"

We need self along with state to access the item itself. Maybe the index as well. For example:

const user = ({ name, surname }, index }) => ({
    name,
    surname,
    message: ({ state, self }) =>
      `${self.name} is the number ${index} of ${state.app}`
})

The types are:

type User = {
    name: string;
    surname: string;
    fullname: DerivedState<State, User, string>
}

type State = {
    users: User[]
}

So it’s not complex. Same for objects:

const user = ({ name, surname }) => ({
    name,
    surname,
    fullname: ({ self }) => `${self.name} ${self.surname}`
})

const state: State = {
    users: object(user)
}

// later on...
state.users.jonSnow = { name: "Jon", surname: "Snow" };
state.users.jonSnow.fullname // === "Jon Snow"

The types are:

type User = {
    name: string;
    surname: string;
    fullname: DerivedState<State, User, string>
}

type State = {
    users: { [key: string]: User }
}

And it can receive the key instead of the index:

const user = ({ name, surname }, key) => ({
    name,
    surname,
    message: ({ state, self }) =>
      `${self.name} with key ${key} of ${state.app}`
})

I’ve tested part of the code and I think it’s doable with proxies and Typescript with something like this, that joins the input of the generator with an optional version of the output:

const array = <U, R>(generator: (item: U) => R): (U & Partial<R>)[] => {
  // ...
}

// later on...
state.push({ name: "Jon", surname: "Snow" }) // fullname is optional

I’ve tested this code and it works great:

const state = {
  number: 1,
  numberPlusOne: state => state.number + 1,
  addNumber: state => number => state.number + number
};

const proxy = new Proxy(state, {
  get: (target, prop) =>
    typeof target[prop] === "function" ? target[prop](state) : target[prop]
});

proxy.number; // === 1
proxy.numberPlusOne; // === 2
proxy.addNumber(2); // === 3

Codesandbox: https://codesandbox.io/s/qv5jnq3q0q

Things I still need to work on:

  • :x: – How to get typings from the state and actions of other packages.
  • :x: – How to use connect and memoize-state together for React rendering.

I’ve discovered that memoize-state expects an immutable, so we need to pass it the each new immutable mixed with the derived state. I’m not sure if that’s a good idea, maybe we should use proxyequal instead.

It looks like neither memoize-state or proxyequal are good ideas because they are designed with immutability in mind and we need the observer part (React) to contain derived state in the same object. I’m checking other more “pure proxy” approaches.

For some reason I was mistaken and this is not true. I really wonder why I was so sure about it in the first place: https://github.com/mweststrate/immer/issues/324

I’ve found a library that does minimize the patches: https://github.com/Starcounter-Jack/JSON-Patch and I have proposed it to improve immer. Let’s see what they say, although I’m not convinced they want to go that route.