Frontity Connect - First thoughts and implementation

I have an interesting proposal for the derived state that is flexible and simple for the user both in JS and TS and supports nested derived state.

Derived State

First, the derived state is like this, inspired by Overmind:

const state = {
    users: [],
    numberOfUsers: ({ state }) => state.users.length
}

// later on...
state.users.push({ name: "Jon", surname: "Snow" });
state.numberOfUsers; // === 1

They get the state and return the value.

Types are like this:

type State = {
  users: User[],
  numberOfUsers: DerivedState<State, number>
}
  1. First argument: the state.
  2. Third argument: the returned type.

I agree with the Overmind designer (Christian Alfoni) that MobxStateTree references are not really needed because you can store references in the state and derive the real value in other property like this:

const state = {
    users: {},
    selectedUserId: 1,
    selectedUser: ({ state }) => state.users[state.selectedUserId]
}

// later on...
state.users[1] = { name: "Jon", surname: "Snow" };
state.selectedUserId = 1;
state.selectedUser.name // === "Jon"

Derived Functions

But Overmind doesn’t support derived functions, and those are quite useful. My proposal is that if your derived state returns a function, then it is a derived function:

const state = {
    users: [],
    userById: ({ state }) => id => state.users.find(user => user.id === id)
}

// later on...
state.users.push({ id: 1, name: "Jon", surname: "Snow" });
state.userById(1).name // === "Jon"

The types are:

type State = {
  users: User[],
  userById: DerivedFunction<State, number, User>
}
  1. First argument: the state.
  2. Second argument: the input of the function.
  3. Third argument: the returned type.

Nested Derived State

Now the tricky part, derived state nested in arrays or objects that are populated dynamically.

My idea is, instead of users: [], use a function like users: array(...). Inside the array() or object() goes a generator function that receives the non-derived state and return the combination of non-derived and derived state. Like this:

const user = ({ name, surname }) => ({
    name,
    surname,
    fullname: ({ self }) => `${self.name} ${self.surname}`
})

const state: State = {
    users: array(user)
}

// later on...
state.users.push({ name: "Jon", surname: "Snow" });
state.users[0].fullname // === "Jon Snow"

We need self along with state to access the item itself. Maybe the index as well. For example:

const user = ({ name, surname }, index }) => ({
    name,
    surname,
    message: ({ state, self }) =>
      `${self.name} is the number ${index} of ${state.app}`
})

The types are:

type User = {
    name: string;
    surname: string;
    fullname: DerivedState<State, User, string>
}

type State = {
    users: User[]
}

So it’s not complex. Same for objects:

const user = ({ name, surname }) => ({
    name,
    surname,
    fullname: ({ self }) => `${self.name} ${self.surname}`
})

const state: State = {
    users: object(user)
}

// later on...
state.users.jonSnow = { name: "Jon", surname: "Snow" };
state.users.jonSnow.fullname // === "Jon Snow"

The types are:

type User = {
    name: string;
    surname: string;
    fullname: DerivedState<State, User, string>
}

type State = {
    users: { [key: string]: User }
}

And it can receive the key instead of the index:

const user = ({ name, surname }, key) => ({
    name,
    surname,
    message: ({ state, self }) =>
      `${self.name} with key ${key} of ${state.app}`
})

I’ve tested part of the code and I think it’s doable with proxies and Typescript with something like this, that joins the input of the generator with an optional version of the output:

const array = <U, R>(generator: (item: U) => R): (U & Partial<R>)[] => {
  // ...
}

// later on...
state.push({ name: "Jon", surname: "Snow" }) // fullname is optional

I’ve tested this code and it works great:

const state = {
  number: 1,
  numberPlusOne: state => state.number + 1,
  addNumber: state => number => state.number + number
};

const proxy = new Proxy(state, {
  get: (target, prop) =>
    typeof target[prop] === "function" ? target[prop](state) : target[prop]
});

proxy.number; // === 1
proxy.numberPlusOne; // === 2
proxy.addNumber(2); // === 3

Codesandbox: https://codesandbox.io/s/qv5jnq3q0q

Things I still need to work on:

  • :x: – How to get typings from the state and actions of other packages.
  • :x: – How to use connect and memoize-state together for React rendering.

I’ve discovered that memoize-state expects an immutable, so we need to pass it the each new immutable mixed with the derived state. I’m not sure if that’s a good idea, maybe we should use proxyequal instead.

It looks like neither memoize-state or proxyequal are good ideas because they are designed with immutability in mind and we need the observer part (React) to contain derived state in the same object. I’m checking other more “pure proxy” approaches.

For some reason I was mistaken and this is not true. I really wonder why I was so sure about it in the first place: https://github.com/mweststrate/immer/issues/324

I’ve found a library that does minimize the patches: https://github.com/Starcounter-Jack/JSON-Patch and I have proposed it to improve immer. Let’s see what they say, although I’m not convinced they want to go that route.

Another idea: what if, instead of comparing the values to get the patches, it iterates over the received object and adds the leaves one by one:

state.user = { name: "Jon", surname: "Targaryen" };
// This triggers a "set" with:
target = state
prop = "user"
value = { name: "Jon", surname: "Targaryen" };

// And runs something like:
for (let key in value) {
  target[prop][key] = value[key]
}

This can happen only if there is an existing object in state.user. If it’s not, it just adds it.

I’m also looking at @nx-js/observer-util and a view made for React to substitute memoize-state.

I’ve found a filter API that I kind of like and it’s possible with observer-util.

This is the implementation:

const filter = (
  trigger: ({ state }: { state: State }) => void,
  mutation: ({ state, initial }: { state: State; initial: any }) => void
) => {
  let initial = trigger({ state });
  const reaction = observe(() => trigger({ state }), {
    scheduler: () => {
      const current = trigger({ state });
      unobserve(reaction);
      mutation({ state: raw(state), initial });
      const final = trigger({ state });
      console.log({
        final,
        initial,
        current
      });
      initial = final;
      filter(trigger, mutation);
    }
  });
  return () => unobserve(reaction);
};

And this is the actual API:

filter(
  ({ state }) => state.user.surname,
  ({ state }) => (state.user.surname = "$" + state.user.surname)
);

To abort a mutation just set it back to it’s original value:

filter(
  ({ state }) => state.user.surname,
  ({ state, oldValue }) => (state.user.surname = oldValue)
);

We can use its system of queues and priorities to trigger the rest of the mutations after the filters have been run. This works well:

const scheduler = new Queue(priorities.HIGH);
observe(() => console.log(state.user.surname), { scheduler });

Even simpler is the addition of reactions:

const reaction = (
  trigger: ({ state }: { state: State }) => void,
  sideEffect: ({ state }: { state: State }) => void
) => {
  const reaction = observe(() => trigger({ state }), {
    scheduler: () => {
      sideEffect({ state });
    }
  });
  return () => unobserve(reaction);
};

const abortReaction = reaction(
  ({ state }) => state.user.name,
  ({ state }) => {
    state.count += 1;
    console.log(state.count);
  }
);

I’ve been studying the code of observer-util and it is extremely simple and well thought. I think we can make a fork and add:

  • The minimal mutation logic.
  • The mutations middleware.
  • The flushes middleware. We can reuse their scheduler logic.
  • The derived state and derived functions.

It’s pretty much that.

I love the idea that they don’t proxify everything, just what is being accessed on the fly. We can use that for the proxy actions. Much cheaper.

I’m also thinking about the filter/reactions API. If it should be more like Mobx or more like normal middleware (redux, express, koa).

For example, this is another idea:

const state = {
  name: "Jon",
  users: []
}
const reactions = {
  // Use reactions for nested derived values.
  userFullname: {
    // Mutation is called on each mutation.
    mutation: ({ paths, op }) => paths[1] === "users" && op === "add",
    // If mutation returns true, the reactions is called.
    reaction: async ({ path, state, next }) => {
      // await until the mutation has finished (Koa style)
      await next();
      state.users[path[1]].fullname = ({ self }) => `${self.name} ${self.surname}`;
    }
  },
  // User reactions for filters.
  dollarToName: {
    mutation: ({ paths }) => paths[1] === "name",
    reaction: (ctx) => {
      // Replace value, Koa style.
      ctx.value = "$" + ctx.value;
    }
  },
  // Abort mutations.
  noNameJohn: {
    mutation: ({ paths }) => paths[1] === "name",
    reaction: ({ value, abort }) => {
      if (value === "John") abort()
    }
  },
  // React to actions instead of mutations.
  sendPageview: {
    action: ({ path }) => path === "analytics.sendPageview",
    reaction: ({ actions, args }) => {
      // Call my own action.
      actions.myAnalytics.sendPageview(...args)
    }
  },
  // Abort actions.
  dontRepeatFetch: {
    action: ({ path }) => path === "source.fetch",
    reaction: ({ state, args, abort }) => {
      // Do not trigger fetch again if data is already fetched
      if (state.source.data[args.name].isFetched) abort();
    }
  }, 
}

It may be more complex than other APIs but with only one concept you have the ultimate hackable tool.

It can also be used to listen to flushes although I don’t know any good reason yet, except for our devtools.

I’ve been giving more thought to this after Michel’s comment in the issue and he is right, patches are a non-deterministic way to describe mutations. And what we need is:

  • Minimize unnecessary rerenders.
  • Preserve object/array references on mutations.
  • A deterministic way to describe mutations for the middleware.

Patches are great for tasks like synchronizing two machines or undoing mutations. We don’t really need those features although they are great additions. I’m going to give it a go to see if I can figure out a way to have those 3 features and a patching system out of them.

I think we are going to need different strategies to accomplish all this:

  • Minimize unnecessary rerenders.
  • Preserve object/array references on mutations.
  • A deterministic way to describe mutations for the middleware.
  • Patches for devtools, generating immutable snapshots or synchronize clients.

This is what I thought yesterday:

Minimize unnecessary rerenders and preserve object/array references

We can achieve this by never replacing an existing object or array. In order to get that, we need to iterate over the added objects/array and take a look at the current state to see if objects/arrays exist in the same positions. If there’s no such object or array, the iteration finishes and the rest of the data is added. This is to ensure we don’t iterate more than we need because iterating objects is pretty slow. This technique may be slow if the dev is constantly adding the same big object over and over again, but that’s not usually the case.

EDIT: This can be done preserving proxies references. It won’t be slow, at least, it won’t be slower than proxies themselves.

A deterministic way to describe mutations for the middleware

The problem with patches is that they are not deterministic. If the dev does:

state.user = { name: "Jon", surname: "Snow" }
// The patch is:
{ op: "replace", path: "state.user", value: { name: "Jon", surname: "Snow" } }

There’s no way to use that for proper filter of the name value, so I think we need something like “atomic mutation patches” which are deterministic at a scalar level.

state.user = { name: "Jon", surname: "Snow" }
// The mutations are:
{ op: "replace", path: "state.user.name", value: "Jon" }
{ op: "replace", path: "state.user.surname", value: "Snow" }

The problem with this is that in order to emit those atomic mutation patches we would need to iterate over the added object. In case of big objects that can be slow.

In order to avoid unnecessary iterations, we can use a subscription model:

// Dev subscribes to atomic mutation patch
subscribe("state.user.name", myCallback);

// Internally, that is saved like this:
const subscriptions[0] = ["state", "user", "name"];

We can use those arrays to avoid iterating over parts of the object where nobody is subscribed.

Actually, instead of strings, those should be regexps for more complex filtering:

// Dev subscribes to atomic mutation patch
subscribe("state.users.\d+.name", myCallback); // i.e. user[0].name

That syntax will allow subscribing to any mutation in a path, something difficult with mobx-like subscriptions:

// Dev subscribes to atomic mutation patch
subscribe("state.users", myCallback); // any state.user.X.Y.Z mutation
// This doesn't work because it is listening to the users reference
reaction(state => state.users) 

We use strings so we lose typescript in that part, but we can use regexps.

Patches for devtools, generating immutable snapshots or synchronize clients

We should still generate regular patches when mutations happen. They serve a different purpose and they are very useful as well. If we don’t need them for either minimizing the rerenders or filtering, they can be just regular, non-deterministic patches like those of immer.

state.user = { name: "Jon", surname: "Snow" }
// Triggers this patch:
{ op: "replace", path: "state.user", value: { name: "Jon", surname: "Snow" } }
// And executes callback for these subscriptions:
["state", "user", "name"]
["state", "user", "surname"]
// If the object "user" doesn't exist, this mutation occur:
state.user = { name: "Jon", surname: "Snow" };
// If the object "user" exists, these mutations occur:
state.user.name = "Jon";
state.user.surname = "Jon";

This is what we are going to implement for the first version:

Actions (without params)

const actions = {
  toggleFilter: ({ state }) => { state.filter = !state.filter },
  fetchTodos: async ({ state }) => { state.todos = await api.getTodos() }
}

actions.toggleFilter()
actions.fetchTodos()

interface Actions {
  toggleFilter: Action<MyPkg>
  fetchTodos: Action<MyPkg>
}

Actions (with params)

const actions = {
  addTodo: ({ state }) => name => { state.todos.push({ name }) }
  fetchTodo: ({ state }) => async id => { state.todos[id] = await api.getTodo(id) }
}

actions.addTodo("finish the state manager")
actions.fetchTodo(374)

interface Actions {
  addTodo: Action<MyPkg, string>
  fetchTodo: Action<MyPkg, number>
}

Derived State

const state = {
  todos: [],
  completedTodos: ({ state }) => state.todos.filter(todo => todo.completed)
}

state.todos
state.completedTodos

interface State {
  todos: Todos,
  completedTodos: Derived<MyPkg, Todos>
}

Derived State Functions

const state = {
  todos: [],
  filteredTodos: ({ state }) => completed =>
    state.todos.filter(todo => todo.completed === completed)
}

state.todos
state.filteredTodos(false)

interface State {
  todos: Todos,
  filteredTodos: Derived<MyPkg, boolean, Todos>
}

We’ve decided to name it Frontity Connect to honor it’s only API: connect.

import connect from "@frontity/connect";
// or...
import { connect } from "frontity";

const MyComp = ({ state, actions }) => (
  <>
    <button onClick={actions.addSomething}>Add Something</button>
    <div>{state.someState}</div>
  </>
);

export default connect(MyComp);

The connect API makes the component “reactive” to state changes and pass state and actions as props.

1 Like

@luisherranz I ran into some doubts while writing the theme. When I am building a component, how do I define the state types in that function, so I can enjoy autocomplete from VSCode?

Also, does connect() support a mapper function or won’t be necessary?

At the moment it does not.

Sorry, I forgot to put it here. Connect works like Action or Derived, you pass the entire package type as the first argument and the other props as the second argument.

import MyTheme from "../../types"; // <- import your package type.
import { Connect } from "frontity/types";

type Props = Connect<MyTheme, { props... }>

const MyComp: React.FC<Props> = ({ state, actions, props... }) => (
...
);

export default connect(MyComp);

1 Like