Cold state and resume

When a project has stopped automatically and the node runs short on disk space, the project's process is shut down and its database is encrypted and stored off-site. This is called the cold state.

The reactivation sequence

When the first request arrives for a project in the cold state, it is reactivated in this order.

1. The front answers the request with 503 cold and starts the reactivation.
2. The encrypted database is downloaded to the node and restored.
3. A new generation is opened and the worker process starts.

This usually takes 6–7 seconds. Requests that arrive while it is in progress get a 503 response along with guidance to retry in a few seconds.

Temporary limitation right after reactivation

Even after the front starts forwarding requests to the new database generation, the node process can take a short time to converge on that state. During this period your application responds, but database requests alone can fail temporarily.

What to do. Retrying 57P03 errors in your application at short intervals is enough to cover this window.

async function withRetry<T>(fn: () => Promise<T>, tries = 4): Promise<T> {
  for (let i = 0; ; i++) {
    try {
      return await fn();
    } catch (e) {
      const code = (e as { code?: string }).code;
      if (code !== "57P03" || i >= tries) throw e;
      await new Promise((r) => setTimeout(r, 500 * (i + 1)));
    }
  }
}

This behaviour is a known limitation and we plan to improve it.

When does a project go cold

What sends a project into the cold state is not idle time but the node running short on disk space. As long as the node has free space, your data stays on the node even if the project is unused for days. In that case nothing is downloaded from external storage and only the worker is started again.