Pipelining Minecraft Chunk Generation

It is very difficult to implement a Minecraft server in another programming language — for example, Rust. A particular focus is often placed on optimizing RAM and CPU usage. A common stumbling block in this process is delegating concurrent processes, as we can see in the PumpkinMC project.
However, sometimes, there is a simpler solution to a difficult problem.

Let’s take Minecraft chunk generation as an example; we know that the entire system is very inflexible and interwoven with asynchronous logic, since there are multiple chunk stages that must be processed sequentially, thus forming a complicated dependency graph. Consequently, from the viewpoint of per-chunk deterministic/isolated generation, Minecraft’s chunk generation is very overhead-heavy.
But I’d like to look at it differently.
Assume that all functions/algorithms belonging to a stage are represented as functions $f_i$, where $i$ is the index of the chunk stage. Then we could approach the problem in terms of function composition instead of sequential execution as follows:

Let $g$ be the “final” chunk-gen function. Then we have

$g = f_1 \circ f_2 \circ \ldots \circ f_n$.

This means that ideally, each chunk would only need to execute $g$ once to obtain the same result as before.
Well, that sounds nice, but it doesn’t reflect reality correctly, because as mentioned, many chunk stages depend on the stages of surrounding chunks (i.e., chunks may perform I/O operations on neighboring chunks, leading to complexity and inefficiency).
To get around this issue, we could split each chunk stage — each $f_i$ — into the part that depends on neighboring chunk stages and the part that doesn’t, as follows:

Let $D$ (for dependent fn) be the set of chunk-stage operations that depend on the state of neighboring chunks, then we have:

$\forall f \in D: f = f' \circ f^{d}$,
*for clarity, I have omitted the indices*

where $f^{d}$ represents the part dependent on neighboring chunk stages, and $f'$ represents the independent part. Consequently, we can combine all $f'_i$ into a single $g'$ and all $f^{d}_i$ into a single $g^{d}$, obtaining:

$g = g' \circ g^{d}$.

This suggests that one could potentially generate one or more chunks simultaneously using a chunk-isolated part $g'$, and only at the end run a non-isolated part $g^{d}$ on top, hopefully yielding correct chunks while greatly improving concurrency and reducing overhead.

Obstacles

A possible problem is the lack of the commutative property for function composition. This could mean that in some cases, it might not be possible to realize this system.

Consider a chunk stage $B$, which depends on a chunk stage $A$, and a chunk stage $C$, which depends on $B$, as well as an independent chunk stage $X$.
This leads to the following dependency graph:

It doesn’t matter when we execute $X$, but it does matter in which order we execute $A, B, C$.
In the sense of our previous reasoning, we could therefore, for example, consider the order (permutation)

$\sigma = \begin{pmatrix} X & A & B \\ X & B & C \end{pmatrix} = \begin{pmatrix} X \\ X \end{pmatrix} \begin{pmatrix} A & B \\ B & C \end{pmatrix} = \sigma' \cdot \sigma^{d}$

Additionally, we introduce another chunk stage $Y$, which depends on $\sigma^{d}$.

It now becomes clear that $Y$ must be executed after $\sigma^{d}$, but $\sigma'$ remains independent.
The biggest challenge, therefore, will be determining when and where to perform this decoupling.

I will add some better examples soon.