Generalize cache system to better handle reuse #3

New issue

Open

opened 2026-02-13 23:46:47 +00:00 by UserCurt · 0 comments

UserCurt commented

2026-02-13 23:46:47 +00:00

(Migrated from codeberg.org)

As a separate point, if needing n-gram metrics while m-gram metrics are already calculated, the system will not re-use the existing cache.

This feature request proposes two changes:

Splitting n-gram cache into 1..n separate caches
Ensuring 1-gram hash is only dependent on the layered binding set and not the position of the bindings.

This will reduce the need to re-generate caches and also remove the need for --cache in optimize.py.

Some layouts share the same set of layered bindings so have common unigram metrics. However, the caching system will often regenerate these metrics despite them already being pre-calculated when using a different layout. As a separate point, if needing n-gram metrics while m-gram metrics are already calculated, the system will not re-use the existing cache. This feature request proposes two changes: - Splitting n-gram cache into 1..n separate caches - Ensuring 1-gram hash is only dependent on the layered binding set and not the position of the bindings. This will reduce the need to re-generate caches and also remove the need for `--cache` in `optimize.py`.