In-memory value joins read the whole attribute once per query (v3) #723

Split from #721 (fix: #722). v3 only.

Gap

An in-memory database has no committed generation: every fact lives in the in-memory fact store, which is ordered by entity and by attribute but not by value. A join on a bound value ([?x :up ?p] with ?p bound, #722's IndexNestedLoopJoin on the value) therefore reads every fact of the attribute once per query to find the facts with a value. File databases probe the committed AVET index and read only their few uncheckpointed facts this way.

Criterion recursion/ancestors/w10_d4 (in memory, 11,110 :up facts, bench profile, bench machine):

The same shape on a checkpointed file is 1.1–1.9 ms from 100K to 10M facts with #722.

Options

• A value index in the in-memory fact store (also used for uncheckpointed facts of a file database). Costs write time and memory on every write; streaming engine 3/9: IndexNestedLoopJoin and HashJoin (#473) #545 (plan decision 3) left it out.
• Let a rule's delta run choose, per round, between the value join and the order before fix(engine): rule delta runs join on a bound variable (#694) #700 (the magic relation first, then an entity probe), from the relation and attribute sizes.
• Accept it: in-memory databases are small by design, and 15 ms for 11K facts is linear in the attribute, read once per query.