Defer descendant propagation to batch pass at end of resolve()#743
Draft
Defer descendant propagation to batch pass at end of resolve()#743
Conversation
During linearize_ancestors, descendant tracking was the #1 performance bottleneck: propagate_descendants ran O(ancestors × descendants) on every cache hit, and per-declaration descendant tracking added O(accumulated_descendants) work at each recursion level. Since descendants are write-only during resolution (read only during invalidation at graph.rs:1118, 1181), we can safely defer all descendant propagation to a single batch pass after the resolution loop completes. This eliminates: 1. The propagate_descendants() call on every cache hit 2. The per-declaration descendant insertion loop during recursion 3. The descendants field from LinearizationContext entirely Instead, after building each ancestor chain we record flat (descendant_id, ancestor_id) pairs, then batch-process all relationships at the end of resolve(). Resolution on Shopify core: ~42s → ~34s (20% faster).
- Add backticks around `descendant_id`, `ancestor_id`, `resolve()` in doc comment to satisfy clippy doc lint - Sort descendant assertions in test_descendants since IdentityHashSet iteration order is non-deterministic
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
linearize_ancestorsto a single batch pass at the end ofresolve()graph.rs:1118,1181) and MCP queries (get_descendants) — both of which happen afterresolve()returns, so deferral is safepropagate_descendants()method,descendantsfield fromLinearizationContext, and the per-declaration descendant insertion loopBefore / After
Measured on Shopify core (127,216 files, 1,470,515 declarations, 1,634,734 definitions).
Why this works
During recursive linearization, every cache hit triggered
propagate_descendants()— an O(ancestors × descendants) nested loop that inserted the current declaration (and its transitive descendants) into every ancestor's descendant set. With 155k classes mostly inheriting from Object, this meant ~155k calls each iterating Object's full ancestor chain.Since descendants are not read during the resolution loop — only during incremental invalidation and MCP queries, both of which happen after
resolve()returns — we can safely collect flat(descendant_id, ancestor_id)pairs during linearization and batch-insert them after the resolution loop completes. This eliminates the hot nested loop entirely.