feat(db): implement createEffect reactive effects API#1221
Open
feat(db): implement createEffect reactive effects API#1221
Conversation
Add reactive effects API that fires handlers when rows enter, exit, or update within a query result, processing deltas without materializing the full result set. - Extract shared helpers (extractCollectionsFromQuery, extractCollectionAliases, buildQueryFromConfig, sendChangesToInput, splitUpdates) into query/live/utils.ts - Implement EffectPipelineRunner with D2 graph, source subscriptions, output accumulation with previousValue tracking, skipInitial support, join support, and transaction-scoped scheduler integration for coalesced graph runs - Implement createEffect with handler/batchHandler invocation, error routing, AbortSignal disposal, and in-flight async handler tracking - Implement useLiveQueryEffect React hook with useEffect lifecycle management - Add 27 core tests and 3 React hook tests Co-authored-by: Cursor <cursoragent@cursor.com>
- Add source error/cleanup detection: EffectPipelineRunner now listens for status:change on source collections and auto-disposes the effect when a source enters error or cleaned-up state - Remove no-op onClear scheduler listener (effects have no pending state to clear, unlike CollectionConfigBuilder) - Add 3 truncate tests verifying correct exit/enter events after truncate, full-clear truncate, and post-truncate re-insertion - Add 5 orderBy+limit tests covering top-K window semantics: initial window, insert displacement, delete backfill, desc ordering, and in-window updates - Add 2 source error handling tests verifying auto-disposal and event suppression on collection cleanup Co-authored-by: Cursor <cursoragent@cursor.com>
🦋 Changeset detectedLatest commit: d2dbb82 The changes in this PR will be included in the next version bump. This PR includes changesets to release 16 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
More templates
@tanstack/angular-db
@tanstack/db
@tanstack/db-ivm
@tanstack/electric-db-collection
@tanstack/offline-transactions
@tanstack/powersync-db-collection
@tanstack/query-db-collection
@tanstack/react-db
@tanstack/rxdb-db-collection
@tanstack/solid-db
@tanstack/svelte-db
@tanstack/trailbase-db-collection
@tanstack/vue-db
commit: |
Contributor
|
Size Change: +5.42 kB (+5.89%) 🔍 Total Size: 97.4 kB
ℹ️ View Unchanged
|
Contributor
|
Size Change: 0 B Total Size: 3.7 kB ℹ️ View Unchanged
|
…join) Co-authored-by: Cursor <cursoragent@cursor.com>
Lazy aliases (marked by the join compiler) should not eagerly load initial state — the join tap operator loads exactly the rows needed on demand. For on-demand collections, the previous behavior would trigger a full server fetch for data meant to be lazily loaded. Co-authored-by: Cursor <cursoragent@cursor.com>
Add loadMoreIfNeeded integration so effects with orderBy+limit queries can pull more data from source collections when the pipeline filters items out of the topK window. This supports on-demand collections where data should be loaded incrementally. Changes: - Pass real optimizableOrderByCollections to compileQuery so the topK operator gets dataNeeded() and index-based cursor support - Add ordered subscription path: requestLimitedSnapshot for initial data, splitUpdates for ordered changes, trackSentValues for cursor positioning - Add loadMoreIfNeeded/loadNextItems called after each graph run step - Pass orderBy/limit hints to unordered subscriptions for server-side optimization - Add 3 tests for lazy loading with filter, hints, and exit-triggered load Co-authored-by: Cursor <cursoragent@cursor.com>
…riber Deduplicate logic used by both EffectPipelineRunner and CollectionSubscriber by extracting four helpers into utils.ts: - filterDuplicateInserts: prevent duplicate D2 inserts via sentKeys - trackBiggestSentValue: cursor tracking for ordered subscriptions - computeSubscriptionOrderByHints: normalize orderBy/limit for subscriptions - computeOrderedLoadCursor: build minValues/loadRequestKey for lazy loading Both consumers now import from the shared module, removing ~200 lines of near-identical inline logic. Co-authored-by: Cursor <cursoragent@cursor.com>
Must-fix correctness issues: 1. batchHandler promise rejections now attach .catch() before tracking to prevent unhandled rejection warnings 2. Two-phase disposal: graph/inputs/pipeline cleanup is deferred when dispose() is called mid-graph-run; while-loop stops if disposed 3. Buffer flush drains robustly per-alias (delete from map before drain), applies ordered tracking/splitting for buffered ordered alias data 4. previousValue for coalesced batches uses first delete (pre-batch state) and last insert (post-batch state) instead of most-recent of each Semantics/DX improvements: 5. Explicitly set includeInitialState: false for ordered aliases 6. Exit events no longer duplicate value as previousValue — previousValue is only present on update events 7. Batch boundary: accumulate across entire while-loop and emit once per scheduler run instead of after each graph.run() step 8. onSourceError callback on EffectConfig lets users handle source errors without relying on console.error; auto-dispose still occurs after Tests added for all 8 fixes (5 new tests). Co-authored-by: Cursor <cursoragent@cursor.com>
Adds a comprehensive "Reactive Effects (createEffect)" section covering delta events, handler types, skipInitial, error handling, disposal, query features, transaction coalescing, and the React useLiveQueryEffect hook. Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
createEffectanduseLiveQueryEffectas specified in the reactive effects RFCcreateEffectattaches an effect handler to a query's delta stream, firingenter,exit, andupdateevents as rows match/unmatch/change within the query resultwhere,join,select,orderBy+limitqueriesbuildQueryFromConfig,extractCollectionAliases,sendChangesToInput,splitUpdates, etc.) fromCollectionConfigBuilderandCollectionSubscriberintosrc/query/live/utils.tsto reduce duplicationskipInitial,onError,batchHandler,handler, and asyncdispose()withAbortSignalfor cancelling in-flight workuseLiveQueryEffectReact hook with proper mount/unmount lifecycle managementKey design decisions
createLiveQueryCollection— avoids materialising results into a collection when only deltas are neededCollectionConfigBuildercould wrapcreateEffectto further reduce duplicationstatus:changeon source collections and auto-disposes on error/cleanup (gap identified during review vsCollectionConfigBuilder)Test plan
onparameter variations (single type, array,delta)batchHandlerbatching semanticsskipInitialsuppression of initial dataQueryBuilderinstance inputorderBy+limit(top-K window, insert displacement, delete backfill, desc, in-window updates)useLiveQueryEffect(lifecycle, dep recreation, event reception)Made with Cursor