Skip to content

feat: BED-7498 batch node update#38

Draft
bsheth711 wants to merge 10 commits intomainfrom
BED-7498-batch-node-update
Draft

feat: BED-7498 batch node update#38
bsheth711 wants to merge 10 commits intomainfrom
BED-7498-batch-node-update

Conversation

@bsheth711
Copy link
Contributor

@bsheth711 bsheth711 commented Mar 4, 2026

This PR adds the ability to batch update nodes.

A couple things to note:

  • I ended up going with the UpdateNodes(nodes []*Node) error signature for the Batch interface because this allows me to use two different implementations for updating depending on the number of nodes being updated. If the update is below a certain threshold, LargeNodeUpdateThreshold (currently set to 1M nodes), we do the typical batched SQL UPDATE...SET.... Above that threshold, we instead do a COPY...FROM... into a temporary table for the transaction, and then MERGE INTO the nodes table. This seems to be marginally faster for 1M+ updates, and should have even better scalability for much larger updates.
  • I added an optional BatchOption parameter to the BatchOperation function signature that allows one to set a BatchWriteSize for the BatchOperation
  • I added a helper function in ops.go: func UpdateNodes(ctx context.Context, graphDB graph.Database, nodes []*graph.Node, batchSize ...int) error that I think should be the first tool users of DAWGS reach for when updating nodes
  • I added a helper function (s *Node) StripAllPropertiesExcept(except ...string). The reason for this is if a developer is working with fully hydrated nodes in memory, but they only change one property and want to update all these nodes, we can cut down on a lot of network traffic to DB by removing all but the properties we care about before sending the nodes to the DB to be updated. The DB also does not need to bother parsing unnecessary information as well.

TODO: Add neo4j implementation

@bsheth711 bsheth711 self-assigned this Mar 4, 2026
@bsheth711 bsheth711 added the go Pull requests that update go code label Mar 4, 2026
@coderabbitai
Copy link

coderabbitai bot commented Mar 4, 2026

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: f87e390a-3ce1-4dd9-83c1-0c4072c8834e

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Post copyable unit tests in a comment
  • Commit unit tests in branch BED-7498-batch-node-update

Tip

Try Coding Plans. Let us write the prompt for your AI agent so you can ship faster (with fewer bugs).
Share your feedback on Discord.


Comment @coderabbitai help to get the list of available commands and usage tips.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

go Pull requests that update go code

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant