feat(new-compiler): migrate metadata storage from lockfile to LMDB #1955
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Contributor
📝 WalkthroughWalkthroughMigrates metadata from a JSON file with file locks to LMDB-backed databases, replacing the MetadataManager class with stateless functions (loadMetadata, saveMetadata, cleanupExistingMetadata, createEmptyMetadata), adding environment-specific dirs (.lingo/metadata-dev, .lingo/metadata-build), and removing proper-lockfile. Changes
Sequence DiagramsequenceDiagram
participant Compiler as Compiler (Plugin)
participant SaveFunc as saveMetadata(dbPath, entries)
participant LoadFunc as loadMetadata(dbPath)
participant DB as LMDB Database
rect rgba(100,150,200,0.5)
Note over Compiler,SaveFunc: Save metadata flow
Compiler->>SaveFunc: saveMetadata(dbPath, entries)
SaveFunc->>DB: openDatabase(dbPath)
DB-->>SaveFunc: db handle
SaveFunc->>DB: write entries (transaction)
SaveFunc->>DB: update __stats__ key
DB-->>SaveFunc: commit
SaveFunc->>DB: close connection
SaveFunc-->>Compiler: confirmation
end
rect rgba(150,200,100,0.5)
Note over Compiler,LoadFunc: Load metadata flow
Compiler->>LoadFunc: loadMetadata(dbPath)
LoadFunc->>DB: openDatabase(dbPath)
DB-->>LoadFunc: db handle
LoadFunc->>DB: read all entries & __stats__
DB-->>LoadFunc: entries + stats
LoadFunc->>DB: close connection
LoadFunc-->>Compiler: MetadataSchema
end
Estimated code review effort🎯 4 (Complex) | ⏱️ ~50 minutes Possibly related PRs
Suggested labels
Suggested reviewers
Poem
🚥 Pre-merge checks | ✅ 3 | ❌ 2❌ Failed checks (1 warning, 1 inconclusive)
✅ Passed checks (3 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing touches
🧪 Generate unit tests (beta)
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Fix all issues with AI agents
In `@packages/new-compiler/src/metadata/manager.test.ts`:
- Around line 309-314: The test is incorrectly treating the synchronous
MetadataManager constructor as async; update the "should throw descriptive error
for invalid path" test to directly assert the synchronous throw by replacing the
await expect(async () => new MetadataManager(invalidPath)).rejects.toThrow()
pattern with a synchronous assertion such as expect(() => new
MetadataManager(invalidPath)).toThrow() (or toThrowError/... if you want to
assert message content), referencing the MetadataManager constructor in that
test.
🧹 Nitpick comments (4)
packages/new-compiler/src/metadata/manager.ts (1)
205-226: Avoid O(n) full scans when updating stats.
Counting all keys on every save scales with database size and can slow large builds. Consider incrementally updatingtotalEntriesper write inside the transaction.♻️ Suggested refactor
- for (const entry of entries) { - this.db.put(entry.hash, entry); - } - - // Count entries explicitly (excluding stats key) for clarity - let entryCount = 0; - for (const { key } of this.db.getRange()) { - if (key !== STATS_KEY) { - entryCount++; - } - } - - const stats = { - totalEntries: entryCount, - lastUpdated: new Date().toISOString(), - }; + const prevStats = + (this.db.get(STATS_KEY) as MetadataSchema["stats"] | undefined) ?? { + totalEntries: 0, + lastUpdated: new Date().toISOString(), + }; + let entryCount = prevStats.totalEntries; + + for (const entry of entries) { + const existed = this.db.get(entry.hash) !== undefined; + this.db.put(entry.hash, entry); + if (!existed) entryCount++; + } + + const stats = { + totalEntries: entryCount, + lastUpdated: new Date().toISOString(), + };packages/new-compiler/src/metadata/manager.test.ts (3)
106-108: Awaiting a synchronous function is unnecessary.
cleanupExistingMetadatareturnsvoid(synchronous), soawaitis a no-op here. While this doesn't break functionality, it's misleading and inconsistent with the function signature.Suggested fix
afterEach(async () => { - await cleanupExistingMetadata(testDbPath); + cleanupExistingMetadata(testDbPath); });
380-384: Avoid returning values fromforEachcallbacks.
worker.kill()returns a boolean, triggering the Biome lint warning. Usefor...ofloop instead for cleaner semantics when you don't need the return values.Suggested fix
} finally { - workers.forEach((w) => w.kill()); + for (const w of workers) { + w.kill(); + } }
237-258: Same issue: awaiting synchronouscleanupExistingMetadatacalls.Lines 243, 303, 304, and 357 also await the synchronous
cleanupExistingMetadatafunction. Consider removingawaitfrom all these call sites for consistency.Also applies to: 286-307
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
Contributor
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Fix all issues with AI agents
In `@packages/new-compiler/src/metadata/manager.test.ts`:
- Around line 213-217: The test "should throw descriptive error for invalid
path" is flaky because it uses a hard-coded /root/... path; change it to create
a guaranteed-failing target using the tmp directory: create a temporary file
(e.g., via fs.writeFileSync or fs.mkdtempSync under os.tmpdir()) and pass that
file path to loadMetadata so mkdirSync will deterministically fail; update the
test code around the it block (the test function invoking loadMetadata) to
create and clean up the temp file and assert that loadMetadata(tmpFilePath)
rejects.
Comment on lines
+213
to
+217
| describe("error handling", () => { | ||
| it("should throw descriptive error for invalid path", async () => { | ||
| const invalidPath = "/root/definitely/cannot/create/this/path"; | ||
| await expect(loadMetadata(invalidPath)).rejects.toThrow(); | ||
| }); |
Contributor
There was a problem hiding this comment.
Make the invalid-path test deterministic across CI environments.
Line 215 uses a hard-coded /root/... path, which can be writable when tests run as root, so loadMetadata may not throw and the test can flake. Use a temp file path to guarantee mkdirSync fails.
🔧 Suggested fix
describe("error handling", () => {
it("should throw descriptive error for invalid path", async () => {
- const invalidPath = "/root/definitely/cannot/create/this/path";
- await expect(loadMetadata(invalidPath)).rejects.toThrow();
+ const invalidPath = path.join(
+ os.tmpdir(),
+ `lmdb-invalid-${Date.now()}-${Math.random().toString(36).slice(2)}`,
+ );
+ fs.writeFileSync(invalidPath, "not a directory");
+ try {
+ await expect(loadMetadata(invalidPath)).rejects.toThrow();
+ } finally {
+ fs.rmSync(invalidPath, { force: true });
+ }
});
});As per coding guidelines, "When you add tests, make sure they pass".
📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| describe("error handling", () => { | |
| it("should throw descriptive error for invalid path", async () => { | |
| const invalidPath = "/root/definitely/cannot/create/this/path"; | |
| await expect(loadMetadata(invalidPath)).rejects.toThrow(); | |
| }); | |
| describe("error handling", () => { | |
| it("should throw descriptive error for invalid path", async () => { | |
| const invalidPath = path.join( | |
| os.tmpdir(), | |
| `lmdb-invalid-${Date.now()}-${Math.random().toString(36).slice(2)}`, | |
| ); | |
| fs.writeFileSync(invalidPath, "not a directory"); | |
| try { | |
| await expect(loadMetadata(invalidPath)).rejects.toThrow(); | |
| } finally { | |
| fs.rmSync(invalidPath, { force: true }); | |
| } | |
| }); | |
| }); |
🤖 Prompt for AI Agents
In `@packages/new-compiler/src/metadata/manager.test.ts` around lines 213 - 217,
The test "should throw descriptive error for invalid path" is flaky because it
uses a hard-coded /root/... path; change it to create a guaranteed-failing
target using the tmp directory: create a temporary file (e.g., via
fs.writeFileSync or fs.mkdtempSync under os.tmpdir()) and pass that file path to
loadMetadata so mkdirSync will deterministically fail; update the test code
around the it block (the test function invoking loadMetadata) to create and
clean up the temp file and assert that loadMetadata(tmpFilePath) rejects.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
Migrate metadata storage from JSON files with lockfile to LMDB database for improved concurrent access and elimination of ELOCKED errors.
Changes
Testing
Business logic tests added:
Visuals
N/A - No UI changes
Checklist
Closes #1636
Summary by CodeRabbit
New Features
Documentation
Tests
Dependencies