Conversation
|
Review requested:
|
nodejs-github-bot
added
lib / src
mcollina
requested review from
Ethan-Arrowood,
RaisinTen,
avivkeller and
joyeecheung
avivkeller
added
fs
|
The
notable-change
Please suggest a text for the release notes if you'd like to include a more detailed summary, then proceed to update the PR description with the text or a link to the notable change suggested text comment. Otherwise, the commit will be placed in the Other Notable Changes section. |
|
Nice! This is a great addition. Since it's such a large PR, this will take me some time to review. Will try to tackle it over the next week. |
| */ | ||
| existsSync(path) { | ||
| // Prepend prefix to path for VFS lookup | ||
| const fullPath = this.#prefix + (StringPrototypeStartsWith(path, '/') ? path : '/' + path); |
| validateObject(files, 'options.files'); | ||
| } | ||
|
|
||
| const { VirtualFileSystem } = require('internal/vfs/virtual_fs'); |
There was a problem hiding this comment.
Shouldn't we import this at the top level / lazy load it at the top level?
| ArrayPrototypePush(this.#mocks, { | ||
| __proto__: null, | ||
| ctx, | ||
| restore: restoreFS, |
There was a problem hiding this comment.
| restore: restoreFS, | |
| restore: ctx.restore, |
nit
lib/internal/vfs/entries.js
Outdated
| * @param {object} [options] Optional configuration | ||
| */ | ||
| addFile(name, content, options) { | ||
| const path = this._directory.path + '/' + name; |
lib/internal/vfs/virtual_fs.js
Outdated
| let entry = current.getEntry(segment); | ||
| if (!entry) { | ||
| // Auto-create parent directory | ||
| const dirPath = '/' + segments.slice(0, i + 1).join('/'); |
lib/internal/vfs/virtual_fs.js
Outdated
| let entry = current.getEntry(segment); | ||
| if (!entry) { | ||
| // Auto-create parent directory | ||
| const parentPath = '/' + segments.slice(0, i + 1).join('/'); |
lib/internal/vfs/virtual_fs.js
Outdated
| } | ||
| } | ||
| callback(null, content); | ||
| }).catch((err) => { |
There was a problem hiding this comment.
| }).catch((err) => { | |
| }, (err) => { |
lib/internal/vfs/virtual_fs.js
Outdated
| const bytesToRead = Math.min(length, available); | ||
| content.copy(buffer, offset, readPos, readPos + bytesToRead); |
lib/internal/vfs/virtual_fs.js
Outdated
| } | ||
|
|
||
| callback(null, bytesToRead, buffer); | ||
| }).catch((err) => { |
There was a problem hiding this comment.
| }).catch((err) => { | |
| }, (err) => { |
|
Left an initial review, but like @Ethan-Arrowood said, it'll take time for a more in depth look |
|
It's nice to see some momentum in this area, though from a first glance it seems the design has largely overlooked the feedback from real world use cases collected 4 years ago: https://github.com/nodejs/single-executable/blob/main/docs/virtual-file-system-requirements.md - I think it's worth checking that the API satisfies the constraints that users of this feature have provided, to not waste the work that have been done by prior contributors to gather them, or having to reinvent it later (possibly in a breaking manner) to satisfy these requirements from real world use cases. |
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #61478 +/- ##
==========================================
- Coverage 89.74% 89.69% -0.06%
==========================================
Files 675 690 +15
Lines 204601 210789 +6188
Branches 39325 40177 +852
==========================================
+ Hits 183616 189057 +5441
- Misses 13273 13984 +711
- Partials 7712 7748 +36
🚀 New features to boost your workflow:
|
|
And why not something like OPFS aka whatwg/fs? const rootHandle = await navigator.storage.getDirectory()
await rootHandle.getFileHandle('config.json', { create: true })
fs.mount('/app', rootHandle) // to make it work with fs
fs.readFileSync('/app/config.json')OR const rootHandle = await navigator.storage.getDirectory()
await rootHandle.getFileHandle('config.json', { create: true })
fs.readFileSync('sandbox:/config.json')fs.createVirtual seems like something like a competing specification |
mcollina
force-pushed
the
vfs
branch
3 times, most recently
from
5e317de to
977cc3d
Compare
I generally prefer not to interleave with WHATWG specs as much as possible for core functionality (e.g., SEA). In my experience, they tend to perform poorly on our codebase and remove a few degrees of flexibility. (I also don't find much fun in working on them, and I'm way less interested in contributing to that.) On an implementation side, the core functionality of this feature will be identical (technically, it's missing writes that OPFS supports), as we would need to impact all our internal fs methods anyway. If this lands, we can certainly iterate on a WHATWG-compatible API for this, but I would not add this to this PR. |
|
Small prior art: https://github.com/juliangruber/subfs |
mcollina
force-pushed
the
vfs
branch
2 times, most recently
from
8d711c1 to
73c18cd
Compare
|
I also worked on this a bit on the side recently: Qard@73b8fc6 That is very much in chaotic ideation stage with a bunch of LLM assistance to try some different ideas, but the broader concept I was aiming for was to have a module.exports = new VirtualFileSystem(new LocalProvider())I intended for it to be extensible for a bunch of different interesting scenarios, so there's also an S3 provider and a zip file provider there, mainly just to validate that the model can be applied to other varieties of storage systems effectively. Keep in mind, like I said, the current state is very much just ideation in a branch I pushed up just now to share, but I think there are concepts for extensibility in there that we could consider to enable a whole ecosystem of flexible storage providers. 🙂 Personally, I would hope for something which could provide both read and write access through an abstraction with swappable backends of some variety, this way we could pass around these virtualized file systems like objects and let an ecosystem grow around accepting any generalized virtual file system for its storage backing. I think it'd be very nice for a lot of use cases like file uploads or archive management to be able to just treat them like any other readable and writable file system. |
just a bit off topic... but this reminds me of why i created this feature request: Would not lie, it would be cool if NodeJS also provided some type of static example that would only work in NodeJS (based on how it works internally) const size = 26
const blobPart = BlobFrom({
size,
stream (start, end) {
// can either be sync or async (that resolves to a ReadableStream)
// return new Response('abcdefghijklmnopqrstuvwxyz'.slice(start, end)).body
// return new Blob(['abcdefghijklmnopqrstuvwxyz'.slice(start, end)]).stream()
return fetch('https://httpbin.dev/range/' + size, {
headers: {
range: `bytes=${start}-${end - 1}`
}
}).then(r => r.body)
}
})
blobPart.text().then(text => {
console.log('a-z', text)
})
blobPart.slice(-3).text().then(text => {
console.log('x-z', text)
})
const a = blobPart.slice(0, 6)
a.text().then(text => {
console.log('a-f', text)
})
const b = a.slice(2, 4)
b.text().then(text => {
console.log('c-d', text)
})An actual working PoC (I would not rely on this unless it became officially supported by nodejs core - this is a hack) const blob = new Blob()
const symbols = Object.getOwnPropertySymbols(blob)
const blobSymbol = symbols.map(s => [s.description, s])
const symbolMap = Object.fromEntries(blobSymbol)
const {
kHandle,
kLength,
} = symbolMap
function BlobFrom ({ size, stream }) {
const blob = new Blob()
if (size === 0) return blob
blob[kLength] = size
blob[kHandle] = {
span: [0, size],
getReader () {
const [start, end] = this.span
if (start === end) {
return { pull: cb => cb(0) }
}
let reader
return {
async pull (cb) {
reader ??= (await stream(start, end)).getReader()
const {done, value} = await reader.read()
cb(done ^ 1, value)
}
}
},
slice (start, end) {
const [baseStart] = this.span
return {
span: [baseStart + start, baseStart + end],
getReader: this.getReader,
slice: this.slice,
}
}
}
return blob
}currently problematic to do: also need to handle properly clone, serialize & deserialize, if this where to be sent of to another worker - then i would transfer a MessageChannel where the worker thread asks main frame to hand back a transferable ReadableStream when it needs to read something. but there are probably better ways to handle this internally in core with piping data directly to and from different destinations without having to touch the js runtime? - if only getReader could return the reader directly instead of needing to read from the ReadableStream using js? |
| this.#autoClose = options.autoClose !== false; | ||
|
|
||
| // Open the file on next tick so listeners can be attached | ||
| process.nextTick(() => this.#openFile()); |
There was a problem hiding this comment.
It's worth adding a short comment here that #openFile will not throw and if it fails the stream will be destroyed.
| return new Glob(pattern, options).globSync(); | ||
| } | ||
|
|
||
|
|
- Use JSON.stringify() instead of string literal in overlay example - Document multiple VirtualFileSystem instances interaction - Clarify mountPoint returns absolute path - Document unmount() is idempotent - Add Windows mount example with drive letters
Add test-vfs-windows.js to verify VFS mounting with Windows drive letter paths. Tests include: - Mounting at paths with drive letters (e.g., C:\temp\vfs-test) - Mounting at drive root (e.g., C:\vfs-test-root) - Verifying mountPoint returns Windows-style absolute path - Require from Windows VFS paths
- Revert unintended whitespace change in lib/fs.js - Use JSON.stringify() in documentation examples for consistency - Add comment in streams.js explaining #openFile error handling - Document path encoding behavior in overlay mode
| the native file system APIs which handle encoding according to platform | ||
| conventions (UTF-8 on most Unix systems, UTF-16 on Windows). This means the | ||
| VFS inherits the underlying file system's encoding behavior for paths that | ||
| fall through, while VFS-internal paths always use UTF-8. |
There was a problem hiding this comment.
I'd extend this to give some warning about it meaning that some paths on disk might not be shadowed, or some might actually be unexpectedly shadowed with a simple example. I just know someone is going to hit this case and it's going to come up as a bug report .
Add security monitoring events that are emitted when a VFS is mounted or unmounted. Applications can listen to these events to detect unauthorized VFS usage or enforce security policies. Events include: - mountPoint: The path where the VFS is mounted - overlay: Whether overlay mode is enabled - readonly: Whether the VFS is read-only
- Use != null instead of !== undefined && !== null in vfs.js - Convert underscore-prefixed methods to private fields in watcher.js
Document that VFS is always case-sensitive internally and explain how this interacts with case-insensitive file systems in overlay mode.
github-actions
bot
removed
the
request-ci
jasnell
left a comment
jasnell
left a comment
There was a problem hiding this comment.
LGTM. There's likely little details here and there still... given that it's such a large PR it's hard to give a 100% thorough review... but given that it's experimental, happy to go with this and iterate.
|
There are roughly ~70 windows failures. Guess I have some |
|
|
||
| ```cjs | ||
| // Require bundled modules directly | ||
| const myModule = require('/sea/lib/mymodule.js'); |
There was a problem hiding this comment.
I think this may be somewhat impractical for real-world apps - they are likely not really requiring from any absolute paths when being developed locally, for example just doing require('pkg') and require('./path') as usual. Given that there are already module loading hooks it would be better to suggest/provide a built-in hook that alters resolution under the hood to prepend /sea and hide this from SEA users.
Also this technically does not fully work, because require('/path/to/addons.node') would go through process.dlopen() instead. That either needs a call out or better be done as a follow up with some "temporarily decompressing to disk" mechanism.
There was a problem hiding this comment.
Also another thing that may need to be called out in the docs - I don't think the current implementation would support useCodeCache for the modules loaded from the SEA blob, that requires actually executing the modules to know "what modules to cache" and build the cache during SEA building, whereas the current SEA plumbing here only runs post-building.
|
I feel like there should be a mention of this API somewhere in the |
lib/internal/main/embedding.js
Outdated
| // Check if the file exists in VFS | ||
| if (seaVfs.existsSync(modulePath)) { | ||
| // Use Module._load to load the module, which will use VFS hooks | ||
| return Module._load(modulePath, embedderRequire.main, false); |
There was a problem hiding this comment.
I think it would make much more sense to do this in a module.registerHooks hook instead of pre-processing before Module._load. That would also allow ESM to work easily with this once the hooks are baked in. Also I think doing a bypassing this way could invalidate user apps that have their own hooks for e.g. instrumentation, plumbing it through another hook would keep them work together.
There was a problem hiding this comment.
Additionally, internally CJS loading should use wrapModuleLoad instead of Module._load directly, otherwise it bypasses diagnostics channels and performance tracing.
|
|
||
| #### Synchronous Methods | ||
|
|
||
| * `vfs.accessSync(path[, mode])` - Check file accessibility |
There was a problem hiding this comment.
Does this need a duplicate list here? It seems better to list "what is not supported by vfs" instead.
On that front, this also seems to lack a list of "what is not supported" e.g. as pointed out in another comment, this won't work with addons. I think child processes are not supported here? Also, is permission model supported (I am not quite sure how it's supposed to behave with permissions though, maybe @RafaelGSS has some idea).
There was a problem hiding this comment.
If I read this PR correctly, all operations do occur in memory-only - so the permission model shouldn't be affected nor have power on this module.
I think that it would be useful to have some sort of "supported/not supported in vfs" annotation to each API in fs.md? It would lower the mental burden for users when they need to look up the docs - I suspect people are more likely to try to confirm whether something works/doesn't work in vfs than trying to directly use vfs APIs and only vfs, not switching to real fs at any point. |
|
I'm not a huge fan of the My thinking was something like: function someApi(fs: FsLike) {
fs.readFile(...)
}
// Works with just the base fs provider itself.
someApi(require('fs'))
// Also works with any provider that conforms to the FsLike contract.
// Because mounting exists on the provider rather than the actual vfs
// additional mounts can't be set from elsewhere without access to the
// original provider.
const mockOverlay = new OverlayProvider(require('fs'))
mockOverlay.mount('/mount', new MockProvider(...))
someApi(new VirtualFileSystem(mockOverlay)) |
|
That hijack is what makes it viable for the ecosystem without changing their current APIs. |
|
Perhaps, but it's also somewhat risky. Perhaps we need to think about how we can apply the permission model controls to that to prevent unintended tampering with the global fs? |
Use wrapModuleLoad instead of Module._load directly to ensure diagnostics channels and performance tracing work properly when loading modules from SEA VFS.
- Add "Limitations" section to vfs.md documenting unsupported features: native addons, child processes, worker threads, fs.watch polling, and SEA code caching - Add code caching limitations section to SEA VFS documentation - Add VFS support section to fs.md with example and link to vfs.md
Use POSIX path normalization for VFS paths starting with '/' to preserve forward slashes on Windows. The platform's path.normalize() converts forward slashes to backslashes on Windows, breaking VFS path matching. Add normalizeVFSPath() helper that uses path.posix.normalize() for Unix-style paths and path.normalize() for Windows drive letter paths. Fixes test-vfs-chdir, test-vfs-real-provider, test-vfs-mount-events, and test-vfs-watch on Windows.
15 participants
A first-class virtual file system module (
node:vfs) with a provider-based architecture that integrates with Node.js's fs module and module loader.Key Features
Provider Architecture - Extensible design with pluggable providers:
MemoryProvider- In-memory file system with full read/write supportSEAProvider- Read-only access to Single Executable Application assetsVirtualProvider- Base class for creating custom providersStandard fs API - Uses familiar
writeFileSync,readFileSync,mkdirSyncinstead of custom methodsMount Mode - VFS mounts at a specific path prefix (e.g.,
/virtual), clear separation from real filesystemModule Loading -
require()andimportwork seamlessly from virtual filesSEA Integration - Assets automatically mounted at
/seawhen running as a Single Executable ApplicationFull fs Support - readFile, stat, readdir, exists, streams, promises, glob, symlinks
Example
SEA Usage
When running as a Single Executable Application, bundled assets are automatically available:
Public API
Disclaimer: I've used a significant amount of Claude Code tokens to create this PR. I've reviewed all changes myself.