Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

proposal: embed: Add Dispose function to embed.FS #57457

Closed
erexo opened this issue Dec 25, 2022 · 2 comments
Closed

proposal: embed: Add Dispose function to embed.FS #57457

erexo opened this issue Dec 25, 2022 · 2 comments
Labels
FrozenDueToAge Proposal
Milestone
Proposal

Comments

@erexo
Copy link

erexo commented Dec 25, 2022 •
edited

As described in docs, the //go:embed directive allows us to embed multiple files into a dedicated File System that satisfies the io.FS interface.
I've encountered a situation where I would like to embed (namely hide) certain large images into the executable to load them into the memory at the very start of my application, but as soon as I decode those images (or use embedded files) I don't need them in my memory anymore.
If I set the value of the embed.FS variable to embed.FS{} it would wipe every embedded file from the memory, but there is no way to remove a particular one.
The fs.File returned by embed.FS's Open() does nothing at Close() which is absolutely understandable, that's why I would like to propose a new function in the embed.FS struct that would erase a file (or dir) at given name, my suggestion would be Dispose(name string) error.
@erexo erexo added the Proposal label Dec 25, 2022
@gopherbot gopherbot added this to the Proposal milestone Dec 25, 2022
@Jorropo
Copy link
Member

Jorropo commented Dec 25, 2022

hide

Just so you know, embed hides absolutely nothing, if your files are recognisable formats it will be trivial to use something like binwalk to extract them.
If they aren't you would need to extract the symbol which is only slightly harder. Someone will probably make some turn key outbed cli that extract all files from a go binary one day.
🙂

That a 👎 for me, embed.FS data is stored in a read only portion of executable, it's not fully loaded into ram it's lazily swapped on demend by the OS, maybe this is broken and data is kept alive more than it should but then the fix would be to change some linking parameters, or maybe madvise this part of memory (with refcounting based on Close), whatever prevents memory from being released.
IMO this is between,

  • everything is happenning as it should (you don't see the memory usage being reduced because your OS does not feel under memory pressure)
  • or a bug prevents the OS from reclaiming the memory.

Having embed.FS being readonly allows to put all the internal data structures in read only part of the exectuable that can be swapped in and out of the ram when needed, if you start having ways to remove files you would need to always keep alive objects telling what is still there and what isn't.

@seankhliao
Copy link
Member

I think this should be rolled into #45757 and whatever delete/remove functionality it would provide.

Right now you could copy whichever files you want to keep in to an alternate io.FS implementation, then wipe out the original embed.FS.

@seankhliao seankhliao closed this as not planned Won't fix, can't repro, duplicate, stale Dec 26, 2022
@golang golang locked and limited conversation to collaborators Dec 26, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
FrozenDueToAge Proposal
Projects
None yet
Milestone
Proposal
Development

No branches or pull requests
4 participants
@erexo @gopherbot @seankhliao @Jorropo