Make manifest cache size configurable and allow for disabling

[kris-gaudel]

Feature Request / Improvement

Discussion in #2325 revealed that the manifest cache leaks memory over time. On-going work to fix the leak is outlined in #2951, but having the ability to configure the cache size and disable it entirely would be beneficial to developers

[kevinjqliu]

Im thinking either set as env var or catalog properties. Leaning towards env var since this might be use in places other than the catalog route. No hard opinions though

cc @Fokko if you have any preferences here

[rambleraptor]

I think an env variable would be the way to go. The Manifest Cache isn't necessarily a catalog level property, but it's really a detail around PyIceberg.

[kevinjqliu]

@kris-gaudel would you have some time to take a look at this? We just merged #2951 (using the original 128 cache size). Would be great to make it configurable for the upcoming 0.11 release

[kris-gaudel]

@kevinjqliu Sure! I was planning on tackling it Friday + this weekend 👍 . Could you assign it to me please?

[kris-gaudel]

PR up!