You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Image explanations haven't been exercised recently. We should verify that they a) still work in the base unfeaturized case, and b) the unpooled/pre-featurized case. The latter would allow for us to include explanations in the search/classify workflow that we have been demonstrating.
The text was updated successfully, but these errors were encountered:
Testing against big earth single produces an unpooled prefeaturized dataset of 10K rows x 32K images. The parquet loading library we use seems to be aggressively chewing through RAM, and probably leaking some in the process of loading. A dataset that is about 5GB ends up occupying around 15GB of RAM after being loaded. It is also much slower than the parquet reader that is part of PANDAS. It also seems like we end up duplicating the data when splitting, and possibly again before search. All of these factors combine to cause the big earth single model creation process to fail, with the server running out of RAM before the search can be started. Parquet read memory optimization #2382 fixed one of the duplications, but that's not sufficient to get everything running.
Image explanations are not being generated in for the un-pooled pre-featurized case. This needs investigation.
#2385 has been opened to track issues observed with parquet loading, and #2386 covers out of memory errors encountered with larger datasets that are unpooled.
Image explanations haven't been exercised recently. We should verify that they a) still work in the base unfeaturized case, and b) the unpooled/pre-featurized case. The latter would allow for us to include explanations in the search/classify workflow that we have been demonstrating.
The text was updated successfully, but these errors were encountered: