Handle non-deterministic pyproj failure

[AdeelH]

Overview

This PR fixes a problem reported in #2257 and #2243 where RasterioSource.get_chip() would result in the error below:

  File "/opt/src/rastervision_core/rastervision/core/data/label_source/semantic_segmentation_label_source.py", line 79, in get_label_arr
    label_arr = self.raster_source.get_chip(window)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/src/rastervision_core/rastervision/core/data/raster_source/rasterio_source.py", line 172, in get_chip
    chip = self._get_chip(window, out_shape=out_shape, bands=bands)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/src/rastervision_core/rastervision/core/data/raster_source/rasterio_source.py", line 147, in _get_chip
    chip = fill_overflow(self.bbox, window, chip)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/src/rastervision_core/rastervision/core/data/utils/raster.py", line 35, in fill_overflow
    chip[..., :, xmax:, :] = fill_value
    ~~~~^^^^^^^^^^^^^^^^^^
TypeError: slice indices must be integers or None or have an __index__ method

This was tracked down to be the result of pyproj.Transformer.transform() non-deterministically returning infs in some cases ... apparently due to network connectivity issues! More discussion:

• PROJ_NETWORK: First transform fails, succeeds afterwards pyproj4/pyproj#705
• First call failiure using transform but then it works? - unsure if it's a bug or something else pyproj4/pyproj#1421
• BUG: to_crs returns inf on first call. second call correct geopandas/geopandas#3433

This PR also adds some validation code in the Box constructor to disallow non-finite values. This will allow similar errors to be caught earlier in the future.

Checklist

• Added unit tests, if applicable
• Updated documentation, if applicable
• Added needs-backport label if the change should be back-ported to the previous release
• PR has a name that won't get you publicly shamed for vagueness

Notes

N/A

Testing Instructions

• This was tested in a copy of the Kaggle notebook shared in RasterioSource.get_chip() raises TypeError: slice indices must be integers or None or have an __index__ method #2257.

---

Closes #2257

[codecov]

Codecov Report

Attention: Patch coverage is 50.00000% with 10 lines in your changes missing coverage. Please review.

Project coverage is 90.31%. Comparing base (10a9c98) to head (59e65e2).
Report is 2 commits behind head on master.

Files with missing lines	Patch %	Lines
...e/data/crs_transformer/rasterio_crs_transformer.py	44.44%	10 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #2259      +/-   ##
==========================================
- Coverage   90.40%   90.31%   -0.09%     
==========================================
  Files         199      199              
  Lines        9908     9925      +17     
==========================================
+ Hits         8957     8964       +7     
- Misses        951      961      +10     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.