Increased capability

[RichardScottOZ]

[RichardScottOZ]

This was actually computing on my laptop.

[RichardScottOZ]

At least it got this far, anyway [#### ] | 12% Completed | 3hr 0min 3.3s
Admittedly a very good laptop

[RichardScottOZ]

The error was CPLE_OpenFailedError: '/vsicurl/https://sentinel-cogs.s3.us-west-2.amazonaws.com/sentinel-s2-l2a-cogs/22/M/EV/2020/7/S2A_22MEV_20200704_0_L2A/B02.tif' not recognized as a supported file format.

Not an out of memory error.

[RichardScottOZ]

These are relatively common too - basically bandwith/internet crapout type errors, where the file actually exists, it just fails to retrieve.

[RichardScottOZ]

So possibly as you mentioned improve Dask graph handling - what you have done is alleviate some memory pressure there? So that would be interesting to investigate, compare to the old way. As something this size would certainly have blown up the memory quickly.

[RichardScottOZ]

I was just leaving whatever chunks it came out at - same for cropping.

[RichardScottOZ]

eg - there's a sat-search bug with paging apparently, so sometimes you get 10000 max even if it should not be from the .found()
Anyway, it is a test.

[RichardScottOZ]

Maybe a couple of days for that? Needs bigger chunks?

[RichardScottOZ]

lazy stacking time: 0:00:01.884090
Stack Size in GB: 1135795.0
<xarray.DataArray 'stackstac-4b8d3b909ebdb566c3a2e2837e62d35d' (time: 412, band: 17, y: 143055, x: 141697)>
dask.array<fetch_raster_window, shape=(412, 17, 143055, 141697), dtype=float64, chunksize=(1, 1, 1024, 1024), chunktype=numpy.ndarray>

[RichardScottOZ]

2 months of South Australia bounding box

[RichardScottOZ]

Stack Size in GB: 465896.0
<xarray.DataArray 'stackstac-5696d5744bf6c3e77d1598e249eebe9b' (time: 169, band: 17, y: 143055, x: 141697)>
dask.array<fetch_raster_window, shape=(169, 17, 143055, 141697), dtype=float64, chunksize=(1, 1, 1024, 1024), chunktype=numpy.ndarray>
Coordinates: (12/25)

• time (time) datetime64[ns] 2021-01-01T01:34:24 ......
  id (time) <U24 'S2A_52JGS_20210101_0_L2A' ... 'S...
• band (band) <U8 'overview' 'visual' ... 'WVP' 'SCL'
• x (x) float64 2.932e+05 2.933e+05 ... 1.71e+06
• y (y) float64 2.741e+06 2.741e+06 ... 1.31e+06
  platform <U11 'sentinel-2a'
  ... ...
  gsd (band) object None 10 60 10 ... None None None
  sentinel:grid_square (time) <U2 'GS' 'GR' 'DS' ... 'UP' 'LH' 'UF'
  title (band) object None ... 'Scene Classification ...
  common_name (band) object None None 'coastal' ... None None
  center_wavelength (band) object None None 0.4439 ... None None
  full_width_half_max (band) object None None 0.027 ... None None None
  Attributes:
  spec: RasterSpec(epsg=3107, bounds=(293240, 1310090, 1710210, 2740...
  crs: epsg:3107
  transform: | 10.00, 0.00, 293240.00|\n| 0.00,-10.00, 2740640.00|\n| 0.0...
  resolution: 10
  Stack Title Info:
  <xarray.DataArray 'title' (band: 17)>
  array([None, 'True color image', 'Band 1 (coastal)', 'Band 2 (blue)',
  'Band 3 (green)', 'Band 4 (red)', 'Band 5', 'Band 6', 'Band 7',
  'Band 8 (nir)', 'Band 8A', 'Band 9', 'Band 11 (swir16)',
  'Band 12 (swir22)', 'Aerosol Optical Thickness (AOT)',
  'Water Vapour (WVP)', 'Scene Classification Map (SCL)'],
  dtype=object)
  Coordinates: (12/13)
• band (band) <U8 'overview' 'visual' ... 'WVP' 'SCL'
  platform <U11 'sentinel-2a'
  sentinel:valid_cloud_cover bool True
  instruments <U3 'msi'
  view:off_nadir int64 0
  sentinel:sequence <U1 '0'
  ... ...
  eo:cloud_cover int64 0
  gsd (band) object None 10 60 10 ... None None None
  title (band) object None ... 'Scene Classification ...
  common_name (band) object None None 'coastal' ... None None
  center_wavelength (band) object None None 0.4439 ... None None
  full_width_half_max (band) object None None 0.027 ... None None None
  <class 'xarray.core.dataarray.DataArray'> (17,)
  cropping time: 0:06:34.146246
  <xarray.DataArray 'stackstac-5696d5744bf6c3e77d1598e249eebe9b' (time: 169, band: 17, y: 135081, x: 120416)>
  dask.array<getitem, shape=(169, 17, 135081, 120416), dtype=float64, chunksize=(1, 1, 1024, 1024), chunktype=numpy.ndarray>
  Coordinates: (12/26)
• y (y) float64 2.665e+06 2.665e+06 ... 1.314e+06
• x (x) float64 3.98e+05 3.98e+05 ... 1.602e+06
• time (time) datetime64[ns] 2021-01-01T01:34:24 ......
  id (time) <U24 'S2A_52JGS_20210101_0_L2A' ... 'S...
• band (band) <U8 'overview' 'visual' ... 'WVP' 'SCL'
  platform <U11 'sentinel-2a'
  ... ...
  sentinel:grid_square (time) <U2 'GS' 'GR' 'DS' ... 'UP' 'LH' 'UF'
  title (band) object None ... 'Scene Classification ...
  common_name (band) object None None 'coastal' ... None None
  center_wavelength (band) object None None 0.4439 ... None None
  full_width_half_max (band) object None None 0.027 ... None None None
  spatial_ref int64 0
  Attributes:
  spec: RasterSpec(epsg=3107, bounds=(293240, 1310090, 1710210, 27...
  resolution: 10
  grid_mapping: spatial_ref
  <xarray.DataArray 'nanmedian-aa322ab12953db1f2144da27ca020bd8' (band: 1, y: 135081, x: 120416)>
  dask.array<nanmedian, shape=(1, 135081, 120416), dtype=float64, chunksize=(1, 1024, 1024), chunktype=numpy.ndarray>
  Coordinates: (12/16)
• y (y) float64 2.665e+06 2.665e+06 ... 1.314e+06
• x (x) float64 3.98e+05 3.98e+05 ... 1.602e+06
• band (band) <U8 'B02'
  platform <U11 'sentinel-2a'
  sentinel:valid_cloud_cover bool True
  instruments <U3 'msi'
  ... ...
  gsd (band) object dask.array<chunksize=(1,), meta=np.ndarray>
  title (band) object dask.array<chunksize=(1,), meta=np.ndarray>
  common_name (band) object dask.array<chunksize=(1,), meta=np.ndarray>
  center_wavelength (band) object dask.array<chunksize=(1,), meta=np.ndarray>
  full_width_half_max (band) object dask.array<chunksize=(1,), meta=np.ndarray>
  spatial_ref int64 0
  [ ] | 0% Completed | 4min 2.0s/home/ubuntu/anaconda3/envs/richard/lib/python3.8/site-packages/numpy/lib/nanfunctions.py:1113: RuntimeWarning: All-NaN slice encountered
  r, k = function_base._ureduce(a, func=_nanmedian, axis=axis, out=out,
  [########################################] | 100% Completed | 3hr 45min 42.6s[[B

[RichardScottOZ]

1 band, 1 million square kilometres.