Confusion matrix in terms of map area proportions instead of sample size

[hannah-rae]

Currently, our intercomparison and other metrics are being computed based on the confusion matrix expressed in terms of sample size. However, the recommendation in Stehman & Foody 2019 is to compute metrics from the confusion matrix expressed in terms of map area proportions of each class (which is what we do for area estimation in src/area_utils.py see lines 563-567 in that file).

This PR adds a new function to compute the mapped area for each class and use these totals to compute the "area matrix". There are a few issues to be addressed still:

• I count the number of pixels in each class by summing the binary values (e.g., if cropland = 1 and noncrop = 0, then the sum of the map is equal to the number of cropland pixels). However, the sum returned by the ee Reducer is a float, while I would expect an integer. [SOLVED] This is because the reduction is weighted by default, so that partial pixels resulting from the clip() operation, e.g., are partial. I changed this to use an unweighted reduction because this is how we do it in our area estimation pipeline too and integers seem more transparent to me. The difference is negligible between the two anyway.
• Getting the number of pixels for the ensemble in a way that lets us be flexible about which map(s) we specify to use for the ensemble is tricky. I am still thinking about the best way to do this. [SOLVED]
• The F1 scores for the Rwanda test case do not look right. I am still investigating this. [SOLVED] This was because the F1 score (and some of the standard errors) were computed using the sample metrics from the sklearn classification report, but others were computed using the population error matrix.
• Update all of the notebooks under maps

[review-notebook-app]

Check out this pull request on 

See visual diffs & provide feedback on Jupyter Notebooks.

---

Powered by ReviewNB

[ivanzvonkov]

Here's an alternative to getting the crop/non-crop proportion without the need for two reduceRegion calls.
It also may be more "accurate" since the proportions are computed using area vs pixel count.

geom = aoi.geometry()
area = geom.area()
crop_proportion = binary_image \
    .multiply(ee.Image.pixelArea()) # Multiplies each pixel by its area
    .divide(area)                   # Divides each pixel by the total area
    .reduceRegion(
        reducer=ee.Reducer.sum(), 
        geometry=geom, 
        scale=self.resolution, 
        maxPixels=1e12,
        bestEffort=True        # Should reduce scale if computation times out
    ).get("crop").getInfo()

w_j = np.array([1-crop_proportion, crop_proportion])
return w_j

[ivanzvonkov]

Can be multiplied by area to get a_j

[ivanzvonkov]

Never mind I see you need the exact amount of pixels for the producer variance estimator 🤦

[ivanzvonkov]

Thanks for adding this!

[ivanzvonkov]

This looks excellent! Looks like worldcover got a bit of a boost from these area weights. Thank you for updating the template notebook and the descriptions in each notebook.