HADOOP-18876. ABFS: Change default for fs.azure.data.blocks.buffer to…

… bytebuffer (#6009)


The default value for fs.azure.data.blocks.buffer is changed from "disk" to "bytebuffer"

This will speed up writing to azure storage, at the risk of running out of memory
-especially if there are many threads writing to abfs at the same time and the
upload bandwidth is limited.

If jobs do run out of memory writing to abfs, change the option back to "disk"

Contributed by Anmol Asrani

hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/constants/ConfigurationKeys.java

@@ -85,7 +85,7 @@ public final class ConfigurationKeys {
   /**
    * What data block buffer to use.
    * <br>
-   * Options include: "disk"(Default), "array", and "bytebuffer".
+   * Options include: "disk", "array", and "bytebuffer"(Default).
    * <br>
    * Default is {@link FileSystemConfigurations#DATA_BLOCKS_BUFFER_DEFAULT}.
    * Value: {@value}

hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/constants/FileSystemConfigurations.java

@@ -132,11 +132,13 @@ public final class FileSystemConfigurations {
    */
   public static final String DATA_BLOCKS_BUFFER_DISK = "disk";
 
+  public static final String DATA_BLOCKS_BYTEBUFFER = "bytebuffer";
+
   /**
    * Default buffer option: {@value}.
    */
   public static final String DATA_BLOCKS_BUFFER_DEFAULT =
-      DATA_BLOCKS_BUFFER_DISK;
+          DATA_BLOCKS_BYTEBUFFER;
 
   /**
    * IO rate limit. Value: {@value}