Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

new: dev,usr: HADTWO-2127: Function to mount lustre as Shuffle Dir #19

Open
wants to merge 3 commits into
base: master
Choose a base branch
from
Open

new: dev,usr: HADTWO-2127: Function to mount lustre as Shuffle Dir #19

Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
33ac17e
new: dev,usr: HADTWO-2127: Added bootstrap function to mount lustre a…
Aug 5, 2019
bdee9ec
Added deletion logic in bootstrap
Aug 8, 2019
060c26a
fix: dev: HADTWO-2127: Install xmlstarlet if required
Aug 23, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
31 changes: 31 additions & 0 deletions hadoop/util.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -73,3 +73,34 @@ function use_java8() {
restart_worker_services
fi
}

# Mount given lustre fsx DNS as Shuffle Directory
# Mandatory functional param -> Lustre Dns Name
function mount_lustre_as_shuffle_dir() {
lustre_dns=$1
if [ -z "$lustre_dns" ]; then
echo "Specifying Lustre DNS is must!"
return 1
else
sudo yum -y install xmlstarlet
cluster_id=`nodeinfo cluster_id`
cluster_inst_id=`nodeinfo cluster_inst_id`
instance_folder_identifier="$cluster_id-$cluster_inst_id"
mkdir -p /lustre/
mount -t lustre ${lustre_dns}@tcp:/fsx /lustre
mkdir -p /lustre/qubole/${instance_folder_identifier}
chmod 777 /lustre/qubole/${instance_folder_identifier}
chmod 777 /lustre/qubole/
chmod 777 /lustre/
xmlstarlet ed --inplace --update "/configuration/property[name='yarn.nodemanager.shuffle-dirs']/value" --value /lustre/qubole/${instance_folder_identifier} /usr/lib/hadoop2/etc/hadoop/yarn-site.xml
delete_previous_shuffle_dirs $cluster_id $instance_folder_identifier &
fi
}

# Delete data of shuffle dirs that were created previously for cluster id's prev instances
# First param -> Cluster id for which shuffle -dir is to be deleted
# Second Param -> For that is to be avoided for deletion
function delete_previous_shuffle_dirs() {
cd /lustre/qubole
ls | grep $1- | grep -v $2 | xargs rm -rf
}