Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ananthakalusivalingam patch 1 #59

Open
wants to merge 13 commits into
base: master
Choose a base branch
from
20 changes: 15 additions & 5 deletions requestToUsage/README.md
Original file line number Diff line number Diff line change
@@ -1,9 +1,19 @@
#requestToUsage.sh
This script is used with the syntax:
```request.sh request.log```
(replace request.log with the name of your request log file)
```requestToUsage.sh artifactory-request.log $Delimiter $Prefix```
(replace artifactory-request.log with the name of your request log file. $Delimiter could be pipe or comma. Default is comma.)

It outputs a file request.csv which you can open in excel.
The furthest right field is your overall usage in gigabytes over the period of the request log. You need to figure out the difference in the dates and turn it into a 30-day figure to get a monthly usage figure.
1.It consolodates the GET requests and generate a CSV file with the naming convention of
{Date}_GET_Requests.CSV.
2.It consolodates the POST requests and generate a CSV file with the naming convention of
{Date}_POST_Requests.CSV
3.It generates DataUsageSummary.txt which lists the individual files and their data
usage size.

#multiLogParseWithCounter.sh
This script is used with the syntax:
```multiLogParseWithCounter.sh ../var/log/archived $Delimiter```
This will recursively go behind all the artifactory-requet-*.log from the archived folder and summarizes
the data usage in the DataUsageSummary.txt file. Every log file will be split with ONE CSV file with GET
requests and Second CSV file with POST requets. $Delimiter could be pipe or comma. Default is comma.

TODO: Automate date difference calculation to monthly statistic
15 changes: 15 additions & 0 deletions requestToUsage/multiLogParseWithCounter.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
#!/bin/bash
LOC=$1
DELIMITER=$2
${LOC:-"."}
if [ ! -d $LOC ]; then
echo "Usage is: ./multiLogParseWithCounter.sh /path/to/logs/ <DELIMITER> <OPTIONAL_PREFIX>"
else
echo "Using directory $LOC as LOC"
fi
COUNTER=1
cd $LOC
for i in artifactory-request.*; do
.\requestToUsage.sh $LOC\\$i $DELIMITER $COUNTER
COUNTER=$(( COUNTER + 1 ))
done
45 changes: 34 additions & 11 deletions requestToUsage/requestToUsage.sh
Original file line number Diff line number Diff line change
@@ -1,27 +1,50 @@
#!/bin/bash
FILE=$1
if [ ! -f $FILE ]; then
echo "Correct usage is: requestToUsage.sh request.log [optional prefix string]"
echo "Correct usage is: requestToUsage.sh request.log [optional delimiter] [optional prefix string]"
fi
PREFIX=$2
DELIMITER=$2

OUTPUT=${PREFIX:+${PREFIX}-}$(head -c 8 $FILE).csv
if [ -z "$2" ]
then
echo "No argument supplied for delimiter. Set comma as delimiter"
DELIMITER=','
fi
PREFIX=$3

OUTPUT=${PREFIX:+${PREFIX}-}$(head -c 10 $FILE)_POST_Requests_Only.csv

awk '!/0$/' $FILE > $OUTPUT
if sed --version
then sed "s/[|]/,/g" $OUTPUT -i
awk '/POST/' $FILE > $OUTPUT
if sed --version
then sed "s/[$DELIMITER]/,/g" $OUTPUT -i
echo "I am in sed with out error"
else
echo "Please ignore the above error message from sed, switching to gsed."
gsed "s/[|]/,/g" $OUTPUT -i
gsed "s/[$DELIMITER]/,/g" $OUTPUT -i
fi
echo "Successfully outputted to $OUTPUT"
if date --version
then echo 'date'
else
else
echo "Please ignore the above error message from date, switching to gdate."
echo 'gdate'
fi
awk -v var=$OUTPUT -F'|' '{sum+=$8;}END{print var "|" sum}' $OUTPUT >> DataUsageSummary.txt

OUTPUT=${PREFIX:+${PREFIX}-}$(head -c 10 $FILE)_GET_Requests_Only.csv

echo "0,0,0,0,0,0,0,0,=SUM(I:I)/(1024^3),0,0" >> $OUTPUT
echo "Added calculation line."
echo "Open $OUTPUT in excel or a similar spreadsheet program"
awk '/GET/' $FILE > $OUTPUT
if sed --version
then sed "s/[$DELIMITER]/,/g" $OUTPUT -i
else
echo "Please ignore the above error message from sed, switching to gsed."
gsed "s/[$DELIMITER]/,/g" $OUTPUT -i
fi
echo "Successfully outputted to $OUTPUT"
if date --version
then echo 'date'
else
echo "Please ignore the above error message from date, switching to gdate."
echo 'gdate'
fi
awk -v var=$OUTPUT -F'|' '{sum+=$9;}END{print var "|" sum}' $OUTPUT >> DataUsageSummary.txt