feat: Fix Cost Calculation for AWS CloudFront Distributions #1083

bishal7679 · 2023-10-11T18:38:34Z

Problem #877

Currently cost for AWS cloudfront is full static and showing wrong.

Solution

This PR is fixing that issue by adding more metrics for lambda-edge-duration, lambda-edge-requests, function-invocations with pricemap fetching.

Changes Made

Fetching price info using AWS SDK price module using AmazonCloudFront service code and appropriate filters.
getting price map using util function GetPriceMap from utils.go
fetching distribution objects using cloudfront client
also fetching lambda-edge-duration, lambda-edge-requests, function-invocations metrics using cloudwatch
Finally added dynamic calculation as per all the metrics

Checklist

Code follows the contributing guidelines
Changes have been thoroughly tested
Documentation has been updated, if necessary
Any dependencies have been added to the project, if necessary

Pricingoutput :-
pricing.json

Signed-off-by: bishal7679 <[email protected]>

bishal7679 · 2023-10-12T02:57:35Z

/cc @AvineshTripathi @Azanul

Azanul · 2023-10-12T04:02:50Z

providers/aws/cloudfront/distributions.go

+func ConvertBytesToTerabytes(bytes int64) float64 {
+	return float64(bytes) / 1099511627776
+}


Can't we convert to GB instead? We do the same by multiplying by 1024 to the result

Yes I thought it would be more efficient not to multiply inline

ok i got it

Azanul · 2023-10-12T04:52:53Z

providers/aws/cloudfront/distributions.go

+			dataTransferToInternetCost := awsUtils.GetCost(priceMap["CloudFront-DataTransfer-In-Bytes"], sizeInTBUpload*1024)
+
+			dataTransferToOriginCost := awsUtils.GetCost(priceMap["CloudFront-DataTransfer-Out-Bytes"], sizeInTBDownload*1024)
+
+			requestsCost := awsUtils.GetCost(priceMap["CloudFront-Requests"], requests/10000)


Where did you get the keys for priceMap?

Should we keep it static like previous???

Let me see if I can find the correct keys

Azanul · 2023-10-12T04:53:17Z

providers/aws/cloudfront/distributions.go


-			// calculate requests cost
-			requestsCost := requests * 0.000001
+			functionInvocationsCost := awsUtils.GetCost(priceMap["AWS-CloudFront-FunctionInvocation"], functionInvocation)


Where did you get this key for priceMap?

Yes! updated the code with a function. Searched about it but was not getting the appropriate key. Give your suggestion on the getRate function now

Signed-off-by: bishal7679 <[email protected]>

bishal7679 · 2023-10-13T06:12:30Z

@Azanul any update??

Azanul · 2023-10-13T06:37:37Z

@Azanul any update??

Yes, I looked into costing of CloudFront. I found out that there are only 3 components to CloudFront cost: CF distributions, CF Functions & CF Origin Shield requests. ref: https://aws.amazon.com/cloudfront/pricing/
Other costs that are being calculated here don't belong here, they belong with their own services such as AWS Lambda.
I'd suggest you look into what a CloudFront distribution is and their review it's cost calculation

bishal7679 · 2023-10-13T06:39:51Z

@Azanul any update??

Yes, I looked into costing of CloudFront. I found out that there are only 3 components to CloudFront cost: CF distributions, CF Functions & CF Origin Shield requests. ref: https://aws.amazon.com/cloudfront/pricing/ Other costs that are being calculated here don't belong here, they belong with their own services such as AWS Lambda. I'd suggest you look into what a CloudFront distribution is and their review it's cost calculation

Yes actually I went through it already

bishal7679 · 2023-10-13T06:40:48Z

I am including lambda-edge as our pricingoutput is giving those
and what the issue with it? if someone use lambda as edge with cloudfront distribution then they will be charged by some amount otherwise it will remain $0
isnt it?

Azanul · 2023-10-13T06:59:44Z

I am including lambda-edge as our pricingoutput is giving those and what the issue with it? if someone use lambda as edge with cloudfront distribution then they will be charged by some amount otherwise it will remain $0 isnt it?

The issue that you're trying to resolve is for CloudFront Distributions, the file that you're making changes to is for CloudFront Distributions not CloudFront Functions or Origin Shield requests.

if someone use lambda as edge with cloudfront distribution then they will be charged by some amount

That'll be accounted for in it's own resource

Azanul · 2023-10-13T07:05:37Z

I think the respective issue description is misleading as it says we do not have the calculation whereas the calculation is there. The only thing off about the calculation seems to be the hard-coded prices.
cc: @mlabouardy @AvineshTripathi @ShubhamPalriwala

bishal7679 · 2023-10-13T07:11:18Z

I think the respective issue description is misleading as it says we do not have the calculation whereas the calculation is there. The only thing off about the calculation seems to be the hard-coded prices. cc: @mlabouardy @AvineshTripathi @ShubhamPalriwala

Yeah I noticed when I took this issue! So should we include all the charges distribution + function. and resolve with this issue only? Otherwise we have to create different issue for those.
WDYT only distribution charges are enough or adding others with it would make it much more efficient @Azanul @mlabouardy @AvineshTripathi

bishal7679 · 2023-10-15T05:29:19Z

@jakepage91 Can we add hold label for this PR

mlabouardy · 2023-10-19T19:15:31Z

@jakepage91 Can we add hold label for this PR

Hey @bishal7679 sorry for jumping late to the thread, is there something I can help with to move forward with your PR?

AvineshTripathi · 2023-10-20T07:30:07Z

@mlabouardy we have intentionally kept this on hold to dicuss the changes on cost calculation to avoid repeatative work!

bishal7679 · 2023-10-21T17:14:51Z

closing this PR as awsutils recent changes are not being rendered onto my feature/correct-cost-for-cloudformation branch locally [PS:- did git pull & sync]
and raising a new PR

fixed cost of cloudfront

e2c34bd

Signed-off-by: bishal7679 <[email protected]>

bishal7679 requested review from mlabouardy, ShubhamPalriwala, AvineshTripathi, greghub and Traxmaxx as code owners October 11, 2023 18:38

Azanul requested changes Oct 12, 2023

View reviewed changes

added function invocation cost of cloudfront

a96976f

Signed-off-by: bishal7679 <[email protected]>

bishal7679 force-pushed the feature/correct-cost-for-cloudfront branch from f1c8da8 to a96976f Compare October 12, 2023 05:01

Merge branch 'develop' into feature/correct-cost-for-cloudfront

bc4cc8a

mlabouardy requested a review from Azanul October 12, 2023 09:29

Merge branch 'develop' into feature/correct-cost-for-cloudfront

258f4f3

Merge branch 'develop' into feature/correct-cost-for-cloudfront

8768cc3

mlabouardy added the aws label Oct 15, 2023

mlabouardy added this to the v3.1.2 milestone Oct 15, 2023

jakepage91 added the hold label Oct 16, 2023

bishal7679 closed this Oct 21, 2023

bishal7679 deleted the feature/correct-cost-for-cloudfront branch October 21, 2023 17:15

This was referenced Oct 21, 2023

feat: Fix Cost Calculation for AWS CloudFront Distributions & Functions #1123

Closed

feat: Fix Cost Calculation for AWS CloudFront Distributions & Functions #1131

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Fix Cost Calculation for AWS CloudFront Distributions #1083

feat: Fix Cost Calculation for AWS CloudFront Distributions #1083

bishal7679 commented Oct 11, 2023

bishal7679 commented Oct 12, 2023

Azanul Oct 12, 2023

bishal7679 Oct 12, 2023

bishal7679 Oct 12, 2023

Azanul Oct 12, 2023

bishal7679 Oct 12, 2023

Azanul Oct 12, 2023

Azanul Oct 12, 2023

bishal7679 Oct 12, 2023

bishal7679 commented Oct 13, 2023

Azanul commented Oct 13, 2023

bishal7679 commented Oct 13, 2023

bishal7679 commented Oct 13, 2023 •

edited

Loading

Azanul commented Oct 13, 2023

Azanul commented Oct 13, 2023

bishal7679 commented Oct 13, 2023 •

edited

Loading

bishal7679 commented Oct 15, 2023

mlabouardy commented Oct 19, 2023 •

edited by Azanul

Loading

AvineshTripathi commented Oct 20, 2023

bishal7679 commented Oct 21, 2023

feat: Fix Cost Calculation for AWS CloudFront Distributions #1083

feat: Fix Cost Calculation for AWS CloudFront Distributions #1083

Conversation

bishal7679 commented Oct 11, 2023

Problem #877

Solution

Changes Made

Checklist

bishal7679 commented Oct 12, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bishal7679 commented Oct 13, 2023

Azanul commented Oct 13, 2023

bishal7679 commented Oct 13, 2023

bishal7679 commented Oct 13, 2023 • edited Loading

Azanul commented Oct 13, 2023

Azanul commented Oct 13, 2023

bishal7679 commented Oct 13, 2023 • edited Loading

bishal7679 commented Oct 15, 2023

mlabouardy commented Oct 19, 2023 • edited by Azanul Loading

AvineshTripathi commented Oct 20, 2023

bishal7679 commented Oct 21, 2023

bishal7679 commented Oct 13, 2023 •

edited

Loading

bishal7679 commented Oct 13, 2023 •

edited

Loading

mlabouardy commented Oct 19, 2023 •

edited by Azanul

Loading