Using The Amazon Cost Explorer API to Conserve Cloud Costs

AWS Cost Explorer is a tool that allows you to view and analyze your costs and usage of the Amazon cloud. It provides two main reports: the Cost and Usage report and the Cost Manager RI report. These reports come with pre-configured views and enable customization for specific needs. You can use the reports to:

View EC2 pricing and cost data from the past 12 months to predict how much you will spend in the next 12 months.
Get advice on purchasing Reserved Instances, in which you make a long-term commitment to purchase Amazon resources in exchange for a discount.
Identify areas that need further investigation and view trends to help you understand your costs.
Understand the cost impact of cloud migrations.

Amazon provides the Cost Management user interface free for all AWS users once they enable it.

The AWS Cost Explorer API

You can also access data programmatically using the AWS Cost Explorer API. This makes it possible to integrate analytics applications to Cost Management and leverage Cost Management data for automated processes in AWS.

The Cost Explorer API allows you to query cost and usage data. You can query summary data such as total monthly cost and total daily usage. In production, you can also query detailed data, such as the number of daily write operations to DynamoDB database tables.

Note there is a fee of $0.01 per paginated API request.

Getting Basic AWS Cost Data: The GetCostAndUsage Function

The basic function of AWS Cost Explorer is to report your actual costs and usage in AWS. You can do this via the API using the GetCostAndUsage() function.

Request arguments:

Filter: Lets you filter costs by dimensions such as the Amazon SERVICE or the AZ (Availability Zone). It supports the operators And, Or, and Not to combine criteria in your query.
Granularity (required): Lets you report MONTHLY, DAILY, or HOURLY costs.
GroupBy: Lets you group costs using up to two criteria, which can include dimensions, tags, or cost categories.
Metrics (required): Specifies which metrics you want to report. Can be one of the following: BlendedCost, NetUnblendedCost, UnblendedCost, AmortizedCost, NetAmortizedCost, NormalizedUsageAmount, and UsageQuantity. Learn more about cost metrics here.
TimePeriod (required): Sets the start date (inclusive) and end date (exclusive) for cost data. Following a recent update, the API supports using the current date as a start date. The date string should be in this format: YYYY-MM-DD.
NextPageToken: Lets you request the next page of results, which incurs additional cost.

The syntax looks like this:

{
   "Filter": { 
      "And": [ 
         "Expression"
      ],
      "CostCategories": { 
         "Key": "string",
         "MatchOptions": [ "string" ],
         "Values": [ "string" ]
      },
      "Dimensions": { 
         "Key": "string",
         "MatchOptions": [ "string" ],
         "Values": [ "string" ]
      },
      "Not": "Expression",
      "Or": [ 
         "Expression"
      ],
      "Tags": { 
         "Key": "string",
         "MatchOptions": [ "string" ],
         "Values": [ "string" ]
      }
   },
   "Granularity": "string",
   "GroupBy": [ 
      { 
         "Key": "string",
         "Type": "string"
      }
   ],
   "Metrics": [ "string" ],
   "NextPageToken": "string",
   "TimePeriod": { 
      "End": "string",
      "Start": "string"
   }
}

Usage Example

The following example shows how to request daily unblended costs for a given month for a specific account, excluding costs of type Refund. The data will be grouped by Availability Zone (AZ).

import boto3
client = boto3.client('ce')
result = client.get_cost_and_usage(
    TimePeriod = {
        'Start': 2021-11-01,
        'End': 2021-12-01
    },
    Granularity = 'DAILY',
    Filter = {
        "And": [{
            "Dimensions": {
                "Key": "LINKED_ACCOUNT",
                "Values": 123456789012
            }
        }, {
            "Not": {
                "Dimensions": {
                    "Key": "RECORD_TYPE",
                    "Values": "Refund"
                }
            }
        }]
    },
    Metrics = ["UnblendedCost"],
    GroupBy = [
        {
            'Type': 'DIMENSION',
            'Key': 'AZ'
        }
    ]
)

Advanced API Operations

The AWS Cost Explorer API provides more than just historical usage data. It uses machine learning algorithms to show cost anomalies and predict future costs. Here are some of the advanced operations available. See the complete list of operations here.

GetAnomalies

Analyzes your data to discover cost anomalies during a specified time period.

Request arguments

DateInterval: Specifies the time frame to search for anomalies. The API returns an anomaly object with an AnomalyEndDate.
Feedback: Lets you filter anomaly results using the feedback field in the returned anomaly object.
MaxResults: Specifies how many entries you want to return.
MonitorArn: Specifies an Amazon Resource Name (ARN) to return cost anomalies for.
NextPageToken: Lets you return the next page of results.
TotalImpact: Lets you view results by their expected financial impact. For example, GREATER_THAN 1000.00 will show anomalies that are expected to impact your account on the order of $1,000 or more.

GetCostForecast

Provides a forecast of future spending over a selected time period that you choose, based on historical cost data.

Request arguments

Filter, Granularity, Metric, TimePeriod: These are the same as in GetCostAndUsage above.
PredictionIntervalLevel: You can use this argument to specify a confidence level for the estimate. The prediction interval is the range of predictions in which the actual value is expected to fall. The wider the prediction interval, the higher the confidence level.

GetRightsizingRecommendation

This function provides recommendations for optimizing idle or poorly utilized Amazon EC2 instances. It can identify an underutilized EC2 instance and recommend whether you should downsize it (because it is underutilized) or terminate it (because it is not utilized at all).

Request arguments

Filter, NextPageToken, PageSize: Same as above.
Configuration: Lets you define which EC2 instances you want to receive recommendations for. You can do this by specifying specific instance families. You can also ask the recommendation to consider current Savings Plans or Reserved Instances.
Service: Currently, the only valid value is AmazonEC2 because this API only provides recommendations for EC2.

Best Practices for the AWS Cost Explorer API

When using the Cost Explorer API, you should consider the following best practices.

Consider API Pagination Costs

Amazon charges you for each paginated request to the Cost Explorer API, so you should determine which dataset you want the API to access before submitting any queries. AWS updates its billing information up to three times per day. Standard use cases and workloads for the Cost Explorer API typically have a pattern of calls once or multiple times a day. To ensure you get the most up-to-date cost data, you should query for the exact time you want to view it.

If you use the Cost Explorer API to create an application, you should architect the application to include a caching layer. You can then update your underlying data regularly for end-users without triggering queries for each time a user in your organization accesses the data.

Secure Cost Explorer API Access

API security should be a primary concern for Cost Explorer API users since cost data is sensitive for most organizations. IAM users can only query the AWS Cost Explorer API if they have explicit permission. If you grant Cost Explorer API access to a user, you allow that user to query any usage or cost data associated with your account. When you configure access to the Cost Explorer API, you should thus create a unique IAM user with programmatic access permission. If you allow multiple users to query the Cost Explorer API, you should create a separate programmatic access role for each IAM user with query access.

Query the Cost Explorer API Using Filters

When you query the Cost Explorer API, you should use filters to refine the queries and ensure that you only get the required data. You can limit the set of results returned by your request using filters, so you will receive the query results faster than if you were accessing a larger data set. Alternatively, you can restrict the time range for the query to a smaller interval.

If you add one or multiple grouping dimensions to a query, it may make the result larger and impact the performance of the query. You may benefit from filtering your data to improve performance, depending on the use case.

Conclusion

In this article, I explained the basics of the Amazon Cost Explorer API and showed how to perform the following operations:

GetCostAndUsage: Receiving filtered cost data from your AWS account.
GetAnomalies: Analyze historical cost data to identify anomalies that can impact your account and result in unexpected costs.
GetCostForecast: Generate a forecast of future costs based on historical data.
GetRightsizingRecommendation: Generate recommendations for optimizing EC2 instances that are underutilized or completely unutilized.

I also provided several recommendations for using the API more effectively, including how to conserve pagination costs and ensure secure access to sensitive cost data.

I hope this will be useful as you expand your ability to estimate and act on AWS cost data.