▸ Guides / Monitoring
CloudWatch log retention: find the log groups that never expire
Updated · Prices checked against AWS pricing · 7 min read
CloudWatch Logs keeps every log event forever unless someone sets a retention period on its log group. Ingestion is a steady monthly cost, but storage is not: every month a log group never expires, it holds more data than the month before, and the bill for keeping it grows with it.
This guide covers what CloudWatch Logs charges, why so many log groups end up set to Never expire, and how to find them and set a retention with the AWS CLI.
[ What it costs ]
CloudWatch Logs bills separately for getting logs in and for keeping them. These are the US East (N. Virginia) prices; other regions differ.
| Ingestion, Standard log class | $0.50 per GB |
|---|---|
| Ingestion, Infrequent Access log class | $0.25 per GB |
| Storage | $0.03 per GB-month |
Storage is the part that grows. Take one log group that adds 100 GB to storage each month (stored logs are compressed, so this is less than it ingests). Left on Never expire, it keeps every month's data:
| Stored after 1 year, Never expire | 1.2 TB · $36/month |
|---|---|
| Stored after 3 years, Never expire | 3.6 TB · $108/month |
| Storage paid over 3 years, Never expire | $1,998 |
| Storage paid over 3 years, 30-day retention | about $108 |
Ingestion is the same in both cases. Retention only changes how much you keep, and with it a storage bill that otherwise never stops rising.
[ Why logs pile up ]
- ▸There is no default retention. A new log group keeps its data indefinitely until someone sets a retention period, and the console shows that as Never expire.
- ▸AWS creates log groups for you. Lambda creates
/aws/lambda/<function>the first time a function runs, and other services do the same. Nobody decided how long to keep those logs, so nobody set a retention. - ▸Nothing breaks. A log group that never expires works exactly like one that does. The only signal is a storage line on the bill that grows a little every month.
[ Find them ]
A log group with no retention has no retentionInDays. This lists those log groups in the region your CLI is set to, biggest first, with the bytes each one stores:
$ aws logs describe-log-groups --query "reverse(sort_by(logGroups[?retentionInDays==null], &storedBytes))[].[logGroupName,storedBytes]" --output table
Log groups are regional. This runs the same check in every region enabled for the account and puts the region in front of each log group:
$ for region in $(aws ec2 describe-regions --query "Regions[].RegionName" --output text); do aws logs describe-log-groups --region "$region" --query "logGroups[?retentionInDays==null].[logGroupName,storedBytes]" --output text | sed "s/^/$region /"; done
Both run in bash or zsh, and in AWS CloudShell, which already has your credentials. The CLI follows the pagination itself, so accounts with thousands of log groups come back complete.
[ Set retention ]
Set a retention period on one log group:
$ aws logs put-retention-policy --log-group-name /aws/lambda/my-function --retention-in-days 30
Retention is a number of days, and CloudWatch accepts only these values: 1, 3, 5, 7, 14, 30, 60, 90, 120, 150, 180, 365, 400, 545, 731, 1096, 1827, 2192, 2557, 2922, 3288 or 3653.
To give every log group in the region that has none a 30-day retention:
$ aws logs describe-log-groups --query "logGroups[?retentionInDays==null].logGroupName" --output text | tr "\t" "\n" | while read -r name; do aws logs put-retention-policy --log-group-name "$name" --retention-in-days 30; done
To stop new log groups starting out as Never expire, define them yourself with a retention instead of letting the service create them: retention_in_days on aws_cloudwatch_log_group in Terraform, or RetentionInDays on AWS::Logs::LogGroup in CloudFormation.
[ Across many accounts ]
The commands above cover one account. Across ten or fifty, finding the log groups that never expire means assuming a role in each, running the loop, and comparing stored sizes across all of them to see which ones matter.
Resource Atlas does that in one scan. It runs inside an AWS account you own, reads every connected account through a read-only role, and lists their log groups in one table with retention, stored size and KMS key, and the ones set to Never expire counted at the top. It covers the region you deploy it to in each account; for other regions, the loop above still does the job. Community is free for up to 2 AWS accounts.
[ See it in the demo ]
The live demo scans a sample estate of three AWS accounts. Its CloudWatch log groups page lists 28 log groups and flags the 10 set to Never expire, with each one's stored size. The largest, /acme/prod/access, holds about 1.5 TB.
Questions
What is the default retention for a CloudWatch log group?
There is none: by default CloudWatch Logs keeps log data indefinitely, which the console shows as Never expire. That includes log groups that AWS services create for you, such as the one Lambda creates the first time a function runs.
How much does CloudWatch Logs cost?
In US East (N. Virginia), ingesting logs costs $0.50 per GB in the Standard log class and $0.25 per GB in Infrequent Access, and storing them costs $0.03 per GB per month. Other regions are priced differently.
Does setting a shorter retention delete old logs straight away?
Not immediately. Log events past the new retention period are deleted within about 72 hours, occasionally longer. Once they are marked for deletion they stop counting toward your storage charges.
Which retention periods can I choose?
1, 3, 5, 7, 14, 30, 60, 90, 120, 150, 180, 365, 400, 545, 731, 1096, 1827, 2192, 2557, 2922, 3288 or 3653 days. To go back to keeping logs forever, delete the retention policy.
How do I keep logs for years without paying CloudWatch storage?
Export the log group to an S3 bucket, where long-term storage costs less, and give the log group a short retention. Check any compliance requirements before shortening retention, because expired log data cannot be recovered.