Skip to content

Commit

Permalink
Feature/cudos new v (#2)
Browse files Browse the repository at this point in the history 
* Update cost-anomalies.yaml (aws-samples#625)

* Tao 3 (aws-samples#626)

* dataset updates for TAO 3

* fix calc field order

---------

Co-authored-by: Prykhodko <[email protected]>

* fix typos (aws-samples#623)

* fix logic (aws-samples#607)

* Minor fixes (aws-samples#610)

* Added automated Athena workgroup creation if missing (aws-samples#618)

* Feature: Allow custom Role for QuickSight DataSet (aws-samples#606)

* bump version (aws-samples#622)

* Tao 3 (aws-samples#627)

* dataset updates for TAO 3

* fix calc field order

* added changelog

---------

Co-authored-by: Prykhodko <[email protected]>

* Update CHANGELOG-tao.md (aws-samples#629)

* bump release (aws-samples#630)

* Release2.28 (aws-samples#631)

* Update CHANGELOG-tao.md (aws-samples#632)

* Update CHANGELOG-tao.md (aws-samples#633)

* fix cfn for data collection lab (aws-samples#636)

* typo and lint (aws-samples#635)

* Allow SQL datasets for export (aws-samples#638)

* fix release script (aws-samples#628)

* Bugfix for workgroup creation in non us-east-1 region (aws-samples#642)

* Add Cloudfront Dashboards (aws-samples#639)

* Feature user add paginator (aws-samples#645)

* Bugfix to handle empty datasetarns (aws-samples#640)

* bump release (aws-samples#643)

* Fixing missing permissions for proper QS functioning after deploy (aws-samples#644)

* Data transfer dashboard initial migration from WAL to Workshopstudio (aws-samples#647)

* Fixed QS Data Source policy name collision (aws-samples#649)

* Adding missing perms, this prevents applying rls on customer_all ds (aws-samples#651)

Co-authored-by: vmindru <[email protected]>

* Expanding load resources (aws-samples#653)

* bump release (aws-samples#654)

* allways allow access to data collection bucket for account map access (aws-samples#656)

* Feature - better status - shows owners and last dataset ingestion   (aws-samples#660)

* Fix cfn - add databases permissions for data collection lab (aws-samples#663)

* Feature catalog (aws-samples#658)

* release/0.2.31 (aws-samples#664)

* Update common.py (aws-samples#665)

* Update resources.yaml (aws-samples#666)

* allow data collection bucket not to exist (aws-samples#667)

* Fix permisions (aws-samples#668)

* allow data collection bucket not to exist

* fix permissions

* fix some titles in cur deploy (aws-samples#670)

* Cudos v5 (aws-samples#661)

* bump release (aws-samples#671)

* cfn changes for cudos v5 (aws-samples#662)

* cfn changes for cudos v5

* bump release (aws-samples#672)

* Update resources.yaml (aws-samples#673)

* bump release (aws-samples#674)

* Release 0.2.35 (aws-samples#676)

* Changelog Update for CUDOS v5 (aws-samples#675)

* chnage names of resources for better update and remove CidVersion (aws-samples#678)

* Terraform: S3 object tags and fix for continous drift with KMS-encrypted bucket (aws-samples#679)

Co-authored-by: Sean Nixon <[email protected]>

* fix hourly_view (aws-samples#681)

* replace deepmerge by custom code (aws-samples#682)

* Update fixes (aws-samples#684)

* fix-empty-categories (aws-samples#677)

* Add product storage fields and better messages if CUR does not contain fields (aws-samples#680)

* bump release (aws-samples#685)

* change test to cudos v5 (aws-samples#686)

* Update CHANGELOG-cudos.md (aws-samples#687)

* Fixed supported dashboards links (aws-samples#690)

* update demo links (aws-samples#691)

* cleanup old files (aws-samples#688)

* release 0.2.37 (aws-samples#694)

* exclude known views and enforce naming (aws-samples#693)

* fix account map creation (aws-samples#689)

* release 0.2.38 (aws-samples#696)

* Fixes queries for sustainability proxy metrics dashboard (aws-samples#697)

* AWS Marketplace Single Pane of Glass (SPG) Dashboard v0.0.1 (aws-samples#692)

* release 0.2.39 (aws-samples#699)

* fixes joins for multiple business metrics per hour (aws-samples#700)

* update boto3 requirements (aws-samples#698)

* Update CHANGELOG-tao.md (aws-samples#701)

* Better error handling (aws-samples#702)

* fix: add glue:GetPartition action to QuickSightDataSourceRole (aws-samples#706)

* add checkov lints script (aws-samples#703)

* fix definitions deploy (aws-samples#707)

* allow choice of category (aws-samples#708)

* Manage CUR fields (aws-samples#710)

* fix non utf8 chars (aws-samples#713)

* release 0.2.40 (aws-samples#709)

* release 0.2.41 (aws-samples#715)

* Update links and bucket name to reflect data collection v3 (aws-samples#716)

* adjusting default s3 path to cid-data- prefix (aws-samples#717)

* release 0.2.42 (aws-samples#718)

* fix update from legacy (aws-samples#720)

* Update CHANGELOG-cudos.md (aws-samples#721)

* Update CHANGELOG-cudos.md

* Update CHANGELOG-cudos.md

* Fix QuickSight dataset refresh schedule creation for hourly_view and resource_view (aws-samples#726)

* release 0.2.43 (aws-samples#727)

* Changed the reference from user to group (aws-samples#728)

* release 0.2.44 (aws-samples#729)

* Update CHANGELOG-cudos.md (aws-samples#730)

* Update CHANGELOG-cudos.md (aws-samples#731)

* support paritions (aws-samples#723)

* more resilence to catalog errors (aws-samples#732)

* release 0.2.45 (aws-samples#733)

* Update CidSpiceRefreshLambda function (aws-samples#736)

* Update CidSpiceRefreshLambda function

Recently we implemented a fix (PR # 726) to resolve the dataset refresh scheduling issue for hourly_view and resource_view in CUDOS v5 dashboard.

In order to implement the fix for existing customers who have already deployed the CUDOS v5 dashboard, I have made changes to the CidSpiceRefreshLambda function by adding hourly_view and resource_view in it.

* Update version

Updated the version 0.2.45 to 0.2.46

* make url compatibile with China regions (aws-samples#738)

* create schedule even if we cannot see it (aws-samples#737)

* Fixes for KMS bucket encryption use case (aws-samples#741)

* Update CHANGELOG-cid.md (aws-samples#744)

* Update cid-cfn.yml (aws-samples#746)

* Update CHANGELOG-cudos.md (aws-samples#747)

* feat: adding RDS Extended Support Cost Projection dashboard definition and catalog entry.

* add context manager for parameters (aws-samples#748)

* add context manager for parameters

* add context manager for parameters

* Voicu aws/refresh datasets command (aws-samples#754)

* Refresh datasets in status command

---------

Co-authored-by: Voicu Chirtes <[email protected]>
Co-authored-by: Voicu <[email protected]>

* Removing actual cost fields, EDP parameter and adding 'About' sheet with legal notice, dashboard author information and links to the CID portal and email.

* removing default choice (aws-samples#757)

* Support theme (aws-samples#758)

* Yprikhodko patch 5 (aws-samples#760)

* Update CHANGELOG-cudos.md

* Update CHANGELOG-cudos.md

* Update CHANGELOG-cudos.md

* Update CHANGELOG-cudos.md

* Update CHANGELOG-cid.md (aws-samples#755)

* allow CN region and add lint (aws-samples#759)

* allow CN region and add lint

* add cfn-nag

* Update .github/workflows/security-scan.yml

* support cur replication for cn (aws-samples#753)

* support cur replication for cn

* Update cur-aggregation.yaml

* release 0.2.47 (aws-samples#742)

* fix managed policy (aws-samples#762)

* fix managed policy

* update python for checkov

* add checkov dep

* add checkov dep

* remove support of python3.7 and 3.8, add 3.12 and Bump boto3  (aws-samples#761)

* rename rds extended support to extended support (aws-samples#765)

* rename rds extended support to extended support

* more re-naming

* Update extended-support-cost-projection.yaml (aws-samples#766)

* Update extended-support-cost-projection.yaml

* Update extended-support-cost-projection.yaml

* Create Crawlers and IAM Roles for Datasource

* wip

* add management of CUR fields

* wip

* better type management

* refactoring

* fixes

* add warning if crawler is not well configured

* sort

* fix import

* refactoring cur

* remove resource id dep

* remove resource_id dep

* more refactoring

* more refactoring

* fixes

* fixes

* add cur creation

* fixes

* fixes

* fixes

* better creation workflow and messages plus database creation

* doc and workflow fixes

* doc and workflow fixes

* add cur.yaml

* add QS DS role management

* refactor iam

* allow customer to create dataset with new QS role

* release 0.3.0

* refactor export

* lint

* lint

* fix cur yaml

* fix role creation

* workaround IAM

* wip

* more fixes

* various fixes for the crawler and roles

* various fixes for the crawler and roles

* Update cid/helpers/quicksight/__init__.py

Co-authored-by: Yuriy Prykhodko <[email protected]>

* Update cid/cli.py

Co-authored-by: Yuriy Prykhodko <[email protected]>

* Update cid/commands/init_qs.py

Co-authored-by: Yuriy Prykhodko <[email protected]>

* Update cid/commands/init_qs.py

Co-authored-by: Yuriy Prykhodko <[email protected]>

* review fixes

* wip

* align roles

* lint

* more fixes

* minor fixes

* align role names

* fix comment

* remove print

* Update cid/common.py

Co-authored-by: Yuriy Prykhodko <[email protected]>

* Update cid/common.py

Co-authored-by: Yuriy Prykhodko <[email protected]>

---------

Co-authored-by: Yuriy Prykhodko <[email protected]>

* Support for Permissionboundary and custom role paths (aws-samples#745)

* graviton-opportunities-dashboard yaml (aws-samples#769)

* graviton dashboard folder

* Add files via upload

* Delete dashboards/graviton-opportunties-dashboard/readme.md

* Add files via upload

updated yaml file to v0.0.4

* Delete dashboards/graviton-opportunties-dashboard/graviton_dashboard_0.0.3.yaml

* Update catalog.yaml

* Rename graviton_dashboard_0.0.4.yaml to graviton_dashboard.yaml

* fix typo in folder name (aws-samples#770)

* Update dashboard id in graviton_dashboard.yaml (aws-samples#771)

* docs: fix README typos (aws-samples#774)

* Add status video (aws-samples#775)

* Update README.md

* Update README.md

* Update graviton_dashboard.yaml (aws-samples#776)

* Update graviton_dashboard.yaml (aws-samples#777)

* fix-policies-duplication (aws-samples#773)

* fix co path (aws-samples#778)

* Create CHANGELOG-aws-marketplace-spg.md (aws-samples#781)

Added changelog for v0.1.1 for AWS Marketplace single pane of glass

* Update last_kpi_tracker_view.sql (aws-samples#779)

* Update last_kpi_tracker_view.sql -  fix duplicate linked account data in the scenario where a linked account is moved from one org to other.

* Fix s3 object hash

* new dashboard scad-cca (aws-samples#786)

* Changed category to "Additional" (aws-samples#788)

* Adding EKS Extended Support Cost Projection dashboard definition (aws-samples#785)

* Adding EKS Extended Support Cost Projection dashboard definition, supporting dataset and view.

* Adding filtering by cluster version action in the main details table. Changing dashboard category back to 'Advanced'.

* Adjusting name for discount parameter and associated calculated fields.

* Added change log for the EKS Extended Support Cost Projection dashboard. (aws-samples#789)

* Better process from 0 (aws-samples#782)

* better choosing of name

* enhancements for flowless cretion of dashboard

* better export

* Fix permissions boundaries + release 0.3.1 (aws-samples#790)

* add-databases-permission (aws-samples#791)

* release 0.3.2 (aws-samples#793)

* fix-ri-sp-without-crawler (aws-samples#792)

---------

Co-authored-by: Iakov GAN <[email protected]>
Co-authored-by: Yuriy Prykhodko <[email protected]>
Co-authored-by: Prykhodko <[email protected]>
Co-authored-by: Gabor Schulz <[email protected]>
Co-authored-by: rohitsinglagithub <[email protected]>
Co-authored-by: Dan Malloy <[email protected]>
Co-authored-by: Voicu <[email protected]>
Co-authored-by: Veaceslav Mindru <[email protected]>
Co-authored-by: Chaitanya Shah <[email protected]>
Co-authored-by: Eric Christensen <[email protected]>
Co-authored-by: vmindru <[email protected]>
Co-authored-by: Sean Nixon <[email protected]>
Co-authored-by: Sean Nixon <[email protected]>
Co-authored-by: Gustavo Carreira <[email protected]>
Co-authored-by: Steffen Grunwald <[email protected]>
Co-authored-by: ramyavij <[email protected]>
Co-authored-by: David Rettie <[email protected]>
Co-authored-by: sbalaji-aws <[email protected]>
Co-authored-by: sumitagarwalHcst <[email protected]>
Co-authored-by: Julio Chaves <[email protected]>
Co-authored-by: Voicu Chirtes <[email protected]>
Co-authored-by: juchavw <[email protected]>
Co-authored-by: Jan-Arve Nygård <[email protected]>
Co-authored-by: Rem Baumann <[email protected]>
Co-authored-by: Maxim Fedchishin <[email protected]>
Co-authored-by: Soumya Vanga <[email protected]>
Co-authored-by: Udi Dahan <[email protected]>
  • Loading branch information
@Npinmor96 @iakov-aws
@yprikhodko @gaborschulz-aws @rohitsinglagithub @dnmll @VoicuAWS @vmindru @chaitanyashah @esc1144 @sean-nixon @gucarreira @steffeng @ramyavij @qmg-drettie @sbalaji-aws @sumitagarwalHcst @juchavw @jannyg @remaws @MaximF @soumyavanga @udid-aws
28 people authored Apr 23, 2024
1 parent 26126a1 commit 9693d35
Show file tree
Hide file tree
Showing 76 changed files with 23,120 additions and 1,642 deletions.
35 changes: 30 additions & 5 deletions .github/workflows/security-scan.yml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -12,8 +12,8 @@ jobs:
strategy:
matrix:
include:
- {python-version: '3.7' }
- {python-version: '3.10' }
- {python-version: '3.9' }
- {python-version: '3.12' }
steps:
- name: Git clone the repository
uses: actions/checkout@v3
Expand Down Expand Up @@ -45,7 +45,7 @@ jobs:
- name: Set up Python
uses: actions/setup-python@v4
with:
python-version: '3.10'
python-version: '3.11'
- name: Install cfn-lint
run: |
pip install pylint
Expand Down Expand Up @@ -105,6 +105,31 @@ jobs:
run: |
cfn-lint ./cfn-templates/cid-admin-policies.yaml
cfn-scan-cur-aggregation:
runs-on: ubuntu-latest
steps:
- name: Git clone the repository
uses: actions/checkout@v3
- name: Set up Python
uses: actions/setup-python@v4
with:
python-version: '3.10'
- name: Set up Ruby
uses: ruby/setup-ruby@v1
with:
ruby-version: '3.1'
- name: Install CFN tools
run: |
gem install cfn-nag
- name: CFN Nag scan
run: |
cfn_nag_scan --input-path ./cfn-templates/cur-aggregation.yaml
- name: Install cfn-lint
run: |
pip install cfn-lint
- name: CFN Lint
run: |
cfn-lint ./cfn-templates/cur-aggregation.yaml
terraform-scan:
runs-on: ubuntu-latest
Expand All @@ -114,10 +139,10 @@ jobs:
- name: Set up Python
uses: actions/setup-python@v4
with:
python-version: '3.10'
python-version: '3.11'
- name: Install checkov
run: |
pip install checkov
pip install -U schema checkov
- name: Checkov scan
run: |
checkov --directory ./terraform-modules
4 changes: 2 additions & 2 deletions .github/workflows/tests.yml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -19,10 +19,10 @@ jobs:
max-parallel: 1
matrix:
include:
- {python-version: '3.7', os: ubuntu-latest, dashboard: cudos }
- {python-version: '3.8', os: macos-latest, dashboard: cudos }
- {python-version: '3.9', os: ubuntu-latest, dashboard: cudos }
- {python-version: '3.10', os: macos-latest, dashboard: cudos }
- {python-version: '3.11', os: ubuntu-latest, dashboard: cudos }
- {python-version: '3.12', os: macos-latest, dashboard: cudos }
steps:
- name: Git clone the repository
uses: actions/checkout@v3
Expand Down
1 change: 1 addition & 0 deletions .gitignore
View file
Original file line number Diff line number Diff line change
Expand Up @@ -57,3 +57,4 @@ venv.bak/

#Local dev and testing files
cfn-templates/parameters.local
sandbox/
46 changes: 30 additions & 16 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
[![PyPI version](https://badge.fury.io/py/cid-cmd.svg)](https://badge.fury.io/py/cid-cmd)

## Welcome to Cloud Intelligence Dashboards (CUDOS Framework) automation repository
This repository contains CloudFormation templates, Terraform modules, and a Command Line tool (cid-cmd) for managing various dashboards provided in AWS Well Architected LAB [Cloud Intelligence Dashboards](https://www.wellarchitectedlabs.com/cost/200_labs/200_cloud_intelligence/).
This repository contains CloudFormation templates, Terraform modules, and a Command Line tool (cid-cmd) for managing various dashboards provided in AWS Well Architected LAB [Cloud Intelligence Dashboards](https://www.wellarchitectedlabs.com/cloud-intelligence-dashboards/).

There are several ways we can manage dashboards:
1. [CloudFormation Template](./cfn-templates/cid-cfn.yml) (using cid-cmd tool in lambda)
Expand All @@ -16,13 +16,16 @@ We recommend cid-cmd tool via [AWS CloudShell](https://console.aws.amazon.com/cl
---
| Dashboard documentation | Demo URL | Prerequisites URL |
| --- | --- | --- |
| [CUDOS Dashboard](https://www.wellarchitectedlabs.com/cost/200_labs/200_cloud_intelligence/cost-usage-report-dashboards/) | [demo](https://d1s0yx3p3y3rah.cloudfront.net/anonymous-embed?dashboard=cudos) | [link](https://wellarchitectedlabs.com/cost/200_labs/200_cloud_intelligence/cost-usage-report-dashboards/dashboards/alternative_deployments/) |
| [Cost Intelligence Dashboard](https://www.wellarchitectedlabs.com/cost/200_labs/200_cloud_intelligence/cost-usage-report-dashboards/) | [demo](https://d1s0yx3p3y3rah.cloudfront.net/anonymous-embed?dashboard=cost_intelligence_dashboard) | [link](https://wellarchitectedlabs.com/cost/200_labs/200_cloud_intelligence/cost-usage-report-dashboards/dashboards/alternative_deployments/) |
| [Trusted Advisor Organisation (TAO) Dashboard](https://www.wellarchitectedlabs.com/cost/200_labs/200_cloud_intelligence/trusted-advisor-dashboards/) | [demo](https://d1s0yx3p3y3rah.cloudfront.net/anonymous-embed?dashboard=e1799d0d-166c-4e61-8fa6-5c927f70c799) | [link](https://wellarchitectedlabs.com/cost/200_labs/200_cloud_intelligence/trusted-advisor-dashboards) |
| [Trends Dashboard](https://www.wellarchitectedlabs.com/cost/200_labs/200_cloud_intelligence/cost-usage-report-dashboards/dashboards/3_additional_dashboards/#trends-dashboard) | [demo](https://d1s0yx3p3y3rah.cloudfront.net/anonymous-embed?dashboard=trends-dashboard) | [link](https://wellarchitectedlabs.com/cost/200_labs/200_cloud_intelligence/cost-usage-report-dashboards) |
| [KPI Dashboard](https://wellarchitectedlabs.com/cost/200_labs/200_cloud_intelligence/cost-usage-report-dashboards/dashboards/deploy_dashboards/) | [demo](https://d1s0yx3p3y3rah.cloudfront.net/anonymous-embed?dashboard=kpi) | [link](https://wellarchitectedlabs.com/cost/200_labs/200_cloud_intelligence/cost-usage-report-dashboards/dashboards/alternative_deployments/) |
| [Compute Optimizer Dashboard](https://www.wellarchitectedlabs.com/cost/200_labs/200_cloud_intelligence/compute-optimizer-dashboards/) | [demo](https://d1s0yx3p3y3rah.cloudfront.net/anonymous-embed?dashboard=compute-optimizer-dashboard) | [link](https://wellarchitectedlabs.com/cost/200_labs/200_cloud_intelligence/compute-optimizer-dashboards) |
| [CUDOS Dashboard](https://catalog.workshops.aws/awscid/en-US/dashboards/foundational/cudos-cid-kpi#cudos-dashboard) | [demo](https://cid.workshops.aws.dev/demo?dashboard=cudos) | [link](https://aws.amazon.com/aws-cost-management/aws-cost-and-usage-reporting) |
| [Cost Intelligence Dashboard](https://catalog.workshops.aws/awscid/en-US/dashboards/foundational/cudos-cid-kpi#cost-intelligence-dashboard-(cid)) | [demo](https://cid.workshops.aws.dev/demo?dashboard=cost_intelligence_dashboard) | [link](https://aws.amazon.com/aws-cost-management/aws-cost-and-usage-reporting) |
| [Trusted Advisor Organisation (TAO) Dashboard](https://catalog.workshops.aws/awscid/en-US/dashboards/advanced/trusted-advisor) | [demo](https://cid.workshops.aws.dev/demo?dashboard=tao-dashboard) | [link](https://catalog.workshops.aws/awscid/en-US/dashboards/advanced/trusted-advisor/prerequisites) |
| [Trends Dashboard](https://catalog.workshops.aws/awscid/en-US/dashboards/additional/trends) | [demo](https://cid.workshops.aws.dev/demo?dashboard=trends-dashboard) | [link](https://aws.amazon.com/aws-cost-management/aws-cost-and-usage-reporting) |
| [KPI Dashboard](https://catalog.workshops.aws/awscid/en-US/dashboards/foundational/cudos-cid-kpi#kpi-dashboard) | [demo](https://cid.workshops.aws.dev/demo?dashboard=kpi) | [link](https://aws.amazon.com/aws-cost-management/aws-cost-and-usage-reporting) |
| [Compute Optimizer Dashboard](https://catalog.workshops.aws/awscid/en-US/dashboards/advanced/compute-optimizer) | [demo](https://cid.workshops.aws.dev/demo?dashboard=compute-optimizer-dashboard) | [link](https://catalog.workshops.aws/awscid/en-US/dashboards/advanced/compute-optimizer/prerequisites) |
| [Cost Anomaly Dashboard](https://catalog.workshops.aws/awscid/en-US/dashboards/advanced/cost-anomaly) | [demo](https://cid.workshops.aws.dev/demo?dashboard=aws-cost-anomalies) | [link](https://catalog.workshops.aws/awscid/en-US/dashboards/advanced/cost-anomaly/prerequisites) |
| [Data Transfer Cost Dashboard](https://catalog.workshops.aws/awscid/en-US/dashboards/additional/data-transfer) | [demo](https://cid.workshops.aws.dev/demo?dashboard=datatransfer-cost-analysis-dashboard) | [link](https://catalog.workshops.aws/awscid/en-US/dashboards/foundational/cudos-cid-kpi) |

See more dashboards on the [workshop page](https://catalog.workshops.aws/awscid/en-US/dashboards).

## Before you start
1. :heavy_exclamation_mark: Complete the prerequisites for respective dashboard (see above).
Expand All @@ -41,7 +44,7 @@ We recommend cid-cmd tool via [AWS CloudShell](https://console.aws.amazon.com/cl

Automation requires Python 3

2. Make sure you have latest pip package installed
2. Make sure you have the latest pip package installed
```bash
python3 -m ensurepip --upgrade
```
Expand All @@ -63,38 +66,49 @@ Update only Dashboard
```bash
cid-cmd update
```
Update dashboard and all dependenies (Datasets and Athena View). WARNING: this will overide any customization of SQL files and Datasets.
Update dashboard and all dependencies (Datasets and Athena View). WARNING: this will override any customization of SQL files and Datasets.
```bash
cid-cmd update --force --recursive
```
#### Show Dashboard status
#### Show Dashboard Status
Show dashboards status

```bash
cid-cmd status
```
[<img width="558" alt="status" src="https://github.com/aws-samples/aws-cudos-framework-deployment/assets/82834333/cae2015f-0f81-4593-80b3-c67ec1200fcd">](https://www.youtube.com/watch?v=ivr1MoGaApM)




#### Share QuickSight resources
```bash
cid-cmd share
```

#### Initialize Amazon QuickSight
One time action to intialize Amazon QuickSight Enerprise Edition.
One time action to initialize Amazon QuickSight Enterprise Edition.

```bash
cid-cmd initqs
cid-cmd init-qs
```

#### Initialize CUR
One time action to initialize Athena table and Crawler from s3 with CUR data.

```bash
cid-cmd init-cur
```

#### Delete Dashboard and all dependencies unused by other
Delete Dashboards and all dependencies unused by other CID-managed dashboards.(including QuickSight datasets, Athena views and tables)
```bash
cid-cmd delete
```

#### Delete Command Options:
```
--dashboard-id TEXT QuickSight dashboard id
--dashboard-id TEXT QuickSight dashboard id
--athena-database TEXT Athena database
```
Expand Down Expand Up @@ -126,10 +140,10 @@ This command generates a SQL file that you can execute. Please mind [Athena Serv
## Cloud Formation
CID is also provided in a form of CloudFormation telmplates. See detailed instuctions in the [Well Architected Labs](https://wellarchitectedlabs.com/cost/200_labs/200_cloud_intelligence/cost-usage-report-dashboards/dashboards/deploy_dashboards/) site.
CID is also provided in a form of CloudFormation templates. See detailed instructions in the [Well Architected Labs](https://wellarchitectedlabs.com/cost/200_labs/200_cloud_intelligence/cost-usage-report-dashboards/dashboards/deploy_dashboards/) site.
## Terraform
CID offers a set of Terraform modules to deploy CUR replicaion and CID dashboards. These modules act as a wrapper around CloudFormation.
CID offers a set of Terraform modules to deploy CUR replication and CID dashboards. These modules act as a wrapper around CloudFormation.
1. Create a bucket for consolidating CUR [terraform-modules/cur-setup-destination/](terraform-modules/cur-setup-destination/)
2. Create a CUR in Payer Account(s) [terraform-modules/cur-setup-source/](terraform-modules/cur-setup-source/)
Expand All @@ -151,4 +165,4 @@ This will produce a log file in the same directory that were at the tile of laun

:heavy_exclamation_mark:Inspect the produced debug log for any sensitive information and anonymize it.

We encourage you to open [new issue](https://github.com/aws-samples/aws-cudos-framework-deployment/issues/new) with description of the problem and attached debug log file.
We encourage you to open [new issue](https://github.com/aws-samples/aws-cudos-framework-deployment/issues/new) with description of the problem and attached debug log file.
29 changes: 29 additions & 0 deletions assets/build_lambda_layer.sh
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
#!/bin/bash
# This script builids a lambda layer. Outpits relative path of layer zip.
export CID_VERSION=$(python3 -c "from cid import _version;print(_version.__version__)")
rm -rf build

function get_hash {
find ./cid -type f -exec md5sum {} + | md5sum | awk '{print $1}'
}

function build_layer {
echo 'Building a layer'
mkdir -p ./python
python3 -m pip install . -t ./python
zip -qr cid-$CID_VERSION.zip ./python
ls -l cid-$CID_VERSION.zip
rm -rf ./python
}

# Check if code has been changed
previous_hash=$(cat cid-$CID_VERSION.hash)
actual_hash=$(get_hash)
if [ "$actual_hash" == "$previous_hash" ] && [ -e "cid-$CID_VERSION.zip" ]; then
echo "No changes in code. Reuse existing zip." 1>&2
else
build_layer 1>&2
echo $actual_hash > cid-$CID_VERSION.hash
fi

ls cid-$CID_VERSION.zip
77 changes: 77 additions & 0 deletions assets/lint.sh
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,77 @@
#!/bin/bash
# shellcheck disable=SC2086,SC2181
# This script runs cfn-lint cfn_nag_scan and checkov for all templates in folder

RED='\033[0;31m'
GREEN='\033[0;32m'
YELLOW='\033[0;33m'
NC='\033[0m' # No Color

folder=$(git rev-parse --show-toplevel)/cfn-templates/
success_count=0
failure_count=0

# CKV_AWS_109: "Ensure IAM policies does not allow permissions management without constraints"
# CKV_AWS_111: "Ensure IAM policies does not allow write access without constraints"
# CKV_AWS_115: "Ensure that AWS Lambda function is configured for function-level concurrent execution limit"
# CKV_AWS_116: "Ensure that AWS Lambda function is configured for a Dead Letter Queue(DLQ)"
# CKV_AWS_117: "Ensure that AWS Lambda function is configured inside a VPC"
# CKV_AWS_173: "Check encryption settings for Lambda environmental variable"
# CKV_AWS_195: "Ensure Glue component has a security configuration associated"
# CKV_AWS_18: "Ensure the S3 bucket has access logging enabled"
# CKV_AWS_21: "Ensure the S3 bucket has versioning enabled"
checkov_skip=CKV_AWS_109,CKV_AWS_111,CKV_AWS_115,CKV_AWS_116,CKV_AWS_117,CKV_AWS_173,CKV_AWS_195,CKV_AWS_18,CKV_AWS_21


export exclude_files=("module-inventory.yaml" "module-pricing.yaml") # For::Each breaks lint :'(

yaml_files=$(find "$folder" -type f \( -name "*.yaml" -o -name "*.yml" \) -exec ls -1t "{}" +;) # ordered by date

for file in $yaml_files; do
echo "Linting $(basename $file)"
fail=0

# checkov
output=$(eval checkov --skip-download --skip-check $checkov_skip --quiet -f "$file")
if [ $? -ne 0 ]; then
echo "$output" | awk '{ print "\t" $0 }'
echo -e "checkov ${RED}KO${NC}" | awk '{ print "\t" $0 }'
fail=1
else
echo -e "checkov ${GREEN}OK${NC}" | awk '{ print "\t" $0 }'
fi

# cfn-lint
output=$(eval cfn-lint -- "$file")
if [ $? -ne 0 ]; then
echo "$output" | awk '{ print "\t" $0 }'
echo -e "cfn-lint ${RED}KO${NC}" | awk '{ print "\t" $0 }'
fail=1
else
echo -e "cfn-lint ${GREEN}OK${NC}" | awk '{ print "\t" $0 }'
fi

# cfn_nag_scan
output=$(eval cfn_nag_scan --input-path "$file")
if [ $? -ne 0 ]; then
echo "$output" | awk '{ print "\t" $0 }'
echo -e "cfn_nag_scan ${RED}KO${NC}" | awk '{ print "\t" $0 }'
fail=1
else
echo -e "cfn_nag_scan ${GREEN}OK${NC}" | awk '{ print "\t" $0 }'
fi

if [ $fail -ne 0 ]; then
((failure_count++))
else
((success_count++))
fi
done

echo "Successful lints: $success_count"
echo "Failed lints: $failure_count"
if [ $failure_count -ne 0 ]; then
exit 1
else
exit 0
fi
Loading

0 comments on commit 9693d35

Please sign in to comment.