This folder contains the CDMC Report Engine, the python application which based on configuration files, configuration tables and BigQuery jobs metadata inspects that BigQuery sensitive data assets follow the CDMC controls.

The repository contains 12 dedicated controls implementations classes which refers to the 14 CDMC controls, being controls 1 and 5 implicitly implemented and not having dedicated classes.

GCP CDMC Report Engine is a Python flask application, with execution based on rest requests.

• Python3 environment;
• Metadata generated by TAG Engine;
• Terraform (see below)

Apply the terraform scripts in this folders, specifying the related variables define in Terraform variables template (terraform/terraform.tfvars.template). For convenience you can use the setup.sh script in this folder to perform variables substitutions.

cd terraform
source setup.sh

Once the setup is completed, use the following commands to create the infrastructure:

pushd terrraform
terraform init
terraform plan
terraform apply
popd

The Terraform scripts create: - A service account with the related roles to access metadata in Organization level, and access to Pub/Sub and BigQuery in project level; - Pub/Sub topics for events and data_assets, with related Pub/Sub schemas and BQ subscribers; - BigQuery dataset and tables for events and data assets logs;

After the execution of the requirements, Report Engine can be deployed in Cloud Run (recommended) or Compute Engine.

For convenience, you can use the deploy_reportengine.sh script to deploy to cloud run. The script will:

• Create a resources/config.ini file using the template as a basis
• Substitute the required variables in the config file (project-id and region)
• Deploy to CloudRun

Execute the cloud run deployment with this command:

source deploy_reportengine.sh

Once the deployment is completed, you can find the URL of the service with this command:

gcloud run services describe cdmc-reportengine --region $REGION --format='value(status.url)'

If you prefer to deploy to Compute Engine, perform the following steps:

• Create a config.ini file with the correct variables using the template. You can use these commands:

  pushd resources
  ## Create a copy of the config.ini.template and substitute values
  cp config.ini.template config.ini
  sed -i "s/<PROJECT_ID_GOV>/$PROJECT_ID_GOV/" config.ini
  sed -i "s/<REGION>/$REGION/" config.ini
  popd
  

• Install all Python requirements from requirements.txt file;

  python3 -m pip install -r requirements.txt
  

• Execute the python3 main.py file

  python3 generate_report.py
  

CDMC Report Engine is a Flask web application. The following request parameters must be considered for the POST method in /generate:

• (Mandatory) orgId= the GCP Organization ID of the inspection;
• (Mandatory) projectId= the GCP Project ID to be inspected;
• (Mandatory) topicProjectId= the GCP Project ID that has the Pub/Sub topic that receive the alerts;
• (Mandatory) topic= the Pub/Sub topic name (cdmc-controls-topic or cdmc-data-assets-topic)
• (Mandatory) projectNumber= the inspected GCP Project number
• (Optional) controls= the list of all CDMC controls to be inspected, the format of list are two digit per control. Ex.: 020312 inspects the controls 02, 03 and 12. The default value is “all”.
• (Optional) assetsScope= informs if the log of inspected assets must be sent to a Pub/Sub topic configured in config.ini ASSETS_SCOPE section. Default value is TRUE

Sample request format (from local client):

curl -H "Authorization: Bearer $(gcloud auth print-identity-token)" -X POST "$ENDPOINT/generate?orgId=$ORGANIZATION_ID&projectId=$PROJECT_ID_DATA&topicProjectId=$PROJECT_ID_GOV&topic=cdmc-controls-topic&projectNumber=$PROJECT_NUMBER_DATA&controls=020608"

It is recommended that Dataplex Catalog templates, Pub/Sub topics and BigQuery configuration tables keep the original values, in case these values must be adjusted, the requirements/config.ini must be considered.

In case the deployment occurs in a deployment of Secure Data Warehouse, which uses VPC- Service Controls, or in a self deployed VPC-Service Control environment, consider the following aspects to be considered in the report-engine deployment:

• Evaluate and disable potential Organization Policies that blocks the creation of Service Accounts in the deployment project;
• Create an egress rule enabling the "sa-report-engine@" service account, created by Terraform, to access data assets from the projects to be scanned;
• Create an Access Level, or update potential existing one, enabling the "sa-report-engine@" service account to access target deployment project/ data-governance;
• Create an Access Level, or update potential existing one, enabling the "sa-report-engine@" service account to access the data confidential project.

In case of Cloud Run deployment, potential Access Level modifications can be necessary, allowing deployments components, such as Cloud Build default Service Account or Pub/Sub default Service Account to BigQuery. This configurations change might be analyzed according to VPC-SC topology and definitions.