Human Systems Dialogic Agent

This project is for the setup/execution of the Human Systems Dialogic Agent.

The solution uses AWS CDK to automate the compilation of CloudFormation code for deployment to Human Systems' Dialogic Agent AWS Environments (i.e. 'HS-DIALOGIC-AGENT-DEV', 'HS-DIALOGIC-AGENT-STAGE' and 'HS-DIALOGIC-AGENT-PROD' accounts).

• Human Systems Dialogic Agent
  • 2. Setup Local Environment for local build/deployment
    • 2.1 Software Pre-requisites/Recommendations
  • 3. CI/CD Initial Setup (Once-off) - Setting up Orchestration
    • 3.1 Set some variables we will reuse for the deployment
    • 3.2 Setup CodeCommit Repo in DEPLOYMENT Account
    • 3.3 Deploy the S3 Bucket for Build Artifacts with Policy + KMS Key to DEPLOYMENT Account
    • 3.4 Get Copy of KMS Key Arn just created
    • 3.5 Setup IAM Roles for CodePipeline to access DEPLOYMENT Account
    • 3.6 Deploy IAM Roles and KMS Trust with TARGET Account
    • 3.6 Deploy Placeholder Secrets used by Application
    • 3.7 Setup CI/CD Infrastructure Pipeline (CodePipeline) to DEPLOYMENT account
  • 4. Testing Locally
    • 4.1 Python Environment for Local Build/Debugging
    • 4.2 AWS CDK
    • 5. Working with the API (FastAPI)
      • Running locally
  • 7. Contact

2. Setup Local Environment for local build/deployment

2.1 Software Pre-requisites/Recommendations

1. An IDE (I recommend VS Code) [https://code.visualstudio.com/]
   Some helpful VS Code Extensions!
   • AWS
     • AWS Toolkit (amazonwebservices.aws-toolkit-vscode) - Provides AWS Explorer for many services, AWS SAM integration
     • CloudFormation:
       • CloudFormation (aws-scripting-guy) – intellisense for CloudFormation templates
       • Indent-rainbow (oderwat) – helps with seeing your indents in Python and YAML
       • YAML (redhat.vscode-yaml) - YAML validation and formatter
       • CloudFormation Linter (kddejong) - Fixes YAML Custom tags. Ironically you can uninstall it once installed (it will still modify your settings.json with the correct CloudFormation custom tags). If you do keep it installed you will also need to install Python and cfn-lint to get working correctly.
2. AWS Account Setup as per 'AWS Control Tower' landing zones. This will provide the best overarching security for your environment. You can proceed without this as long as you have minimally two aws accounts DEPLOYMENT (sweat-deployment) and a application account (sweat-production) but it is not recommended.
3. Atlassian Sourcetree [https://www.sourcetreeapp.com/]
4. AWS CLI [https://aws.amazon.com/cli/]
5. AWS SAM CLI [https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/serverless-sam-cli-install.html]
6. AWS CDK [https://docs.aws.amazon.com/cdk/latest/guide/home.html]
7. Setup your AWS profiles. To start by running:

aws configure

This step is mostly just to create a 'config' and 'credentials' text file located:
C:\Users\MYUSER.aws (assume your running this in Windows)
set your default region in the 'config' file

This project assumes the following AWS profiles setup on your machine:
[hs-deployment] - the code and ci/cd account
[hs-dialogic-agent-dev] - target DEV environment account
[hs-dialogic-agent-stage] - target STAGE environment account
[hs-dialogic-agent-prod] - target PROD environment account

1. Create a local build folders (using Bash Terminal)
   mkdir .build .build/dev .build/stage .build/prod .build/cdk

3. CI/CD Initial Setup (Once-off) - Setting up Orchestration

3.1 Set some variables we will reuse for the deployment

*DEV VARIABLES - OLD ONE - NEEDED TEMPORARILY

projectFriendlyName="Human Systems Dialogic Agent"
projectResourcePrefix="hs-dialogic-agent"
environmentType="dev"
deploymentAccountNumber="851725322837" 
targetAccountNumber1="891377036533"
profileDeploymentAccount="hs-deployment"
profileTargetAccount1="hs-dialogic-agent-dev"
awsregion="us-east-1"
repoBranch="dev"
artifactsBucketName="${projectResourcePrefix}-${environmentType}-buildartifacts"
enableDeletionProtection="False"
printf "Done"

*DEV VARIABLES

projectFriendlyName="Human Systems Dialogic Agent"
projectResourcePrefix="hs-dialogic-agent"
environmentType="dev-main"
deploymentAccountNumber="851725322837" 
targetAccountNumber1="891377036533"
profileDeploymentAccount="hs-deployment"
profileTargetAccount1="hs-dialogic-agent-dev"
awsregion="us-east-1"
repoBranch="dev"
artifactsBucketName="${projectResourcePrefix}-${environmentType}-buildartifacts"
enableDeletionProtection="False"
printf "Done"

*DEV/C3L VARIABLES

projectFriendlyName="Human Systems Dialogic Agent"
projectResourcePrefix="hs-dialogic-agent"
environmentType="dev-c3l"
deploymentAccountNumber="851725322837" 
targetAccountNumber1="891377036533"
profileDeploymentAccount="hs-deployment"
profileTargetAccount1="hs-dialogic-agent-dev"
awsregion="us-east-1"
repoBranch="dev-c3l"
artifactsBucketName="${projectResourcePrefix}-${environmentType}-buildartifacts"
enableDeletionProtection="False"
printf "Done"

*DEV/Figaro VARIABLES

projectFriendlyName="Human Systems Dialogic Agent"
projectResourcePrefix="hs-dialogic-agent"
environmentType="dev-figaro"
deploymentAccountNumber="851725322837" 
targetAccountNumber1="891377036533"
profileDeploymentAccount="hs-deployment"
profileTargetAccount1="hs-dialogic-agent-dev"
awsregion="us-east-1"
repoBranch="dev-figaro"
artifactsBucketName="${projectResourcePrefix}-${environmentType}-buildartifacts"
enableDeletionProtection="False"
printf "Done"

3.2 Setup CodeCommit Repo in DEPLOYMENT Account

1. Compile the CloudFormation script

aws cloudformation package --template-file ./cf/setup/01_codecommit_repo.yaml --output-template-file "./.build/_01_codecommit_repo.yaml" --s3-bucket NOTUSED --profile $profileDeploymentAccount

2. Deploy the CloudFormation script

aws cloudformation deploy --template-file "./.build/_01_codecommit_repo.yaml" --stack-name "${projectResourcePrefix}-setup-codecommit" --profile $profileDeploymentAccount --region $awsregion --capabilities CAPABILITY_NAMED_IAM --parameter-overrides ProjectFriendlyName="$projectFriendlyName" ProjectResourcePrefix=$projectResourcePrefix

3.3 Deploy the S3 Bucket for Build Artifacts with Policy + KMS Key to DEPLOYMENT Account

1. Compile the CloudFormation script

aws cloudformation package --template-file ./cf/setup/02_deployment_artifacts_and_kms.yaml --output-template-file "./.build/${environmentType}/_02_deployment_artifacts_and_kms.yaml" --s3-bucket NOTUSED --profile $profileDeploymentAccount

2. Deploy the CloudFormation script

aws cloudformation deploy --template-file "./.build/${environmentType}/_02_deployment_artifacts_and_kms.yaml" --stack-name "${projectResourcePrefix}-setup-kms-artif-${environmentType}" --profile $profileDeploymentAccount --region $awsregion --capabilities CAPABILITY_NAMED_IAM --parameter-overrides EnvironmentType=$environmentType TargetAccountNumber1=$targetAccountNumber1 ProjectFriendlyName="$projectFriendlyName" ProjectResourcePrefix=$projectResourcePrefix

3.4 Get Copy of KMS Key Arn just created

This command will copy to a local variable the KMS Key Arn for step 3

get_cmk_command="aws cloudformation describe-stacks --stack-name "${projectResourcePrefix}-setup-kms-artif-${environmentType}" --profile $profileDeploymentAccount --region $awsregion --query \"Stacks[0].Outputs[?OutputKey=='CodePipelineKMSKeyArn'].OutputValue\" --output text"
CodePipelineKMSKeyArn=$(eval $get_cmk_command)
printf "Got CMK ARN: $CodePipelineKMSKeyArn"

3.5 Setup IAM Roles for CodePipeline to access DEPLOYMENT Account

1. Compile the CloudFormation script

aws cloudformation package --template-file ./cf/setup/03_iam_role_codepipeline.yaml --output-template-file "./.build/${environmentType}/_03_iam_role_codepipeline.yaml" --s3-bucket NOTUSED --profile $profileDeploymentAccount

2. Deploy the CloudFormation script

aws cloudformation deploy --template-file "./.build/${environmentType}/_03_iam_role_codepipeline.yaml" --stack-name "${projectResourcePrefix}-setup-cp-roles-${environmentType}" --profile $profileDeploymentAccount --region $awsregion --capabilities CAPABILITY_NAMED_IAM --parameter-overrides EnvironmentType=$environmentType DeploymentAccountNumber=$deploymentAccountNumber KMSKeyArn=$CodePipelineKMSKeyArn ProjectFriendlyName="$projectFriendlyName" ProjectResourcePrefix=$projectResourcePrefix

3.6 Deploy IAM Roles and KMS Trust with TARGET Account

1. Compile the CloudFormation script

aws cloudformation package --template-file ./cf/setup/04_target_deploy_roles.yaml --output-template-file "./.build/${environmentType}/_04_target_deploy_roles.yaml" --s3-bucket NOTUSED --profile $profileDeploymentAccount

2. Deploy the CloudFormation script

aws cloudformation deploy --template-file "./.build/${environmentType}/_04_target_deploy_roles.yaml" --stack-name "${projectResourcePrefix}-setup-deployroles-${environmentType}" --profile $profileTargetAccount1 --region $awsregion --capabilities CAPABILITY_NAMED_IAM --parameter-overrides EnvironmentType=$environmentType DeploymentAccountNumber=$deploymentAccountNumber KMSKeyArn=$CodePipelineKMSKeyArn ProjectFriendlyName="$projectFriendlyName" ProjectResourcePrefix=$projectResourcePrefix

3.6 Deploy Placeholder Secrets used by Application

3.7 Setup CI/CD Infrastructure Pipeline (CodePipeline) to DEPLOYMENT account

1. Compile the CloudFormation script

aws cloudformation package --template-file ./cf/cicd/01_pipeline.yaml --output-template-file "./.build/${environmentType}/_01_pipeline.yaml" --s3-bucket NOTUSED --profile $profileDeploymentAccount

2. Deploy the CloudFormation script

aws cloudformation deploy --template-file "./.build/${environmentType}/_01_pipeline.yaml" --stack-name "${projectResourcePrefix}-pipeline-${environmentType}" --profile $profileDeploymentAccount --region $awsregion --capabilities CAPABILITY_NAMED_IAM --parameter-overrides EnvironmentType=$environmentType TargetAccountNumber1=$targetAccountNumber1 ProjectFriendlyName="$projectFriendlyName" ProjectResourcePrefix=$projectResourcePrefix CFTemplateName="cdkapp.template.json" BranchName=$repoBranch RepoAccountNumber=$deploymentAccountNumber EnableDeletionProtection=$enableDeletionProtection

4. Testing Locally

4.1 Python Environment for Local Build/Debugging

This project is set up like a standard Python project. The initialization process also creates
a virtualenv within this project, stored under the .venv directory. To create the virtualenv
it assumes that there is a python3 executable in your path with access to the venv package.
If for any reason the automatic creation of the virtualenv fails, you can create the virtualenv
manually once the init process completes.

To manually create a virtualenv on MacOS and Linux:

sudo apt install python3.8-venv
cd py-cdk/src
python3 -m venv .venv

Install and Update NodeJS (required for CDK)

sudo apt-get install -y nodejs

curl -fsSL https://deb.nodesource.com/setup_current.x | sudo -E bash -
sudo apt-get install -y nodejs

After the init process completes and the virtualenv is created, you can use the following
step to activate your virtualenv.

cd py-cdk/src
source .venv/bin/activate

If you are a Windows platform, you would activate the virtualenv like this:

% .venv\Scripts\activate.bat

Once the virtualenv is activated, you can install the required dependencies.

$ pip install -r requirements.txt

4.2 AWS CDK

Provided you have the AWS CDK installed locally you can test generated stacks built with this framework by providing the build script a local AWS Profile.

Run the AWS CDK build script from THE ROOT project folder on a bash terminal with:

./cf/cicd/build_aws_cdk.sh -p $profileTargetAccount1 -s $artifactsBucketName -d $enableDeletionProtection

Outputs from the CDK will be built to the (./.build/cdk) folder for reviewing.

5. Working with the API (FastAPI)

Running locally

cd '{projectdirectory}/lambda/src/api'
pip install -r requirements.txt
source ./env/bin/activate
export AWS_PROFILE=hs-saas-platform-dev
uvicorn index:app --reload

Navigate to docs to get swagger site:
http://localhost:8001/docs

7. Contact

For more information please contact:

• Damien Coyle (Comunet Pty Ltd) - Initial work OCT 2023 - Comunet