IDEA Release 3.1.4

CHANGELOG.md

@@ -5,29 +5,59 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
 
+## [3.1.4] - 2023-07-25
+
+:heavy_exclamation_mark: - *Please note the IDEA ECR Repository location has changed as of `3.1.4`*
+
+Users of older `idea-admin.sh` and `idea-admin-windows.ps1` may need to manually update these files for the new repo location (`public.ecr.aws/h5i3y8y1/idea-administrator`).
+
+
+### Features
+* Added support for `Launch Tenancy` for eVDI software stacks. This allows the IDEA administrator to configure [EC2 Launch Tenancy](https://docs.aws.amazon.com/autoscaling/ec2/userguide/auto-scaling-dedicated-instances.html).
+
+
+### Changes
+* Reduced IAM actions for SQS and SNS in scheduler, VDC broker, and VDC Host to least required privileges
+* eVDI session tiles now display the Project of the associated eVDI session (along with putting this information in boxes for clarity)
+* Updates to the ECR Repo alias/location
+* Add additional logging during (`debug` logging profile) for some latency sensitive operations such as Active Directory lookups.
+
+
+### Bug Fixes
+* In Active Directory (AD) environments - the DynamoDB AD automation table would continue to grow and not properly expire old entries.
+* In Active Directory environments - an incoming Windows eVDI session was not using the same domain-controller that `cluster-manager` pre-created the object. This could lead to a race condition where the incoming client would fail to join the domain properly as the AD object had not replicated within the domain.
+* In Active Directory environments - users were not displayed properly as members of a group on the Active Directory `Member Of` tab within Active Directory Users and Computers.
+* Restore AMI IDs for `us-gov-west-1` region (missing since `3.1.3`)
+* Fixed a bug that prevented `userdata_customizations.sh` from executing for jobs that require EFA driver to be installed
+* Fixed a syntax error in the `robots.txt` on `cluster-manager` that allowed bots to index IDEA.
+* Expand skipping service quotas to batch queues when `scheduler.job_provisioning.service_quotas` is set to `False`
+* Added missing IAM actions to installer policies
+* Project titles were allowed to exceed character limits in some cases. Project titles are now restricted to `3-32` characters.
+* Under certain conditions - the `cluster-manager` task manager could have difficulty keeping up with task execution requirements. Additional configuration parameters have been added to address this.
+
+
 ## [3.1.3] - 2023-06-16
 
 ### Features
 
 * Added support for `Red Hat Enterprise Linux 8.7` and `Rocky Linux 8.7` as supported operating systems for VDI and compute nodes.
   * **NOTE:** Subscription to `Rocky Linux 8` in AWS Marketplace is required to access Rocky Linux AMIs.
 
-* `ideactl`  now supports adding multiple users to multiple named groups in a single command. For example: `ideactl groups add-user-to-group --username user1 --username user2 --groupname group1 --groupname group2`  will add both `user1`  and `user2`  to groups `group1`  and `group2`
-* Added support for new instance families: `i4g, inf2, trn1n, c6in, m6in, m6idn, r6in,  rdidn`
+* `ideactl` now supports adding multiple users to multiple named groups in a single command. For example: `ideactl groups add-user-to-group --username user1 --username user2 --groupname group1 --groupname group2`  will add both `user1`  and `user2`  to groups `group1` and `group2`
+* Added support for new instance families: `i4g`, `inf2`, and `trn1n`
 * eVDI session startup now validates that there is remaining budget for the associated project. If there is no remaining budget a session cannot be started for the project. This is controlled via the new configuration setting `vdc.controller.enforce_project_budgets`  (defaults to `True` ).
 
 ### Changes
 
-* Update boto3 from 1.26.61 to 1.26.138
-* Update requests module from 2.27.1 to 2.31.0
+* Update boto3 from `1.26.61` to `1.26.138`
+* Update requests module from `2.27.1` to `2.31.0`
 * ALB deployments will now set the option to Drop Invalid Headers
-* Default Web/API page size increased from 20 to 50
+* Default Web/API page size increased from `20` to `50`
 * Update AMI IDs for all supported operating systems
-* Update AWS EFA Installer from 1.22.1  to 1.23.1
-* Update DCV Server from 2023.0-14852 to 2023.0-15065, DCV Session Manager from 2023.0-642 to 2023.0-675, and DCV viewer from 2023.0.5388 to 2023.0.5483
-
+* Update AWS EFA Installer from `1.22.1` to `1.23.1`
+* Update DCV Server from `2023.0-14852` to `2023.0-15065`, DCV Session Manager from `2023.0-642` to `2023.0-675`, and DCV viewer from `2023.0.5388` to `2023.0.5483`
 * Reduce the default DCV idle disconnect from 24-hours to 4-hours
-* Update Nvidia drivers from 510.47.03 to 525.105.17
+* Update Nvidia drivers from `510.47.03` to `525.105.17`
 
 ### Bug Fixes
 * `ideactl ldap search-nnn `on `cluster-manager` did not properly return all results when the results spanned multiple pages of results. This has been fixed.
@@ -37,7 +67,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Known Caveats
 
-* `Red Hat Enterprise Linux  8.7` and `Rocky Linux 8.7` do not launch VDI sessions on `G4ad` instances due to AMD GPU driver kernel version dependencies.
+* `Red Hat Enterprise Linux 8.7` and `Rocky Linux 8.7` do not launch VDI sessions on `G4ad` instances due to AMD GPU driver kernel version dependencies.
 * AWS EFA installer doesn't install successfully on `Rocky Linux 8.7`
 
 
@@ -120,7 +150,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 * Amazon OpenSearch Service - Default engine version updated to OpenSearch 2.3  for new installations.
 * Enhanced `delete-cluster` to delete CloudWatch Log Groups.
 * Added option `--delete-all` for `delete-cluster`. This will delete bootstrap, backups, dynamodb tables, and cloudwatch log groups.
-* Support for new instances:   `C6in`, `M6in`, `M6idn`, `R6in` and `R6idn`
+* Support for new instance families:   `c6in`, `m6in`, `m6idn`, `r6in` and `r6idn`
 * (`cluster-manager`) - Added `groups`  subcommands for group add/delete/enable/disable/listing
 
 

IDEA_VERSION.txt

@@ -1 +1 @@
-3.1.3
+3.1.4

deployment/ecr/idea-administrator/Dockerfile

@@ -0,0 +1,47 @@
+FROM public.ecr.aws/docker/library/python:3.9.16-slim
+
+WORKDIR /root
+
+RUN apt-get update && \
+    apt-get -y install \
+    curl \
+    tar \
+    unzip \
+    locales \
+    && apt-get clean
+
+
+ENV DEBIAN_FRONTEND=noninteractive
+ENV LC_ALL="en_US.UTF-8" \
+    LC_CTYPE="en_US.UTF-8" \
+    LANG="en_US.UTF-8"
+
+RUN sed -i -e "s/# $LANG.*/$LANG UTF-8/" /etc/locale.gen \
+    && locale-gen "en_US.UTF-8" \
+    && dpkg-reconfigure locales
+
+# install aws cli
+RUN curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip" && \
+    unzip -qq awscliv2.zip && \
+    ./aws/install && \
+    rm -rf ./aws awscliv2.zip
+
+# install nvm and node
+RUN curl -sL https://deb.nodesource.com/setup_16.x | bash - \
+    && apt-get install -y nodejs \
+    && apt-get clean all
+
+# add all packaged artifacts to container
+ARG PUBLIC_ECR_TAG
+ENV PUBLIC_ECR_TAG=${PUBLIC_ECR_TAG}
+ADD all-*.tar.gz /root/.idea/downloads/
+
+# install administrator app
+RUN mkdir -p /root/.idea/downloads/idea-administrator-${PUBLIC_ECR_TAG} && \
+    tar -xvf /root/.idea/downloads/idea-administrator-*.tar.gz -C /root/.idea/downloads/idea-administrator-${PUBLIC_ECR_TAG} && \
+    /bin/bash /root/.idea/downloads/idea-administrator-${PUBLIC_ECR_TAG}/install.sh && \
+    rm -rf /root/.idea/downloads/idea-administrator-${PUBLIC_ECR_TAG}
+
+CMD ["bash"]
+
+

idea-admin-windows.ps1

@@ -38,8 +38,8 @@ function Verify-Command($type,$message,$command) {
 $IDEADevMode = if ($Env:IDEA_DEV_MODE) {$Env:IDEA_DEV_MODE} else {""}
 $VirtualEnv = if ($Env:VIRTUAL_ENV) {$Env:VIRTUAL_ENV} else {""}
 $ScriptDir = $PSScriptRoot
-$IDEARevision = if ($Env:IDEA_REVISION) {$Env:IDEA_REVISION} else {"v3.1.3"}
-$IDEADockerRepo = "public.ecr.aws/g8j8s8q8"
+$IDEARevision = if ($Env:IDEA_REVISION) {$Env:IDEA_REVISION} else {"v3.1.4"}
+$IDEADockerRepo = "public.ecr.aws/h5i3y8y1"
 $DocumentationError = "https://ide-on-aws.com"
 $AWSProfile = if ($Env:AWS_PROFILE) {$Env:AWS_PROFILE} else {"default"}
 $AWSRegion= if ($Env:AWS_REGION) {$Env:AWS_REGION} else {"us-east-1"}

idea-admin.sh

@@ -28,8 +28,8 @@
 # * IDEA_DEV_MODE - Set to "true" if you are working with IDEA sources
 
 SCRIPT_DIR=$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )
-IDEA_REVISION=${IDEA_REVISION:-"v3.1.3"}
-IDEA_DOCKER_REPO=${IDEA_DOCKER_REPO:-"public.ecr.aws/g8j8s8q8/idea-administrator"}
+IDEA_REVISION=${IDEA_REVISION:-"v3.1.4"}
+IDEA_DOCKER_REPO=${IDEA_DOCKER_REPO:-"public.ecr.aws/h5i3y8y1/idea-administrator"}
 IDEA_ECR_CREDS_RESET=${IDEA_ECR_CREDS_RESET:-"true"}
 IDEA_ADMIN_AWS_CREDENTIAL_PROVIDER=${IDEA_ADMIN_AWS_CREDENTIAL_PROVIDER:=""}
 IDEA_ADMIN_ENABLE_CDK_NAG_SCAN=${IDEA_ADMIN_ENABLE_CDK_NAG_SCAN:-"true"}
@@ -102,8 +102,9 @@ verify_command "Docker is installed on the system but it does not seems to be ru
 if [[ "${IDEA_ECR_CREDS_RESET}" == "true" ]]; then
   # Check if user is connected to internet an can ping ECR repo
   DIG_BIN=$(command -v dig)
-  ${DIG_BIN} +tries=1 +time=3 ${IDEA_DOCKER_REPO} >> /dev/null 2>&1
-  verify_command "Unable to query ECR. Are you connected to internet?"
+  IDEA_DOCKER_REPO_HOSTNAME=$(echo "${IDEA_DOCKER_REPO}" | cut -d '/' -f 1)
+  ${DIG_BIN} +tries=1 +time=3 ${IDEA_DOCKER_REPO_HOSTNAME} >> /dev/null 2>&1
+  verify_command "Unable to query ECR host ${IDEA_DOCKER_REPO_HOSTNAME} . Are you connected to internet?"
 
   ${DOCKER_BIN} logout public.ecr.aws >> /dev/null 2>&1
   verify_command "Failed to refresh ECR credentials. docker logout public.ecr.aws failed"

requirements/dev.txt

@@ -27,7 +27,6 @@ coverage[toml]==6.5.0
 cryptography==38.0.4
 dataset==1.5.2
 decorator==5.1.1
-defusedxml==0.7.1
 dill==0.3.5.1
 exceptiongroup==1.1.1
 fastcounter==1.1.0

requirements/idea-administrator.in

@@ -5,4 +5,3 @@ sanic~=23.3.0
 aws-cdk-lib
 cdk-nag
 prettytable
-defusedxml

requirements/idea-administrator.txt

@@ -24,7 +24,6 @@ constructs==10.1.10
 cryptography==38.0.4
 dataset==1.5.2
 decorator==5.1.1
-defusedxml==0.7.1
 exceptiongroup==1.0.0rc6
 fastcounter==1.1.0
 greenlet==1.1.2

requirements/tests.in

@@ -2,4 +2,3 @@ pytest>=7.2.0
 pytest-mock
 pytest-cov
 memory_profiler
-defusedxml

requirements/tests.txt

@@ -1,5 +1,4 @@
 coverage[toml]==6.5.0
-defusedxml==0.7.1
 exceptiongroup==1.1.1
 iniconfig==1.1.1
 memory-profiler==0.60.0

source/idea/idea-administrator/resources/config/region_ami_config.yml

@@ -147,6 +147,10 @@ us-east-2:
   rhel7: ami-00342897eb8ba6355
   rhel8: ami-057094267c651958e
   rocky8: ami-02fb9384e880ed67c
+us-gov-west-1:
+  amazonlinux2: ami-0d6bf8ceb4766ef71
+  rhel7: ami-09616d1f2ecd77e9b
+  rhel8: ami-0206165c7b00c4adf
 us-west-1:
   amazonlinux2: ami-0ee3e1e65adeef858
   centos7: ami-0bcd12d19d926f8e9

source/idea/idea-administrator/resources/config/templates/cluster-manager/settings.yml

@@ -101,3 +101,15 @@ notifications:
   # email notifications are supported at the moment. slack, sms and other channels will be supported in a future release.
   email:
     enabled: true
+
+#
+# Control Task Manager settings
+#
+task_manager:
+  min_workers: 1    # Minimum number of task executor worker threads (Min 1)
+  max_workers: 5    # Maximum number of task executor worker threads (Max 10)
+  polling_messages_max: 1 # Max number of Messages to poll from SQS queue
+  polling_visibility_timeout: 30 # Visibility timeout for SQS messages
+  sqs_wait_time: 20 # SQS polling / Wait time (in seconds). Used to active short or long SQS polling. (Min 0, Max 20)
+  debug: false      # Debug task_manager (extra information in logs, logging profile must be 'debug' to see)
+

source/idea/idea-administrator/resources/config/templates/directoryservice/_templates/aws_managed_activedirectory.yml

@@ -12,13 +12,8 @@ directory_id: {{ directory_id or '~' }}
 ad_short_name: "IDEA"
 
 # AWS Managed Microsoft AD Edition. Must be one of: [Standard, Enterprise]
-# Note: Enterprise edition is not tested/supported yet, and additional configurations may be required and/or cdk stack needs to be updated.
 ad_edition: "Standard"
 
-# added for future use - not supported yet.
-# primary_region: "{{aws_region}}"
-# replica_region: "{{aws_region}}"
-
 # Password Max Age in Days. Used by Cluster IDP such as Cognito UserPool or KeyCloak in JWT Claims
 # Authenticated API requests will be rejected if the password has expired.
 # see: https://docs.aws.amazon.com/directoryservice/latest/admin-guide/ms_ad_password_policies.html

source/idea/idea-administrator/resources/installer_policies/idea-admin-cdk-toolkit-policy.yml

@@ -7,6 +7,7 @@ Statement:
       - s3:PutBucketVersioning
       - s3:PutBucketPublicAccessBlock
       - s3:PutBucketPolicy
+      - s3:PutBucketTagging
       - s3:GetBucketPolicy
       - s3:CreateBucket
     Resource: arn:aws:s3:::idea*
@@ -16,12 +17,15 @@ Statement:
       - ecr:CreateRepository
       - ecr:DescribeRepositories
       - ecr:SetRepositoryPolicy
+      - ecr:TagResource
       - ec2:DescribeAccountAttributes
       - ssm:GetParameter
       - ssm:GetParameters
       - ssm:PutParameter
       - ssm:DeleteParameter
       - ssm:GetParametersByPath
+      - ssm:SendCommand
+      - ssm:ListCommandInvocations
       - iam:CreateRole
       - iam:CreateServiceLinkedRole
       - iam:GetRole

source/idea/idea-administrator/resources/installer_policies/idea-admin-delete-policy.yml

@@ -60,9 +60,14 @@ Statement:
       - s3:DeleteBucket
       - s3:ListAllMyBuckets
       - s3:ListBucketVersions
+      - s3:DeleteObjectVersion
       - route53:DeleteHostedZone
       - acm:DescribeCertificate
       - acm:ListCertificates
       - acm:RequestCertificate
       - tag:GetResources
+      - cognito-idp:ListUserPools
+      - cognito-idp:DescribeUserPool
+      - cognito-idp:UpdateUserPool
+      - cognito-idp:DeleteUserPool
     Resource: '*'