[Resolve #1114] Resolvable Template Handler configs and the !stack_at…

…tr resolver (#1189)

This pull request adds two related things:
1. Template handler configurations are now fully resolvable.
2. You can now use the `!stack_attr` resolver to reference other configurations on the stack and StackGroup 

The need for these two things came about in my testing on the recent changes to Sceptre's resolvers. 

## Why is this necessary?
Previously, I had this on one of my stacks:

```yaml
template:
    type: sam
    path: beatbox/template.yaml
    artifact_bucket_name: {{ template_bucket_name }}
    artifact_prefix: {{ template_key_prefix }}
```

^^ This has been working just fine until I set `template_bucket_name` in my project to a `!stack_output` resolver. Once I did that, however, the template handler blew up because Jinja was attempting to cast the StackOutput resolver to string.

With the changes in this PR, I can now do this:

```yaml
template:
    type: sam
    path: beatbox/template.yaml
    # This will access and resolve the value of template_bucket_name from the StackGroup 
    # and then access that value as the resolved value for handler.
    artifact_bucket_name: !stack_attr template_bucket_name
    artifact_prefix: {{ template_key_prefix }}
```

docs/_source/conf.py

@@ -209,7 +209,9 @@
     ('py:class', 'TextIO'),
     ('py:class', '_io.StringIO'),
     ('py:class', 'yaml.loader.SafeLoader'),
-    ('py:class', 'yaml.dumper.Dumper')
+    ('py:class', 'yaml.dumper.Dumper'),
+    ('py:class',  'cfn_tools.odict.ODict'),
+    ('py:class',  'T_Container')
 ]
 
 set_type_checking_flag = True

docs/_source/docs/permissions.rst

@@ -125,10 +125,10 @@ Tips for working with Sceptre, IAM, and a CI/CD system
     permissions of the deployment role to only certain paths, preventing the deployment role from
     elevating its own permissions or modifying unrelated roles and policies.
   * Using ``aws:CalledVia`` and ``aws:CalledViaFirst`` conditions matching against
-    ``"cloudformation.amazonaws.com"`` on non-cloudformation actions to ensure that the deployment
-    role can only execute changes via CloudFormation and not on its own. Note: Some actions are
-    taken by Sceptre directly and not via cloudformation (see the section below on this). Those
-    actions should *not* have a CalledVia condition applied.
+    ``"cloudformation.amazonaws.com"`` to ensure that the deployment role can only execute changes
+    via CloudFormation and not on its own. Note: Some actions are taken by Sceptre directly and not
+    via cloudformation (see the section below on this). Those actions should *not* have a CalledVia
+    condition applied.
 
 * If you define your deployment role (and any other related resources) using Sceptre and then
   reference it on all *other* stacks using ``iam_role: !stack_output ...``, this means that your

docs/_source/docs/resolvers.rst

@@ -79,6 +79,53 @@ A resolver to execute any shell command.
 
 Refer to `sceptre-resolver-cmd <https://github.com/Sceptre/sceptre-resolver-cmd/>`_ for documentation.
 
+.. _stack_attr_resolver:
+
+stack_attr
+~~~~~~~~~~
+
+This resolver resolves to the values of other fields on the same Stack Config or those
+inherited from StackGroups in which the current Stack Config exists, even when those other fields are
+also resolvers.
+
+To understand why this is useful, consider a stack's ``template_bucket_name``. This is usually set on
+the highest level StackGroup Config. Normally, you could reference the template_bucket_name that was
+set in an outer StackGroup Config with Jinja using ``{{template_bucket_name}}`` or, more explicitly, with
+``{{stack_group_config.template_bucket_name}}``.
+
+However, if the value of ``template_bucket_name`` is set with a resolver, using Jinja won't work.
+This is due to the :ref:`resolution_order` on a Stack Config. Jinja configs are rendered *before*
+resolvers are constructed or resolved, so you can't resolve a resolver from a StackGroup Config via
+Jinja. That's where !stack_attr is useful. It's a resolver that resolves to the value of another stack
+attribute (which could be another resolver).
+
+.. code-block:: yaml
+
+   template:
+       type: sam
+       path: path/from/my/cwd/template.yaml
+       # template_bucket_name could be set by a resolver in the StackGroup.
+       artifact_bucket_name: !stack_attr template_bucket_name
+
+The argument to this resolver is the full attribute "path" from the Stack Config. You can access
+nested values in dicts and lists using "." to separate key/index segments. For example:
+
+.. code-block:: yaml
+
+   sceptre_user_data:
+       key:
+           - "some random value"
+           - "the value we want to select"
+
+   iam_role: !stack_output roles.yaml::RoleArn
+
+   parameters:
+       # This will pass the value of "the value we want to select" for my_parameter
+       my_parameter: !stack_attr sceptre_user_data.key.1
+       # You can also access the value of another resolvable property like this:
+       use_role_arn: !stack_attr iam_role
+
+
 stack_output
 ~~~~~~~~~~~~
 

docs/_source/docs/stack_config.rst

@@ -47,7 +47,7 @@ from the Stack config filename.
 
 template
 ~~~~~~~~
-* Resolvable: No
+* Resolvable: Yes
 * Can be inherited from StackGroup: No
 
 Configuration for a template handler. Template handlers can take in parameters
@@ -363,6 +363,76 @@ Syntax:
 When compiled, ``sceptre_user_data`` would be the dictionary
 ``{"iam_policy_file": "/path/to/policy.json"}``.
 
+.. _resolution_order:
+
+Resolution order of values
+--------------------------
+
+Stack Configs allow you to pull together values from a variety of sources to configure a
+CloudFormation stack. These values are retrieved and applied in phases. Understanding these phases can
+be very helpful when designing your Stack Configs.
+
+When launching a stack (or performing other stack actions), values are gathered and accessed in this
+order:
+
+1. User variables (from ``--var`` and ``--var-file`` arguments) are gathered when the CLI first runs.
+2. StackGroup Configs are read from the highest level downward, rendered with Jinja and then loaded
+   into yaml. The key/value pairs from these configs are layered on top of each other, with more nested
+   configs overriding higher-level ones. These key/value pairs will be "inherited" by the Stack
+   Config. These variables are made available when rendering a StackGroup Config:
+
+   * User variables (via ``{{ var }}``)
+   * Environment variables (via ``{{ environment_variable }}``)
+   * StackGroup configurations from *higher* level StackGroup Configs are available by name. Note:
+     more nested configuration values will overshadow higher-level ones by the same key.
+
+3. With the layered StackGroup Config variables, the Stack Config file will be read and then rendered
+   with Jinja. These variables are made available when the Stack Config is being rendered with Jinja:
+
+   * User variables (via ``{{ var }}``)
+   * Environment variables (via ``{{ environment_variable }}``)
+   * All StackGroup configurations are available by name directly as well as via ``{{ stack_group_config }}``
+
+   **Important:** If any StackGroup configuration values were set with resolvers, accessing them via
+   Jinja will not resolve them, since resolvers require a Stack object, which has not yet been
+   assembled yet. **Resolvers will not be accessible until a later phase.**
+4. Once rendered via Jinja into a string, the Stack Config will be loaded into yaml. This is when the
+   resolver instances on the Stack config will be **constructed** (*not* resolved).
+5. The Stack instance will be constructed with the key/value pairs from the loaded yaml layered on
+   top of the key/value pairs from the StackGroup configurations. This is when all resolver instances,
+   both those inherited from StackGroup Configs and those from the present Stack Config, will be
+   connected to the Stack instance and thus *ready* to be resolved.
+6. The first time a resolvable configuration is *accessed* is when the resolver(s) at that
+   configuration will be resolved and replaced with their resolved value. This is normally done at
+   the very last moment, right when it is needed (and not before).
+
+"Render Time" vs. "Resolve Time"
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+A common point of confusion tends to be around the distinction between **"render time"** (phase 3, when
+Jinja logic is applied) and **"resolve time"** (phase 6, when resolvers are resolved). You cannot use
+a resolver via Jinja during "render time", since the resolver won't exist or be ready to use yet. You can,
+however, use Jinja logic to indicate *whether*, *which*, or *how* a resolver is configured.
+
+For example, you **can** do something like this:
+
+.. code-block:: yaml
+
+   parameters:
+     {% if var.use_my_parameter %}
+       my_parameter: !stack_output {{ var.stack_name }}::{{ var.output_name }}
+     {% endif %}
+
+Accessing resolved values in other fields
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Sometimes you might want to reference the resolved value of one field in another field. Since you cannot
+use Jinja to access resolved values, there is another way to this. The :ref:`stack_attr_resolver`
+resolver is meant for addressing just this need. It's a resolver that will resolve to the value of
+another Stack Config field value. See the linked documentation for more details on that resolver and
+its use.
+
+
 Examples
 --------
 
@@ -377,7 +447,7 @@ Examples
 
 .. code-block:: yaml
 
-   template
+   template:
      path: templates/example.yaml
      type: file
    dependencies:

sceptre/config/reader.py

@@ -13,6 +13,8 @@
 import fnmatch
 import logging
 from os import environ, path, walk
+from typing import Set
+
 from pkg_resources import iter_entry_points
 from pathlib import Path
 import yaml
@@ -186,15 +188,14 @@ def resolve_node_tag(self, loader, node):
         node.tag = loader.resolve(type(node), node.value, (True, False))
         return node
 
-    def construct_stacks(self):
+    def construct_stacks(self) -> Set[Stack]:
         """
         Traverses the files under the command path.
         For each file encountered, a Stack is constructed
         using the correct config. Dependencies are traversed
         and a final set of Stacks is returned.
 
         :returns: A set of Stacks.
-        :rtype: set
         """
         stack_map = {}
         command_stacks = set()
@@ -340,21 +341,17 @@ def read(self, rel_path, base_config=None):
         self.logger.debug("Config: %s", config)
         return config
 
-    def _recursive_read(self, directory_path, filename, stack_group_config):
+    def _recursive_read(self, directory_path: str, filename: str, stack_group_config: dict) -> dict:
         """
         Traverses the directory_path, from top to bottom, reading in all
         relevant config files. If config attributes are encountered further
         down the StackGroup they are merged with the parent as defined in the
         `CONFIG_MERGE_STRATEGIES` dict.
 
         :param directory_path: Relative directory path to config to read.
-        :type directory_path: str
         :param filename: File name for the config to read.
-        :type filename: dict
         :param stack_group_config: The loaded config file for the StackGroup
-        :type stack_group_config: dict
         :returns: Representation of inherited config.
-        :rtype: dict
         """
 
         parent_directory = path.split(directory_path)[0]

sceptre/resolvers/stack_attr.py

@@ -0,0 +1,62 @@
+from typing import Any, List
+
+from sceptre.resolvers import Resolver
+
+
+class StackAttr(Resolver):
+    """Resolves to the value of another field on the Stack Config, including other resolvers.
+
+    The argument for this resolver should be the "key path" from the stack object, which can access
+    nested keys/indexes using a "." to separate segments.
+
+    For example, given this Stack Config structure...
+
+        sceptre_user_data:
+            nested_list:
+              - first
+              - second
+
+    Using "!stack_attr sceptre_user_data.nested_list.1" on your stack would resolve to "second".
+    """
+
+    # These are all the attributes on Stack Configs whose names are changed when they are assigned
+    # to the Stack instance.
+    STACK_ATTR_MAP = {
+        'template': 'template_handler_config',
+        'protect': 'protected',
+        'stack_name': 'external_name',
+        'stack_tags': 'tags'
+    }
+
+    def resolve(self) -> Any:
+        """Returns the resolved value of the field referenced by the resolver's argument."""
+        segments = self.argument.split('.')
+
+        # Remap top-level attributes to match stack config
+        first_segment = segments[0]
+        segments[0] = self.STACK_ATTR_MAP.get(first_segment, first_segment)
+
+        if self._key_is_from_stack_group_config(first_segment):
+            obj = self.stack.stack_group_config
+        else:
+            obj = self.stack
+
+        result = self._recursively_resolve_segments(obj, segments)
+        return result
+
+    def _key_is_from_stack_group_config(self, key: str):
+        return key in self.stack.stack_group_config and not hasattr(self.stack, key)
+
+    def _recursively_resolve_segments(self, obj: Any, segments: List[str]):
+        if not segments:
+            return obj
+
+        attr_name, *rest = segments
+        if isinstance(obj, dict):
+            value = obj[attr_name]
+        elif isinstance(obj, list):
+            value = obj[int(attr_name)]
+        else:
+            value = getattr(obj, attr_name)
+
+        return self._recursively_resolve_segments(value, rest)

sceptre/stack.py

@@ -132,6 +132,11 @@ class Stack(object):
         "s3_details",
         PlaceholderType.none
     )
+    template_handler_config = ResolvableContainerProperty(
+        'template_handler_config',
+        PlaceholderType.alphanum
+    )
+
     template_bucket_name = ResolvableValueProperty(
         "template_bucket_name",
         PlaceholderType.none
@@ -169,7 +174,6 @@ def __init__(
         self.required_version = required_version
         self.external_name = external_name or get_external_stack_name(self.project_code, self.name)
         self.template_path = template_path
-        self.template_handler_config = template_handler_config
         self.dependencies = dependencies or []
         self.protected = protected
         self.on_failure = on_failure
@@ -187,6 +191,7 @@ def __init__(
         self.tags = tags or {}
         self.role_arn = role_arn
         self.template_bucket_name = template_bucket_name
+        self.template_handler_config = template_handler_config
 
         self.s3_details = s3_details
         self.parameters = parameters or {}

setup.py

@@ -73,7 +73,8 @@ def get_version(rel_path):
             "stack_output = sceptre.resolvers.stack_output:StackOutput",
             "stack_output_external ="
             "sceptre.resolvers.stack_output:StackOutputExternal",
-            "no_value = sceptre.resolvers.no_value:NoValue"
+            "no_value = sceptre.resolvers.no_value:NoValue",
+            "stack_attr = sceptre.resolvers.stack_attr:StackAttr"
         ],
         "sceptre.template_handlers": [
             "file = sceptre.template_handlers.file:File",