WIP: spike consolidating how we run Smokey test suite

README.md

@@ -55,6 +55,7 @@ You can use the following environment variables to configure the tests:
 * `SIGNON_EMAIL`: email of a Signon user in $ENVIRONMENT
 * `SIGNON_PASSWORD`: password of a Signon user in $ENVIRONMENT
 * `RATE_LIMIT_TOKEN`: (optional) a token used to bypass rate limiting if present on apps.
+* `PROXY_PROFILE` (optional) name of profile to use in proxy.rb. See [proxy tests](#proxy)
 
 Smokey cannot use a Signon account which has multi-factor authentication
 enabled.
@@ -67,6 +68,33 @@ k get secret smokey-signon-account -oyaml | yq .data.password | base64 -d
 k get secret smokey-signon-account -oyaml | yq .data.email | base64 -d
 ```
 
+### Proxy tests
+
+The Smokey test suite can be told to run against our primary CDN (Fastly), our failover CDN (Cloudfront), or any one of the GOV.UK mirrors. To do so, you'll need to run the proxy in a separate process, and ensure you specify the `PROXY_PROFILE` environment variable.
+
+Example of running Smokey tests against the main GOV.UK mirror:
+
+```shell
+shell1$ env PROXY_PROFILE=mirrorS3 ruby proxy.rb
+
+shell2$ env PROXY_PROFILE=mirrorS3 ENVIRONMENT=production bundle exec cucumber --tags="@worksonmirror"
+```
+
+Example of running Smokey tests against the failover CDN (NB: the 'secret' Cloudfront value can be [found in govuk-dns-tf](https://github.com/alphagov/govuk-dns-tf/pull/69)):
+
+```shell
+shell1$ env PROXY_PROFILE=failoverCDN FAILOVER_CDN_HOST="secret.cloudfront.net" ruby proxy.rb
+
+shell2$ env PROXY_PROFILE=failoverCDN ENVIRONMENT=production bundle exec cucumber --tags="not @notreplatforming and not @notcloudfront"
+```
+
+Debug mode for checking you're setting headers correctly:
+
+```shell
+env PROXY_PROFILE=debug ruby proxy.rb
+# Visit http://127.0.0.1:8080/ to see your request headers
+```
+
 ## Further documentation
 
 - [deployment](docs/deployment.md)

features/apps/collections.feature

@@ -1,12 +1,14 @@
 @app-collections @replatforming
 Feature: Collections
 
+  @worksonmirror
   Scenario: Check the frontend can talk to Content Store
     When I visit "/browse/driving"
     And I should see "Teaching people to drive"
     And I click on the section "Teaching people to drive"
     Then I should see "Apply to become a driving instructor"
 
+  @worksonmirror
   Scenario: Check the frontend can talk to Search API
     When I visit "/government/organisations/hm-revenue-customs/services-information"
     Then I see links to pages per topic

features/apps/feedback.feature

@@ -1,10 +1,12 @@
 @replatforming @app-feedback
 Feature: Feedback
 
+  @worksonmirror
   Scenario: Check the frontend can talk to Content Store
     When I visit "/contact/govuk"
     Then I should see "Contact GOV.UK"
 
+  @notcloudfront
   Scenario: Check "is this page useful?" email survey
     When I visit "/"
     And I click to say the page is not useful

features/apps/finder_frontend.feature

@@ -1,5 +1,6 @@
 @app-finder-frontend @replatforming
 Feature: Finder Frontend
+  @worksonmirror
   Scenario: Check the frontend can talk to Content Store
     When I visit "/government/people"
     Then I should see "All ministers and senior officials on GOV.UK"
@@ -12,6 +13,7 @@ Feature: Finder Frontend
     And I choose the checkbox "Statistics (published)" and click on "Continue"
     Then I should see "How often do you want to get emails?"
 
+  @notcloudfront
   Scenario Outline: Check the frontend can talk to Search API
     Given I consent to cookies
     When I search for "<keywords>"

features/apps/frontend.feature

@@ -1,14 +1,17 @@
 @replatforming @app-frontend
 Feature: Frontend
 
+  @worksonmirror
   Scenario: Check help page loads
     When I visit "/help"
     Then I should see "Help using GOV.UK"
 
+  @worksonmirror
   Scenario: Check homepage loads
     When I visit "/"
     Then I should see "Welcome to GOV.UK"
 
+  @notcloudfront
   Scenario: Check the client can talk to Google Analytics
     When I visit "/"
     And I consent to cookies
@@ -21,6 +24,7 @@ Feature: Frontend
     When I try to post to "/find-licences/busking-licence" with "postcode=E20+2ST"
     Then I should see "Busking licence"
 
+  @worksonmirror
   Scenario: Check the frontend can talk to Asset Manager with media path
     When I visit "/media/5a7b9f8ced915d4147621960/passport-impact-indicat.csv/preview"
     Then I should see "Passport impact indicators - CSV version"
@@ -45,6 +49,7 @@ Feature: Frontend
     When I try to post to "/ukonline-centre-internet-access-computer-training" with "postcode=WC2B+6NH"
     Then I should see "Holborn Library"
 
+  @worksonmirror
   Scenario: Check the travel advice index page loads
     When I visit "/foreign-travel-advice"
     Then I should see "Foreign travel advice"

features/apps/government_frontend.feature

@@ -1,6 +1,7 @@
 @app-government-frontend @replatforming
 Feature: Government Frontend
 
+  @worksonmirror
   Scenario: Check the frontend can talk to Content Store
     When I visit "/government/get-involved"
     Then I should see "Get involved"
@@ -21,16 +22,19 @@ Feature: Government Frontend
     And I click on the button "Continue"
     Then I should see "How often do you want to get emails?"
 
+  @worksonmirror
   Scenario: Check the frontend can talk to Search API
     When I visit "/government/get-involved"
     Then I should see "Recently opened"
     And it should be populated with three open consultations
 
+  @worksonmirror
   Scenario: Check a travel advice country page loads
     When I visit "/foreign-travel-advice/luxembourg"
     Then I should see "Luxembourg"
     And I should see "Summary"
 
+  @worksonmirror
   Scenario: Check that Service Manuals load
     When I visit "/service-manual"
     Then I should see "Service Manual"

features/apps/search_api.feature

@@ -1,4 +1,4 @@
-@app-search-api
+@app-search-api @worksonmirror
 Feature: Search API
   Scenario: Check the app is routable
     When I visit "/sitemap.xml"

features/apps/smartanswers.feature

@@ -9,6 +9,7 @@ Feature: Smart Answers
     | /vat-payment-deadlines/y/2000-01-31         |
     | /vat-payment-deadlines/y/2000-01-31/cheque  |
 
+  @worksonmirror
   Scenario: Check the frontend can talk to Worldwide API
     When I visit "/check-uk-visa/y"
     Then I should see a populated country select

features/apps/static.feature

@@ -1,5 +1,6 @@
 Feature: Static
 
+  @notcloudfront
   Scenario: Check the feedback component loads
     When I visit "/help"
     And I click to report a problem with the page

features/cdn.feature

@@ -1,9 +1,11 @@
 Feature: CDN
+  @notcloudfront
   Scenario: Check all A/B test variants work
     Given I consent to cookies
     When multiple new users visit "/help/ab-testing"
     Then we have shown them all versions of the A/B test
 
+  @notcloudfront
   Scenario: Check an A/B test is persistent
     Given I consent to cookies
     And I do not have any A/B testing cookies set
@@ -13,13 +15,13 @@ Feature: CDN
     And the bucket is reported to Google Analytics
     And I stay on the same bucket when I keep visiting "/help/ab-testing"
 
-  @replatforming
+  @replatforming @notcloudfront
   Scenario: Check caching behaviour for POST requests
     When I try to post to "/find-local-council" with "postcode=WC2B+6NH" without following redirects
     Then I should not hit the cache
     Then I should see "camden"
 
-  @replatforming
+  @replatforming @notcloudfront
   Scenario: Check caching behaviour for GET requests
     When I request "/"
     Then I should hit the cache

features/origin.feature

@@ -5,12 +5,12 @@ Feature: Origin
     When I send a GET request to "/robots.txt"
     Then I should get a 403 status code
 
-  @replatforming
+  @replatforming @worksonmirror
   Scenario: Check robots.txt loads
     When I visit "/robots.txt"
     Then I should see "User-agent:"
 
-  @replatforming
+  @replatforming @worksonmirror
   Scenario: Check redirects work
     When I visit "/workplacepensions"
     Then I should be at a location path of "/workplace-pensions"
@@ -20,7 +20,7 @@ Feature: Origin
     When I visit a non-existent page
     Then I should see "Page not found"
 
-  @replatforming
+  @replatforming @worksonmirror
   Scenario: Check pages are rendered using UTF-8
     When I request "/"
     Then I should get a "Content-Type" header of "text/html; charset=utf-8"

features/step_definitions/smokey_steps.rb

@@ -150,13 +150,10 @@ def should_see(text)
 end
 
 Then /^I should see "(.*)"$/ do |content|
-  if @responses
-    @responses.each do |response|
-      expect(response.body).to include(content)
-    end
-  elsif @response
+  if @response
     expect(@response.body).to include(content)
   elsif page
+    wait_for_any_redirects_to_govuk
     expect(page.body).to include(content)
   end
 end
@@ -166,6 +163,7 @@ def should_see(text)
     uri = URI(@response['location'])
     expect(uri.path).to eq(location_path)
   else
+    wait_for_any_redirects_to_govuk
     uri = URI(page.current_url)
     expect(uri.path).to eq(location_path)
   end

features/support/env.rb

@@ -3,21 +3,7 @@
 require 'plek'
 require 'selenium-webdriver'
 require 'uri'
-
-# Set up environment
-case ENV["ENVIRONMENT"]
-when "integration"
-  ENV["GOVUK_APP_DOMAIN"] ||= "integration.publishing.service.gov.uk"
-  ENV["GOVUK_WEBSITE_ROOT"] ||= "https://www.integration.publishing.service.gov.uk"
-when "staging"
-  ENV["GOVUK_APP_DOMAIN"] ||= "staging.publishing.service.gov.uk"
-  ENV["GOVUK_WEBSITE_ROOT"] ||= "https://www.staging.publishing.service.gov.uk"
-when "production"
-  ENV["GOVUK_APP_DOMAIN"] ||= "publishing.service.gov.uk"
-  ENV["GOVUK_WEBSITE_ROOT"] ||= "https://www.gov.uk"
-else
-  raise "ENVIRONMENT should be one of integration, staging, production"
-end
+require_relative './env_vars'
 
 # Set up error reporting (using SENTRY_CURRENT_ENV for the environment).
 GovukError.configure

features/support/env_vars.rb

@@ -0,0 +1,19 @@
+# Set up environment
+case ENV["ENVIRONMENT"]
+when "integration"
+  ENV["GOVUK_APP_DOMAIN"] ||= "integration.publishing.service.gov.uk"
+  ENV["GOVUK_WEBSITE_HOST"] ||= "www.integration.publishing.service.gov.uk"
+  ENV["PROXY_PROFILE"] ||= "primaryCDN"
+when "staging"
+  ENV["GOVUK_APP_DOMAIN"] ||= "staging.publishing.service.gov.uk"
+  ENV["GOVUK_WEBSITE_HOST"] ||= "www.staging.publishing.service.gov.uk"
+  ENV["PROXY_PROFILE"] ||= "primaryCDN"
+when "production"
+  ENV["GOVUK_APP_DOMAIN"] ||= "publishing.service.gov.uk"
+  ENV["GOVUK_WEBSITE_HOST"] ||= "www.gov.uk"
+  ENV["PROXY_PROFILE"] ||= "primaryCDN"
+else
+  raise "ENVIRONMENT should be one of integration, staging, production"
+end
+
+ENV["GOVUK_WEBSITE_ROOT"] = "http://127.0.0.1:8080"

features/support/visiting_pages.rb

@@ -7,3 +7,7 @@ def visit_path(path)
     visit "#{path}?smokey_cachebust=#{rand.to_s}"
   end
 end
+
+def wait_for_any_redirects_to_govuk
+  page.has_css?(".govuk-template")
+end