Merge branch 'release/1.0-rc7'

# Please enter a commit message to explain why this merge is necessary, # especially if it merges an updated upstream into a topic branch. # # Lines starting with '#' will be ignored, and an empty message aborts # the commit.
art-institute-of-chicago · Dec 20, 2019 · 241e1b6 · 241e1b6
2 parents 40cc121 + 4f89e5d
commit 241e1b6
Show file tree

Hide file tree

Showing 12 changed files with 213 additions and 42 deletions.
diff --git a/README.md b/README.md
@@ -3,38 +3,59 @@
 # Data Aggregator
 > A central location of data collected from several different systems and offered up through an API
 
-The Data Aggregator is part of a large internal project to consolidate data across many disparate systems
-at the Art Institute of Chicago into a single, unified source. This offers our products a rich set of
-data that can be accessed in one way, in one location.
+The aggregator is the core of our data hub — a large internal project to consolidate data across many disparate systems at the Art Institute of Chicago into a single, unified source. This offers our products a rich set of data that can be accessed in one way, in one location. For more information about our data hub, please peruse the following paper:
+
+[Moskvin, Illya and trivedi, nikhil. "Building a Data Hub: Microservices, APIs, and System Integration at the Art Institute of Chicago." Museums and the Web 2019. Published January 15, 2019.](https://mw19.mwconf.org/paper/building-a-data-hub-microservices-apis-and-system-integration-at-the-art-institute-of-chicago/)
+
+
+## Looking for our data?
+
+You are in the right place! The aggregator contains all of our public APIs, which power our public-facing applications, such as our website and mobile app. As part of our [Open Access](https://www.artic.edu/open-access) efforts, we are making them available to the general public.
+
+For example, here's an endpoint that lists all of our published artworks:
+
+https://aggregator-data.artic.edu/api/v1/artworks
+
+...and here's a query that shows identifiers, titles, and last modified dates for all artworks that have been updated in our collections system in the past seven days from this moment, sorted in reverse chronological order:
+
+https://aggregator-data.artic.edu/api/v1/artworks/search?fields=id,title,last_updated_source&query[range][last_updated_source][gte]=now-7d&sort[last_updated_source][order]=desc
+
+Our API is a wrapper around [Elasticsearch's Query DSL](https://www.elastic.co/guide/en/elasticsearch/reference/6.0/query-dsl.html). Depending on your needs, these queries can get quite complex.
+
+Here are some resources to get you started:
+
+* [Art Institute of Chicago — API Documentation](https://aggregator-data.artic.edu/home) (fields and endpoints)
+* [Elasticsearch 6.0 — Query DSL](https://www.elastic.co/guide/en/elasticsearch/reference/6.0/query-dsl.html) (query syntax)
+* [Art Institute of Chicago — Open Access — Public API](https://www.artic.edu/open-access/public-api) (example projects)
+
+We are currently working on improving our documentation. In the meantime, feel free to open an issue here or reach out to [email protected] with any questions. We would love to hear about any projects you pursue with our API.
 
 
 ## Features
 
-* All data available via a JSON-based REST API
-* Large lists are paginated
+* All data is available via a JSON-based RESTful API
+* Most data is searchable via an Elasticsearch wrapper
 * Complex data types can be "included" in requests
-* Seed data provided for all resources
+* Large lists are paginated
 * Unit tests for all endpoints
 
 
 ## Overview
 
-The Data Aggregator interfaces with several internal APIs to collect its data. All data is imported and served
-up locally so that at runtime the API doesn't have dependencies on other systems. `artisan` commands have
-been set up to import data from various sources, either en masse or incrementally. One of the greatest benefits
-of an aggregator like this one is the ability to provide relationship between resources across systems. Our `/artworks`
-endpoint is a great example, as you can see relationships they have to a number of different things, like mobile tours,
-digital publications, and historic static sites.
+The aggregator interfaces with several internal APIs to collect its data. All data is imported and served up locally so that at runtime the API doesn't have dependencies on other systems. `artisan` commands have been set up to import data from various sources, either en masse or incrementally. One of the greatest benefits of an aggregator like this one is the ability to provide relationship between resources across systems. Our `/artworks` endpoint is a great example, as you can see relationships they have to a number of different things, like mobile tours, digital publications, and historic static sites.
 
 
 ## Requirements
 
 The project has been built in Laravel, and includes the following requirements:
 
-* Laravel 5.6
-* PHP 7.1 (may work in earlier versions but hasn't been tested)
-* MySQL 5.7 (may work in earlier versions but hasn't been tested)
+* Laravel 5.8
+* PHP 7.1
+* MySQL 5.7
 * [Composer](https://getcomposer.org/)
+* Elasticsearch 6.0
+
+For development, we recommend that you use [Laravel Homestead](https://laravel.com/docs/5.8/homestead). It includes everything you need to run this project. Note that you will need to [enable the optional Elasticsearch feature](https://laravel.com/docs/5.8/homestead#installing-optional-features) in your Homestead.yaml.
 
 
 ## Installing
@@ -55,8 +76,7 @@ composer install
 
 ## Developing
 
-First you'll need to create a `.env` file and update it to reflect your environment. We've provided an
-example file to get you started:
+First you'll need to create a `.env` file and update it to reflect  your environment. We've provided an example file to get you started:
 
 ```shell
 # Copy the example file
@@ -72,14 +92,12 @@ Then, to create the database tables and seed them with fake data, run:
 php artisan migrate --seed
 ```
 
-This will create all the tables and relationships, and fill the tables with data from the
-[Faker](https://github.com/fzaninotto/Faker) PHP library.
+This will create all the tables and relationships, and fill the tables with data from the [Faker](https://github.com/fzaninotto/Faker) PHP library.
 
 
 ### Importing real data
 
-We've created a series of `artisan` tasks to import data from source systems. You can see all the available
-imports like so:
+We've created a series of `artisan` tasks to import data from source systems. You can see all the available imports like so:
 
 ```shell
 php artisan list import
@@ -91,10 +109,6 @@ To import all data from all systems, run:
 php artisan import:all
 ```
 
-### Adding a new data source
-
-See [here](ADD_NEW_DATA_SOURCE.md) for details on adding new data sources to the Aggregator.
-
 
 ## Contributing
 
@@ -119,20 +133,15 @@ git checkout -b feature/good-short-description
 git push origin feature/good-short-description
 ```
 
-Then on github.com, create a Pull Request to merge your changes into our
-`develop` branch.
+Then on GitHub, create a [Pull Request](https://help.github.com/en/github/collaborating-with-issues-and-pull-requests/about-pull-requests) to merge your changes into our `develop` branch.
 
-Our internal team uses [`php-cs-fixer`](https://github.com/FriendsOfPHP/PHP-CS-Fixer) to ensure
-our code meets various PHP Standards Recommendations. You're welcome to integrate `php-cs-fixer`
-into your workflow as you work on this project, but is not required to make a contribution.
+Our internal team uses [`php-cs-fixer`](https://github.com/FriendsOfPHP/PHP-CS-Fixer) to ensure our code meets various [PHP Standards Recommendations](https://www.php-fig.org/psr/). You're welcome to integrate `php-cs-fixer` into your workflow as you work on this project, but it is not required to make a contribution.
 
-This project is released with a Contributor Code of Conduct. By participating in
-this project you agree to abide by its [terms](CODE_OF_CONDUCT.md).
+This project is released with a Contributor Code of Conduct. By participating in this project you agree to abide by its [terms](CODE_OF_CONDUCT.md).
 
 We welcome bug reports and questions under GitHub's [Issues](issues). For other concerns, you can reach our engineering team at [[email protected]](mailto:[email protected])
 
 
 ## Licensing
 
-This project is licensed under the [GNU Affero General Public License
-Version 3](LICENSE).
+This project is licensed under the [GNU Affero General Public License Version 3](LICENSE).
diff --git a/app/Models/Web/Event.php b/app/Models/Web/Event.php
@@ -26,6 +26,7 @@ class Event extends WebModel
         'alt_event_types' => 'array',
         'alt_audiences' => 'array',
         'programs' => 'array',
+        'test_emails' => 'array',
         'publish_start_date' => 'datetime',
         'publish_end_date' => 'datetime',
     ];

diff --git a/app/Models/Web/EventEmailSeriesPivot.php b/app/Models/Web/EventEmailSeriesPivot.php
@@ -11,6 +11,13 @@ class EventEmailSeriesPivot extends BasePivot
 
     protected $table = 'event_email_series';
 
+    protected $casts = [
+        'send_affiliate_test' => 'boolean',
+        'send_member_test' => 'boolean',
+        'send_sustaining_fellow_test' => 'boolean',
+        'send_nonmember_test' => 'boolean',
+    ];
+
     public function event()
     {
         return $this->belongsTo('App\Models\Web\Event', 'event_id', 'id');

diff --git a/app/Transformers/Inbound/Web/Event.php b/app/Transformers/Inbound/Web/Event.php
@@ -14,6 +14,7 @@ protected function getExtraFields(Datum $datum)
             'start_date' => $datum->datetime('start_date'),
             'end_date' => $datum->datetime('end_date'),
             'type' => $datum->event_type,
+            'test_emails' => $this->getTestEmails($datum),
 
             // TODO: Move these to trait?
             'publish_start_date' => $datum->date('publish_start_date'),
@@ -28,6 +29,11 @@ protected function getSync(Datum $datum, $test = false)
         ];
     }
 
+    private function getTestEmails(Datum $datum)
+    {
+        return array_map('trim', array_filter(explode(',', rtrim($datum->test_emails, ','))));
+    }
+
     private function getSyncEmailSeries(Datum $datum)
     {
         return $this->getSyncPivots($datum, 'email_series', 'email_series_id', function ($pivot) {
@@ -37,6 +43,10 @@ private function getSyncEmailSeries(Datum $datum)
                     'member_copy',
                     'sustaining_fellow_copy',
                     'nonmember_copy',
+                    'send_affiliate_test',
+                    'send_member_test',
+                    'send_sustaining_fellow_test',
+                    'send_nonmember_test',
                 ])),
             ];
         });

diff --git a/app/Transformers/Inbound/WebTransformer.php b/app/Transformers/Inbound/WebTransformer.php
@@ -15,10 +15,11 @@ class WebTransformer extends BaseTransformer
      */
     protected function getDates(Datum $datum)
     {
-        return array_merge(parent::getDates($datum),
-                           [
-                               'source_modified_at' => $datum->date('last_updated'),
-                           ]
+        return array_merge(
+            parent::getDates($datum),
+            [
+                'source_modified_at' => $datum->date('last_updated'),
+            ]
         );
     }
 }
diff --git a/app/Transformers/Outbound/Web/Event.php b/app/Transformers/Outbound/Web/Event.php
@@ -309,6 +309,12 @@ protected function getFields()
                     return $item->sponsor->id ?? null;
                 },
             ],
+            'test_emails' => [
+                'doc' => 'Email addresses to target for email series tests',
+                'type' => 'array',
+                'elasticsearch' => 'text',
+                'is_restricted' => true,
+            ],
         ];
     }
 

diff --git a/app/Transformers/Outbound/Web/EventEmailSeriesPivot.php b/app/Transformers/Outbound/Web/EventEmailSeriesPivot.php
@@ -43,6 +43,22 @@ protected function getFields()
                 'doc' => 'Copy to use for Non-Members when communicating this event in this email series',
                 'type' => 'string',
             ],
+            'send_affiliate_test' => [
+                'doc' => 'Whether to send the Affiliate Members test email during next check',
+                'type' => 'boolean',
+            ],
+            'send_member_test' => [
+                'doc' => 'Whether to send the Members test email during next check',
+                'type' => 'boolean',
+            ],
+            'send_sustaining_fellow_test' => [
+                'doc' => 'Whether to send the Sustaining Fellows test email during next check',
+                'type' => 'boolean',
+            ],
+            'send_nonmember_test' => [
+                'doc' => 'Whether to send the Non-Members test email during next check',
+                'type' => 'boolean',
+            ],
         ];
     }
 

diff --git a/config/aic.php b/config/aic.php
@@ -12,7 +12,7 @@
     | or any other location as required by the application or its packages.
     */
 
-    'version' => '1.0-rc6',
+    'version' => '1.0-rc7',
 
     /*
     |--------------------------------------------------------------------------

diff --git a/database/migrations/2019_12_13_094516_add_source_modified_at_to_web_tables.php b/database/migrations/2019_12_13_094516_add_source_modified_at_to_web_tables.php
@@ -0,0 +1,51 @@
+<?php
+
+use Illuminate\Support\Facades\Schema;
+use Illuminate\Database\Schema\Blueprint;
+use Illuminate\Database\Migrations\Migration;
+
+class AddSourceModifiedAtToWebTables extends Migration
+{
+    private $tables = [
+        // 'tags', // has, dropped
+        // 'locations' => 'published', // needs, dropped
+        // 'hours', // has
+        'closures' => 'type', // needs
+        'web_exhibitions' => 'is_featured', // needs
+        'events' => 'publish_end_date', // needs
+        // 'event_occurrences', // has
+        'event_programs' => 'is_event_host', // needs
+        'articles' => 'publish_end_date', // needs
+        // 'email_series', // has
+        // 'selections', // has
+        'web_artists' => 'datahub_id', // needs
+        // 'static_pages' => 'web_url', // hardcoded?
+        'generic_pages' => 'imgix_uuid', // needs
+        'press_releases' => 'imgix_uuid', // needs
+        // 'research_guides' => 'imgix_uuid', // needs, dropped
+        'educator_resources' => 'imgix_uuid', // needs
+        'digital_catalogs' => 'imgix_uuid', // needs
+        'printed_catalogs' => 'imgix_uuid', // needs
+        // 'interactive_features', // has
+        // 'experiences', // has
+        'sponsors' => 'published', // needs
+    ];
+
+    public function up()
+    {
+        foreach ($this->tables as $table => $previousColumn) {
+            Schema::table($table, function (Blueprint $table) use ($previousColumn) {
+                $table->timestamp('source_modified_at')->nullable()->after($previousColumn);
+            });
+        }
+    }
+
+    public function down()
+    {
+        foreach ($this->tables as $table => $previousColumn) {
+            Schema::table($table, function (Blueprint $table) {
+                $table->dropColumn('source_modified_at');
+            });
+        }
+    }
+}
diff --git a/database/migrations/2019_12_17_144919_add_send_test_fields_to_event_email_series.php b/database/migrations/2019_12_17_144919_add_send_test_fields_to_event_email_series.php
@@ -0,0 +1,40 @@
+<?php
+
+use Illuminate\Support\Facades\Schema;
+use Illuminate\Database\Schema\Blueprint;
+use Illuminate\Database\Migrations\Migration;
+
+class AddSendTestFieldsToEventEmailSeries extends Migration
+{
+    /**
+     * Run the migrations.
+     *
+     * @return void
+     */
+    public function up()
+    {
+        Schema::table('event_email_series', function (Blueprint $table) {
+            $table->boolean('send_affiliate_test')->nullable()->after('affiliate_copy');
+            $table->boolean('send_member_test')->nullable()->after('send_affiliate_test');
+            $table->boolean('send_sustaining_fellow_test')->nullable()->after('send_member_test');
+            $table->boolean('send_nonmember_test')->nullable()->after('send_sustaining_fellow_test');
+        });
+    }
+
+    /**
+     * Reverse the migrations.
+     *
+     * @return void
+     */
+    public function down()
+    {
+        Schema::table('event_email_series', function (Blueprint $table) {
+            $table->dropColumn([
+                'send_affiliate_test',
+                'send_member_test',
+                'send_sustaining_fellow_test',
+                'send_nonmember_test',
+            ]);
+        });
+    }
+}
diff --git a/database/migrations/2019_12_17_160039_add_test_emails_to_events.php b/database/migrations/2019_12_17_160039_add_test_emails_to_events.php
@@ -0,0 +1,32 @@
+<?php
+
+use Illuminate\Support\Facades\Schema;
+use Illuminate\Database\Schema\Blueprint;
+use Illuminate\Database\Migrations\Migration;
+
+class AddTestEmailsToEvents extends Migration
+{
+    /**
+     * Run the migrations.
+     *
+     * @return void
+     */
+    public function up()
+    {
+        Schema::table('events', function (Blueprint $table) {
+            $table->text('test_emails')->nullable()->after('image_url');
+        });
+    }
+
+    /**
+     * Reverse the migrations.
+     *
+     * @return void
+     */
+    public function down()
+    {
+        Schema::table('events', function (Blueprint $table) {
+            $table->dropColumn('test_emails');
+        });
+    }
+}
diff --git a/resources/views/home.blade.php b/resources/views/home.blade.php
@@ -34,9 +34,7 @@
 
     <h2>Getting started</h2>
 
-    <p>API requests are made by accessing various endpoints at <a href="/api/v1">{{ config('app.url') }}/api/v1</a>. Read more about our <a href="/docs/endpoints">endpoints</a> and <a href="/docs/fields">fields</a> in our documentation to learn about the parameters you can use to manipulate your query.</p>
-
-    <p>Most data is accessible without authentication. If you're developing a high-traffic app, you can create an account on this site to create personal tokens that can allow you to access the API more frequently.</p>
+    <p>API requests are made by accessing various endpoints at <a href="/api/v1">{{ config('app.url') }}/api/v1</a>. Read more about our <a href="/docs/endpoints">endpoints</a> and <a href="/docs/fields">fields</a> to learn about the parameters you can use to manipulate your query.</p>
 
     <h2>What's an API?</h2>