Squashed commit of the following:

commit d9db33c Merge: 71b3fc0 6d08730 Author: Juha-Matti Lehtinen <[email protected]> Date: Mon Jun 4 10:28:22 2018 +0300 Merge pull request #271 from CSCfi/CSCMETAX-280 CSCMETAX-280:[FIX] proper handling of language field data. commit 6d08730 Merge: 70cb0ea 71b3fc0 Author: Juha-Matti Lehtinen <[email protected]> Date: Mon Jun 4 10:05:03 2018 +0300 Merge branch 'test' into CSCMETAX-280 commit 70cb0ea Merge: 8106f62 3a4ffdf Author: jkesanie <[email protected]> Date: Mon Jun 4 09:56:29 2018 +0300 Merge branch 'CSCMETAX-280' of github.com:CSCfi/metax-api into CSCMETAX-280 commit 8106f62 Author: jkesanie <[email protected]> Date: Mon Jun 4 09:38:10 2018 +0300 CSCMETAX-280:[FIX] proper handling of language field data. commit 3a4ffdf Author: jkesanie <[email protected]> Date: Mon Jun 4 09:38:10 2018 +0300 CSCMETAX-280:[FIX] proper handling of language field data. commit 71b3fc0 Merge: 7544057 4a7c1e5 Author: hannu40k <[email protected]> Date: Fri Jun 1 16:41:04 2018 +0300 Merge pull request #270 from CSCfi/CSCMETAX-280 CSCMETAX-280:[ADD] datacatalog harvesting commit 4a7c1e5 Merge: 71cf1e2 7544057 Author: hannu40k <[email protected]> Date: Fri Jun 1 16:27:48 2018 +0300 Merge branch 'test' into CSCMETAX-280 commit 71cf1e2 Author: jkesanie <[email protected]> Date: Fri Jun 1 15:18:15 2018 +0300 CSCMETAX-280:[FIX] according to review comments. commit 73353bc Author: Joonas Kesäniemi <[email protected]> Date: Fri Jun 1 12:35:38 2018 +0300 Update minimal_api.py commit 6fceb94 Author: jkesanie <[email protected]> Date: Fri Jun 1 12:23:44 2018 +0300 CSCMETAX-280:[ADD] datacatalog harvesting commit 7544057 Merge: 619b05f a0a4177 Author: Juha-Matti Lehtinen <[email protected]> Date: Fri Jun 1 10:01:59 2018 +0300 Merge pull request #269 from CSCfi/CSCMETAX-61-schema Etsim migration related schema changes (part2) commit a0a4177 Author: jkesanie <[email protected]> Date: Fri Jun 1 08:37:59 2018 +0300 Etsim migration related schema changes (part2) commit 619b05f Merge: e630a33 5698ef8 Author: Juha-Matti Lehtinen <[email protected]> Date: Thu May 31 09:53:05 2018 +0300 Merge pull request #268 from CSCfi/CSCMETAX-61-schema Schema changes required by the Etsin migration. commit 5698ef8 Author: jkesanie <[email protected]> Date: Thu May 31 09:33:11 2018 +0300 Schema changes required by the Etsin migration. commit e630a33 Merge: 3cd96f4 68c329a Author: Juha-Matti Lehtinen <[email protected]> Date: Wed May 30 18:36:03 2018 +0300 Merge pull request #264 from CSCfi/CSCMETAX-394-remove-sensitive-data-in-oai-pmh CSCMETAX-394: [ADD] Remove sensitive fields (email, phone, telephone)… commit 68c329a Merge: f27ada1 3cd96f4 Author: Juha-Matti Lehtinen <[email protected]> Date: Wed May 30 17:30:21 2018 +0300 Merge branch 'test' into CSCMETAX-394-remove-sensitive-data-in-oai-pmh commit 3cd96f4 Merge: 2e75cb6 840679f Author: Juha-Matti Lehtinen <[email protected]> Date: Wed May 30 17:00:43 2018 +0300 Merge pull request #267 from CSCfi/CSCMETAX-408-begin-storing-django-db-migration-files CSCMETAX-408: [ADD] Remove exclusion of migrations/ directory from .g… commit 840679f Merge: ac84393 2e75cb6 Author: Juha-Matti Lehtinen <[email protected]> Date: Wed May 30 16:37:22 2018 +0300 Merge branch 'test' into CSCMETAX-408-begin-storing-django-db-migration-files commit ac84393 Author: Hannu Kamarainen <[email protected]> Date: Wed May 30 16:12:16 2018 +0300 CSCMETAX-408: [ADD] Remove exclusion of migrations/ directory from .gitignore. Generate first migration files. Fix .flake8 to ignore migrations/ directory since there are files that are autogenerated commit 2e75cb6 Merge: a7d893a 253f3f4 Author: hannu40k <[email protected]> Date: Wed May 30 14:38:18 2018 +0300 Merge pull request #265 from CSCfi/CSCMETAX-61-load-initial-data-fix CSCMETAX-61: [FIX] Try a fix to an error in initial data loading duri… commit 253f3f4 Author: Hannu Kamarainen <[email protected]> Date: Wed May 30 14:37:40 2018 +0300 CSCMETAX-61: [FIX] Try a fix to an error in initial data loading during travis deployment commit f27ada1 Author: Hannu Kamarainen <[email protected]> Date: Wed May 30 14:24:31 2018 +0300 CSCMETAX-394: [FIX] Uncomment the part that does the actual cleaning... In tests, search for known sensitive values instead of field names commit bb654a6 Author: Hannu Kamarainen <[email protected]> Date: Wed May 30 13:42:27 2018 +0300 CSCMETAX-394: [ADD] Remove sensitive fields (email, phone, telephone) from OAI-PMH api outputs commit a7d893a Merge: 92a4b96 d8fad2e Author: Juha-Matti Lehtinen <[email protected]> Date: Wed May 30 12:22:52 2018 +0300 Merge pull request #263 from CSCfi/CSCMETAX-395-add-fields-params-for-datasets-pid-files-api CSCMETAX-395: [FIX] Datasets api /datasets/pid/files now also support… commit d8fad2e Merge: 019542d 92a4b96 Author: Juha-Matti Lehtinen <[email protected]> Date: Wed May 30 12:05:30 2018 +0300 Merge branch 'test' into CSCMETAX-395-add-fields-params-for-datasets-pid-files-api commit 92a4b96 Merge: 8a816a2 f8cec67 Author: Juha-Matti Lehtinen <[email protected]> Date: Wed May 30 12:02:02 2018 +0300 Merge pull request #262 from CSCfi/CSCMETAX-400-allow-etsin-pid-override CSCMETAX-400: [ADD] Datasets api: Add query param ?migration_override… commit 019542d Author: Hannu Kamarainen <[email protected]> Date: Wed May 30 11:43:16 2018 +0300 CSCMETAX-395: [FIX] Datasets api /datasets/pid/files now also supports ?file_fields=x,y,z parameter commit f8cec67 Author: Hannu Kamarainen <[email protected]> Date: Tue May 29 16:08:17 2018 +0300 CSCMETAX-400: [ADD] Datasets api: Add query param ?migration_override=bool which enables passing a custom preferred_identifier when creating datasets commit 8a816a2 Merge: 224efb6 04348f4 Author: Juha-Matti Lehtinen <[email protected]> Date: Tue May 29 13:53:10 2018 +0300 Merge pull request #261 from CSCfi/CSCMETAX-61-change-travis-deploy-user CSCMETAX-61: [REF] Change Travis deploy user commit 04348f4 Author: Juha-Matti Lehtinen <[email protected]> Date: Tue May 29 13:43:43 2018 +0300 CSCMETAX-61: [REF] Change Travis deploy user commit 224efb6 Merge: a46bc27 daf89b4 Author: Juha-Matti Lehtinen <[email protected]> Date: Tue May 29 12:25:43 2018 +0300 Merge pull request #260 from CSCfi/CSCMETAX-398-allow-file-uptade-from-0-to-n CSCMETAX-398: [ADD] Datasets api: Allow the first addition of files t… commit daf89b4 Author: Hannu Kamarainen <[email protected]> Date: Tue May 29 12:13:35 2018 +0300 CSCMETAX-398: [ADD] Datasets api: Allow the first addition of files to an empty dataset to occur without creating a new dataset version (update from 0 files to N files) commit a46bc27 Merge: 2fbcd7c 934f759 Author: Juha-Matti Lehtinen <[email protected]> Date: Tue May 29 10:30:00 2018 +0300 Merge pull request #259 from CSCfi/CSCMETAX-407-metadata-owner-org-filter CSCMETAX-407: [ADD] Query parameter to filter datasets by field metad… commit 934f759 Author: Hannu Kamarainen <[email protected]> Date: Tue May 29 09:29:57 2018 +0300 CSCMETAX-407: [ADD] Query parameter to filter datasets by field metadata_owner_org commit 2fbcd7c Merge: be8d86c c535c71 Author: Juha-Matti Lehtinen <[email protected]> Date: Mon May 28 17:33:49 2018 +0300 Merge pull request #258 from CSCfi/CSCMETAX-394-remove-sensitive-fields CSCMETAX-394: Remove sensitive fields from datasets apis commit c535c71 Author: Hannu Kamarainen <[email protected]> Date: Mon May 28 16:48:54 2018 +0300 CSCMETAX-61: [FIX] Restrict /datasets/pid/files from public commit 64a626e Author: Hannu Kamarainen <[email protected]> Date: Mon May 28 16:48:15 2018 +0300 CSCMETAX-394: [ADD] Remove sensitive fields from datasets apis commit be8d86c Merge: 2c1a726 679fa92 Author: hannu40k <[email protected]> Date: Mon May 28 15:04:03 2018 +0300 Merge pull request #257 from CSCfi/CSCMETAX-359-relation-entity-type-refdata CSCMETAX-359: [ADD] research_dataset.relation.entity.type value is au… commit 679fa92 Author: Juha-Matti Lehtinen <[email protected]> Date: Mon May 28 14:54:04 2018 +0300 CSCMETAX-359: [ADD] research_dataset.relation.entity.type value is auto-populated from resource_type reference data, if given. Adjust test data and tests accordingly. commit 2c1a726 Merge: addf26c 0fc52cb Author: hannu40k <[email protected]> Date: Mon May 28 09:44:26 2018 +0300 Merge pull request #256 from CSCfi/CSCMETAX-401 CSCMETAX-401:[ADD] more data to the oai_dc metadataformat output commit 0fc52cb Merge: 34e23f5 addf26c Author: Juha-Matti Lehtinen <[email protected]> Date: Fri May 25 13:02:05 2018 +0300 Merge branch 'test' into CSCMETAX-401 commit addf26c Merge: 2968d16 863d154 Author: Juha-Matti Lehtinen <[email protected]> Date: Fri May 25 13:01:38 2018 +0300 Merge pull request #255 from CSCfi/CSCMETAX-393-restrict-api-access CSCMETAX-393: [ADD] Restrict service access per api by reading app_config commit 34e23f5 Author: Joonas Kesäniemi <[email protected]> Date: Fri May 25 12:53:36 2018 +0300 Update minimal_api.py commit 156f492 Author: jkesanie <[email protected]> Date: Fri May 25 12:45:28 2018 +0300 CSCMETAX-401:[ADD] more data to the oai_dc metadataformat output commit 863d154 Author: Hannu Kamarainen <[email protected]> Date: Fri May 25 12:23:35 2018 +0300 CSCMETAX-393: [ADD] Add permission class ServicePermissions, which reads app_config and controls general read and write access for services per each api commit 779543b Author: Hannu Kamarainen <[email protected]> Date: Fri May 25 11:21:49 2018 +0300 CSCMETAX-61: [REF] Refactor modifying the Request object in CommonViewSet to be more sensible commit 2968d16 Merge: 900992d b82a9cb Author: hannu40k <[email protected]> Date: Thu May 24 13:19:21 2018 +0300 Merge pull request #253 from CSCfi/CSCMETAX-379 CSCMETAX-379:[ADD] special handling of syke datasets for urnresolver commit b82a9cb Author: Joonas Kesäniemi <[email protected]> Date: Thu May 24 12:52:11 2018 +0300 Update syke.py commit 4c7b2cb Author: jkesanie <[email protected]> Date: Thu May 24 12:47:00 2018 +0300 CSCMETAX-379:[ADD] special handling of syke datasets for urnresolver commit 900992d Merge: f5b31b3 8c2ab79 Author: hannu40k <[email protected]> Date: Tue May 22 10:24:26 2018 +0300 Merge pull request #252 from CSCfi/CSCMETAX-61-update-requirements CSCMETAX-61: [REF] Update requirements.txt. Brings about a warning ab… commit 8c2ab79 Author: Juha-Matti Lehtinen <[email protected]> Date: Tue May 22 10:10:16 2018 +0300 CSCMETAX-61: [REF] Update requirements.txt. Brings about a warning about gevent monkey-patching, but according to gevent/gevent#1016 should not affect since we do not import gevent in the code. commit f5b31b3 Merge: 82836c0 43f9fd3 Author: hannu40k <[email protected]> Date: Mon May 21 16:47:18 2018 +0300 Merge pull request #251 from CSCfi/CSCMETAX-37 Changed datacite output according to comments commit 43f9fd3 Author: jkesanie <[email protected]> Date: Mon May 21 15:34:49 2018 +0300 Changed datacite output according to comments
CSCfi · Jun 4, 2018 · cbfd4af · cbfd4af
1 parent 3d99be2
commit cbfd4af
Show file tree

Hide file tree

Showing 50 changed files with 1,510 additions and 200 deletions.
diff --git a/.flake8 b/.flake8
@@ -27,5 +27,5 @@ ignore =
     # do not use bare except
     E722
 
-# often contains "unused" imports
-exclude = __init__.py, src/metax_api/migrations, src/static
+# often contains "unused" imports, too long lines (generated files), and such
+exclude = __init__.py,src/metax_api/migrations/*,src/static
diff --git a/.gitignore b/.gitignore
@@ -6,6 +6,4 @@
 /src/.coverage
 .idea/
 ubuntu-xenial-16.04-cloudimg-console.log
-/src/metax_api/migrations/*
-!/src/metax_api/migrations/*__keep*
 .ropeproject/
diff --git a/.travis-deploy.sh b/.travis-deploy.sh
@@ -13,11 +13,11 @@ cd metax-ops/ansible/
 if [[ "$TRAVIS_BRANCH" == "test" && "$TRAVIS_PULL_REQUEST" == "false" ]]; then
     echo "Deploying to test.."
     ansible-galaxy -r requirements.yml install --roles-path=roles
-    ansible-playbook -vv -i inventories/test/hosts site_deploy.yml --extra-vars "ssh_user=metax-user"
+    ansible-playbook -vv -i inventories/test/hosts site_deploy.yml --extra-vars "ssh_user=metax-deploy-user"
 elif [[ "$TRAVIS_BRANCH" == "stable" && "$TRAVIS_PULL_REQUEST" == "false" ]]; then
     echo "Deploying to stable.."
     ansible-galaxy -r requirements.yml install --roles-path=roles
-    ansible-playbook -vv -i inventories/stable/hosts site_deploy.yml --extra-vars "ssh_user=metax-user"
+    ansible-playbook -vv -i inventories/stable/hosts site_deploy.yml --extra-vars "ssh_user=metax-deploy-user"
 fi
 
 # Make sure the last command to run before this part is the ansible-playbook command

diff --git a/.travis.yml b/.travis.yml
@@ -17,7 +17,7 @@ services:
 - postgresql
 
 before_install:
-- openssl aes-256-cbc -K $encrypted_596a6d1c4f83_key -iv $encrypted_596a6d1c4f83_iv -in deploy-key.enc -out deploy-key -d
+- openssl aes-256-cbc -K $encrypted_62ed3fb8af4c_key -iv $encrypted_62ed3fb8af4c_iv -in deploy-key.enc -out deploy-key -d
 - rm deploy-key.enc
 - chmod 600 deploy-key
 - mv deploy-key ~/.ssh/id_rsa

diff --git a/deploy-key.enc b/deploy-key.enc
diff --git a/requirements.txt b/requirements.txt
@@ -1,24 +1,24 @@
 coveralls==1.3.0 # code coverage reportin in travis
 dicttoxml==1.7.4
-python-dateutil==2.7.1
+python-dateutil==2.7.3
 Django==2.0  # BSD-license
 elasticsearch<6.0.0
 hiredis==0.2.0 # Used by redis (redis-py) for parser
 djangorestframework==3.8.2  # BSD-license
 django-rainbowtests==0.6.0 # colored test output
 flake8==3.5.0 # MIT-license
-gevent==1.2.2 # gunicorn dep
-gunicorn==19.7.1 # MIT-license
+gevent==1.3.1 # gunicorn dep
+gunicorn==19.8.1 # MIT-license
 ipdb==0.11 # dev tool
 jsonschema==2.6.0
 lxml==4.2.1
 pika==0.11.2
 psycopg2-binary==2.7.4  # LGPL with exceptions or ZPL
 pyoai==2.5.0
 python-simplexquery==1.0.5.3
-pytz==2018.3
+pytz==2018.4
 pyyaml==3.12
 redis==2.10.6
 requests==2.18.4  # Apache 2.0-license
-simplejson==3.13.2  # MIT-license
+simplejson==3.15.0  # MIT-license
 urllib3==1.22
diff --git a/src/metax_api/api/oaipmh/base/metax_oai_server.py b/src/metax_api/api/oaipmh/base/metax_oai_server.py
@@ -7,14 +7,16 @@
 from oaipmh.error import IdDoesNotExistError
 from oaipmh.error import BadArgumentError
 
-from metax_api.models.catalog_record import CatalogRecord
+from metax_api.models.catalog_record import CatalogRecord, DataCatalog
 from metax_api.services import CatalogRecordService as CRS
 
+syke_url_prefix_template = 'http://metatieto.ymparisto.fi:8080/geoportal/catalog/search/resource/details.page?uuid=%s'
+
 
 class MetaxOAIServer(ResumptionOAIPMH):
 
     def _is_valid_set(self, set):
-        if not set or set == 'urnresolver' or set in settings.OAI['SET_MAPPINGS']:
+        if not set or set in ['urnresolver', 'datacatalogs'] or set in settings.OAI['SET_MAPPINGS']:
             return True
         return False
 
@@ -30,16 +32,20 @@ def _get_filtered_records(self, set, cursor, batch_size, from_=None, until=None)
         if not self._is_valid_set(set):
             raise BadArgumentError('invalid set value')
 
-        query_set = CatalogRecord.objects.all()
+        proxy = CatalogRecord
+        if set == 'datacatalogs':
+            proxy = DataCatalog
+
+        query_set = proxy.objects.all()
         if from_ and until:
-            query_set = CatalogRecord.objects.filter(date_modified__gte=from_, date_modified__lte=until)
+            query_set = proxy.objects.filter(date_modified__gte=from_, date_modified__lte=until)
         elif from_:
-            query_set = CatalogRecord.objects.filter(date_modified__gte=from_)
+            query_set = proxy.objects.filter(date_modified__gte=from_)
         elif until:
-            query_set = CatalogRecord.objects.filter(date_modified__lte=until)
+            query_set = proxy.objects.filter(date_modified__lte=until)
 
         if set:
-            if set == 'urnresolver':
+            if set in ['urnresolver', 'datacatalogs']:
                 pass
             else:
                 query_set = query_set.filter(
@@ -48,38 +54,154 @@ def _get_filtered_records(self, set, cursor, batch_size, from_=None, until=None)
             query_set = query_set.filter(data_catalog__catalog_json__identifier__in=self._get_default_set_filter())
         return query_set[cursor:batch_size]
 
+    def _handle_syke_urnresolver_metadata(self, record):
+        identifiers = []
+        preferred_identifier = record.research_dataset.get('preferred_identifier')
+        identifiers.append(preferred_identifier)
+        for id_obj in record.research_dataset.get('other_identifier', []):
+            if id_obj.get('notation', '').startswith('{'):
+                uuid = id_obj['notation']
+                identifiers.append(syke_url_prefix_template % uuid)
+        return identifiers
+
     def _get_oai_dc_urnresolver_metadata(self, record):
         """
         Preferred identifier is added only for ida and att catalog records
         other identifiers are added for all.
+
+        Special handling for SYKE catalog.
         """
+
         identifiers = []
-        identifiers.append(settings.OAI['ETSIN_URL_TEMPLATE'] % record.identifier)
 
-        # assuming ida and att catalogs are not harvested
-        if not record.catalog_is_harvested():
-            preferred_identifier = record.research_dataset.get('preferred_identifier')
-            identifiers.append(preferred_identifier)
-        for id_obj in record.research_dataset.get('other_identifier', []):
-            if id_obj.get('notation', '').startswith('urn:nbn:fi:csc-kata'):
-                other_urn = id_obj['notation']
-                identifiers.append(other_urn)
+        data_catalog = record.data_catalog.catalog_json.get('identifier')
+        if data_catalog == 'urn:nbn:fi:att:data-catalog-harvest-syke':
+            identifiers = self._handle_syke_urnresolver_metadata(record)
+
+        else:
+            identifiers.append(settings.OAI['ETSIN_URL_TEMPLATE'] % record.identifier)
+
+            # assuming ida and att catalogs are not harvested
+            if not record.catalog_is_harvested():
+                preferred_identifier = record.research_dataset.get('preferred_identifier')
+                identifiers.append(preferred_identifier)
+            for id_obj in record.research_dataset.get('other_identifier', []):
+                if id_obj.get('notation', '').startswith('urn:nbn:fi:csc-kata'):
+                    other_urn = id_obj['notation']
+                    identifiers.append(other_urn)
 
         meta = {
             'identifier':  identifiers
         }
         return meta
 
-    def _get_oai_dc_metadata(self, record):
-        identifier = record.research_dataset.get('preferred_identifier')
+    def _get_oaic_dc_value(self, value, lang=None):
+        valueDict = {}
+        valueDict['value'] = value
+        if lang:
+            valueDict['lang'] = lang
+        return valueDict
+
+    def _get_oai_dc_metadata(self, record, json, type):
+        identifier = []
+        if 'preferred_identifier' in json:
+            identifier.append(self._get_oaic_dc_value(json.get('preferred_identifier')))
+        if 'identifier' in json:
+            identifier.append(self._get_oaic_dc_value(json.get('identifier')))
+
+        title = []
+        title_data = json.get('title', {})
+        for key, value in title_data.items():
+            title.append(self._get_oaic_dc_value(value, key))
+
+        creator = []
+        creator_data = json.get('creator', [])
+        for value in creator_data:
+            if 'name' in value:
+                creator.append(self._get_oaic_dc_value(value.get('name')))
+
+        subject = []
+        subject_data = json.get('keyword', [])
+        for value in subject_data:
+            subject.append(self._get_oaic_dc_value(value))
+        subject_data = json.get('field_of_science', [])
+        for value in subject_data:
+            for key, value2 in value.get('pref_label', {}).items():
+                subject.append(self._get_oaic_dc_value(value2, key))
+        subject_data = json.get('theme', [])
+        for value in subject_data:
+            for key, value2 in value.get('pref_label', {}).items():
+                subject.append(self._get_oaic_dc_value(value2, key))
+
+        desc = []
+        desc_data = json.get('description', [])
+        for value in desc_data:
+            for key, value2 in value.items():
+                desc.append(self._get_oaic_dc_value(value2, key))
+
+        publisher = []
+        publisher_data = json.get('publisher', {})
+        for key, value in publisher_data.get('name', {}).items():
+            publisher.append(self._get_oaic_dc_value(value, key))
+
+        contributor = []
+        contributor_data = json.get('contributor', [])
+        for value in contributor_data:
+            if 'name' in value:
+                contributor.append(self._get_oaic_dc_value(value.get('name')))
+
+        date = self._get_oaic_dc_value(str(record.date_created))
+
+        language = []
+        language_data = json.get('language', [])
+        for value in language_data:
+            if 'identifier' in value:
+                language.append(self._get_oaic_dc_value(value['identifier']))
+
+        relation = []
+        relation_data = json.get('relation', [])
+        for value in relation_data:
+            if 'identifier'in value.get('entity', {}):
+                relation.append(self._get_oaic_dc_value(value['entity']['identifier']))
+
+        coverage = []
+        coverage_data = json.get('spatial', [])
+        for value in coverage_data:
+            if 'geographic_name' in value:
+                coverage.append(self._get_oaic_dc_value(value['geographic_name']))
+
+        rights = []
+        rights_data = json.get('access_rights', {})
+        for value in rights_data.get('description', []):
+            for key, value2 in value.items():
+                rights.append(self._get_oaic_dc_value(value2, key))
+        for value in rights_data.get('license', []):
+            if 'identifier' in value:
+                rights.append(self._get_oaic_dc_value(value['identifier']))
+
+        types = []
+        types.append(self._get_oaic_dc_value(type))
+
         meta = {
-            'identifier':  [identifier]
+            'identifier':  identifier,
+            'title': title,
+            'creator': creator,
+            'subject': subject,
+            'description': desc,
+            'publisher': publisher,
+            'contributor': contributor,
+            'date': [date],
+            'type': types,
+            'language': language,
+            'relation': relation,
+            'coverage': coverage,
+            'rights': rights
         }
         return meta
 
-    def _get_oai_datacite_metadata(self, record):
+    def _get_oai_datacite_metadata(self, json):
         datacite_xml = CRS.transform_datasets_to_format(
-            {'research_dataset': record.research_dataset}, 'datacite', False
+            {'research_dataset': json}, 'datacite', False
         )
         meta = {
             'datacentreSymbol': 'Metax',
@@ -88,13 +210,17 @@ def _get_oai_datacite_metadata(self, record):
         }
         return meta
 
-    def _get_metadata_for_record(self, record, metadata_prefix):
+    def _get_metadata_for_record(self, record, json, type, metadata_prefix):
         meta = {}
+        json = CRS.strip_catalog_record(json)
+
         if metadata_prefix == 'oai_dc':
-            meta = self._get_oai_dc_metadata(record)
+            meta = self._get_oai_dc_metadata(record, json, type)
         elif metadata_prefix == 'oai_datacite':
-            meta = self._get_oai_datacite_metadata(record)
+            meta = self._get_oai_datacite_metadata(json)
         elif metadata_prefix == 'oai_dc_urnresolver':
+            # This is a special case. Only identifier values are retrieved from the record,
+            # so strip_catalog_record is not applicable here.
             meta = self._get_oai_dc_urnresolver_metadata(record)
         return self._fix_metadata(meta)
 
@@ -106,9 +232,14 @@ def _get_header_timestamp(self, record):
             timestamp = record.date_created
         return timezone.make_naive(timestamp)
 
-    def _get_oai_item(self, record, metadata_prefix):
-        identifier = record.identifier
-        metadata = self._get_metadata_for_record(record, metadata_prefix)
+    def _get_oai_item(self, identifier, record, metadata_prefix):
+        metadata = self._get_metadata_for_record(record, record.research_dataset, 'Dataset', metadata_prefix)
+        item = (common.Header('', identifier, self._get_header_timestamp(record), ['metax'], False),
+                common.Metadata('', metadata), None)
+        return item
+
+    def _get_oai_catalog_item(self, identifier, record, metadata_prefix):
+        metadata = self._get_metadata_for_record(record, record.catalog_json, 'Datacatalog', metadata_prefix)
         item = (common.Header('', identifier, self._get_header_timestamp(record), ['metax'], False),
                 common.Metadata('', metadata), None)
         return item
@@ -161,18 +292,24 @@ def listMetadataFormats(self, identifier=None):
 
     def listSets(self, cursor=None, batch_size=None):
         """Implement OAI-PMH verb ListSets."""
-        data = []
+        data = [('datacatalogs', 'datacatalog', '')]
         for set_key in settings.OAI['SET_MAPPINGS'].keys():
             data.append((set_key, set_key, ''))
         return data
 
+    def _get_record_identifier(self, record, set):
+        if set == 'datacatalogs':
+            return record.catalog_json['identifier']
+        else:
+            return record.identifier
+
     def listIdentifiers(self, metadataPrefix=None, set=None, cursor=None,
                         from_=None, until=None, batch_size=None):
         """Implement OAI-PMH verb listIdentifiers."""
         records = self._get_filtered_records(set, cursor, batch_size, from_, until)
         data = []
         for record in records:
-            identifier = record.research_dataset.get('preferred_identifier')
+            identifier = self._get_record_identifier(record, set)
             data.append(common.Header('', identifier, self._get_header_timestamp(record), ['metax'], False))
         return data
 
@@ -182,18 +319,27 @@ def listRecords(self, metadataPrefix=None, set=None, cursor=None, from_=None,
         data = []
         records = self._get_filtered_records(set, cursor, batch_size, from_, until)
         for record in records:
-            data.append(self._get_oai_item(record, metadataPrefix))
+            identifier = self._get_record_identifier(record, set)
+            if set == 'datacatalogs':
+                data.append(self._get_oai_catalog_item(identifier, record, metadataPrefix))
+            else:
+                data.append(self._get_oai_item(identifier, record, metadataPrefix))
         return data
 
     def getRecord(self, metadataPrefix, identifier):
         """Implement OAI-PMH verb GetRecord."""
         try:
-            record = CatalogRecord.objects.get(
-                data_catalog__catalog_json__identifier__in=self._get_default_set_filter(),
-                identifier__exact=identifier
-            )
+            record = CatalogRecord.objects.get(identifier__exact=identifier)
+            json = record.research_dataset
+            type = 'Dataset'
         except CatalogRecord.DoesNotExist:
-            raise IdDoesNotExistError("No dataset with id %s available through the OAI-PMH interface." % identifier)
-        metadata = self._get_metadata_for_record(record, metadataPrefix)
+            try:
+                record = DataCatalog.objects.get(catalog_json__identifier__exact=identifier)
+                json = record.catalog_json
+                type = 'Datacatalog'
+            except DataCatalog.DoesNotExist:
+                raise IdDoesNotExistError("No record with id %s available." % identifier)
+
+        metadata = self._get_metadata_for_record(record,  json, type, metadataPrefix)
         return (common.Header('', identifier, self._get_header_timestamp(record), ['metax'], False),
                 common.Metadata('', metadata), None)