[MongoDB Atlas] Process data stream (#9552)

* add integration package

* update data stream

* add code owner

* update permissions

* update system test

* address review comments

* adress review comments

* address review comment

.github/CODEOWNERS

@@ -237,6 +237,7 @@
 /packages/mimecast @elastic/security-service-integrations
 /packages/modsecurity @elastic/sec-deployment-and-devices
 /packages/mongodb @elastic/obs-infraobs-integrations
+/packages/mongodb_atlas @elastic/obs-infraobs-integrations
 /packages/mysql @elastic/obs-infraobs-integrations
 /packages/mysql_enterprise @elastic/sec-windows-platform
 /packages/nagios_xi @elastic/obs-infraobs-integrations

packages/mongodb_atlas/_dev/build/build.yml

@@ -0,0 +1,3 @@
+dependencies:
+  ecs:
+    reference: [email protected]

packages/mongodb_atlas/_dev/build/docs/README.md

@@ -0,0 +1,66 @@
+# MongoDB Atlas Integration
+
+## Overview
+
+[MongoDB Atlas](https://www.mongodb.com/atlas), the leading multi-cloud developer data platform, offers the easiest way to run MongoDB, enabling you to work with your code's objects directly through its document-based data model, which allows for flexible schema and easy scalability.
+
+Use the MongoDB Atlas integration to:
+
+- Collect metrics related to process.
+- Create informative visualizations to track usage trends, measure key metrics, and derive actionable business insights.
+- Set up alerts to minimize Mean Time to Detect (MTTD) and Mean Time to Resolve (MTTR) by quickly referencing relevant logs during troubleshooting.
+
+## Data streams
+
+The MongoDB Atlas integration collects metrics.
+
+Metrics give you insight into the statistics of the MongoDB Atlas. The `Metric` data stream collected by the MongoDB Atlas integration is `process` so that the user can monitor and troubleshoot the performance of the MongoDB Atlas instance.
+
+Data streams:
+
+- `process` : This data stream collects host metrics per process for all the hosts of the specified group. Metrics like measurements for the host, such as CPU usage, number of I/O operations and memory are available on this data stream.
+
+Note:
+- Users can monitor and see the metrics inside the ingested documents for MongoDB Atlas in the `logs-*` index pattern from `Discover`.
+
+## Prerequisites
+
+You can store and search your data using Elasticsearch and visualize and manage it with Kibana. We recommend using our hosted Elasticsearch Service on Elastic Cloud or self-managing the Elastic Stack on your own hardware.
+
+## Setup
+
+### To collect data from MongoDB Atlas, the following parameters from your MongoDB Atlas instance are required:
+
+1. Public Key
+2. Private Key
+3. GroupId
+
+### Steps to obtain Public Key, Private Key and GroupId:
+
+1. Generate programmatic API Keys with project owner permissions using the instructions in the Atlas [documentation](https://www.mongodb.com/docs/atlas/configure-api-access/#create-an-api-key-for-a-project). Then, copy the public key and private key. These serve the same function as a username and API Key respectively.
+2. You can find your Project ID (Group ID) in the Atlas UI. To do this, navigate to your project, click on Settings, and copy the Project ID (Group ID). You can also programmatically find it using the Atlas Admin API or Atlas CLI as described in this Atlas [document](https://www.mongodb.com/docs/atlas/app-services/apps/metadata/#find-a-project-id).
+
+### Important terms of MongoDB Atlas API:
+
+1. Granularity: Duration that specifies the interval at which Atlas reports the metrics.
+2. Period: Duration over which Atlas reports the metrics.
+
+Note: Both of above attributes can be set by using `period` in configuration parameters.
+
+### Enabling the integration in Elastic:
+
+1. In Kibana go to Management > Integrations
+2. In "Search for integrations" search bar, type MongoDB Atlas
+3. Click on the "MongoDB Atlas" integration from the search results.
+4. To add the integration, click on the "Add MongoDB Atlas" button.
+5. Enter all the necessary configuration parameters, including Public Key, Private Key, and GroupId.
+6. Finally, save the integration.
+
+## Metrics reference
+
+### Process
+This data stream collects host metrics per process for all the hosts of the specified group. Metrics like measurements for the host, such as CPU usage, number of I/O operations and memory are available on this data stream.
+
+{{event "process"}}
+
+{{fields "process"}}

packages/mongodb_atlas/_dev/deploy/docker/Dockerfile

@@ -0,0 +1,6 @@
+FROM golang:1.19
+
+COPY ./test /go
+CMD ["./test"]
+
+HEALTHCHECK --interval=1s --retries=90 CMD curl localhost:7780/api/atlas/v2/groups/mongodb-group1/processes --digest -u admin:MongoDB@123

packages/mongodb_atlas/_dev/deploy/docker/docker-compose.yml

@@ -0,0 +1,8 @@
+version: '3.0'
+services:
+  mongodbatlas:
+    build:
+      context: .
+      dockerfile: Dockerfile
+    ports:
+      - 7780

packages/mongodb_atlas/changelog.yml

@@ -0,0 +1,6 @@
+# newer versions go on top
+- version: "0.0.1"
+  changes:
+    - description: MongoDB Atlas integration package with "process" data stream.
+      type: enhancement
+      link: https://github.com/elastic/integrations/pull/9552

packages/mongodb_atlas/data_stream/process/_dev/test/pipeline/test-common-config.yml

@@ -0,0 +1,2 @@
+dynamic_fields:
+  "event.ingested": ".*"

packages/mongodb_atlas/data_stream/process/_dev/test/pipeline/test-process-metrics.json

@@ -0,0 +1,144 @@
+{
+    "events": [
+        {
+            "groupId": "646f4379c47da356740d14ad",
+            "hostId": "atlas-ccx4uc-shard-00-00.q5ljb.mongodb.net:27017",
+            "response": {
+                "ASSERT_MSG": 0.213805548,
+                "ASSERT_REGULAR": 0.33244343545,
+                "ASSERT_USER": 0.34219922212309,
+                "ASSERT_WARNING": 0.33245678912,
+                "BACKGROUND_FLUSH_AVG": 0.224438123,
+                "CACHE_DIRTY_BYTES": 12567.12,
+                "CACHE_BYTES_READ_INTO": 132153.9,
+                "CACHE_USED_BYTES": 821456778.500023,
+                "CACHE_BYTES_WRITTEN_FROM": 1328.7865221233,
+                "CONNECTIONS": 38,
+                "MAX_PROCESS_NORMALIZED_CPU_CHILDREN_KERNEL": 0.33455633344,
+                "MAX_PROCESS_CPU_CHILDREN_KERNEL": 0.182233244,
+                "PROCESS_CPU_CHILDREN_KERNEL": 0.223444567789,
+                "MAX_PROCESS_CPU_CHILDREN_USER": 0.2789877666,
+                "PROCESS_CPU_CHILDREN_USER": 0.127876554,
+                "MAX_PROCESS_CPU_KERNEL": 0.198877766,
+                "PROCESS_CPU_KERNEL": 0.23777324612388449,
+                "PROCESS_NORMALIZED_CPU_CHILDREN_KERNEL": 0.21343458834492,
+                "MAX_PROCESS_NORMALIZED_CPU_CHILDREN_USER": 0.2426637278,
+                "PROCESS_NORMALIZED_CPU_CHILDREN_USER": 0.434455522443,
+                "MAX_PROCESS_NORMALIZED_CPU_KERNEL": 0.98763331334444322,
+                "PROCESS_NORMALIZED_CPU_KERNEL": 0.07326222334291,
+                "MAX_PROCESS_NORMALIZED_CPU_USER": 3.9237183371832,
+                "PROCESS_NORMALIZED_CPU_USER": 0.6547433822844,
+                "MAX_PROCESS_CPU_USER": 8.934677738289,
+                "PROCESS_CPU_USER": 1.0725321162733222133,
+                "CURSORS_TOTAL_OPEN": 3,
+                "CURSORS_TOTAL_TIMED_OUT": 2.987333711237,
+                "DB_DATA_SIZE_TOTAL": 12352234432,
+                "DB_STORAGE_TOTAL": 41235435,
+                "DOCUMENT_METRICS_DELETED": 0.35222371123144,
+                "DOCUMENT_METRICS_INSERTED": 0.298877313444,
+                "DOCUMENT_METRICS_RETURNED": 0.2333067316131,
+                "DOCUMENT_METRICS_UPDATED": 0.039932412345432,
+                "FTS_PROCESS_CPU_KERNEL": 0.22325346765433,
+                "FTS_PROCESS_NORMALIZED_CPU_KERNEL": 0.2137811999382,
+                "FTS_PROCESS_NORMALIZED_CPU_USER": 0.321445664321,
+                "FTS_PROCESS_CPU_USER": 0.24566543211,
+                "FTS_DISK_UTILIZATION": 0.1228391993818,
+                "FTS_MEMORY_MAPPED": 0.328123372383,
+                "FTS_MEMORY_RESIDENT": 0.12444445223,
+                "FTS_MEMORY_VIRTUAL": 0.1433342324553,
+                "GLOBAL_ACCESSES_NOT_IN_MEMORY": 3227372,
+                "GLOBAL_LOCK_CURRENT_QUEUE_READERS": 263737882,
+                "GLOBAL_LOCK_CURRENT_QUEUE_TOTAL": 4127712,
+                "GLOBAL_LOCK_CURRENT_QUEUE_WRITERS": 2153363,
+                "GLOBAL_PAGE_FAULT_EXCEPTIONS_THROWN": 3,
+                "EXTRA_INFO_PAGE_FAULTS": 1241,
+                "INDEX_COUNTERS_BTREE_ACCESSES": 14221,
+                "INDEX_COUNTERS_BTREE_HITS": 267712392,
+                "INDEX_COUNTERS_BTREE_MISS_RATIO": 1.623372,
+                "INDEX_COUNTERS_BTREE_MISSES": 132543,
+                "JOURNALING_COMMITS_IN_WRITE_LOCK": 11322,
+                "JOURNALING_MB": 1.2836627,
+                "JOURNALING_WRITE_DATA_FILES_MB": 23.23442,
+                "MAX_SYSTEM_NORMALIZED_CPU_USER": 23.87636632771,
+                "COMPUTED_MEMORY": 34.238,
+                "MEMORY_MAPPED": 21.32477234,
+                "MEMORY_RESIDENT": 190.4,
+                "MEMORY_VIRTUAL": 3242.1122,
+                "NETWORK_BYTES_IN": 313.872898381,
+                "NETWORK_BYTES_OUT": 20923.523387712,
+                "NETWORK_NUM_REQUESTS": 13.287773311191,
+                "OPCOUNTER_CMD": 13.472788221236,
+                "OPCOUNTER_DELETE": 0.8374367234442,
+                "OPCOUNTER_GETMORE": 1.34425467654,
+                "OPCOUNTER_INSERT": 0.24442323654,
+                "OPCOUNTER_QUERY": 0.1456555474455228,
+                "OPCOUNTER_REPL_CMD": 0.244556655421,
+                "OPCOUNTER_REPL_DELETE": 0.00456765432333333,
+                "OPCOUNTER_REPL_INSERT": 0.003333222133,
+                "OPCOUNTER_REPL_UPDATE": 0.05837348438,
+                "OPCOUNTER_UPDATE": 0.047555753224433,
+                "OP_EXECUTION_TIME_COMMANDS": 0.07347823122113,
+                "OP_EXECUTION_TIME_READS": 0.16234554981,
+                "OP_EXECUTION_TIME_WRITES": 0.24552345,
+                "OPERATIONS_SCAN_AND_ORDER": 0.36373718112,
+                "OPLOG_MASTER_LAG_TIME_DIFF": 2786665.435567,
+                "OPLOG_MASTER_TIME": 2771288.8755,
+                "OPLOG_RATE_GB_PER_HOUR": 0.0001341,
+                "OPLOG_REPLICATION_LAG": 0.00134322,
+                "OPLOG_SLAVE_LAG_MASTER_TIME": 21.23442,
+                "QUERY_EXECUTOR_SCANNED": 0.0033322145545511,
+                "QUERY_EXECUTOR_SCANNED_OBJECTS": 0.2663211245776,
+                "QUERY_TARGETING_SCANNED_OBJECTS_PER_RETURNED": 1.16235487878865,
+                "QUERY_TARGETING_SCANNED_PER_RETURNED": 0.007346657774388,
+                "RESTARTS_IN_LAST_HOUR": 6,
+                "MAX_SWAP_USAGE_FREE": 4193787.0000000004,
+                "SWAP_USAGE_FREE": 4193787.89324442122,
+                "SWAP_USAGE_USED": 511.8778888886,
+                "MAX_SWAP_USAGE_USED": 511.9999888888,
+                "MAX_SYSTEM_CPU_GUEST": 231.8657881111,
+                "SYSTEM_CPU_GUEST": 265.7363623321122,
+                "MAX_SYSTEM_CPU_IOWAIT": 111.335453221,
+                "SYSTEM_CPU_IOWAIT": 1.33334554555446,
+                "MAX_SYSTEM_CPU_IRQ": 1.243556311245,
+                "SYSTEM_CPU_IRQ": 2.233235555,
+                "MAX_SYSTEM_CPU_KERNEL": 13.3839493839467,
+                "SYSTEM_CPU_KERNEL": 1.413822809117,
+                "SYSTEM_CPU_NICE": 3.12455667777,
+                "MAX_SYSTEM_CPU_SOFTIRQ": 1.3880999000999,
+                "SYSTEM_CPU_SOFTIRQ": 0.09578823373226,
+                "MAX_SYSTEM_CPU_STEAL": 13.9995777339999,
+                "SYSTEM_CPU_STEAL": 0.7125933648776,
+                "MAX_SYSTEM_CPU_USER": 0.233311345555,
+                "SYSTEM_CPU_USER": 19.94005994777666,
+                "SYSTEM_MEMORY_AVAILABLE": 1297178.4,
+                "MAX_SYSTEM_MEMORY_AVAILABLE": 1287180,
+                "SYSTEM_MEMORY_FREE": 140886,
+                "MAX_SYSTEM_MEMORY_FREE": 149138.8,
+                "SYSTEM_MEMORY_USED": 567885.3,
+                "MAX_SYSTEM_MEMORY_USED": 1852490,
+                "SYSTEM_NETWORK_IN": 1345.89298711,
+                "MAX_SYSTEM_NETWORK_IN": 176129.9111111111002,
+                "MAX_SYSTEM_NETWORK_OUT": 11789.7,
+                "SYSTEM_NETWORK_OUT": 737.89400083167,
+                "MAX_SYSTEM_NORMALIZED_CPU_GUEST": 123.4435467577443,
+                "SYSTEM_NORMALIZED_CPU_GUEST": 122.332344335578,
+                "MAX_SYSTEM_NORMALIZED_CPU_IOWAIT": 0.7,
+                "SYSTEM_NORMALIZED_CPU_IOWAIT": 0.076625581546717,
+                "MAX_SYSTEM_NORMALIZED_CPU_IRQ": 0.086543389996543,
+                "SYSTEM_NORMALIZED_CPU_IRQ": 0.187666323010,
+                "MAX_SYSTEM_NORMALIZED_CPU_KERNEL": 8.78943256778111,
+                "SYSTEM_NORMALIZED_CPU_KERNEL": 0.9222388999125745,
+                "MAX_SYSTEM_NORMALIZED_CPU_NICE": 0.89631899,
+                "SYSTEM_NORMALIZED_CPU_NICE": 0.23488931133323,
+                "MAX_SYSTEM_NORMALIZED_CPU_SOFTIRQ": 0.59970029934565434,
+                "SYSTEM_NORMALIZED_CPU_SOFTIRQ": 0.06667564629406,
+                "MAX_SYSTEM_NORMALIZED_CPU_STEAL": 3.0995004995004996,
+                "SYSTEM_NORMALIZED_CPU_STEAL": 0.4948211119340082,
+                "SYSTEM_NORMALIZED_CPU_USER": 2.09939880605,
+                "TICKETS_AVAILABLE_READS": 127,
+                "TICKETS_AVAILABLE_WRITE": 126
+            },
+            "processId": "atlas-ccx4uc-shard-00-00.q5ljb.mongodb.net:27017"
+        }
+    ]
+}