[api/frontend] Implement data curation and entities merging (#246, #480)

opencti-documentation/docs/installation/auto.md

@@ -4,6 +4,7 @@ title: Automatic installation
 sidebar_label: Automatic installation
 ---
 
+> For production deployment, we advise you to deploy `Grakn` and `ElasticSearch` manually in a dedicated environment and then to start the other components using `Docker`.
 
 # Virtual machine template
 
@@ -33,11 +34,9 @@ $ cd docker
 
 ### Configure the environment
 
-Before running the docker-compose command, please change the admin token (this token must be a [valid UUID](https://www.uuidgenerator.net/)) and password of the application in the file `.env`. 
+Before running the `docker-compose` command, don't forget to change the admin token (this token must be a [valid UUID](https://www.uuidgenerator.net/)) and the password in the file `.env`.  There is a file `.env.example` with a preset of variables for a demonstration purpose only.
 
-They are a `.env.example` with indications of differents variables (if it's UUID or just text).
-
-If you cannot or don't want to use the `.env`, please, edit the file `docker-compose.yml`:
+If you cannot or don't want to use the `.env`, please edit the file `docker-compose.yml` with:
 
 ```yaml
 - APP__ADMIN__PASSWORD=ChangeMe

opencti-documentation/website/versioned_docs/version-3.0.3/installation/auto.md

@@ -0,0 +1,167 @@
+---
+id: version-3.0.3-auto
+title: Automatic installation
+sidebar_label: Automatic installation
+original_id: auto
+---
+
+> For production deployment, we advise you to deploy `Grakn` and `ElasticSearch` manually in a dedicated environment and then to start the other components using `Docker`.
+
+# Virtual machine template
+
+OpenCTI could be deployed for **testing purposes** with a provided OVA file.
+
+## Download the OVA file
+
+The OVA file is available on the [Luatix Google Drive folder](https://drive.google.com/open?id=1bvB6RmdQNHMW_3h-88KbAit9GRZlL5Bj). It has been pre-configured with the minimal requirements of memory and CPU.
+
+## Launch the virtual machine
+
+Then open the OVA file with VirtualBox or convert the OVA to another type of virtual machine image and launch it. You can login within the VM or connect in SSH with the default login `opencti` and the default password `opencti`.
+
+> Once the virtual machine is launched, the **OpenCTI platform can take 3 to 5 minutes to start the first time**. Then you have access to the plaform using the URL **http://{IP_ADDRESS}:8080** and credentials `[email protected]` / `admin`.
+
+# Using Docker
+
+OpenCTI could be deployed using the *docker-compose* command.
+
+## Clone the repository
+
+```bash
+$ mkdir /path/to/your/app && cd /path/to/your/app
+$ git clone https://github.com/OpenCTI-Platform/docker.git
+$ cd docker
+```
+
+### Configure the environment
+
+Before running the `docker-compose` command, don't forget to change the admin token (this token must be a [valid UUID](https://www.uuidgenerator.net/)) and the password in the file `.env`.  There is a file `.env.example` with a preset of variables for a demonstration purpose only.
+
+If you cannot or don't want to use the `.env`, please edit the file `docker-compose.yml` with:
+
+```yaml
+- APP__ADMIN__PASSWORD=ChangeMe
+- APP__ADMIN__TOKEN=ChangeMe
+```
+
+And change the variable `OPENCTI_TOKEN` (for the `worker` and all connectors) according to the value of `APP__ADMIN__TOKEN`
+
+```yaml
+- OPENCTI_TOKEN=ChangeMe
+```
+
+As OpenCTI has a dependency to ElasticSearch and Grakn, you have to set the `vm.max_map_count` before running the containers, as mentioned in the [ElasticSearch documentation](https://www.elastic.co/guide/en/elasticsearch/reference/current/docker.html#docker-cli-run-prod-mode).
+
+```bash
+$ sysctl -w vm.max_map_count=1048575
+```
+
+To make this parameter persistent, please update your file `/etc/sysctl.conf` and add the line:
+```bash
+$ vm.max_map_count=1048575
+```
+
+## Run
+
+In order to have the best experience with Docker, we recommend to use the Docker stack feature. In this mode we will have the capacity to easily scale your deployment.
+
+```bash
+$ env $(cat .env | grep ^[A-Z] | xargs) docker stack deploy --compose-file docker-compose.yml opencti
+```
+
+> In some configuration, Grakn could fail to start with the following error: `Starting Storage.....FAILED!`
+> You can restart it by using the command `$ docker service update --force opencti_grakn`.
+
+You can also deploy with the standard Docker command:
+
+```bash
+$ docker-compose --compatibility up
+```
+
+You can now go to http://localhost:8080 and log in with the credentials configured in your environment variables.
+
+### Update the stack or delete the stack
+
+```bash
+$ docker service update --force service_name
+$ docker stack rm opencti
+```
+
+### Behind a reverse proxy
+
+If you want to use OpenCTI behind a reverse proxy with a context path, like `https://myproxy.com/opencti`, please change the base_path configuration.
+
+```yaml
+- APP__BASE_PATH=/opencti
+```
+By default OpenCTI use Websockets so dont forget to configure your proxy for this usage.
+
+
+## Data persistence
+
+If you wish your OpenCTI data to be persistent in production, you should be aware of the  `volumes` section for `Grakn`, `ElasticSearch` and `MinIO` services in the `docker-compose.yml`.
+
+Here is an example of volumes configuration:
+
+```yaml
+volumes:
+  grakndata:
+    driver: local
+    driver_opts:
+      o: bind
+      type: none
+  esdata:
+    driver: local
+    driver_opts:
+      o: bind
+      type: none
+  s3data:
+    driver: local
+    driver_opts:
+      o: bind
+      type: none      
+```
+
+## Memory configuration
+
+OpenCTI default `docker-compose.yml` file does not provide any specific memory configuration. But if you want to adapt some dependencies configuration, you can find some links below.
+
+### OpenCTI - Platform
+
+OpenCTI platform is based on a NodeJS runtime, with a memory limit of **512MB by default**. We do not provide any option to change this limit today. If you encounter any `OutOfMemory` exception, please open a [Github issue](https://github.com/OpenCTI-Platform/opencti/issues/new?assignees=&labels=&template=bug_report.md&title=).
+
+### OpenCTI - Workers and connectors
+
+OpenCTI workers and connectors are Python processes. If you want to limit the memory of the process we recommend to directly use Docker to do that. You can find more information in the [official Docker documentation](https://docs.docker.com/compose/compose-file/). 
+
+> If you do not use Docker stack, think about `--compatibility` option.
+
+### Grakn 
+
+Grakn is a JAVA process that rely on Cassandra (also a JAVA process). In order to setup the JAVA memory allocation, you can use the environment variable `SERVER_JAVAOPTS` and `STORAGE_JAVAOPTS`. 
+
+> The current recommendation is `-Xms4G` for both options.
+
+You can find more information in the [official Grakn documentation](https://dev.grakn.ai/docs).
+
+### ElasticSearch
+
+ElasticSearch is also a JAVA process. In order to setup the JAVA memory allocation, you can use the environment variable `ES_JAVA_OPTS`. 
+
+> The minimal recommended option today is `-Xms512M -Xmx512M`.
+
+You can find more information in the [official ElasticSearch documentation](https://www.elastic.co/guide/en/elasticsearch/reference/current/docker.html).
+
+### Redis
+
+Redis has a very small footprint and only provides an option to limit the maximum amount of memory that can be used by the process. You can use the option `--maxmemory` to limit the usage. 
+
+You can find more information in the [Redis docker hub](https://hub.docker.com/r/bitnami/redis/).
+
+### MinIO
+
+MinIO is a small process and does not require a high amount of memory. More information are available for Linux here on the [Kernel tuning guide](https://github.com/minio/minio/tree/master/docs/deployment/kernel-tuning).
+
+### RabbitMQ
+
+The RabbitMQ memory configuration can be find in the [RabbitMQ official documentation](https://www.rabbitmq.com/memory.html). Basically RabbitMQ will consumed memory until a specific threshold. So it should be configure along with the Docker memory limitation.

opencti-documentation/website/versions.json

@@ -1,4 +1,5 @@
 [
+  "3.0.3",
   "3.0.2",
   "3.0.1",
   "3.0.0",

opencti-platform/opencti-front/package.json

@@ -1,6 +1,6 @@
 {
   "name": "opencti-front",
-  "version": "3.0.2",
+  "version": "3.0.3",
   "author": "Luatix",
   "license": "Apache-2.0",
   "dependencies": {

opencti-platform/opencti-front/src/components/list_lines/ListLines.js

@@ -120,6 +120,8 @@ class ListLines extends Component {
       classes,
       handleSearch,
       handleChangeView,
+      disableCards,
+      enableDuplicates,
       handleRemoveFilter,
       handleToggleExports,
       openExports,
@@ -184,7 +186,7 @@ class ListLines extends Component {
             ) : (
               ''
             )}
-            {typeof handleChangeView === 'function' ? (
+            {typeof handleChangeView === 'function' && !disableCards ? (
               <Tooltip title={t('Cards view')}>
                 <IconButton
                   color="primary"
@@ -208,6 +210,18 @@ class ListLines extends Component {
             ) : (
               ''
             )}
+            {typeof handleChangeView === 'function' && enableDuplicates ? (
+              <Tooltip title={t('Detect duplicates')}>
+                <IconButton
+                  color="secondary"
+                  onClick={handleChangeView.bind(this, 'duplicates')}
+                >
+                  <TableChart />
+                </IconButton>
+              </Tooltip>
+            ) : (
+              ''
+            )}
             <Security needs={[KNOWLEDGE_KNGETEXPORT]}>
               {typeof handleToggleExports === 'function' ? (
                 <Tooltip title={t('Exports panel')}>
@@ -293,6 +307,8 @@ ListLines.propTypes = {
   handleSearch: PropTypes.func,
   handleSort: PropTypes.func.isRequired,
   handleChangeView: PropTypes.func,
+  disableCards: PropTypes.bool,
+  enableDuplicates: PropTypes.bool,
   handleRemoveFilter: PropTypes.func,
   handleToggleExports: PropTypes.func,
   openExports: PropTypes.bool,

opencti-platform/opencti-front/src/components/list_lines/ListLinesContent.js

@@ -1,7 +1,7 @@
 /* eslint-disable no-underscore-dangle */
 import React, { Component } from 'react';
 import * as PropTypes from 'prop-types';
-import { compose, differenceWith } from 'ramda';
+import { compose, differenceWith, propOr } from 'ramda';
 import { withStyles } from '@material-ui/core/styles';
 import {
   AutoSizer,
@@ -37,7 +37,16 @@ class ListLinesContent extends Component {
       this.props.dataList,
       prevProps.dataList,
     );
-    if (diff.length > 0) {
+    let selection = false;
+    if (this.props.selectedElements) {
+      if (
+        Object.keys(this.props.selectedElements).length
+        !== Object.keys(propOr({}, 'selectedElements', prevProps)).length
+      ) {
+        selection = true;
+      }
+    }
+    if (diff.length > 0 || selection) {
       this.listRef.forceUpdateGrid();
     }
   }
@@ -85,6 +94,8 @@ class ListLinesContent extends Component {
       entityLink,
       me,
       onTagClick,
+      selectedElements,
+      onToggleEntity,
     } = this.props;
     const edge = dataList[index];
     if (!edge) {
@@ -106,6 +117,8 @@ class ListLinesContent extends Component {
           entityLink,
           me,
           onTagClick,
+          selectedElements,
+          onToggleEntity,
         })}
       </div>
     );
@@ -183,9 +196,8 @@ ListLinesContent.propTypes = {
   paginationOptions: PropTypes.object,
   entityLink: PropTypes.string,
   onTagClick: PropTypes.func,
+  selectedElements: PropTypes.object,
+  onToggleEntity: PropTypes.func,
 };
 
-export default compose(
-  inject18n,
-  withStyles(styles),
-)(ListLinesContent);
+export default compose(inject18n, withStyles(styles))(ListLinesContent);

opencti-platform/opencti-front/src/private/Root.js

@@ -20,7 +20,7 @@ import RootWorkspace from './components/workspaces/Root';
 import Reports from './components/reports/Reports';
 import RootReport from './components/reports/Root';
 import ExternalReferences from './components/common/external_references/ExternalReferences';
-import Connectors from './components/connectors/Connectors';
+import RootData from './components/data/Root';
 import Profile from './components/Profile';
 import Message from '../components/Message';
 import { NoMatch, BoundaryRoute } from './components/Error';
@@ -165,9 +165,8 @@ const Root = () => {
                         )}
                       />
                       <BoundaryRoute
-                        exact
-                        path="/dashboard/connectors"
-                        render={(routeProps) => <Connectors {...routeProps} />}
+                        path="/dashboard/data"
+                        render={(routeProps) => <RootData {...routeProps} />}
                       />
                       <BoundaryRoute
                         path="/dashboard/settings"

opencti-platform/opencti-front/src/private/components/Dashboard.js

@@ -56,9 +56,6 @@ const styles = (theme) => ({
     maxHeight: 60,
     paddingRight: 0,
   },
-  itemIcon: {
-    color: theme.palette.primary.main,
-  },
   itemText: {
     whiteSpace: 'nowrap',
     overflow: 'hidden',
@@ -458,10 +455,8 @@ class Dashboard extends Component {
                                 component={Link}
                                 to={`/dashboard/reports/all/${report.id}`}
                               >
-                                <ListItemIcon
-                                  classes={{ root: classes.itemIcon }}
-                                >
-                                  <Description />
+                                <ListItemIcon>
+                                  <Description color="primary" />
                                 </ListItemIcon>
                                 <ListItemText
                                   classes={{ root: classes.itemText }}
@@ -528,10 +523,8 @@ class Dashboard extends Component {
                                   component={Link}
                                   to={`/dashboard/signatures/observables/${stixObservable.id}`}
                                 >
-                                  <ListItemIcon
-                                    classes={{ root: classes.itemIcon }}
-                                  >
-                                    <HexagonOutline />
+                                  <ListItemIcon>
+                                    <HexagonOutline color="primary" />
                                   </ListItemIcon>
                                   <ListItemText
                                     classes={{ root: classes.itemText }}