Create a new dashboards page that makes use of embeddable dashboards by reference

[chantal-kelm]

Description

The work on the new engine will generate events with a different schema. This will require to design a user-friendly interface to view the history of an alert, including all associated elements and their interactions, to facilitate better understanding and analysis.

We want to start with what we currently have implemented, but we need to introduce some changes to ensure the UI coherence.

• Design a standard container for dashboard, so the viewport is adjusted the same on all dashboards
• Design a responsive way to adjust the compoments of the dashboard in the same way on all dashboards
• Contemplate the inclusion of a left menu, included in the dashboard section, so we don't need to abuse the global menu
• Create a new dashboards page that makes use of embeddable dashboards by reference so they can be used in the reporting plugin
• Have the dashboards pre-loaded when the application is initialised.
• Give the user the option to undo changes.

[chantal-kelm]

I am researching the use and application of embeddable dashboards by reference.

I have found an example in the Opensearch repository on embeddable dashboards by value but there is no example on embeddable dashboards by reference

[chantal-kelm]

As there is no documentation on embeddable dashboards by reference I am following the Opensearch source code.
I am coding the first dashboard by reference, I started with the vulnerabilities module

[chantal-kelm]

I have found an example of embeddable dashboards by reference in the Opensearch repo and based on that I am using it as a guide.

[chantal-kelm]

I've been doing some research and I found this conversation in the Opensearch dashboard community where the use of this plugin is recommended: https://github.com/opensearch-project/OpenSearch-Dashboards/tree/main/src/plugins/expressions

After doing some research on embeddables by reference, I tried to use the savedObjectId property to display a dashboard using embeddable dashboards by value, but it gave me formatting errors and I consulted my colleague @Desvelao, who proposed another solution that takes a dashboard specification and transforms it so that it can be used to render a dashboard.

[chantal-kelm]

Embeddable dashboards by reference, taking a dashboard specification and transforming it so that it can be used to render a dashboard:

[chantal-kelm]

Working on dashboard automation.

Currently when we create a dashboard we have its definition in the source code, since it does not benefit us to have it this way, I am working on using embeddable dashboards as a reference.

When using an ID to display a dashboard, we need this dashboard to be created before referencing it.

Proposed solution

I am working on a solution that allows us to automate the creation or import of dashboards when the application is initialized, and then display them in their respective modules.

I am using the Kibana API to create or import dashboards at application initialization time; For example, when the health check is performed and the vulnerability index is loaded, it could also load the dashboards.

• https://www.elastic.co/guide/en/kibana/7.10/saved-objects-api-create.html
• https://www.elastic.co/guide/en/kibana/7.10/dashboard-import-api.html

Things to define:

• If I create the dashboards via API and then reference them in each module, I would have to create all the dashboards again; If I instead import the dashboards via API, I could make some adjustments to our current definitions to convert them to an ndjson file so I can import them.

• If I create the dashboards via API, we must take into account the user's permissions, they must be able to write.

• Once the dashboards are loaded, they would be available in the Opensearch plugin, so the user could edit them and that would modify the dashboard view in a module as they are referenced. With this in mind, do we want this to happen? What happens if the user touches this without having knowledge of this? Do we want there to be a button that undoes the changes? Do we want changes made by the user to be undone when they go through the health check?

• Do we want the dashboard load to be a check item within the health-check? to notify the user (like we do today with the vulnerability index)

[chantal-kelm]

Definitions update:

1. If I create the dashboards via API and then reference them in each module, I would have to create all the dashboards again; If I instead import the dashboards via API, I could make some adjustments to our current definitions to convert them to an ndjson file so I can import them.

   • It was decided to move forward with the option to import dashboards via api.

2. If I create the dashboards via API, we must take into account the user's permissions, they must be able to write.

   • We will use Wazuh RBAC

3. Once the dashboards are loaded, they would be available in the Opensearch plugin, so the user could edit them and that would modify the dashboard view in a module as they are referenced. With this in mind, do we want this to happen? What happens if the user touches this without having knowledge of this? Do we want there to be a button that undoes the changes? Do we want changes made by the user to be undone when they go through the health check?

   • The user will be given the option to undo the changes.

4. Do we want the dashboard load to be a check item within the health-check? to notify the user (like we do today with the vulnerability index)

   • It will be as in vulnerabilities

[chantal-kelm]

When entering a dashboard the check is done using validateVulnerabilitiesStateDataSources which checks if the dashboard is created, otherwise it creates it with a specific id.

In this image in the console you can see how the dashboard with id 588dd588-ebfc-40c9-8427-84ca36ccd6d5 has not been created.

after seeing that the dashboard has not been created, here you can see in the console how the dashboard with id 588dd588-ebfc-40c9-8427-84ca36ccd6d5 has been created.

[chantal-kelm]

Seeing undesired behaviour in the case of when the dashboard does not exist, it creates it, shows it for the first time, it is seen correctly, but if for example I change page when I return to the previous dashboard it shows me the dashboard but it tells me that it cannot locate the visualisations or some of the visualisations.

[chantal-kelm]

Update on the functioning of the poc so far:

Grabacion.de.pantalla.2024-08-07.a.la.s.6.07.59.p.m.mov

[chantal-kelm]

When loading the visualisations of the whole dashboard I have experienced the following problems:

When loading the visualizations using the endpoints to create dashboard and create visualizations:

/api/saved_objects/dashboard/{id}

/api/saved_objects/visualization/{id}

I have seen with my partner Antonio that you have to make a request per visualization, so I decided to try with another endpoint that allows to load several saved objects:

/api/saved_objects/_bulk_create

But this one did not load visualizations inside a dashboard but it created the dashboard and the visualizations apart, so I tried with another endpoint:

/api/saved_objects/_import

and with this one I had problems to send in a POST a ndjson file. I have been working on a script that converts the ndjson file to binary to be sent in the POST but I am having errors when generating the blob to convert the file to binary.
I have been watching with my partner Julio this error.

[chantal-kelm]

With the help of my colleague Antonio I have fixed the bug I had with the following endpoint to convert the ndjson to binary to send it in the POST.

I've also been working on the consumption of the ndjson file from the backend and I'm still working on moving the logic to a global instance since it will be used in all the modules of the application. Unlike before, now a single request loads the whole dashboard, which is much more efficient.

/api/saved_objects/_import

[chantal-kelm]

I am working is being done on code reuse and ordering.

Uploading components to a global scope to be used by other child components.

I am bringing methods into common files to keep the logic encapsulated in certain files.

[chantal-kelm]

Complete video of the operation

• When you enter a dashboard, if the dashboard is not created, it creates it and displays it.
• If the dashboard is already created, it displays it
• If the user made changes in the dashboard and now wants to undo them, he can do it by pressing the restart button.

Grabacion.de.pantalla.2024-08-20.a.la.s.11.24.35.a.m.mov