Fleet system package installs 15 Dashboards, 120+ Visualizations and 2 Kibana index-patterns

[kobelb]

7.13+ ESS deployments of Kibana with APM and Fleet-Server enabled have 15 Dashboards, 120+ Visualizations 2 Kibana index-patterns force installed.

When on-prem users navigate to the Fleet management UI, these aforementioned dashboards, visualizations and index-patterns are automatically created.

While this is great for users who want to use our Stack to look at system logs and metrics; however, not all of our users use the Stack for this purpose. We have a very diverse user base, and these dashboards, visualizations, and index patterns are not relevant to all users.

[elasticmachine]

Pinging @elastic/ingest-management (Team:Ingest Management)

[ruflin]

This is indeed to make sure we have a smooth getting started. You mention the dashboards and visualizations but is also logs ingest pipeplines and templates. What it does is installing the system and endpoing package.

I don't think we should stop doing this as this is also helpful for example if someone runs the Elastic Agent in standalone. But the dashboards, visualizations created should be tagged or read only to make it clear these are managed by the Stack.

[ph]

I am not sure I would consider this a bug, without installing them I am not sure how we could reconcile the Standalone vs the fleet inexperience. @kobelb Do you think it should be a human triggered action?

[kobelb]

@kobelb Do you think it should be a human triggered action?

I do. If the user is choosing to create all of these assets, that makes sense to me. I'm primarily concerned that we're creating them without prompting the user to see if that's something that they really want. The first time I stumbled upon this it was because I was trying to click "Stack Management" and mis-clicked, this led to around 20 saved-objects being created which I then had to delete manually.

[alexfrancoeur]

@mostlyjason it seems I wasn't the only one who noticed this issue.

Adding my two cents here. It feels odd that clicking into an application loads dashboards and visualizations without context. I've run into this accidental experience as well. Is there a better point at which we can load the dashboards? When fleet or agent are downloaded (though that still feels off), once you start receiving system data, etc.

Is there another point at which we can pre-load these dashboards so we don't hinder the onboarding experience while at the same time making it clear to the user that dashboards have been added from fleet?

[mostlyjason]

@ph @ruflin what about when the user clicks to initialize fleet? Would that be a good time to install the default integrations? I think today we click to initialize central management, but we could expand the scope to initialize Fleet for both managed and standalone mode.

[ruflin]

At the moment either a package is installed or not. To have a default policy, we need the system package installed. This means all assets for the system package will be installed including dashboards.

Lets look at it from a different perspective: What if Kibana itself would ship with these assets like Canvas templates. They would always already be installed. The problem here is that these are "normal" dashboards and not "managed" dashboards which the user can't modify.

[kobelb]

Lets look at it from a different perspective: What if Kibana itself would ship with these assets like Canvas templates. They would always already be installed. The problem here is that these are "normal" dashboards and not "managed" dashboards which the user can't modify.

Why would we ship these assets by default? Looking at the visualizations that are currently created when clicking on Fleet, they don't seem relevant to all users of Kibana. We have a vast user-base, that are using our Stack for a multitude of use-cases. Why would our users want visualizations about CPU usage and packet loss if they were using the Stack to store recipes?

[ph]

I've added as a point in our sync lets discuss it.

[alexfrancoeur]

++ thanks @ph, looking forward to the outcome. Feel free to invite any of us if you'd like some additional perspective from our side.

[nchaulet]

Also from a user perspective the loading time the first time can be long maybe if we have a user action to do the setup, we can maybe explore what is happening under the hood.

[ph]

@alexfrancoeur and @kobelb We understand the situation an would like to raise what we are trying to do here. With the indexing strategy, the changes from Beats to Elastic-Agent we are becoming more and more opinionated in that regards we want to preinstall things to make sure the system is ready to be used from day 1. This is goal for both the Agent standalone and the managed by fleet.

We are still discussing it how we could takle this, but this is not a priority for the short them. I think it's something we should fix for GA or even 8.0.

We have throw a few ideas of how we could solve that:

• Hiding of assets and protect the assets from Agent.
• Could we be even be more lazy and only install at the last minutes?
• Allow users to unsetup things.

This all come from trade of as we arent the only application, as much stuff is built on top of integration like the security app there will be a need to setup assets before everything.

I will keep it assigned to me and see how critical is to move this forward.

[kobelb]

Thanks @ph, I appreciate you pushing this forward. I think it makes sense a lot of sense for us to ease the burden for users to begin using Fleet and the new indexing strategy. However, I think it's important for us to not lose sight of the diverse usage of the Stack. We have a bunch of people building solutions on top of Kibana in many different formats, and we want to ensure that they all work together in harmony without "stepping on each other's toes".

[skh]

Another option would be to not enable the Fleet plugin by default, but require the user to set xpack.fleet.enabled to true in the Kibana config.

[ph]

Another idea would be to have a minimal package installed by default that doesn't include visualization or dashboard and have a more advanced package that is installed at the last minute that contains the required assets.

[ruflin]

@ph From my perspective we should treat ES and KB assets the same. A KB user is surprised about the dashboards but an ES user about the ingest pipeline. If we solve the problem, we should probably the same solution for both?

@skh If I understand this correctly, it would mean a user has to restart Kibana to start using Fleet?

[kobelb]

This situation is even worse in Cloud now. Starting in 7.13, there are 15 Dashboards, 120+ Visualizations and 2 Kibana index-patterns installed without the user even clicking into Fleet. I've updated the issue description to reflect this.

[alexfrancoeur]

Does this happen on upgrade as well? Or is it just net-new deployments?

[kobelb]

This should only happen when a cluster has the apm/fleet slider enabled.

When I upgrade an ESS 7.12 cluster to 7.13, Fleet is enabled automatically and I don't see a way to opt-out.

This is necessary for th agent/fleet-server to fully work and be ready to use.

Why do the dashboards/visualizations and index-patterns have to be created for Fleet to startup?

If this should be changed then I suggest to revisit the elastic-agent/Fleet setup behavior in general, not specific to ESS/ECE (I believe this issue covers both - on-prem and cloud behavior).

As far as I can tell, this behavior does not happen to the on-prem deployments at the moment. However, it will affect the on-prem deployments potentially as soon as 7.15 once we figure out how to allow Kibana to automatically install the system integration package.

[simitt]

When I upgrade an ESS 7.12 cluster to 7.13, Fleet is enabled automatically and I don't see a way to opt-out.

Only when you already had the apm deployment enabled - from 7.13 on that deployment always also contains elastic-agent and fleet-server.

This is necessary for th agent/fleet-server to fully work and be ready to use.

Why do the dashboards/visualizations and index-patterns have to be created for Fleet to startup?

Let me rephrase to be more precise - the Fleet setup is required for the agent/fleet-server on ESS/ECE to work. Whatever this Fleet setup contains is something that needs to be discussed with the elastic-agent team; I let them chim in here.

Regarding on-prem vs. cloud - afaik the first time the user navigates to Fleet UI the same setup command is run. The difference is that on cloud, when someone enables the apm/fleet deployment, we assume they also want to use the deployment and therefore, partially for user convenience, already trigger the setup.

[kobelb]

Only when you already had the apm deployment enabled - from 7.13 on that deployment always also contains elastic-agent and fleet-server.

Thanks for the clarification @simitt. I'm seeing APM enabled by default with the General purpose, Observability and Security "templates" but not the Enterprise Search templates. However, I can explicitly disable APM from any of them. It's good to know that this doesn't affect all deployments

Regarding on-prem vs. cloud - afaik the first time the user navigates to Fleet UI the same setup command is run. The difference is that on cloud, when someone enables the apm/fleet deployment, we assume they also want to use the deployment and therefore, partially for user convenience, already trigger the setup.

You're right on this as well. Thanks, @simitt .

[ruflin]

All the assets of a package are installed as soon as it is first used in a policy. In addition to this we have some "required" packages that were always installed but we are moving away from this currently.

To make it easier for users to get started we are creating a "Default policy". This default policy contains the system integration which means the system package is installed. In addition, for monitoring the elastic_agent package is required which adds more assets.

Historically we also tried to install the system package always as when the Elastic Agent in standalone mode is started it contains a default configuration which ships the system metrics. To make sure the mappings exist for this scenario without using a policy, the system package must be installed. As our getting started flow is more and more through Kibana maybe we could disable this and find other ways to tell users to install the system package.

Few ideas to improve the situation

• Managed assets in Kibana / ES: Being able to indicate that an assets is managed and also have a separate list for these assets would help to not overwhelm users when the assets are there.
• Do not install dashboards by default: Today either a package is fully installed with all its assets or not. The ES assets are required but Kibana assets could also only be installed in a second step, for example when data exists.
• Smart install packages: Because of the base templates most of the data shipped for by Elastic Agent would be mapped correctly even before the mappings are in place. In case KB could detect when data for a certain package exists it could then install the package.
• Only install assets when first Elastic Agent is enrolled: Instead of installing the assets when the policy is created, it would be possible to install assts as soon as first Elastic Agent is enrolled as this is when the assets are needed. The problem here is that the enrollement does not happen in Kibana but fleet-server so KB does not know about it.

We should remove the Cloud discussion from this issue as in Cloud there is always an Elastic Agent with fleet-server already running and shipping data as pointed out in the previous comments.

[mostlyjason]

@ruflin thanks for writing up the options. I'd like to add one more:

• Install the assets when the user adds their first integration or agent. If we remove cloud from the discussion, most users will add an integration or an agent though the Kibana UI or API. When the user clicks to add their first integration, we create the default agent policy with the system integration. Alternatively, when the user clicks the button to add an agent, we create the default agent policy with the system integration. API users can call a setup endpoint directly to generate the default policy. If they are using agent standalone mode without using the UI, they do not need a default agent policy in Fleet. They still need to install the system package, but that is the case for any package used in standalone mode.

[snide]

Popping on this thread to bump this from a UX perspective. To a new Cloud user these empty assets come across as broken. More importantly, large parts of our current UX flow are built around the question "do they have data yet"? Currently those flows aren't firing because technically they do have data. This directly hurts our KPIs.

I think the solution here is primarily an engineering decision so I won't weigh on too much on the options (though I'd lean into it being tied to an active user action as suggested). Mostly just making sure this is prioritized.

[ruflin]

I had an offline discussion around this that went even further on the granularity of installation of assets. If a users runs on Linux, why are the Windows dashboards installed? Compared to Beats where the dashboards for all modules were loaded, we made already a massive improvement by just loading the ones that the user might likely use.

Going back to the Windows example and the point from @snide about "do they have data yet"? Thanks to the data stream naming scheme this is actually something we could achive per dashboard. A dashboard contains a list of visualisation and each visualisation normally depends on a single data stream. Only if one of the data streams exists the dashboard likely be loaded. This check can now done pretty efficient. It means in theory (ignoring all the technical challenges around it) we could install a dashboard only if there is data for at least a single visualisation inside it.

[alexfrancoeur]

I think we need to prioritize solving this problem as soon as possible. This is impacting onboarding flows. I just ran through creating a net new deployment in Cloud and came across this bug. I then tried to take a path I imagine is not uncommon, Kibana => Dashboard => create visualization. In doing so, we typically have an empty state that leads towards data ingestion. Because there are indices now automatically added and index patterns automatically available, those flows are now gone. And instead, you're left with a dataless experience with no next steps.

Jul-28-2021.10-22-55.mp4

While one could argue we could add some more complex logic to handling empty states, I still don't think it makes sense to provide these assets or indices without explicit user intent. We're willing to help on the Kibana side here, but I think we need to figure this out ASAP. This experience is extremely confusing for net new users.

I know we've decoupled this from the fleet working group, so maybe we should address separately. How can we prioritize a solution in the next release or so?

Adding @mfinkle @VijayDoshi for visibility

[VijayDoshi]

I've been reviewing onboarding flows. Users are really struggling with not having the guidance we used to have for the empty state and are also regularly hitting scenarios where the product feels broken and we aren't even giving minimal guidance to help them out since our current logic of "no index patterns" - installing all of these dashboards, visualizations and index patterns is causing a very broken experience. I believe we need to fix this ASAP, treating it like a bug. There are other related issues around having no-data but having objects we need to address with similar priority (captured in related issues) added above but thought I'd summarize my perspective:

• In Discover, we should show the old "no data" state when there are no documents at all in the system.
• In Dashboards, Visualize Library and if we are in the state where there is no data at all we should not be listing the objects.
• We should be showing the "interstitial" to add data.
• There are other places within solutions that also feel broken because of no data; however, the presence of index patterns is causing an "empty" state vs. a "no data" state. For example; in metrics, we give no indication about what to do...

7.14 default state:

7.12.1 default state:

@snide @alexfrancoeur

[alexfrancoeur]

@joshdover is doing some digging to provide some updated logic around what it means for Kibana to be in an "empty state" (see #107020 (comment) for the latest). Do we plan to leverage this logic for the dashboard / index pattern / visualizations in the short term or will we modify when these Kibana assets get created? Circling back to Vijay's comment, when I say short term I'm thinking 7.14.x / 7.15.0.

cc: @AlonaNadler @ghudgins in case they weren't aware of this issue

[alvarolobato]

@alexfrancoeur I think we need to do both. In the short term improve how we check the empty state, what Josh is doing and start working on the installation workflow of the assets.
I believe we need the work being done in #107020 for two reasons, first timing, we won't be able to do the long term solution for 7.15 and second because on cloud, by definition there will always be some data given that there is always an automatically installed agent shipping some data.

[osmanis]

@linyaru @lucagennari-es @mwtyang with default dashboards, visualizations, index patterns and indices set up with each new deployment, we may need to do a health check on our KPIs to ensure we're not over counting on some metrics like ingest rate which takes into account # of documents in ES.

[linyaru]

I think we should be safe for data ingestion metrics as we use actual documents loaded into ES instead of patterns or artifacts created. However this does significantly impact our activity level indicators (num of dashes, vizes, etc), so we should warn the COS team.

[joshdover]

@alexfrancoeur I think we need to do both. In the short term improve how we check the empty state, what Josh is doing and start working on the installation workflow of the assets.
I believe we need the work being done in #107020 for two reasons, first timing, we won't be able to do the long term solution for 7.15 and second because on cloud, by definition there will always be some data given that there is always an automatically installed agent shipping some data.

@joshdover is doing some digging to provide some updated logic around what it means for Kibana to be in an "empty state" (see #107020 (comment) for the latest). Do we plan to leverage this logic for the dashboard / index pattern / visualizations in the short term or will we modify when these Kibana assets get created? Circling back to Vijay's comment, when I say short term I'm thinking 7.14.x / 7.15.0.

FYI fixes have been merged and backported for both the 7.15.0 and 7.14.1 releases that update the "empty state" logic for both the "add data" onboarding interstitial and the Analytics apps' 'no index pattern' flows. This new logic will now show these UIs whenever the default Fleet assets are installed in Cloud and on-prem. This fix will be testable on Cloud staging once BC2 is available for 7.15.0 and once 7.14.1 BC1 is created. Both should be available later this week.

[alexfrancoeur]

@joshdover can this now be closed with #108193?

[joshdover]

@alexfrancoeur I don't believe so as we are still installing these assets by default. The difference is that now when a user opens one of these dashboards, if there isn't any data yet ingested, they'll be redirected to the index pattern setup flow (which will prompt them to add data).

I believe we still need to explore the UX options in #108456 before we will be able to close this issue.

[jen-huang]

I believe we can close this now as the work in #108456 is complete. System package is no longer installed by default from 8.1 onwards.