Retool Integration (DataDog#863)

Add retool integration

.github/CODEOWNERS

@@ -62,6 +62,7 @@
 /reboot_required/       @montdidier [email protected]
 /redis_sentinel/        @krasnoukhov
 /resin/                 @brentm5
+/retool/                @jamiecuffe
 /riak_repl/             @abtreece
 /rigor/                 [email protected]
 /rookout/               [email protected]

retool/CHANGELOG.md

@@ -0,0 +1,3 @@
+# CHANGELOG - retool_retool
+## Latest
+* https://updates.retool.com/en

retool/README.md

@@ -0,0 +1,39 @@
+# Retool
+
+## Overview
+Monitoring and analytics offer mission-critical insights, but developers often have to jump between separate, siloed, and often custom tools to take action on the insights—leading to inefficient or ineffective responses to those insights. 
+
+Retool helps developers create custom apps that embed directly into a Datadog dashboard, giving you the ability to take action and automate workflows without having to leave Datadog. 
+
+![Screenshot1](https://raw.githubusercontent.com/DataDog/integrations-extras/master/retool/images/1.png)
+
+### Metrics
+Datadog’s embedded Retool app for Elasticsearch Management combines existing visibility into key Elasticsearch metrics and logs with the power to manage clusters, accounts, and more without leaving your Datadog dashboard.
+
+### Dashboards
+Retool built an embedded app for Elasticsearch Management. You can already monitor Elasticsearch metrics, traces, and logs in Datadog today. With our embedded app, developers can take action on their rich Datadog insights directly in the Datadog dashboard. These actions include:
+
+- Add a new index with shards and replicas
+- Manage nodes by rerouting shards and excluding indexes
+- Create new snapshots and restore indexes
+
+## Setup
+The Retool integration comes with an out-of-the-box dashboard, which allows you to sign up or log into Retool through an iframe.
+
+You’ll be prompted to connect to your ElasticSearch cluster with a connection string. This app will automatically be added to you instance. You then need to click resources in the navbar and create a new Datadog resource (adding your api and application keys). Finally connect your Datadog resource to the two Datadog queries by selecting it from the select resource dropdown in the query editor. 
+
+And that’s it - return to Datadog and the app will be up and running in your dashboard. You can edit the app anytime to customize it for your DevOps workflows.
+
+## Data Collected
+
+### Metrics
+The Retool integration does not include any metrics at this time.
+
+### Events
+The Retool integration does not include any events at this time.
+
+### Service Checks
+The Retool does not include any service checks at this time.
+
+## Troubleshooting
+Need help? Contact [Datadog support](https://docs.datadoghq.com/help/)

retool/assets/dashboards/retool_retool_overview.json

@@ -0,0 +1 @@
+{"title":"Retool + Datadog: ElasticSearch Action Console","description":"This dashboard provides a high-level overview of your Elasticsearch clusters, so you can track health status, search and indexing performance, and resource utilization metrics from all your nodes and be better prepared to address potential issues. Further reading on Elasticsearch monitoring:\n\n- [Datadog's guide to key Elasticsearch metrics](https://www.datadoghq.com/blog/monitor-elasticsearch-performance-metrics/)\n\n- [How to collect Elasticsearch metrics with native and open source tools](https://www.datadoghq.com/blog/collect-elasticsearch-metrics/)\n\n- [How to monitor Elasticsearch with Datadog](https://www.datadoghq.com/blog/monitor-elasticsearch-datadog/)\n\n- [How to solve 5 Elasticsearch performance and scaling problems](https://www.datadoghq.com/blog/elasticsearch-performance-scaling-problems/)\n\n- [How to resolve unassigned shards in Elasticsearch](https://www.datadoghq.com/blog/elasticsearch-unassigned-shards/)\n\n- [Datadog's Elasticsearch integration docs](https://docs.datadoghq.com/integrations/elastic/)\n\nClone this template dashboard to make changes and add your own graph widgets. (cloned)","widgets":[{"id":0,"layout":{"x":0,"y":0,"width":29,"height":12},"definition":{"type":"image","url":"/static/images/logos/elasticsearch_large.svg","sizing":"zoom"}},{"id":1,"layout":{"x":0,"y":26,"width":29,"height":51},"definition":{"type":"note","content":"Retool is a no-code platform for writing applications, that includes a native integration with Datadog for using any Datadog API in your Retool apps.  In this example, the Retool application can be connected to your on-prem or hosted Elasticsearch clusters, your s3 account for snapshotting, and your Datadog account.\n\nThe embedded Retool Elasticsearch Management Template, to the right, helps you manage your ES clusters.  If you see slow queries to one of your ES Indexes, move shards to different hosts on the cluster, to alleviate resource contention!  Before you move a shard to a new host, make sure there is enough disk space on that host for the shard to grow, so that you are not constantly rebalancing!\n\nThis ES template is just one of the many ways Retool can be used to create internal applications for dev teams by using Datadog queries.  See our blog post (link TK) for more detail.\n\nIn this app, you will be able to create and move indicies and shards, manage which nodes each are assigned to, and backup and restore snapshots of the database.\n","background_color":"white","font_size":"14","text_align":"left","show_tick":false,"tick_pos":"50%","tick_edge":"left"}},{"id":2,"layout":{"x":94,"y":0,"width":31,"height":5},"definition":{"type":"note","content":"Overview","background_color":"vivid_blue","font_size":"18","text_align":"center","show_tick":false,"tick_pos":"50%","tick_edge":"bottom"}},{"id":4,"layout":{"x":110,"y":6,"width":15,"height":10},"definition":{"title":"Nodes by cluster status","title_size":"13","title_align":"center","type":"check_status","check":"elasticsearch.cluster_health","grouping":"cluster","group_by":[],"tags":[]}},{"id":5,"layout":{"x":127,"y":6,"width":31,"height":14},"definition":{"title":"Avg query and fetch latency over $elastic_cluster","title_size":"16","title_align":"left","show_legend":false,"legend_size":"0","type":"timeseries","requests":[{"q":"(sum:elasticsearch.search.query.time{$elastic_cluster}/sum:elasticsearch.search.query.total{$elastic_cluster})","style":{"palette":"green","line_type":"solid","line_width":"normal"},"display_type":"line"},{"q":"(sum:elasticsearch.search.fetch.time{$elastic_cluster}/sum:elasticsearch.search.fetch.total{$elastic_cluster})","style":{"palette":"green","line_type":"solid","line_width":"normal"},"display_type":"line"}],"yaxis":{"scale":"linear","label":"","include_zero":true,"min":"auto","max":"auto"}}},{"id":6,"layout":{"x":127,"y":0,"width":63,"height":5},"definition":{"type":"note","content":"Cluster Performance","background_color":"blue","font_size":"18","text_align":"center","show_tick":false,"tick_pos":"50%","tick_edge":"left"}},{"id":8,"layout":{"x":94,"y":52,"width":15,"height":10},"definition":{"title":"Active shards over $elastic_cluster","title_size":"16","title_align":"left","type":"query_value","requests":[{"q":"max:elasticsearch.active_shards{$elastic_cluster}","aggregator":"max"}],"autoscale":true,"precision":0}},{"id":9,"layout":{"x":110,"y":52,"width":15,"height":10},"definition":{"title":"Initializing shards","title_size":"16","title_align":"left","type":"query_value","requests":[{"q":"max:elasticsearch.initializing_shards{$elastic_cluster}","aggregator":"max","conditional_formats":[{"comparator":">","palette":"red_on_white","value":0},{"comparator":">=","palette":"green_on_white","value":0}]}],"autoscale":false,"custom_unit":"","precision":0}},{"id":10,"layout":{"x":126,"y":52,"width":15,"height":10},"definition":{"title":"Relocating shards","title_size":"16","title_align":"left","type":"query_value","requests":[{"q":"max:elasticsearch.relocating_shards{$elastic_cluster}","aggregator":"max","conditional_formats":[{"comparator":">","palette":"red_on_white","value":0},{"comparator":"<=","palette":"green_on_white","value":0}]}],"autoscale":true,"precision":0}},{"id":11,"layout":{"x":142,"y":52,"width":15,"height":10},"definition":{"title":"Unassigned shards","title_size":"16","title_align":"left","type":"query_value","requests":[{"q":"max:elasticsearch.unassigned_shards{$elastic_cluster}","aggregator":"max","conditional_formats":[{"comparator":">","palette":"red_on_white","value":0},{"comparator":">=","palette":"green_on_white","value":0}]}],"autoscale":true,"precision":0}},{"id":12,"layout":{"x":94,"y":63,"width":31,"height":14},"definition":{"title":"Active shards (total and primary) over $elastic_cluster","title_size":"16","title_align":"left","show_legend":false,"legend_size":"0","type":"timeseries","requests":[{"q":"max:elasticsearch.active_shards{$elastic_cluster}","style":{"palette":"dog_classic","line_type":"solid","line_width":"normal"},"display_type":"line"},{"q":"max:elasticsearch.active_primary_shards{$elastic_cluster}","style":{"palette":"dog_classic","line_type":"solid","line_width":"normal"},"display_type":"line"}],"yaxis":{"scale":"linear","label":"","include_zero":true,"min":"auto","max":"auto"}}},{"id":13,"layout":{"x":94,"y":46,"width":63,"height":5},"definition":{"type":"note","content":"Shards","background_color":"blue","font_size":"18","text_align":"center","show_tick":false,"tick_pos":"50%","tick_edge":"left"}},{"id":14,"layout":{"x":126,"y":63,"width":31,"height":14},"definition":{"title":"Shards initializing, relocating, unassigned over $elastic_cluster","title_size":"16","title_align":"left","show_legend":false,"legend_size":"0","type":"timeseries","requests":[{"q":"max:elasticsearch.initializing_shards{$elastic_cluster}","style":{"palette":"dog_classic","line_type":"solid","line_width":"normal"},"display_type":"line"},{"q":"max:elasticsearch.unassigned_shards{$elastic_cluster}","style":{"palette":"dog_classic","line_type":"solid","line_width":"normal"},"display_type":"line"},{"q":"max:elasticsearch.relocating_shards{$elastic_cluster}","style":{"palette":"dog_classic","line_type":"solid","line_width":"normal"},"display_type":"line"}],"yaxis":{"scale":"linear","label":"","include_zero":true,"min":"auto","max":"auto"}}},{"id":15,"layout":{"x":127,"y":21,"width":31,"height":14},"definition":{"title":"Nodes of $elastic_cluster with most indexing activity (top 10 hosts)","title_size":"16","title_align":"left","type":"toplist","requests":[{"q":"top(diff(avg:elasticsearch.indexing.index.total{$elastic_cluster} by {host}), 10, 'sum', 'desc')"}]}},{"id":16,"layout":{"x":158,"y":52,"width":32,"height":44},"definition":{"type":"log_stream","indexes":[],"query":"source:elasticsearch $elastic_cluster","sort":{"column":"time","order":"desc"},"columns":["core_host","core_service"],"show_date_column":true,"show_message_column":true,"message_display":"expanded-lg"}},{"id":20,"layout":{"x":158,"y":46,"width":32,"height":5},"definition":{"type":"note","content":"Elasticsearch logs","background_color":"blue","font_size":"18","text_align":"center","show_tick":false,"tick_pos":"50%","tick_edge":"left"}},{"id":21,"layout":{"x":159,"y":21,"width":31,"height":14},"definition":{"title":"Nodes of $elastic_cluster with most queries (top 10 hosts)","title_size":"16","title_align":"left","type":"toplist","requests":[{"q":"top(diff(avg:elasticsearch.search.query.total{$elastic_cluster} by {host}), 10, 'sum', 'desc')"}]}},{"id":22,"layout":{"x":94,"y":6,"width":15,"height":10},"definition":{"title":"Disk space used","title_size":"16","title_align":"left","type":"query_value","requests":[{"q":"(1-(sum:elasticsearch.fs.total.available_in_bytes{$elastic_cluster} by {host}/sum:elasticsearch.fs.total.total_in_bytes{$elastic_cluster} by {host}))*100","aggregator":"avg","conditional_formats":[{"comparator":"<","palette":"green_on_white","value":80},{"comparator":">","palette":"yellow_on_white","value":80},{"comparator":">","palette":"red_on_white","value":90}]}],"custom_unit":"%","precision":1}},{"id":23,"layout":{"x":159,"y":6,"width":31,"height":14},"definition":{"title":"Total number of pending task in $elastic_cluster","title_size":"16","title_align":"left","show_legend":false,"legend_size":"0","type":"timeseries","requests":[{"q":"sum:elasticsearch.pending_tasks_total{$elastic_cluster}","style":{"palette":"dog_classic","line_type":"solid","line_width":"normal"},"display_type":"line"}],"yaxis":{"scale":"linear","label":"","include_zero":true,"min":"auto","max":"auto"}}},{"id":27,"layout":{"x":94,"y":78,"width":63,"height":18},"definition":{"type":"note","content":"If the number of unassigned shard is not 0, it often means that an action should be taken. Among the possible source of error, the two main ones are the following.\n\nIt is possible that the shard allocation is purposefully delayed, in that case a logs with the message `delaying allocation for [54] unassigned shards, next check in [1m]` should appear too.\n\nOther possibility is that there are too many shards for not enough nodes. In that case, make sure that every index in your cluster is initialized with fewer replicas per primary shard than the number of nodes in your cluster.\n","background_color":"gray","font_size":"14","text_align":"center","show_tick":false,"tick_pos":"50%","tick_edge":"left"}},{"id":30,"layout":{"x":94,"y":18,"width":31,"height":10},"definition":{"type":"note","content":"If disk space for the entire cluster is limited, or the cluster is in a \"yellow\" or \"red\" health state, you might need to add more nodes to the cluster, and quickly!","background_color":"gray","font_size":"14","text_align":"left","show_tick":true,"tick_pos":"50%","tick_edge":"top"}},{"id":2327440509042346,"layout":{"x":30,"y":0,"width":63,"height":96},"definition":{"type":"iframe","url":"https://mikezvi.retool.com/apps/Datadog%20Elasticsearch%20Template%20V%202.0%20(Launch%201)"}},{"id":2441766761924720,"layout":{"x":0,"y":13,"width":29,"height":12},"definition":{"type":"image","url":"https://retool.com/logo.png","sizing":"cover"}},{"id":5898267523188546,"layout":{"x":127,"y":36,"width":63,"height":9},"definition":{"type":"note","content":"Query performance directly impacts customers;  if this is over an acceptable threshold, re-balance indexes.\n\nIf the number of pending tasks are too high, move shard from nodes with the most queries to nodes with fewer.","background_color":"gray","font_size":"14","text_align":"left","vertical_align":"top","show_tick":true,"tick_pos":"50%","tick_edge":"top","has_padding":true}},{"id":8603829269002102,"layout":{"x":0,"y":78,"width":29,"height":18},"definition":{"type":"note","content":"\nFor other Elasticsearch monitoring resources that will help inform the way you manage your clusters with this app, the following guides are also available:\n\n- [Datadog's guide to key Elasticsearch metrics](https://www.datadoghq.com/blog/monitor-elasticsearch-performance-metrics/)\n\n- [How to collect Elasticsearch metrics with native and open source tools](https://www.datadoghq.com/blog/collect-elasticsearch-metrics/)\n\n- [How to monitor Elasticsearch with Datadog](https://www.datadoghq.com/blog/monitor-elasticsearch-datadog/)\n\n- [How to solve 5 Elasticsearch performance and scaling problems](https://www.datadoghq.com/blog/elasticsearch-performance-scaling-problems/)\n\n- [How to resolve unassigned shards in Elasticsearch](https://www.datadoghq.com/blog/elasticsearch-unassigned-shards/)\n\n- [Datadog's Elasticsearch integration docs](https://docs.datadoghq.com/integrations/elastic/)","background_color":"white","font_size":"14","text_align":"left","vertical_align":"top","show_tick":false,"tick_pos":"50%","tick_edge":"left","has_padding":true}}],"template_variables":[{"name":"elastic_cluster","default":"*","prefix":"elastic_cluster"},{"name":"node_name","default":"*","prefix":"node_name"}],"layout_type":"free","is_read_only":true,"notify_list":[],"id":"828-d8e-68s"}

retool/assets/service_checks.json

@@ -0,0 +1 @@
+[]