Skip to content

Commit

Permalink
Replace PCP Redis datasource with PCP Valkey
Browse files Browse the repository at this point in the history
  • Loading branch information
sfeifer committed Nov 6, 2024
1 parent 77809e9 commit 211bb5f
Show file tree
Hide file tree
Showing 62 changed files with 216 additions and 216 deletions.
2 changes: 1 addition & 1 deletion .github/ISSUE_TEMPLATE/bug_report.md
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ If applicable, add screenshots to help explain your problem.
- Performance Co-Pilot:
- Grafana:
- grafana-pcp:
- Redis (if applicable):
- Valkey (if applicable):

**Additional context**
Add any other context about the problem here.
2 changes: 1 addition & 1 deletion .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -72,7 +72,7 @@ jobs:
name: build
path: build

- name: Start PCP and Redis in a pod
- name: Start PCP and Valkey in a pod
run: |
# workaround required on GitHub Actions to allow running systemd in rootless containers
# https://github.com/actions/virtual-environments/issues/3536
Expand Down
18 changes: 9 additions & 9 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -36,11 +36,11 @@ watch-frontend: deps-frontend build-dashboards ## Auto rebuilt frontend on file
yarn run watch

dev-backend: deps-backend
go build -race -o ./dist/datasources/redis/pcp_redis_datasource_$$(go env GOOS)_$$(go env GOARCH) -tags netgo -ldflags -w ./pkg
go build -race -o ./dist/datasources/valkey/pcp_valkey_datasource_$$(go env GOOS)_$$(go env GOARCH) -tags netgo -ldflags -w ./pkg

restart-backend: ## Rebuild and restart backend data source (as root)
sudo -u "$$(stat -c '%U' .)" make dev-backend
killall pcp_redis_datasource_$$(go env GOOS)_$$(go env GOARCH)
killall pcp_valkey_datasource_$$(go env GOOS)_$$(go env GOARCH)


##@ Build
Expand All @@ -61,10 +61,10 @@ GO_LD_FLAGS := -w -s -extldflags "-static"
build-backend: deps-backend ## Build backend data source
#mage buildAll
for arch in amd64 arm arm64 s390x ppc64le 386; do \
CGO_ENABLED=0 GOOS=linux GOARCH=$${arch} go build -o dist/datasources/redis/pcp_redis_datasource_linux_$${arch} -ldflags '$(GO_LD_FLAGS)' ./pkg; \
CGO_ENABLED=0 GOOS=linux GOARCH=$${arch} go build -o dist/datasources/valkey/pcp_valkey_datasource_linux_$${arch} -ldflags '$(GO_LD_FLAGS)' ./pkg; \
done
CGO_ENABLED=0 GOOS=darwin GOARCH=amd64 go build -o dist/datasources/redis/pcp_redis_datasource_darwin_amd64 -ldflags '$(GO_LD_FLAGS)' ./pkg
CGO_ENABLED=0 GOOS=windows GOARCH=amd64 go build -o dist/datasources/redis/pcp_redis_datasource_windows_amd64.exe -ldflags '$(GO_LD_FLAGS)' ./pkg
CGO_ENABLED=0 GOOS=darwin GOARCH=amd64 go build -o dist/datasources/valkey/pcp_valkey_datasource_darwin_amd64 -ldflags '$(GO_LD_FLAGS)' ./pkg
CGO_ENABLED=0 GOOS=windows GOARCH=amd64 go build -o dist/datasources/valkey/pcp_valkey_datasource_windows_amd64.exe -ldflags '$(GO_LD_FLAGS)' ./pkg

build: build-dashboards build-frontend build-backend ## Build everything

Expand Down Expand Up @@ -99,21 +99,21 @@ test: test-frontend test-backend ## Run all tests

##@ UI tests

test-ui-start-pod: ## Start PCP and Redis in a pod
test-ui-start-pod: ## Start PCP and Valkey in a pod
-podman pod rm -f grafana-pcp-tests
podman pod create --name grafana-pcp-tests -p 3001:3000
podman run --pod grafana-pcp-tests --name grafana-pcp-tests-pcp -d --systemd always quay.io/performancecopilot/pcp
podman run --pod grafana-pcp-tests --name grafana-pcp-tests-redis -d docker.io/library/redis:6
podman run --pod grafana-pcp-tests --name grafana-pcp-tests-valkey -d docker.io/valkey/valkey:7

test-ui-start-grafana-dist: ## Start Grafana with grafana-pcp from the dist/ folder
podman run --pod grafana-pcp-tests --name grafana-pcp-tests-grafana -d --replace \
-e GF_PLUGINS_ALLOW_LOADING_UNSIGNED_PLUGINS="performancecopilot-pcp-app,performancecopilot-redis-datasource,performancecopilot-vector-datasource,performancecopilot-bpftrace-datasource,performancecopilot-flamegraph-panel,performancecopilot-breadcrumbs-panel,performancecopilot-troubleshooting-panel" \
-e GF_PLUGINS_ALLOW_LOADING_UNSIGNED_PLUGINS="performancecopilot-pcp-app,performancecopilot-valkey-datasource,performancecopilot-vector-datasource,performancecopilot-bpftrace-datasource,performancecopilot-flamegraph-panel,performancecopilot-breadcrumbs-panel,performancecopilot-troubleshooting-panel" \
-v $$(pwd)/dist:/var/lib/grafana/plugins/performancecopilot-pcp-app \
$(GRAFANA_IMAGE)

test-ui-start-grafana-build: ## Start Grafana with grafana-pcp from build/performancecopilot-pcp-app-*.zip
podman run --pod grafana-pcp-tests --name grafana-pcp-tests-grafana -d --replace \
-e GF_PLUGINS_ALLOW_LOADING_UNSIGNED_PLUGINS="performancecopilot-pcp-app,performancecopilot-redis-datasource,performancecopilot-vector-datasource,performancecopilot-bpftrace-datasource,performancecopilot-flamegraph-panel,performancecopilot-breadcrumbs-panel,performancecopilot-troubleshooting-panel" \
-e GF_PLUGINS_ALLOW_LOADING_UNSIGNED_PLUGINS="performancecopilot-pcp-app,performancecopilot-valkey-datasource,performancecopilot-vector-datasource,performancecopilot-bpftrace-datasource,performancecopilot-flamegraph-panel,performancecopilot-breadcrumbs-panel,performancecopilot-troubleshooting-panel" \
-e GF_INSTALL_PLUGINS="/tmp/plugin.zip;performancecopilot-pcp-app" \
-v $$(pwd)/build/$$(basename build/performancecopilot-pcp-app-*.zip):/tmp/plugin.zip \
$(GRAFANA_IMAGE)
Expand Down
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@ It presents a unifying abstraction for all of the performance data in a system,
* support for repeated panels
* sample dashboards for all data sources

[1] PCP Redis
[1] PCP Valkey
[2] PCP Vector
[3] PCP bpftrace

Expand Down
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
describe('PCP Redis data source', () => {
describe('PCP Valkey data source', () => {
before(() => {
cy.task('grafana:reset');
cy.login();
cy.enablePlugin();
cy.addDatasource('performancecopilot-redis-datasource', 'PCP Redis data source');
cy.addDatasource('performancecopilot-valkey-datasource', 'PCP Valkey data source');
});

beforeEach(() => {
Expand All @@ -12,9 +12,9 @@ describe('PCP Redis data source', () => {

it('should import bundled dashboards', () => {
cy.visit('/datasources');
cy.contains('PCP Redis').click();
cy.contains('PCP Valkey').click();
cy.contains('Dashboards').click({ force: true });
cy.contains('td', 'PCP Redis: Host Overview').siblings().contains('Import').click({ force: true });
cy.contains('td', 'PCP Valkey: Host Overview').siblings().contains('Import').click({ force: true });
cy.contains('Dashboard Imported');
});

Expand All @@ -27,7 +27,7 @@ describe('PCP Redis data source', () => {
cy.contains('Add a new panel').click();
} else {
cy.contains('Add visualization').click();
cy.get('div.scrollbar-view').contains('PCP Redis').click();
cy.get('div.scrollbar-view').contains('PCP Valkey').click();
}
});

Expand Down
4 changes: 2 additions & 2 deletions cypress/e2e/setup.cy.ts
Original file line number Diff line number Diff line change
Expand Up @@ -13,9 +13,9 @@ describe('grafana-pcp setup', () => {
cy.get('button').should('include.text', 'Disable');
});

it('should setup PCP Redis data source', () => {
it('should setup PCP Valkey data source', () => {
cy.visit('/datasources/new');
cy.contains('PCP Redis').click();
cy.contains('PCP Valkey').click();
cy.get('input[placeholder="http://localhost:44322"]').type('http://localhost:44322');
cy.get('button[type=submit]').click();
cy.contains('Data source is working');
Expand Down
8 changes: 4 additions & 4 deletions docs/architecture.rst
Original file line number Diff line number Diff line change
Expand Up @@ -14,18 +14,18 @@ Monitored hosts run the **Performance Metrics Collector Daemon (PMCD)**, which c
Each **PMDA** is responsible for gathering metrics of one specific domain - e.g., the kernel, services (e.g., PostgreSQL), or other instrumented applications.
The **pmlogger** daemon records metrics from **pmcd** and stores them in archive files on the hard drive.

Since **PCP 5** metrics can also be stored in the redis database, which allows multi-host performance analysis, the **pmproxy** daemon discovers new archives (created by **pmlogger**) and stores them in a redis database.
Since **PCP 5** metrics can also be stored in the valkey database, which allows multi-host performance analysis, the **pmproxy** daemon discovers new archives (created by **pmlogger**) and stores them in a valkey database.

Dashboards
----------

Performance Co-Pilot metrics can be analyzed with Grafana dashboards, using the **grafana-pcp** plugin.
There are two modes available:

* historical metrics across multiple hosts using the :doc:`datasources/redis` data source
* historical metrics across multiple hosts using the :doc:`datasources/valkey` data source
* live, on-host metrics using the :doc:`datasources/vector` data source

The :doc:`datasources/redis` data source sends :ref:`pmseries <pmseries-query-language>` queries to **pmproxy**, which in turn queries the redis database for metrics.
The :doc:`datasources/valkey` data source sends :ref:`pmseries <pmseries-query-language>` queries to **pmproxy**, which in turn queries the valkey database for metrics.
The :doc:`datasources/vector` data source connects to **pmproxy**, which in turn requests live metrics directly from a local or remote **PMCD**.
In this case, metrics are stored temporarily in the browser, and metric values are lost when the browser tab is refreshed.
The :doc:`PCP Redis data source <datasources/redis>` is required for persistence.
The :doc:`PCP Valkey data source <datasources/valkey>` is required for persistence.
2 changes: 1 addition & 1 deletion docs/datasources/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ Overview

.. include:: ../refs.rst

PCP Redis
PCP Valkey
---------

This data source queries the fast, scalable time series capabilities provided by the `pmseries`_ functionality.
Expand Down
10 changes: 5 additions & 5 deletions docs/datasources/redis.rst → docs/datasources/valkey.rst
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
PCP Redis
PCP Valkey
=========

.. include:: ../refs.rst
Expand All @@ -8,15 +8,15 @@ Introduction

This data source provides a native interface between `Grafana <https://grafana.com/>`_ and `Performance Co-Pilot <https://pcp.io>`_ (PCP), allowing PCP metric data to be presented in Grafana panels, such as graphs, tables, heatmaps, etc.
Under the hood, the data source makes REST API query requests to the PCP `pmproxy`_ service, which can be running either locally or on a remote host.
The pmproxy daemon can be local or remote and uses the Redis time-series database (local or remote) for persistent storage.
The pmproxy daemon can be local or remote and uses the Valkey time-series database (local or remote) for persistent storage.

Setup Redis and PCP daemons
Setup Valkey and PCP daemons
---------------------------

.. code-block:: console
$ sudo dnf install redis
$ sudo systemctl start redis pmlogger pmproxy
$ sudo dnf install valkey
$ sudo systemctl start valkey pmlogger pmproxy
.. _pmseries-query-language:

Expand Down
4 changes: 2 additions & 2 deletions docs/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@ Features
* support for repeated panels
* sample dashboards for all data sources

[1] PCP Redis
[1] PCP Valkey
[2] PCP Vector
[3] PCP bpftrace

Expand Down Expand Up @@ -58,7 +58,7 @@ Getting started
:hidden:

datasources/authentication
datasources/redis
datasources/valkey
datasources/vector
datasources/bpftrace

Expand Down
6 changes: 3 additions & 3 deletions docs/installation.rst
Original file line number Diff line number Diff line change
Expand Up @@ -5,14 +5,14 @@ Minimum Software Requirements
-----------------------------

==== ===== ======= ====================
PCP Redis Grafana grafana-pcp
PCP Valkey Grafana grafana-pcp
==== ===== ======= ====================
5.2+ 5+ 7.x 3.x
5.2+ 5+ 8.x 4.x
5.2+ 5+ 9.x 5.x
==== ===== ======= ====================

Note: Redis is only required for the :doc:`datasources/redis` data source.
Note: Valkey is only required for the :doc:`datasources/valkey` data source.

Distribution Package
--------------------
Expand Down Expand Up @@ -73,7 +73,7 @@ The `yarn package manager <https://yarnpkg.com>`_, `Go compiler <https://golang.
$ git clone https://github.com/performancecopilot/grafana-pcp.git
$ make build
$ sudo ln -s $(pwd) /var/lib/grafana/plugins
$ sudo sed -i 's/;allow_loading_unsigned_plugins =/allow_loading_unsigned_plugins = performancecopilot-pcp-app,pcp-redis-datasource,pcp-vector-datasource,pcp-bpftrace-datasource,pcp-flamegraph-panel,pcp-breadcrumbs-panel,pcp-troubleshooting-panel,performancecopilot-redis-datasource,performancecopilot-vector-datasource,performancecopilot-bpftrace-datasource,performancecopilot-flamegraph-panel,performancecopilot-breadcrumbs-panel,performancecopilot-troubleshooting-panel/' /etc/grafana/grafana.ini
$ sudo sed -i 's/;allow_loading_unsigned_plugins =/allow_loading_unsigned_plugins = performancecopilot-pcp-app,pcp-valkey-datasource,pcp-vector-datasource,pcp-bpftrace-datasource,pcp-flamegraph-panel,pcp-breadcrumbs-panel,pcp-troubleshooting-panel,performancecopilot-valkey-datasource,performancecopilot-vector-datasource,performancecopilot-bpftrace-datasource,performancecopilot-flamegraph-panel,performancecopilot-breadcrumbs-panel,performancecopilot-troubleshooting-panel/' /etc/grafana/grafana.ini
$ sudo systemctl restart grafana-server
To list all available Makefile targets, run ``make help``.
2 changes: 1 addition & 1 deletion docs/pcp-redis.md → docs/pcp-valkey.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,3 @@
# Moved

This document has moved to https://grafana-pcp.readthedocs.io/en/latest/datasources/redis.html
This document has moved to https://grafana-pcp.readthedocs.io/en/latest/datasources/valkey.html
6 changes: 3 additions & 3 deletions docs/quickstart.rst
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@ Data Sources

Before using grafana-pcp, you need to configure the data sources.
Open the Grafana configuration, go to Data Sources and add the
:doc:`datasources/redis`,
:doc:`datasources/valkey`,
:doc:`datasources/vector` and/or
:doc:`datasources/bpftrace` data sources.

Expand All @@ -42,8 +42,8 @@ You can import them by navigating to the *Dashboards* tab on top of the settings
Make sure the *URL* text box actually contains a value (font color should be white) and you're not looking at the placeholder value (light grey text).

.. note::
The Redis and bpftrace data sources need additional configuration on the collector host.
See :doc:`datasources/redis` and :doc:`datasources/bpftrace`.
The Valkey and bpftrace data sources need additional configuration on the collector host.
See :doc:`datasources/valkey` and :doc:`datasources/bpftrace`.

Dashboards
----------
Expand Down
10 changes: 5 additions & 5 deletions docs/troubleshooting.rst
Original file line number Diff line number Diff line change
Expand Up @@ -16,15 +16,15 @@ HTTP Error 502: Bad Gateway, please check the datasource and pmproxy settings
* make sure that pmproxy was built with time-series (libuv) support enabled. You can verify that by reading the logfile in ``/var/log/pcp/pmproxy/pmproxy.log``


PCP Redis
PCP Valkey
---------

Grafana doesn't show any data
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

* Make sure that `pmlogger`_ is up and running, and writing archives to the disk (``/var/log/pcp/pmlogger/<host>/*``)
* Verify that `pmproxy`_ is running, time series support is enabled and a connection to Redis is established: check the logfile at ``/var/log/pcp/pmproxy/pmproxy.log`` and make sure that it contains the following text: ``Info: Redis slots, command keys, schema version setup``
* Check if the Redis database contains any keys: ``redis-cli dbsize``
* Check if any PCP metrics are in the Redis database: ``pmseries disk.dev.read``
* Check if PCP metric values are in the Redis database: ``pmseries 'disk.dev.read[count:10]'``
* Verify that `pmproxy`_ is running, time series support is enabled and a connection to Valkey is established: check the logfile at ``/var/log/pcp/pmproxy/pmproxy.log`` and make sure that it contains the following text: ``Info: Valkey slots, command keys, schema version setup``
* Check if the Valkey database contains any keys: ``valkey-cli dbsize``
* Check if any PCP metrics are in the Valkey database: ``pmseries disk.dev.read``
* Check if PCP metric values are in the Valkey database: ``pmseries 'disk.dev.read[count:10]'``
* Check the Grafana logs: ``journalctl -e -u grafana-server``
Loading

0 comments on commit 211bb5f

Please sign in to comment.