From 62ba5aaf95b3351cc6e35150a696e9ebc7789879 Mon Sep 17 00:00:00 2001 From: Stephen Lawrence Date: Mon, 6 Jan 2025 17:11:40 +0000 Subject: [PATCH 1/9] VISS/VISSR: update VISSR git submodule to commit 7447c11 (VISS v3 WIP) Currently CDSP points at a VISSR commit that implements VISS v2. Since then the upstream VISS community has been working towards VISS v3 and updated the VISSR implementation of it accordingly. Allow CDSP users to experiment with this new work by updating the CDSP git submodule to point at the latest upstream VISSR commit on the master branch. Integration was sanity tested using the VISSR HTML client to get/set VSS data into the Apache IoTDB DB. Signed-off-by: Stephen Lawrence --- cdsp/vissr | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/cdsp/vissr b/cdsp/vissr index 27d6de0..7447c11 160000 --- a/cdsp/vissr +++ b/cdsp/vissr @@ -1 +1 @@ -Subproject commit 27d6de031f69c520ee6b9be49b519768a600343b +Subproject commit 7447c113ce4ef1c077347e080221eb4747a31f68 From 36a3ce7de07a1073f11d24b4b2d064ad11393c5d Mon Sep 17 00:00:00 2001 From: Stephen Lawrence Date: Fri, 17 Jan 2025 17:30:24 +0000 Subject: [PATCH 2/9] online doc site: add VISS and VISSR examples Extend the documentation site examples page to have links to the upstream VISSR hello-world tutorial and the examples in the VISS specification. Signed-off-by: Stephen Lawrence --- docs/docs-gen/content/examples/_index.md | 15 +++++++++++---- 1 file changed, 11 insertions(+), 4 deletions(-) diff --git a/docs/docs-gen/content/examples/_index.md b/docs/docs-gen/content/examples/_index.md index cc4cb9c..6e23408 100644 --- a/docs/docs-gen/content/examples/_index.md +++ b/docs/docs-gen/content/examples/_index.md @@ -35,25 +35,32 @@ Topic examples: Virtual signal platforms, VISSR etc. |------|-------------| | [RemotiveLabs feeder](https://github.com/COVESA/cdsp/tree/main/examples/remotivelabs-feeder) | Example bridge that streams vehicle data from the RemotiveLabs cloud platform into the IoTDB data store | - ## COVESA Touchpoints Topic examples: Low level vehicle abstraction, Mobile devices, Car2Cloud / Cloud etc. ## COVESA Technologies Topic examples: vsome/ip (SOME/IP), uServices, Vehicle API, VISS etc. +| Name | Relationship to the category | +|------|-------------| +| [VISSR/VISS hello-world](https://covesa.github.io/vissr/build-system/hello-world/) | hello-world tutorial for making VISS requests using VISSR in the upstream VISSR project | +| [VISS transport examples](https://raw.githack.com/COVESA/vehicle-information-service-specification/main/spec/VISSv3.0_TransportExamples.html) | Examples in the VISS Specification for making VISS requests and the responses for various transport protocols | + ## Databases Topic examples: Apache IoTDB, MongoDB Realm, Redis/SQLite/memcache etc. - Name | Relationship to the category | +| Name | Relationship to the category | |------|-------------| | [vehicle-speed-downsample-iotdb](https://github.com/COVESA/cdsp/tree/main/examples/vehicle-speed-downsample-iotdb) | Using the IoTDB [Data Quality Library ]({{< ref "apache-iotdb#data-processing-functions" >}} "IoTDB Data Quality Library") for advanced (VSS) timeseries data processing| -| [RemotiveLabs feeder](https://github.com/COVESA/cdsp/tree/main/examples/remotivelabs-feeder) | Example of streaming (writing) southbound (VSS) timeseries data into IoTDB| - +| [RemotiveLabs feeder](https://github.com/COVESA/cdsp/tree/main/examples/remotivelabs-feeder) | Example of streaming (writing) southbound (VSS) timeseries data into IoTDB | ## Frameworks / Protocols Topic examples: vsome/ip (SOME/IP), VISS, uServices/uProtocol/Capabilities, Vehicle API, MQTT, Kafka, Apache Zeppelin etc. +| Name | Relationship to the category | +|------|-------------| +| VISS | Find examples in the COVESA Technologies section and or search for VISS/VISSR in this page | + ## Big data Topic examples: Hadoop, Flink, Spark, Cloud DB, Nifi etc. From 0748ae6fb17fbf3bcf25b38c673aefc477159b9d Mon Sep 17 00:00:00 2001 From: Stephen Lawrence Date: Fri, 17 Jan 2025 19:26:41 +0000 Subject: [PATCH 3/9] docker readme.md: Remove VISSR vss_vissv2.binary and vsspathlist notes Update the docker readme.md to reflect changes brought about by CDSP now using VISSR v3: 1. VISSR now provides a default binary list of VSS nodes so remove the note about having to generate vss_vissv2.binary 2. VISSR no longer serves a list of the VSS nodes supported at the URL http://:8081/vsspathlist. Remove the associated note. Signed-off-by: Stephen Lawrence --- docker/README.md | 9 --------- 1 file changed, 9 deletions(-) diff --git a/docker/README.md b/docker/README.md index 1913f13..439122c 100644 --- a/docker/README.md +++ b/docker/README.md @@ -50,13 +50,6 @@ There is a current issue with the upstream VISSR VISS Server Dockerfile in which If your project requires Access Grant support please discuss enabling it with the VISSR community. -### Generate `vss_vissv2.binary` -The VISSR server component requires a file called `vss_vissv2.binary` to understand the VSS tree it must work with. Unfortunately, VISSR provides no default file and you must therefore generate it yourself. - -Instructions for doing that can be found in the VISSR documentation site [here](https://covesa.github.io/vissr/server/#vss-tree-configuration) - -Tip: The playground maintainers have found that the method involving running `make binary` in a git clone of the VSS source tree to generate the file is straight forward. Note: check the VSS readme for the python requirements for the tooling. - ### Tip: Corporate CA security (download error "tls: failed to verify certificate:") If you are working behind a corporate security system that places a _man-in-the-middle_ between your host and the internet you may see security errors when artifacts are downloaded as part of the build process. @@ -172,5 +165,3 @@ The following example uses the javascript HTML client from `vissr/client/client- Server: readyState=3, status=200 Server: {"data":{"dp":{"ts":"2024-01-10T14:56:48Z","value":"Data-not-found"},"path":"Vehicle.Speed"},"ts":"2024-01-10T14:56:48Z"} ``` - -You can query what VSS nodes the server understands by asking for the VSS path list using the URL `http://localhost:8081/vsspathlist`. Entering that URL in your web browser will typically give you a graphical rendering of the JSON data returned. \ No newline at end of file From 814361cae66ad6bf5665016c058e4db1f767d403 Mon Sep 17 00:00:00 2001 From: Stephen Lawrence Date: Wed, 29 Jan 2025 14:56:49 +0000 Subject: [PATCH 4/9] docker compose: fix IoTDB build arg passing Apache IoTDB build time arguments (args) were not being correctly passed to the Dockerfile due to a capitalisation mismatch. Fix that. Signed-off-by: Stephen Lawrence --- docker/docker-compose-cdsp.yml | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docker/docker-compose-cdsp.yml b/docker/docker-compose-cdsp.yml index d882886..2f2755d 100644 --- a/docker/docker-compose-cdsp.yml +++ b/docker/docker-compose-cdsp.yml @@ -9,8 +9,8 @@ services: build: context: iotdb # CDSP IoTDB image args: - iotdb_version: 1.2.2 - service_hostname: iotdb-service + IOTDB_VERSION: 1.2.2 + SERVICE_HOSTNAME: iotdb-service restart: on-failure:3 ports: - "6667:6667" From c49e2544adbfb92baa92cd35fc4ebfe1fd4bb9bc Mon Sep 17 00:00:00 2001 From: Stephen Lawrence Date: Wed, 29 Jan 2025 15:12:09 +0000 Subject: [PATCH 5/9] docker compose: update to Apache IoTDB v1.3.3 Currently we use Apache IoTDB v1.2.2. Update to use v1.3.3 instead. Signed-off-by: Stephen Lawrence --- docker/docker-compose-cdsp.yml | 7 ++----- 1 file changed, 2 insertions(+), 5 deletions(-) diff --git a/docker/docker-compose-cdsp.yml b/docker/docker-compose-cdsp.yml index 2f2755d..1d74a7c 100644 --- a/docker/docker-compose-cdsp.yml +++ b/docker/docker-compose-cdsp.yml @@ -9,7 +9,7 @@ services: build: context: iotdb # CDSP IoTDB image args: - IOTDB_VERSION: 1.2.2 + IOTDB_VERSION: 1.3.3 SERVICE_HOSTNAME: iotdb-service restart: on-failure:3 ports: @@ -18,8 +18,7 @@ services: - cn_internal_address=iotdb-service - cn_internal_port=10710 - cn_consensus_port=10720 -# - cn_seed_config_node=iotdb-service:10710 - - cn_target_config_node_list=iotdb-service:10710 + - cn_seed_config_node=iotdb-service:10710 - dn_rpc_address=iotdb-service - dn_internal_address=iotdb-service - dn_rpc_port=6667 @@ -27,8 +26,6 @@ services: - dn_schema_region_consensus_port=10750 - dn_data_region_consensus_port=10760 - dn_seed_config_node=iotdb-service:10710 -# - dn_seed_config_node=iotdb-service:10710 - - dn_target_config_node_list=iotdb-service:10710 volumes: - ./data:/iotdb/data - ./logs:/iotdb/logs From 2b712596e6573658a49de1d8359ecca25c13b9fc Mon Sep 17 00:00:00 2001 From: Stephen Lawrence Date: Wed, 29 Jan 2025 15:28:19 +0000 Subject: [PATCH 6/9] CDSP IoTDB Dockerfile: update for IoTDB v1.3.3 The CDSP IoTDB Dockerfile extends the upstream IoTDB docker image by adding the optional UDF Data Quality library and enabling the REST API to allow Grafana connections. This commit updates the Dockerfile for changes in IoTDB v1.3.3: 1. UDF Data Quality library archive internal structure has changed 2. REST: Config iotdb-common.properties has been merged into iotdb-system.properties. The latter no longer contains all the config entries so the entry to enable the REST API must be added. Signed-off-by: Stephen Lawrence --- docker/iotdb/Dockerfile | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/docker/iotdb/Dockerfile b/docker/iotdb/Dockerfile index b69ca48..faa968e 100644 --- a/docker/iotdb/Dockerfile +++ b/docker/iotdb/Dockerfile @@ -4,7 +4,7 @@ # Dockerfile for the CDSP build of Apache IoTDB. # --------------------------------------------------------------------------- -ARG IOTDB_VERSION=1.2.2 +ARG IOTDB_VERSION=1.3.3 ARG IOTDB_IMAGE=apache/iotdb:${IOTDB_VERSION}-standalone FROM ${IOTDB_IMAGE} AS cdsp-extras @@ -15,9 +15,9 @@ ARG IOTDB_IMAGE # UDF Data Quality library ARG UDF_ARCHIVE_NAME=apache-iotdb-${IOTDB_VERSION}-library-udf-bin.zip ARG UDF_ARCHIVE_URL=https://archive.apache.org/dist/iotdb/${IOTDB_VERSION}/${UDF_ARCHIVE_NAME} -ARG UDF_LIB_PATH=apache-iotdb-${IOTDB_VERSION}-library-udf-bin/ext/udf/library-udf.jar +ARG UDF_LIB_PATH=apache-iotdb-${IOTDB_VERSION}-library-udf-bin/library-udf.jar ARG UDF_REG_SCRIPT_NAME=register-UDF.sh -ARG UDF_REG_SCRIPT_PATH=apache-iotdb-${IOTDB_VERSION}-library-udf-bin/tools/${UDF_REG_SCRIPT_NAME} +ARG UDF_REG_SCRIPT_PATH=apache-iotdb-${IOTDB_VERSION}-library-udf-bin/${UDF_REG_SCRIPT_NAME} ARG SERVICE_HOSTNAME=iotdb-service # General configuration @@ -35,7 +35,7 @@ RUN unzip -j -o ${UDF_ARCHIVE_NAME} ${UDF_LIB_PATH} ${UDF_REG_SCRIPT_PATH} \ WORKDIR ${IOTDB_HOME}/conf # Enable IoTDB REST API as it is used by the IoTDB Grafana Connector for queries -RUN sed -i 's/^# enable_rest_service=false/enable_rest_service=true/g' iotdb-common.properties +RUN echo "enable_rest_service=true" >> iotdb-system.properties # Set the entry point path to the IoTDB sbin directory where the IoTDB admin scripts are. # This mimics the upstream image. From 7c8a0a376059bdb1e3142b312b6084f9d4988065 Mon Sep 17 00:00:00 2001 From: Stephen Lawrence Date: Wed, 29 Jan 2025 15:43:00 +0000 Subject: [PATCH 7/9] docker compose: rename local volume names so they are unique to a service To ease maintenance rename local volume names so that it is clear which service they belong too. Signed-off-by: Stephen Lawrence --- docker/docker-compose-cdsp.yml | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docker/docker-compose-cdsp.yml b/docker/docker-compose-cdsp.yml index 1d74a7c..f099762 100644 --- a/docker/docker-compose-cdsp.yml +++ b/docker/docker-compose-cdsp.yml @@ -27,8 +27,8 @@ services: - dn_data_region_consensus_port=10760 - dn_seed_config_node=iotdb-service:10710 volumes: - - ./data:/iotdb/data - - ./logs:/iotdb/logs + - ./iotdb-data:/iotdb/data + - ./iotdb-logs:/iotdb/logs # VISSR acting as VISS Data Server @@ -70,6 +70,6 @@ services: - "0.0.0.0:8887:8887" - "0.0.0.0:8600:8600" volumes: - - ./logs:/app/logs + - ./vissr-logs:/app/logs volumes_from: - tmp From a3d64c2ffa50361c6b5d7b0f3c617f67763b22b8 Mon Sep 17 00:00:00 2001 From: Stephen Lawrence Date: Thu, 30 Jan 2025 18:52:43 +0000 Subject: [PATCH 8/9] doc: update for IoTDB v1.3.3 Update any documentation affected by the upgrade to Apache IoTDB v1.3.3. Accounting for changes to the upstream documentation URLs and improvements to the data import tools. Signed-off-by: Stephen Lawrence --- docker/README.md | 8 ++++---- docs/docs-gen/content/manuals/apache-iotdb.md | 11 ++++++----- .../vehicle-speed-downsample-iotdb/README.md | 18 ++++++++++-------- 3 files changed, 20 insertions(+), 17 deletions(-) diff --git a/docker/README.md b/docker/README.md index 439122c..446cf1a 100644 --- a/docker/README.md +++ b/docker/README.md @@ -123,10 +123,10 @@ $ sudo docker compose -f docker-compose-cdsp.yml down Listing should show three running containers as shown below: ``` $ sudo docker ps -NAME IMAGE COMMAND SERVICE CREATED STATUS PORTS -app_redis redis "docker-entrypoint.s…" redis 10 minutes ago Up 10 minutes 6379/tcp -iotdb-service apache/iotdb:1.2.2-standalone "/usr/bin/dumb-init …" iotdb-service 10 minutes ago Up 10 minutes 0.0.0.0:6667->6667/tcp, :::6667->6667/tcp -vissv2server cdsp-vissv2server "/app/vissv2server -…" vissv2server 10 minutes ago Up 4 seconds 0.0.0.0:8081->8081/tcp, 0.0.0.0:8600->8600/tcp, 0.0.0.0:8887->8887/tcp, 127.0.0.1:8888->8888/tcp +CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES +f43ed0c6ba0a cdsp-iotdb-service "/usr/bin/dumb-init …" About a minute ago Up About a minute 0.0.0.0:6667->6667/tcp, :::6667->6667/tcp iotdb-service +7829813bdcb8 cdsp-vissv2server "/app/vissv2server -…" 5 days ago Up 8 seconds 0.0.0.0:8081->8081/tcp, 0.0.0.0:8600->8600/tcp, 0.0.0.0:8887->8887/tcp, 127.0.0.1:8888->8888/tcp vissv2server +8e21a556e398 redis "docker-entrypoint.s…" 5 days ago Up 5 days 6379/tcp app_redis ``` #### Apache IoTDB You can confirm the Apache IoTDB server is running by connecting to it with the IoTDB CLI client (_quit_ to exit the client): diff --git a/docs/docs-gen/content/manuals/apache-iotdb.md b/docs/docs-gen/content/manuals/apache-iotdb.md index afdce0e..c84cd1d 100644 --- a/docs/docs-gen/content/manuals/apache-iotdb.md +++ b/docs/docs-gen/content/manuals/apache-iotdb.md @@ -14,7 +14,7 @@ The IoTDB project [website](https://iotdb.apache.org/) has extensive documentati Info: The intention is to add more information such as a guide for feeder integration as the project progresses. ## Integrating VSS data into the IoTDB data model -The ["Basic Concept"](https://iotdb.apache.org/UserGuide/latest/Basic-Concept/Data-Model-and-Terminology.html) section of the IoTDB documentation introduces the IoTDB data model, data types, encoding and compression. +The ["Basic Functions"](https://iotdb.apache.org/UserGuide/latest/Basic-Concept/Data-Model-and-Terminology.html) section of the IoTDB documentation introduces the IoTDB data model, data types, encoding and compression. In IoTDB terminology *measurement* is the key in a key/value pair. In VSS terms the leaf node name. The timeseries is the record of the measurement on the time axis. A timeseries is a series of time/value data points. @@ -127,19 +127,20 @@ IoTDB has an extensive collection of built-in data processing functions covering - Sampling Functions, such as `M4`. - Change Points Function, such as `CHANGE_POINTS`. -A full function list with examples can be found in the upstream [IoTDB Function reference manual](https://iotdb.apache.org/UserGuide/latest/Reference/Function-and-Expression.html). +A full function list with examples can be found in the upstream [IoTDB Function reference manual](https://iotdb.apache.org/UserGuide/latest/SQL-Manual/Function-and-Expression.html). ### Data Quality Library functions The IoTDB project also maintains a Data Quality Library which provides an additional collection of functions covering: -- Data Quality, such as `Accuracy`. +- Data Quality, such as `Completeness`. - Data Profiling, such as `Sample`. - Anomaly Detection, such as `Outlier`. - Frequency Domain Analysis, such as `HighPass`. +- Data Matching, such as `Cov`. - Data Repair, such as `TimestampRepair`. - Series Discovery, such as `ConsecutiveSequences`. - Machine Learning, such as `AR`. -A full function list with examples can be found in the upstream [IoTDB Data Quality Library reference manual](https://iotdb.apache.org/UserGuide/latest/Reference/UDF-Libraries.html). +A full function list with examples can be found in the upstream [IoTDB UDF Data Quality Library reference manual](https://iotdb.apache.org/UserGuide/latest/SQL-Manual/UDF-Libraries_apache.html). #### Setup In the upstream IoTDB project the library is an optional install. @@ -156,7 +157,7 @@ Steps: ### User Defined functions -IoTDB also allows you intergrate your own functions as User Defined Functions (UDF). The [UDF section](https://iotdb.apache.org/UserGuide/latest/User-Manual/Database-Programming.html#user-defined-function-udf) of the IoTDB documentation explains how to develop and register your own. +IoTDB also allows you to integrate your own functions as User Defined Functions (UDF). The [UDF section](https://iotdb.apache.org/UserGuide/latest/User-Manual/UDF-development.html) of the IoTDB documentation explains how to develop and register your own. ## VISSR (VISS) integration As part of the initial development of the playground the team extended VISSR to support connections to Apache IoTDB as a VISSR data store backend and upstreamed the support. diff --git a/examples/vehicle-speed-downsample-iotdb/README.md b/examples/vehicle-speed-downsample-iotdb/README.md index e678fb2..1574084 100644 --- a/examples/vehicle-speed-downsample-iotdb/README.md +++ b/examples/vehicle-speed-downsample-iotdb/README.md @@ -35,10 +35,10 @@ The dataset contains just over 13500 values recorded over approximately 5 minute ### Load In the example we will import the dataset into a VSS `Vehicle.Speed` timeseries within IoTDB. -IoTDB provides [import/export tools](https://iotdb.apache.org/UserGuide/latest/Tools-System/TsFile-Import-Export-Tool.html) for its native TsFile file format and CSV and which are included in the image. We will use the [`import-csv.sh`](https://iotdb.apache.org/UserGuide/latest/Tools-System/TsFile-Import-Export-Tool.html#usage-of-import-csv-sh) tool to perform the import. +IoTDB provides import/export tools for its native TsFile file format, SQL and CSV and which are included in the runtime image. We will use the [`import-data.sh`](https://iotdb.apache.org/UserGuide/latest/Tools-System/Data-Import-Tool.html) tool to perform the import. ### Transform -IoTDB has a library of Data Quality functions which includes the function [`Sample`](https://iotdb.apache.org/UserGuide/latest/Reference/UDF-Libraries.html#sample) for sampling. Sample has three sampling methods: `Reservoir`, `Isometric` and `Triangle`. +IoTDB has a library of Data Quality functions which includes the function [`Sample`](https://iotdb.apache.org/UserGuide/latest/SQL-Manual/UDF-Libraries_apache.html#sample) for sampling. Sample has three sampling methods: `Reservoir`, `Isometric` and `Triangle`. Sample function: @@ -64,7 +64,7 @@ The following screenshot shows the results graphed in Grafana. The green graph i *Figure 1: Grafana visualisation of the results. Key: Green=input data, yellow=down-sampled result* Tips: -+ Documentation for each Sample method and the other functions can be found in the [IoTDB library documentation](https://iotdb.apache.org/UserGuide/latest/Reference/UDF-Libraries.html). ++ Documentation for each Sample method and the other functions can be found in the [IoTDB library documentation](https://iotdb.apache.org/UserGuide/latest/SQL-Manual/UDF-Libraries_apache.html). + Triangle uses a *Largest-Triangle-Three-Buckets (LTTB)* algorithm to calculate the output timeseries. Details of which can be found in the originating academic research: [Downsampling Time Series for Visual Representation, Sveinn Steinarsson, 2013](https://skemman.is/bitstream/1946/15343/3/SS_MSthesis.pdf) @@ -106,6 +106,8 @@ $ sudo docker exec -ti iotdb-service /iotdb/sbin/register-UDF.sh Info: See the online documentation site for [details](https://covesa.github.io/cdsp/manuals/apache-iotdb/#setup) +Tip: If for some reason you are failing to register the Data Quality Library, substitute one of the [built-in sample functions](https://iotdb.apache.org/UserGuide/latest/SQL-Manual/Operator-and-Expression.html#sample-functions) rather than `Sample()` in the SQL below. + ### Import the dataset 1. Create the database in IoTDB into which we will import @@ -120,19 +122,19 @@ Info: See the online documentation site for [details](https://covesa.github.io/c 2.1. Copy the dataset file to a volume visible within the IoTDB image: ~~~ - $ cp vehicle_speed_rl_dataset.csv ../../docker/data + $ sudo cp vehicle_speed_rl_dataset.csv ../../docker/iotdb-data ~~~ 2.2. Import the dataset - To import the dataset into IoTDB we need to execute the `import-csv.sh` tool in the IoTDB image. That can be done from the host terminal using the docker `exec` command. + To import the dataset into IoTDB we need to execute the `import-data.sh` tool in the IoTDB image. That can be done from the host terminal using the docker `exec` command. ~~~ - $ docker exec -ti iotdb-service /iotdb/tools/import-csv.sh -h iotdb-service -p 6667 -u root -pw root -f /iotdb/data/vehicle_speed_rl_dataset.csv + $ sudo docker exec -ti iotdb-service /iotdb/tools/import-data.sh -h iotdb-service -p 6667 -u root -pw root -s /iotdb/data/vehicle_speed_rl_dataset.csv ~~~ Example successful execution: ~~~ - $ docker exec -ti iotdb-service /iotdb/tools/import-csv.sh -h iotdb-service -p 6667 -u root -pw root -f /iotdb/data/vehicle_speed_rl_dataset.csv + $ sudo docker exec -ti iotdb-service /iotdb/tools/import-data.sh -h iotdb-service -p 6667 -u root -pw root -s /iotdb/data/vehicle_speed_rl_dataset.csv ------------------------------------------ Starting IoTDB Client Import Script ------------------------------------------ @@ -325,7 +327,7 @@ It costs 0.028s ### Suggested next steps + Repeat the query with the same input data, but with different `k` parameter values to see what affects the number of samples has on the accuracy of the trace compared to the input data. -+ Amend the query with other [SQL clauses](https://iotdb.apache.org/UserGuide/latest/User-Manual/Query-Data.html) to shape what you are interested in, e.g. use `WHERE` to define a time filter. ++ Amend the query with other [SQL clauses](https://iotdb.apache.org/UserGuide/latest/Basic-Concept/Query-Data.html) to shape what you are interested in, e.g. use `WHERE` to define a time filter. + A great way to explore these sampling queries is by using Grafana as discussed below. + Try the other [IoTDB data processing functions](https://covesa.github.io/cdsp/manuals/apache-iotdb/#data-processing-functions) From ea7cd6a167f2fd3772d432565374e26373484f4a Mon Sep 17 00:00:00 2001 From: Stephen Lawrence Date: Fri, 31 Jan 2025 19:32:26 +0000 Subject: [PATCH 9/9] doc: minor updates to hello-world example and Apache IoTDB online doc. Minor improvements to the listed documentation. Signed-off-by: Stephen Lawrence --- docs/docs-gen/content/manuals/apache-iotdb.md | 4 ++-- examples/cdsp-hello-world/README.md | 5 +++++ 2 files changed, 7 insertions(+), 2 deletions(-) diff --git a/docs/docs-gen/content/manuals/apache-iotdb.md b/docs/docs-gen/content/manuals/apache-iotdb.md index c84cd1d..4314aee 100644 --- a/docs/docs-gen/content/manuals/apache-iotdb.md +++ b/docs/docs-gen/content/manuals/apache-iotdb.md @@ -18,7 +18,7 @@ The ["Basic Functions"](https://iotdb.apache.org/UserGuide/latest/Basic-Concept/ In IoTDB terminology *measurement* is the key in a key/value pair. In VSS terms the leaf node name. The timeseries is the record of the measurement on the time axis. A timeseries is a series of time/value data points. -The IoTDB data model supports hierarchical partitioning and like VSS uses a dot notation to separate the levels. This means if we simply appended a VSS leaf node name like `Vehicle.CurrentLocation.Longitude` as the measurement (key) name to the end of a IoTDB path such as `root.test2.dev1` the `Vehicle.CurrentLocation.` IoTDB would treat it as part of the IoTDB data model partitioning which could cause unwanted issues when scaling over millions of vehicles. +The IoTDB data model supports hierarchical partitioning and like VSS uses a dot notation to separate the levels. This means if we simply appended a VSS leaf node name like `Vehicle.CurrentLocation.Longitude` as the measurement (key) name to the end of a IoTDB path such as `root.test2.dev1` IoTDB would treat the VSS path prefix `Vehicle.CurrentLocation.` as part of the IoTDB data model partitioning which could cause unwanted issues when scaling over millions of vehicles. We have separated those two concepts by quoting the VSS leaf node name using backticks when processing the name in IoTDB. As shown below: ``` @@ -157,7 +157,7 @@ Steps: ### User Defined functions -IoTDB also allows you to integrate your own functions as User Defined Functions (UDF). The [UDF section](https://iotdb.apache.org/UserGuide/latest/User-Manual/UDF-development.html) of the IoTDB documentation explains how to develop and register your own. +IoTDB also allows you to integrate your own functions as User Defined Functions (UDF). The [UDF development section](https://iotdb.apache.org/UserGuide/latest/User-Manual/UDF-development.html) of the IoTDB documentation explains how to develop and register your own. ## VISSR (VISS) integration As part of the initial development of the playground the team extended VISSR to support connections to Apache IoTDB as a VISSR data store backend and upstreamed the support. diff --git a/examples/cdsp-hello-world/README.md b/examples/cdsp-hello-world/README.md index 78b1272..2e00a4c 100644 --- a/examples/cdsp-hello-world/README.md +++ b/examples/cdsp-hello-world/README.md @@ -1,9 +1,14 @@ # Central Data Service Playground Hello World + +> [!NOTE] +> Apologies. As you can see the team has been busy adding features and not got around to finishing this example yet. In the meantime, please take a look at the deployment section of the [Docker README.md](https://github.com/COVESA/cdsp/blob/main/docker/README.md#deploy-with-docker-compose) which gives guidance on starting the Playground and sanity testing it is running correctly. + This is a simple "hello-world" example for the Central Data Service Playground (CDSP). Goals: 1. Act as a sanity test that the Playground is running correctly 2. Provide a simplistic example of VSS and VISS use in the Playground. + ## Logical/Implementation Concept This example does not expand the concept of the Playground. Therefore please see the Playground documentation for a description of the Logical and Implementation concepts behind it.