From fd81b1bb90d546d35cbe1ba98f727a0ee070a91b Mon Sep 17 00:00:00 2001 From: J Stickler Date: Wed, 4 Dec 2024 10:52:56 -0500 Subject: [PATCH] docs: fixing some Vale linter errors (#15212) (#15259) --- docs/sources/configure/bp-configure.md | 10 +- docs/sources/configure/storage.md | 6 +- docs/sources/setup/upgrade/_index.md | 389 +++++++++++++------------ 3 files changed, 203 insertions(+), 202 deletions(-) diff --git a/docs/sources/configure/bp-configure.md b/docs/sources/configure/bp-configure.md index 28feb68b1ba4c..3d00867201815 100644 --- a/docs/sources/configure/bp-configure.md +++ b/docs/sources/configure/bp-configure.md @@ -6,7 +6,7 @@ weight: 100 --- # Configuration best practices -Grafana Loki is under active development, and we are constantly working to improve performance. But here are some of the most current best practices for configuration that will give you the best experience with Loki. +Grafana Loki is under active development, and the Loki team is constantly working to improve performance. But here are some of the most current best practices for configuration that will give you the best experience with Loki. ## Configure caching @@ -36,7 +36,7 @@ If Loki received these two lines which are for the same stream, everything would {job="syslog"} 00:00:01 i'm a syslog! <- Rejected out of order! ``` -What can we do about this? What if this was because the sources of these logs were different systems? We can solve this with an additional label which is unique per system: +What can you do about this? What if this was because the sources of these logs were different systems? You can solve this with an additional label which is unique per system: ``` {job="syslog", instance="host1"} 00:00:00 i'm a syslog! @@ -56,9 +56,9 @@ Using `chunk_target_size` instructs Loki to try to fill all chunks to a target _ Other configuration variables affect how full a chunk can get. Loki has a default `max_chunk_age` of 2h and `chunk_idle_period` of 30m to limit the amount of memory used as well as the exposure of lost logs if the process crashes. -Depending on the compression used (we have been using snappy which has less compressibility but faster performance), you need 5-10x or 7.5-10MB of raw log data to fill a 1.5MB chunk. Remembering that a chunk is per stream, the more streams you break up your log files into, the more chunks that sit in memory, and the higher likelihood they get flushed by hitting one of those timeouts mentioned above before they are filled. +Depending on the compression used (Loki has been using snappy which has less compressibility but faster performance), you need 5-10x or 7.5-10MB of raw log data to fill a 1.5MB chunk. Remembering that a chunk is per stream, the more streams you break up your log files into, the more chunks that sit in memory, and the higher likelihood they get flushed by hitting one of those timeouts mentioned above before they are filled. -Lots of small, unfilled chunks negatively affect Loki. We are always working to improve this and may consider a compactor to improve this in some situations. But, in general, the guidance should stay about the same: try your best to fill chunks. +Lots of small, unfilled chunks negatively affect Loki. The team is always working to improve this and may consider a compactor to improve this in some situations. But, in general, the guidance should stay about the same: try your best to fill chunks. If you have an application that can log fast enough to fill these chunks quickly (much less than `max_chunk_age`), then it becomes more reasonable to use dynamic labels to break that up into separate streams. @@ -68,4 +68,4 @@ Loki and Promtail have flags which will dump the entire config object to stderr `-print-config-stderr` works well when invoking Loki from the command line, as you can get a quick output of the entire Loki configuration. -`-log-config-reverse-order` is the flag we run Loki with in all our environments. The configuration entries are reversed, so that the order of the configuration reads correctly top to bottom when viewed in Grafana's Explore. +`-log-config-reverse-order` is the flag Grafana runs Loki with in all our environments. The configuration entries are reversed, so that the order of the configuration reads correctly top to bottom when viewed in Grafana's Explore. diff --git a/docs/sources/configure/storage.md b/docs/sources/configure/storage.md index c0178b67bdcbb..276834a8f8445 100644 --- a/docs/sources/configure/storage.md +++ b/docs/sources/configure/storage.md @@ -193,7 +193,7 @@ When a new schema is released and you want to gain the advantages it provides, y First, you'll want to create a new [period_config](https://grafana.com/docs/loki//configure/#period_config) entry in your [schema_config](https://grafana.com/docs/loki//configure/#schema_config). The important thing to remember here is to set this at some point in the _future_ and then roll out the config file changes to Loki. This allows the table manager to create the required table in advance of writes and ensures that existing data isn't queried as if it adheres to the new schema. -As an example, let's say it's 2023-07-14 and we want to start using the `v13` schema on the 20th: +As an example, let's say it's 2023-07-14 and you want to start using the `v13` schema on the 20th: ```yaml schema_config: @@ -214,7 +214,7 @@ schema_config: period: 24h ``` -It's that easy; we just created a new entry starting on the 20th. +It's that easy; you just created a new entry starting on the 20th. ## Retention @@ -485,7 +485,7 @@ schema_config: ### On premise deployment (MinIO Single Store) -We configure MinIO by using the AWS config because MinIO implements the S3 API: +You configure MinIO by using the AWS config because MinIO implements the S3 API: ```yaml storage_config: diff --git a/docs/sources/setup/upgrade/_index.md b/docs/sources/setup/upgrade/_index.md index 1c82cb6c349fe..0cd6bd6f192bf 100644 --- a/docs/sources/setup/upgrade/_index.md +++ b/docs/sources/setup/upgrade/_index.md @@ -22,7 +22,7 @@ If possible try to stay current and do sequential updates. If you want to skip v Using docker you can check changes between 2 versions of Loki with a command like this: -``` +```bash export OLD_LOKI=2.9.4 export NEW_LOKI=3.0.0 export CONFIG_FILE=local-config.yaml @@ -33,7 +33,6 @@ The `tr -d '\r'` is likely not necessary for most people, seems like WSL2 was sn The output is incredibly verbose as it shows the entire internal config struct used to run Loki, you can play around with the diff command if you prefer to only show changes or a different style output. - ## Main / Unreleased ### HTTP API @@ -76,10 +75,9 @@ Their YAML counterparts in the `limits_config` block are kept identical. All other CLI arguments (and their YAML counterparts) prefixed with `-bloom-compactor.` have been removed. - ## 3.3.0 -### Loki +### Loki 3.3.0 #### Experimental Bloom Filters @@ -104,16 +102,18 @@ Loki 3.0 is a major version increase and comes with several breaking changes. Here is the shortlist of things we think most people may encounter: - * Structured metadata is enabled by default and requires `tsdb` and `v13` schema or Loki won't start. Refer to [Structured Metadata, Open Telemetry, Schemas and Indexes](#structured-metadata-open-telemetry-schemas-and-indexes). - * The `shared_store` config is removed. Refer to [Removed `shared_store` and `shared_store_key_prefix` from shipper configuration](#removed-shared_store-and-shared_store_key_prefix-from-shipper-configuration). - * Loki now enforces a max line size of 256KB by default (you can disable this or increase this but this is how Grafana Labs runs Loki). Refer to [Changes to default configure values](#changes-to-default-configuration-values-in-30). - * Loki now enforces a max label limit of 15 labels per series, down from 30. Extra labels inflate the size of the index and reduce performance, you should almost never need more than 15 labels. Refer to [Changes to default configure values](#changes-to-default-configuration-values-in-30). - * Loki will automatically attempt to populate a `service_name` label on ingestion. Refer to [`service_name` label](#service_name-label). - * There are many metric name changes. Refer to [Distributor metric changes](#distributor-metric-changes), [Embedded cache metric changes](#embedded-cache-metric-changes), and [Metrics namespace](#metrics-namespace). +* Structured metadata is enabled by default and requires `tsdb` and `v13` schema or Loki won't start. Refer to [Structured Metadata, Open Telemetry, Schemas and Indexes](#structured-metadata-open-telemetry-schemas-and-indexes). +* The `shared_store` config is removed. Refer to [Removed `shared_store` and `shared_store_key_prefix` from shipper configuration](#removed-shared_store-and-shared_store_key_prefix-from-shipper-configuration). +* Loki now enforces a max line size of 256KB by default (you can disable this or increase this but this is how Grafana Labs runs Loki). Refer to [Changes to default configure values](#changes-to-default-configuration-values-in-30). +* Loki now enforces a max label limit of 15 labels per series, down from 30. Extra labels inflate the size of the index and reduce performance, you should almost never need more than 15 labels. Refer to [Changes to default configure values](#changes-to-default-configuration-values-in-30). +* Loki will automatically attempt to populate a `service_name` label on ingestion. Refer to [`service_name` label](#service_name-label). +* There are many metric name changes. Refer to [Distributor metric changes](#distributor-metric-changes), [Embedded cache metric changes](#embedded-cache-metric-changes), and [Metrics namespace](#metrics-namespace). If you would like to see if your existing configuration will work with Loki 3.0: + 1. In an empty directory on your computer, copy you configuration into a file named `loki-config.yaml`. -1. Run this command from that directory: +1. Run this command from that directory: + ```bash docker run --rm -t -v "${PWD}":/config grafana/loki:3.0.0 -config.file=/config/loki-config.yaml -verify-config=true ``` @@ -126,16 +126,16 @@ If you introduce a new schema_config entry it may cause additional validation er If you configure `path_prefix` in the `common` config section this can help save a lot of configuration. Refer to the [Common Config Docs](https://grafana.com/docs/loki//configure/#common). {{< /admonition >}} - The **Helm chart** has gone through some significant changes and has a separate upgrade guide: [Upgrading to Helm 6.x](https://grafana.com/docs/loki//setup/upgrade/upgrade-to-6x/). -### Loki +### Loki 3.0.0 #### Structured Metadata, Open Telemetry, Schemas and Indexes A flagship feature of Loki 3.0 is native support for the Open Telemetry Protocol (OTLP). This is made possible by a new feature in Loki called [Structured Metadata](https://grafana.com/docs/loki//get-started/labels/structured-metadata/), a place for metadata which doesn't belong in labels or log lines. OTel resources and attributes are often a great example of data which doesn't belong in the index nor in the log line. Structured Metadata is enabled by default in Loki 3.0, however, it requires your active schema be using both the `tsdb` index type AND the `v13` storage schema. If you are not using both of these you have two options: + * Upgrade your index version and schema version before updating to 3.0, see [schema config upgrade](https://grafana.com/docs/loki//operations/storage/schema/). * Disable Structured Metadata (and therefore OTLP support) and upgrade to 3.0 and perform the schema migration after. This can be done by setting `allow_structured_metadata: false` in the `limits_config` section or set the command line argument `-validation.allow-structured-metadata=false`. @@ -145,17 +145,17 @@ Loki 3.0 will automatically assign a `service_name` label to all ingested logs b Loki will attempt to create the `service_name` label by looking for the following labels in this order: - - service_name - - service - - app - - application - - name - - app_kubernetes_io_name - - container - - container_name - - component - - workload - - job +- service_name +- service +- app +- application +- name +- app_kubernetes_io_name +- container +- container_name +- component +- workload +- job If no label is found matching the list, a value of `unknown_service` is applied. @@ -170,6 +170,7 @@ If you are already using a `service_label`, Loki will not make a new assignment. #### Removed `shared_store` and `shared_store_key_prefix` from shipper configuration The following CLI flags and the corresponding YAML settings to configure shared store for TSDB and BoltDB shippers are now removed: + - `-boltdb.shipper.shared-store` - `-tsdb.shipper.shared-store` @@ -180,9 +181,8 @@ We are removing the shared store setting in an effort to simplify storage config {{% admonition type="warning" %}} With this change Loki no longer allows storing chunks and indexes for a given period in different storage buckets. -This is a breaking change for setups that store chunks and indexes in different storage buckets by setting `-boltdb.shipper.shared-store` or `-tsdb.shipper.shared-store` to a value -different from `object_store` in `period_config`. -{{% /admonition %}} +This is a breaking change for setups that store chunks and indexes in different storage buckets by setting `-boltdb.shipper.shared-store` or `-tsdb.shipper.shared-store` to a value different from `object_store` in `period_config`. +{{< /admonition >}} - If you have not configured `-boltdb.shipper.shared-store`,`-tsdb.shipper.shared-store` or their corresponding YAML setting before, no changes are required as part of the upgrade. - If you have configured `-boltdb.shipper.shared-store` or its YAML setting: @@ -195,12 +195,14 @@ different from `object_store` in `period_config`. - To make these indexes queryable, index tables need to moved or copied to the store configured in `object_store`. The following CLI flags and the corresponding YAML settings to configure a path prefix for TSDB and BoltDB shippers are now removed: + - `-boltdb.shipper.shared-store.key-prefix` - `-tsdb.shipper.shared-store.key-prefix` Path prefix for storing the index can now be configured by setting `path_prefix` under `index` key in [period_config](/docs/loki//configure/#period_config). This enables users to change the path prefix by adding a new period config. -``` + +```yaml period_config: index: path_prefix: "index/" @@ -220,6 +222,7 @@ period_config: #### Removed `shared_store` and `shared_store_key_prefix` from compactor configuration The following CLI flags and the corresponding YAML settings to configure the shared store and path prefix for compactor are now removed: + - `-boltdb.shipper.compactor.shared-store` - `-boltdb.shipper.compactor.shared-store.key-prefix` @@ -243,16 +246,17 @@ additionally the `Background` configuration also lest you set `writeback_size_li The already deprecated handler `/ingester/flush_shutdown` is removed in favor of `/ingester/shutdown?flush=true`. -#### Ingester configuration `max_transfer_retries` is removed. +#### Ingester configuration `max_transfer_retries` is removed The setting `max_transfer_retries` (`-ingester.max-transfer-retries`) is removed in favor of the Write Ahead log (WAL). It was used to allow transferring chunks to new ingesters when the old ingester was shutting down during a rolling restart. Alternatives to this setting are: + - **A. (Preferred)** Enable the WAL and rely on the new ingester to replay the WAL. - Optionally, you can enable `flush_on_shutdown` (`-ingester.flush-on-shutdown`) to flush to long-term storage on shutdowns. - **B.** Manually flush during shutdowns via [the ingester `/shutdown?flush=true` endpoint](https://grafana.com/docs/loki//reference/api/#flush-in-memory-chunks-and-shut-down). -#### Removed the `default` section of the runtime overrides config file. +#### Removed the `default` section of the runtime overrides config file This was introduced in 2.9 and likely not widely used. This only affects you if you run Loki with a runtime config file AND you had populated the new `default` block added in 2.9. @@ -283,19 +287,6 @@ The previous default value `false` is applied. 1. `boltdb.shipper.compactor.deletion-mode` CLI flag and the corresponding YAML setting are removed. You can instead configure the `compactor.deletion-mode` CLI flag or `deletion_mode` YAML setting in [Limits Config](/docs/loki//configuration/#limits_config). 1. Compactor CLI flags that use the prefix `boltdb.shipper.compactor.` are removed. You can instead use CLI flags with the `compactor.` prefix. -#### Legacy ingester shutdown handler is removed - -The already deprecated handler `/ingester/flush_shutdown` is removed in favor of `/ingester/shutdown?flush=true`. - -#### Ingester configuration `max_transfer_retries` is removed. - -The setting `max_transfer_retries` (`-ingester.max-transfer-retries`) is removed in favor of the Write Ahead log (WAL). -It was used to allow transferring chunks to new ingesters when the old ingester was shutting down during a rolling restart. -Alternatives to this setting are: -- **A. (Preferred)** Enable the WAL and rely on the new ingester to replay the WAL. - - Optionally, you can enable `flush_on_shutdown` (`-ingester.flush-on-shutdown`) to flush to long-term storage on shutdowns. -- **B.** Manually flush during shutdowns via [the ingester `/shutdown?flush=true` endpoint](https://grafana.com/docs/loki//reference/loki-http-api#flush-in-memory-chunks-and-shut-down). - #### Distributor metric changes The `loki_distributor_ingester_append_failures_total` metric has been removed in favour of `loki_distributor_ingester_append_timeouts_total`. @@ -303,9 +294,10 @@ This new metric will provide a more clear signal that there is an issue with ing #### Changes to default configuration values in 3.0 -{{% responsive-table %}} +{{< responsive-table >}} + | configuration | new default | old default | notes | -| ------------------------------------------------------ | ----------- | ----------- | -------- +| ------------------------------------------------------ | ----------- | ----------- | --------| | `compactor.delete-max-interval` | 24h | 0 | splits the delete requests into intervals no longer than `delete_max_interval` | | `distributor.max-line-size` | 256KB | 0 | - | | `ingester.sync-period` | 1h | 0 | ensures that the chunk cuts for a given stream are synchronized across the ingesters in the replication set. Helps with deduplicating chunks. | @@ -323,7 +315,8 @@ This new metric will provide a more clear signal that there is an issue with ing | `query-scheduler.max-outstanding-requests-per-tenant` | 32000 | 100 | - | | `validation.max-label-names-per-series` | 15 | 30 | - | | `legacy-read-mode` | false | true | Deprecated. It will be removed in the next minor release. | -{{% /responsive-table %}} + +{{< /responsive-table >}} #### Automatic stream sharding is enabled by default @@ -338,6 +331,7 @@ The TSDB index type has support for caching results for 'stats' and 'volume' que All of these are cached to the `results_cache` which is configured in the `query_range` config section. By default, an in memory cache is used. #### Write dedupe cache is deprecated + Write dedupe cache is deprecated because it not required by the newer single store indexes ([TSDB](https://grafana.com/docs/loki//operations/storage/tsdb/) and [boltdb-shipper](https://grafana.com/docs/loki//operations/storage/boltdb-shipper/)). If you using a [legacy index type](https://grafana.com/docs/loki//configure/storage/#index-storage), consider migrating to TSDB (recommended). @@ -360,74 +354,73 @@ If you using a [legacy index type](https://grafana.com/docs/loki// Some Loki metrics started with the prefix `cortex_`. In this release they will be changed so they start with `loki_`. To keep them at `cortex_` change the `metrics_namespace` from the default `loki` to `cortex`. These metrics will be changed: - - `cortex_distributor_ingester_clients` - - `cortex_dns_failures_total` - - `cortex_dns_lookups_total` - - `cortex_dns_provider_results` - - `cortex_frontend_query_range_duration_seconds_bucket` - - `cortex_frontend_query_range_duration_seconds_count` - - `cortex_frontend_query_range_duration_seconds_sum` - - `cortex_ingester_flush_queue_length` - - `cortex_kv_request_duration_seconds_bucket` - - `cortex_kv_request_duration_seconds_count` - - `cortex_kv_request_duration_seconds_sum` - - `cortex_member_consul_heartbeats_total` - - `cortex_prometheus_last_evaluation_samples` - - `cortex_prometheus_notifications_alertmanagers_discovered` - - `cortex_prometheus_notifications_dropped_total` - - `cortex_prometheus_notifications_errors_total` - - `cortex_prometheus_notifications_latency_seconds` - - `cortex_prometheus_notifications_latency_seconds_count` - - `cortex_prometheus_notifications_latency_seconds_sum` - - `cortex_prometheus_notifications_queue_capacity` - - `cortex_prometheus_notifications_queue_length` - - `cortex_prometheus_notifications_sent_total` - - `cortex_prometheus_rule_evaluation_duration_seconds` - - `cortex_prometheus_rule_evaluation_duration_seconds_count` - - `cortex_prometheus_rule_evaluation_duration_seconds_sum` - - `cortex_prometheus_rule_evaluation_failures_total` - - `cortex_prometheus_rule_evaluations_total` - - `cortex_prometheus_rule_group_duration_seconds` - - `cortex_prometheus_rule_group_duration_seconds_count` - - `cortex_prometheus_rule_group_duration_seconds_sum` - - `cortex_prometheus_rule_group_interval_seconds` - - `cortex_prometheus_rule_group_iterations_missed_total` - - `cortex_prometheus_rule_group_iterations_total` - - `cortex_prometheus_rule_group_last_duration_seconds` - - `cortex_prometheus_rule_group_last_evaluation_timestamp_seconds` - - `cortex_prometheus_rule_group_rules` - - `cortex_query_frontend_connected_schedulers` - - `cortex_query_frontend_queries_in_progress` - - `cortex_query_frontend_retries_bucket` - - `cortex_query_frontend_retries_count` - - `cortex_query_frontend_retries_sum` - - `cortex_query_scheduler_connected_frontend_clients` - - `cortex_query_scheduler_connected_querier_clients` - - `cortex_query_scheduler_inflight_requests` - - `cortex_query_scheduler_inflight_requests_count` - - `cortex_query_scheduler_inflight_requests_sum` - - `cortex_query_scheduler_queue_duration_seconds_bucket` - - `cortex_query_scheduler_queue_duration_seconds_count` - - `cortex_query_scheduler_queue_duration_seconds_sum` - - `cortex_query_scheduler_queue_length` - - `cortex_query_scheduler_running` - - `cortex_ring_member_heartbeats_total` - - `cortex_ring_member_tokens_owned` - - `cortex_ring_member_tokens_to_own` - - `cortex_ring_members` - - `cortex_ring_oldest_member_timestamp` - - `cortex_ring_tokens_total` - - `cortex_ruler_client_request_duration_seconds_bucket` - - `cortex_ruler_client_request_duration_seconds_count` - - `cortex_ruler_client_request_duration_seconds_sum` - - `cortex_ruler_clients` - - `cortex_ruler_config_last_reload_successful` - - `cortex_ruler_config_last_reload_successful_seconds` - - `cortex_ruler_config_updates_total` - - `cortex_ruler_managers_total` - - `cortex_ruler_ring_check_errors_total` - - `cortex_ruler_sync_rules_total` - +- `cortex_distributor_ingester_clients` +- `cortex_dns_failures_total` +- `cortex_dns_lookups_total` +- `cortex_dns_provider_results` +- `cortex_frontend_query_range_duration_seconds_bucket` +- `cortex_frontend_query_range_duration_seconds_count` +- `cortex_frontend_query_range_duration_seconds_sum` +- `cortex_ingester_flush_queue_length` +- `cortex_kv_request_duration_seconds_bucket` +- `cortex_kv_request_duration_seconds_count` +- `cortex_kv_request_duration_seconds_sum` +- `cortex_member_consul_heartbeats_total` +- `cortex_prometheus_last_evaluation_samples` +- `cortex_prometheus_notifications_alertmanagers_discovered` +- `cortex_prometheus_notifications_dropped_total` +- `cortex_prometheus_notifications_errors_total` +- `cortex_prometheus_notifications_latency_seconds` +- `cortex_prometheus_notifications_latency_seconds_count` +- `cortex_prometheus_notifications_latency_seconds_sum` +- `cortex_prometheus_notifications_queue_capacity` +- `cortex_prometheus_notifications_queue_length` +- `cortex_prometheus_notifications_sent_total` +- `cortex_prometheus_rule_evaluation_duration_seconds` +- `cortex_prometheus_rule_evaluation_duration_seconds_count` +- `cortex_prometheus_rule_evaluation_duration_seconds_sum` +- `cortex_prometheus_rule_evaluation_failures_total` +- `cortex_prometheus_rule_evaluations_total` +- `cortex_prometheus_rule_group_duration_seconds` +- `cortex_prometheus_rule_group_duration_seconds_count` +- `cortex_prometheus_rule_group_duration_seconds_sum` +- `cortex_prometheus_rule_group_interval_seconds` +- `cortex_prometheus_rule_group_iterations_missed_total` +- `cortex_prometheus_rule_group_iterations_total` +- `cortex_prometheus_rule_group_last_duration_seconds` +- `cortex_prometheus_rule_group_last_evaluation_timestamp_seconds` +- `cortex_prometheus_rule_group_rules` +- `cortex_query_frontend_connected_schedulers` +- `cortex_query_frontend_queries_in_progress` +- `cortex_query_frontend_retries_bucket` +- `cortex_query_frontend_retries_count` +- `cortex_query_frontend_retries_sum` +- `cortex_query_scheduler_connected_frontend_clients` +- `cortex_query_scheduler_connected_querier_clients` +- `cortex_query_scheduler_inflight_requests` +- `cortex_query_scheduler_inflight_requests_count` +- `cortex_query_scheduler_inflight_requests_sum` +- `cortex_query_scheduler_queue_duration_seconds_bucket` +- `cortex_query_scheduler_queue_duration_seconds_count` +- `cortex_query_scheduler_queue_duration_seconds_sum` +- `cortex_query_scheduler_queue_length` +- `cortex_query_scheduler_running` +- `cortex_ring_member_heartbeats_total` +- `cortex_ring_member_tokens_owned` +- `cortex_ring_member_tokens_to_own` +- `cortex_ring_members` +- `cortex_ring_oldest_member_timestamp` +- `cortex_ring_tokens_total` +- `cortex_ruler_client_request_duration_seconds_bucket` +- `cortex_ruler_client_request_duration_seconds_count` +- `cortex_ruler_client_request_duration_seconds_sum` +- `cortex_ruler_clients` +- `cortex_ruler_config_last_reload_successful` +- `cortex_ruler_config_last_reload_successful_seconds` +- `cortex_ruler_config_updates_total` +- `cortex_ruler_managers_total` +- `cortex_ruler_ring_check_errors_total` +- `cortex_ruler_sync_rules_total` The `metrics_namespace` setting is deprecated already. It will be removed in the next minor release. The default prefix will be `loki` then. @@ -440,7 +433,7 @@ A new CLI flag `-schema-store` is introduced as a replacement to configure the s ## 2.9.0 -### Loki +### Loki 2.9.0 #### Index gateway shuffle sharding @@ -475,6 +468,7 @@ in the logs on startup and the `/ingester/prepare_shutdown` endpoint will return #### Compactor multi-store support In previous releases, setting `-boltdb.shipper.compactor.shared-store` configured the following: + - store used for managing delete requests. - store on which index compaction should be performed. @@ -483,11 +477,13 @@ If `-boltdb.shipper.compactor.shared-store` was not set, it used to default to t Compactor now supports index compaction on multiple buckets/object stores. And going forward loki will not set any defaults on `-boltdb.shipper.compactor.shared-store`, this has a couple of side effects detailed as follows: -##### store on which index compaction should be performed: +##### store on which index compaction should be performed + If `-boltdb.shipper.compactor.shared-store` is configured by the user, loki would run index compaction only on the store specified by the config. If not set, compaction would be performed on all the object stores that contain either a boltdb-shipper or tsdb index. -##### store used for managing delete requests: +##### store used for managing delete requests + A new config option `-boltdb.shipper.compactor.delete-request-store` decides where delete requests should be stored. This new option takes precedence over `-boltdb.shipper.compactor.shared-store`. In the case where neither of these options are set, the `object_store` configured in the latest `period_config` that uses either a tsdb or boltdb-shipper index is used for storing delete requests to ensure pending requests are processed. @@ -501,9 +497,9 @@ If you have a use-case that relies on strict parsing where you expect the parser logfmt parser doesn't include standalone keys(keys without a value) in the resulting label set anymore. You can use `--keep-empty` flag to retain them. -### Jsonnet +### Jsonnet 2..9.0 -##### Deprecated PodDisruptionBudget definition has been removed +#### Deprecated PodDisruptionBudget definition has been removed The `policy/v1beta1` API version of PodDisruptionBudget is no longer served as of Kubernetes v1.25. To support the latest versions of the Kubernetes, it was necessary to replace `policy/v1beta1` with the new definition `policy/v1` that is available since v1.21. @@ -512,10 +508,9 @@ No impact is expected if you use Kubernetes v1.21 or newer. Please refer to [official migration guide](https://kubernetes.io/docs/reference/using-api/deprecation-guide/#poddisruptionbudget-v125) for more details. - ## 2.8.0 -### Loki +### Loki 2.8.0 #### Change in LogQL behavior @@ -525,6 +520,7 @@ was kept. #### Default retention_period has changed This change will affect you if you have: + ```yaml compactor: retention_enabled: true @@ -537,6 +533,7 @@ In this release the default has been changed to `0s`. A value of `0s` is the same as "retain forever" or "disable retention". If, **and only if**, you wish to retain the previous default of 744h, apply this config. + ```yaml limits_config: retention_period: 744h @@ -563,7 +560,7 @@ Instead, now you can use `splits` to see how many split by time intervals were c Currently not every query can be sharded and a shards value of zero is a good indicator the query was not able to be sharded. {{% /admonition %}} -### Promtail +### Promtail 2.8.0 #### The go build tag `promtail_journal_enabled` was introduced @@ -573,6 +570,7 @@ If you need Journal support you will need to run go build with tag `promtail_jou ```shell go build --tags=promtail_journal_enabled ./clients/cmd/promtail ``` + Introducing this tag aims to relieve Linux/CentOS users with CGO enabled from installing libsystemd-dev/systemd-devel libraries if they don't need Journal support. ### Ruler @@ -597,6 +595,7 @@ This is relevant only if you are using [jsonnet for deploying Loki in Kubernetes {{% /admonition %}} The `query-frontend` Kubernetes service was previously headless and was used for two purposes: + * Distributing the Loki query requests amongst all the available Query Frontend pods. * Discover IPs of Query Frontend pods from Queriers to connect as workers. @@ -604,14 +603,16 @@ The problem here is that a headless service does not support load balancing and Additionally, a load-balanced service does not let us discover the IPs of the underlying pods. To meet both these requirements, we have made the following changes: + * Changed the existing `query-frontend` Kubernetes service from headless to load-balanced to have a fair load distribution on all the Query Frontend instances. * Added `query-frontend-headless` to discover QF pod IPs from queriers to connect as workers. If you are deploying Loki with Query Scheduler by setting [query_scheduler_enabled](https://github.com/grafana/loki/blob/cc4ab7487ab3cd3b07c63601b074101b0324083b/production/ksonnet/loki/config.libsonnet#L18) config to `true`, then there is nothing to do here for this change. If you are not using Query Scheduler, then to avoid any issues on the Read path until the rollout finishes, it would be good to follow below steps: -* Create just the `query-frontend-headless` service without applying any changes to the `query-frontend` service. -* Rollout changes to `queriers`. -* Roll out the rest of the changes. + +- Create just the `query-frontend-headless` service without applying any changes to the `query-frontend` service. +- Rollout changes to `queriers`. +- Roll out the rest of the changes. ### General @@ -621,7 +622,7 @@ Statistics are now logged in `metrics.go` lines about how long it takes to downl Example (note the `*_download_time` fields): -``` +```bash level=info ts=2022-12-20T15:27:54.858554127Z caller=metrics.go:147 component=frontend org_id=docker latency=fast query="sum(count_over_time({job=\"generated-logs\"}[1h]))" query_type=metric range_type=range length=6h17m48.865587821s start_delta=6h17m54.858533178s end_delta=5.99294552s step=1m30s duration=5.990829396s status=200 limit=30 returned_lines=0 throughput=123MB total_bytes=738MB total_entries=1 store_chunks_download_time=2.319297059s queue_time=2m21.476090991s subqueries=8 cache_chunk_req=81143 cache_chunk_hit=32390 cache_chunk_bytes_stored=1874098 cache_chunk_bytes_fetched=94289610 cache_chunk_download_time=56.96914ms cache_index_req=994 cache_index_hit=710 cache_index_download_time=1.587842ms cache_result_req=7 cache_result_hit=0 cache_result_download_time=380.555µs ``` @@ -629,7 +630,7 @@ These statistics are also displayed when using `--stats` with LogCLI. ## 2.7.0 -### Loki +### Loki 2.7.0 ### Loki Canary Permission @@ -639,12 +640,12 @@ So if you run Loki behind some proxy with different authorization policies to re ### `engine.timeout` and `querier.query_timeout` are deprecated Previously, we had two configurations to define a query timeout: `engine.timeout` and `querier.query-timeout`. -As they were conflicting and `engine.timeout` isn't as expressive as `querier.query-tiomeout`, +As they were conflicting and `engine.timeout` isn't as expressive as `querier.query-timeout`, we're deprecating it and moving it to [Limits Config](/docs/loki//configuration/#limits_config) `limits_config.query_timeout` with same default values. #### `fifocache` has been renamed -The in-memory `fifocache` has been renamed to `embedded-cache`. This allows us to replace the implementation (currently a simple FIFO datastructure) with something else in the future without causing confusion +The in-memory `fifocache` has been renamed to `embedded-cache`. This allows us to replace the implementation (currently a simple FIFO data structure) with something else in the future without causing confusion #### Evenly spread Memcached pods for chunks across kubernetes nodes @@ -682,6 +683,7 @@ This value now defaults to 3100, so the Loki process doesn't require special pri The docker-compose [setup](https://github.com/grafana/loki/blob/main/production/docker) has been updated to **v2.6.0** and includes many improvements. Notable changes include: + - authentication (multi-tenancy) is **enabled** by default; you can disable it in `production/docker/config/loki.yaml` by setting `auth_enabled: false` - storage is now using Minio instead of local filesystem - move your current storage into `.data/minio` and it should work transparently @@ -715,7 +717,7 @@ ConfigDB was disallowed as a Ruler storage option back in 2.0. The config struct Can no longer specify a remote write client for the ruler. -### Promtail +### Promtail 2.7.0 #### `gcp_push_target_parsing_errors_total` has a new `reason` label @@ -723,15 +725,15 @@ The `gcp_push_target_parsing_errors_total` GCP Push Target metrics has been adde #### Windows event logs: now correctly includes `user_data` -The contents of the `user_data` field was erroneously set to the same value as `event_data` in previous versions. This was fixed in [#7461](https://github.com/grafana/loki/pull/7461) and log queries relying on this broken behaviour may be impacted. +The contents of the `user_data` field was erroneously set to the same value as `event_data` in previous versions. This was fixed in [#7461](https://github.com/grafana/loki/pull/7461) and log queries relying on this broken behavior may be impacted. ## 2.6.0 -### Loki +### Loki 2.6.0 #### Implementation of unwrapped `rate` aggregation changed -The implementation of the `rate()` aggregation function changed back to the previous implemention prior to [#5013](https://github.com/grafana/loki/pulls/5013). +The implementation of the `rate()` aggregation function changed back to the previous implementation prior to [#5013](https://github.com/grafana/loki/pulls/5013). This means that the rate per second is calculated based on the sum of the extracted values, instead of the average increase over time. If you want the extracted values to be treated as [Counter](https://prometheus.io/docs/concepts/metric_types/#counter) metric, you should use the new `rate_counter()` aggregation function, which calculates the per-second average rate of increase of the vector. @@ -742,9 +744,9 @@ This value now defaults to `loki`, it was previously set to `cortex`. If you are ## 2.5.0 -### Loki +### Loki 2.5.0 -#### `split_queries_by_interval` yaml configuration has moved. +#### `split_queries_by_interval` yaml configuration has moved It was previously possible to define this value in two places @@ -755,7 +757,7 @@ query_range: and/or -``` +```yaml limits_config: split_queries_by_interval: 10m ``` @@ -776,6 +778,7 @@ In case you're still using the legacy format, take a look at on how to write alerting rules in the new format. For reference, the newer format follows a structure similar to the one below: + ```yaml groups: - name: example @@ -790,6 +793,7 @@ For reference, the newer format follows a structure similar to the one below: ``` Meanwhile, the legacy format is a string in the following format: + ``` ALERT IF @@ -800,7 +804,7 @@ Meanwhile, the legacy format is a string in the following format: #### Changes to default configuration values -* `parallelise_shardable_queries` under the `query_range` config now defaults to `true`. +* `parallelize_shardable_queries` under the `query_range` config now defaults to `true`. * `split_queries_by_interval` under the `limits_config` config now defaults to `30m`, it was `0s`. * `max_chunk_age` in the `ingester` config now defaults to `2h` previously it was `1h`. * `query_ingesters_within` under the `querier` config now defaults to `3h`, previously it was `0s`. Any query (or subquery) that has an end time more than `3h` ago will not be sent to the ingesters, this saves work on the ingesters for data they normally don't contain. If you regularly write old data to Loki you may need to return this value to `0s` to always query ingesters. @@ -808,19 +812,18 @@ Meanwhile, the legacy format is a string in the following format: * `match_max_concurrent` under the `frontend_worker` config now defaults to true, this supersedes the `parallelism` setting which can now be removed from your config. Controlling query parallelism of a single process can now be done with the `querier` `max_concurrent` setting. * `flush_op_timeout` under the `ingester` configuration block now defaults to `10m`, increased from `10s`. This can help when replaying a large WAL on Loki startup, and avoid `msg="failed to flush" ... context deadline exceeded` errors. -### Promtail +### Promtail 2.5.0 #### `gcplog` labels have changed - - Resource labels have been moved from `__` to `__gcp_resource_labels_` - e.g. if you previously used `__project_id` then you'll need to update your relabel config to use `__gcp_resource_labels_project_id`. - - `resource_type` has been moved to `__gcp_resource_type` +- Resource labels have been moved from `__` to `__gcp_resource_labels_` for example, if you previously used `__project_id` then you'll need to update your relabel config to use `__gcp_resource_labels_project_id`. +- `resource_type` has been moved to `__gcp_resource_type` -#### `promtail_log_entries_bytes_bucket` histogram has been removed. +#### `promtail_log_entries_bytes_bucket` histogram has been removed This histogram reports the distribution of log line sizes by file. It has 8 buckets for every file being tailed. -This creates a lot of series and we don't think this metric has enough value to offset the amount of series genereated so we are removing it. +This creates a lot of series and we don't think this metric has enough value to offset the amount of series generated so we are removing it. While this isn't a direct replacement, two metrics we find more useful are size and line counters configured via pipeline stages, an example of how to configure these metrics can be found in the [metrics pipeline stage docs](https://grafana.com/docs/loki//send-data/promtail/stages/metrics/#counter). @@ -829,7 +832,7 @@ While this isn't a direct replacement, two metrics we find more useful are size If you have dashboards that depended on the log level, change them to search for the `msg="added Docker target"` property. -### Jsonnet +### Jsonnet 2.5.0 #### Compactor config defined as command line args moved to yaml config @@ -850,7 +853,7 @@ Following 2 compactor configs that were defined as command line arguments in jso The following are important changes which should be reviewed and understood prior to upgrading Loki. -### Loki +### Loki 2.4.0 The following changes pertain to upgrading Loki. @@ -869,7 +872,7 @@ Anyone in situation #2, you have two options, the first (and not recommended) is The second and recommended solution, is to use deletes via the compactor: -``` +```yaml compactor: retention_enabled: true limits_config: @@ -878,7 +881,7 @@ limits_config: See the [retention docs](https://grafana.com/docs/loki//operations/storage/retention/) for more info. -#### Log messages on startup: proto: duplicate proto type registered: +#### Log messages on startup: proto: duplicate proto type registered PR [#3842](https://github.com/grafana/loki/pull/3842) **cyriltovena**: Fork cortex chunk storage into Loki. @@ -886,7 +889,7 @@ Since Cortex doesn't plan to use the `chunk` package anymore, we decided to fork be able to evolve and modify it easily. However, as a side-effect, we still vendor Cortex which includes this forked code and protobuf files resulting in log messages like these at startup: -``` +```bash 2021-11-04 15:30:02.437911 I | proto: duplicate proto type registered: purgeplan.DeletePlan 2021-11-04 15:30:02.437936 I | proto: duplicate proto type registered: purgeplan.ChunksGroup 2021-11-04 15:30:02.437939 I | proto: duplicate proto type registered: purgeplan.ChunkDetails @@ -936,7 +939,7 @@ If you would like to disable these caches or change this memory limit: Disable: -``` +```yaml chunk_store_config: chunk_cache_config: enable_fifocache: false @@ -948,7 +951,7 @@ query_range: Resize: -``` +```yaml chunk_store_config: chunk_cache_config: enable_fifocache: true @@ -980,6 +983,7 @@ This changes a few default values, resulting in the ingester WAL now being on by and chunk transfer retries are disabled by default. Note, this now means Loki will depend on local disk by default for its WAL (write ahead log) directory. This defaults to `wal` but can be overridden via the `--ingester.wal-dir` or via `path_prefix` in the common configuration section. Below are config snippets with the previous defaults, and another with the new values. Previous defaults: + ```yaml ingester: max_transfer_retries: 10 @@ -988,6 +992,7 @@ ingester: ``` New defaults: + ```yaml ingester: max_transfer_retries: 0 @@ -998,6 +1003,7 @@ ingester: Using the write ahead log (WAL) is recommended and is now the default. However using the WAL is incompatible with chunk transfers, if you have explicitly configured `ingester.max-transfer-retries` to a non-zero value, you must set it to 0 to disable transfers. #### Memberlist config now automatically applies to all non-configured rings + * [4400](https://github.com/grafana/loki/pull/4400) **trevorwhitney**: Config: automatically apply memberlist config too all rings when provided This change affects the behavior of the ingester, distributor, and ruler rings. Previously, if you wanted to use memberlist for all of these rings, you @@ -1040,19 +1046,22 @@ ruler: ``` #### Changed defaults for some GRPC server settings + * [4435](https://github.com/grafana/loki/pull/4435) **trevorwhitney**: Change default values for two GRPC settings so querier can connect to frontend/scheduler This changes two default values, `grpc_server_min_time_between_pings` and `grpc_server_ping_without_stream_allowed` used by the GRPC server. *Previous Values*: -``` + +```yaml server: grpc_server_min_time_between_pings: '5m' grpc_server_ping_without_stream_allowed: false ``` *New Values*: -``` + +```yaml server: grpc_server_min_time_between_pings: '10s' grpc_server_ping_without_stream_allowed: true @@ -1064,7 +1073,7 @@ server: * [#3842](https://github.com/grafana/loki/pull/3842)/[#4253](https://github.com/grafana/loki/pull/4253) **jordanrushing**: Metrics related to chunk storage and runtime config have changed their prefixes from `cortex_` to `loki_`. -``` +```yaml cortex_runtime_config* -> loki_runtime_config* cortex_chunks_store* -> loki_chunks_store* ``` @@ -1080,17 +1089,17 @@ per-tenant WAL can be found [here](/docs/loki//operations/recordin The `ruler` now requires persistent storage - see the [Operations](/docs/loki//operations/recording-rules/#deployment) page for more details about deployment. -### Promtail +### Promtail 2.4.0 The following changes pertain to upgrading Promtail. #### Promtail no longer insert `promtail_instance` label when scraping `gcplog` target -* [4556](https://github.com/grafana/loki/pull/4556) **james-callahan**: Remove `promtail_instance` label that was being added by promtail when scraping `gcplog` target. +* [4556](https://github.com/grafana/loki/pull/4556) **james-callahan**: Remove `promtail_instance` label that was being added by promtail when scraping `gcplog` target. ## 2.3.0 -### Loki +### Loki 2.3.0 #### Query restriction introduced for queries which do not have at least one equality matcher @@ -1114,10 +1123,9 @@ This difference may seem subtle but if we break it down `.` matches any characte The reasoning for this change has to do with how index lookups work in Loki, if you don't have at least one equality matcher Loki has to perform a complete index table scan which is an expensive and slow operation. - ## 2.2.0 -### Loki +### Loki 2.2.0 **Be sure to upgrade to 2.0 or 2.1 BEFORE upgrading to 2.2** @@ -1127,9 +1135,9 @@ In Loki 2.2 we changed the internal version of our chunk format from v2 to v3, t This makes it important to first upgrade to 2.0, 2.0.1, or 2.1 **before** upgrading to 2.2 so that if you need to rollback for any reason you can do so easily. -{{% admonition type="note" %}} -2.0 and 2.0.1 are identical in every aspect except 2.0.1 contains the code necessary to read the v3 chunk format. Therefor if you are on 2.0 and ugrade to 2.2, if you want to rollback, you must rollback to 2.0.1. -{{% /admonition %}} +{{< admonition type="note" >}} +2.0 and 2.0.1 are identical in every aspect except 2.0.1 contains the code necessary to read the v3 chunk format. Therefor if you are on 2.0 and upgrade to 2.2, if you want to rollback, you must rollback to 2.0.1. +{{< /admonition >}} ### Loki Config @@ -1147,9 +1155,7 @@ You could consider multiplying your current `max_query_parallelism` setting by 1 **Also be aware to make sure `max_outstanding_per_tenant` is always greater than `max_query_parallelism` or large queries will automatically fail with a 429 back to the user.** - - -### Promtail +### Promtail 2.2.0 For 2.0 we eliminated the long deprecated `entry_parser` configuration in Promtail configs, however in doing so we introduced a very confusing and erroneous default behavior: @@ -1157,7 +1163,7 @@ If you did not specify a `pipeline_stages` entry you would be provided with a de In [3404](https://github.com/grafana/loki/pull/3404), we corrected this behavior -**If you are using docker, and any of your `scrape_configs` are missing a `pipeline_stages` definition**, you should add the following to obtain the correct behaviour: +**If you are using docker, and any of your `scrape_configs` are missing a `pipeline_stages` definition**, you should add the following to obtain the correct behavior: ```yaml pipeline_stages: @@ -1168,15 +1174,15 @@ pipeline_stages: The upgrade from 2.0.0 to 2.1.0 should be fairly smooth, be aware of these two things: -### Helm charts have moved! +### Helm charts have moved -Helm charts are now located at: https://github.com/grafana/helm-charts/ +Helm charts are now located in the [Grafana Helm charts repo](https://github.com/grafana/helm-charts/). -The helm repo URL is now: https://grafana.github.io/helm-charts +The helm repo URL is now: [https://grafana.github.io/helm-charts](https://grafana.github.io/helm-charts). ### Fluent Bit plugin renamed -Fluent bit officially supports Loki as an output plugin now! WoooHOOO! +Fluent bit officially supports Loki as an output plugin now. However this created a naming conflict with our existing output plugin (the new native output uses the name `loki`) so we have renamed our plugin. @@ -1184,14 +1190,14 @@ In time our plan is to deprecate and eliminate our output plugin in favor of the Old: -``` +```shell [Output] Name loki ``` New: -``` +```shell [Output] Name grafana-loki ``` @@ -1215,11 +1221,12 @@ For the most part, there are very few impactful changes and for most this will b The default config file in the docker image, as well as the default helm values.yaml and jsonnet for Tanka all specify a schema definition to make things easier to get started. ->**If you have not specified your own config file with your own schema definition (or you do not have a custom schema definition in your values.yaml), upgrading to 2.0 will break things!** +{{< admonition type="caution >}} +If you have not specified your own config file with your own schema definition (or you do not have a custom schema definition in your values.yaml), upgrading to 2.0 will break things! +{{< /admonition >}} In 2.0 the defaults are now v11 schema and the `boltdb-shipper` index type. - If you are using an index type of `aws`, `bigtable`, or `cassandra` this means you have already defined a custom schema and there is _nothing_ further you need to do regarding the schema. You can consider however adding a new schema entry to use the new `boltdb-shipper` type if you want to move away from these separate index stores and instead use just one object store. @@ -1277,14 +1284,13 @@ This likely only affects a small portion of tanka users because the default sche If you had set `index_period_hours` to a value other than 168h (the previous default) you must update this in the above config `period:` to match what you chose. {{% /admonition %}} - -{{% admonition type="note" %}} +{{< admonition type="note" >}} We have changed the default index store to `boltdb-shipper` it's important to add `using_boltdb_shipper: false,` until you are ready to change (if you want to change) {{% /admonition %}} Changing the jsonnet config to use the `boltdb-shipper` type is the same as [below](#upgrading-schema-to-use-boltdb-shipper-andor-v11-schema) where you need to add a new schema section. -**HOWEVER** Be aware when you change `using_boltdb_shipper: true` the deployment type for the ingesters and queriers will change to statefulsets! Statefulsets are required for the ingester and querier using boltdb-shipper. +**HOWEVER** Be aware when you change `using_boltdb_shipper: true` the deployment type for the ingesters and queriers will change to StatefulSets! StatefulSets are required for the ingester and querier using boltdb-shipper. ##### Docker (e.g. docker-compose) @@ -1300,8 +1306,7 @@ docker run -d --name=loki --mount type=bind,source="path to loki-config.yaml",ta The Loki docker image is expecting to find the config file at `/etc/loki/local-config.yaml` - -### IMPORTANT: boltdb-shipper upgrade considerations. +### IMPORTANT: boltdb-shipper upgrade considerations Significant changes have taken place between 1.6.0 and 2.0.0 for boltdb-shipper index type, if you are already running this index and are upgrading some extra caution is warranted. @@ -1313,6 +1318,7 @@ The chunks directory should not need any special backups. If you have an environment to test this in, do so before upgrading against critical data. There are 2 significant changes warranting the backup of this data because they will make rolling back impossible: + * A compactor is included which will take existing index files and compact them to one per day and remove non compacted files * All index files are now gzipped before uploading @@ -1336,7 +1342,7 @@ Ingesters now expose a new RPC method that queriers use when the index type is ` Queriers generally roll out faster than ingesters, so if new queriers query older ingesters using the new RPC, the queries would fail. To avoid any query downtime during the upgrade, rollout ingesters before queriers. -#### If running the compactor, ensure it has delete permissions for the object storage. +#### If running the compactor, ensure it has delete permissions for the object storage The compactor is an optional but suggested component that combines and deduplicates the boltdb-shipper index files. When compacting index files, the compactor writes a new file and deletes unoptimized files. Ensure that the compactor has appropriate permissions for deleting files, for example, s3:DeleteObject permission for AWS S3. @@ -1375,6 +1381,7 @@ schema_config: prefix: index_ period: 24h ⑤ ``` + ① Make sure all of these match your current schema config ② Make sure this matches your previous schema version, Helm for example is likely v9 ③ Make sure this is a date in the **FUTURE** keep in mind Loki only knows UTC so make sure it's a future UTC date @@ -1383,7 +1390,6 @@ schema_config: There are more examples on the [Storage description page](https://grafana.com/docs/loki//configure/storage/#examples) including the information you need to setup the `storage` section for boltdb-shipper. - ## 1.6.0 ### Important: Ksonnet port changed and removed NET_BIND_SERVICE capability from Docker image @@ -1398,11 +1404,10 @@ the capability was removed. It is now no longer possible for the Loki to be started with a port less than 1024 with the published docker image. -The default for Helm has always been port 3100, and Helm users should be unaffect unless they changed the default. +The default for Helm has always been port 3100, and Helm users should be unaffected unless they changed the default. **Ksonnet users however should closely check their configuration, in PR 2294 the loki port was changed from 80 to 3100** - ### IMPORTANT: If you run Loki in microservices mode, special rollout instructions A new ingester GRPC API has been added allowing to speed up metric queries, to ensure a rollout without query errors **make sure you upgrade all ingesters first.** @@ -1414,10 +1419,9 @@ This will only affect reads(queries) and not writes and only for the duration of ### IMPORTANT: Scrape config changes to both Helm and Ksonnet will affect labels created by Promtail -PR [2091](https://github.com/grafana/loki/pull/2091) Makes several changes to the Promtail scrape config: +Loki PR [2091](https://github.com/grafana/loki/pull/2091) makes several changes to the Promtail scrape config: -```` -This is triggered by https://github.com/grafana/jsonnet-libs/pull/261 +This is triggered by jsonnet-libs PR [261](https://github.com/grafana/jsonnet-libs/pull/261): The above PR changes the instance label to be actually unique within a scrape config. It also adds a pod and a container target label @@ -1429,7 +1433,6 @@ the container_name label. It is the same as the container label and was already added to Loki previously. However, the container_name label is deprecated and has disappeared in Kubernetes 1.16, so that it will soon become useless for direct joining. -```` TL;DR @@ -1438,7 +1441,6 @@ The following label have been changed in both the Helm and Ksonnet Promtail scra `instance` -> `pod` `container_name` -> `container` - ### Experimental boltdb-shipper changes PR [2166](https://github.com/grafana/loki/pull/2166) now forces the index to have a period of exactly `24h`: @@ -1465,6 +1467,7 @@ schema_config: prefix: index_ period: 24h <--- This must be 24h ``` + If you are not on `schema: v11` this would be a good opportunity to make that change _in the new schema config_ also. {{% admonition type="note" %}} @@ -1532,7 +1535,7 @@ Defaulting to `gcs,bigtable` was confusing for anyone using ksonnet with other s The required upgrade path outlined for version 1.4.0 below is still true for moving to 1.5.0 from any release older than 1.4.0 (e.g. 1.3.0 -> 1.5.0 needs to also look at the 1.4.0 upgrade requirements). {{% /admonition %}} -### Breaking config changes! +### Breaking config changes Loki 1.5.0 vendors Cortex v1.0.0 (congratulations!), which has a [massive list of changes](https://cortexmetrics.io/docs/changelog/#1-0-0-2020-04-02). @@ -1563,8 +1566,7 @@ Referencing the [list of diffs](https://cortexmetrics.io/docs/changelog/#config- + dynamodb: ``` -Also several other AWS related configs changed and would need to udpate those as well. - +Also several other AWS related configs changed and would need to update those as well. ### Loki Docker Image User and File Location Changes @@ -1590,14 +1592,14 @@ The location Loki is looking for files with the provided config in the docker im In 1.4.0 and earlier the included config file in the docker container was using directories: -``` +```shell /tmp/loki/index /tmp/loki/chunks ``` In 1.5.0 this has changed: -``` +```shell /loki/index /loki/chunks ``` @@ -1614,7 +1616,7 @@ This would mount a docker volume named `loki-data` to the `/tmp/loki` folder whi To move to 1.5.0 I can do the following (note that your container names and paths and volumes etc may be different): -``` +```bash docker stop loki docker rm loki docker run --rm --name="loki-perm" -it --mount source=loki-data,target=/mnt ubuntu /bin/bash @@ -1629,7 +1631,6 @@ Notice the change in the `target=/loki` for 1.5.0 to the new data directory loca The intermediate step of using an ubuntu image to change the ownership of the Loki files to the new user might not be necessary if you can easily access these files to run the `chown` command directly. That is if you have access to `/var/lib/docker/volumes` or if you mounted to a different local filesystem directory, you can change the ownership directly without using a container. - ### Loki Duration Configs If you get an error like: