Merge branch 'use-soft-deletes-api-call' of github.com:elastic/connec…

…tors-python into use-soft-deletes-api-call

.backportrc.json

@@ -3,8 +3,7 @@
     { "name": "main", "checked": true },
     "8.x",
     "8.17",
-    "8.16",
-    "8.15"
+    "8.16"
   ],
   "fork": false,
   "targetPRLabels": ["backport"],

Dockerfile.ftest.wolfi

@@ -1,4 +1,4 @@
-FROM docker.elastic.co/wolfi/python:3.11-dev@sha256:4ce83a86c3e27c6f4c308f477024d85b62edcf236f209b6ec64556f87f3e5ae5
+FROM docker.elastic.co/wolfi/python:3.11-dev@sha256:430c6d1353a2be51c1b011e1eb806f5d7f92e5c5e0b15cb921f31f0250c02757
 USER root
 COPY . /connectors
 WORKDIR /connectors

Dockerfile.wolfi

@@ -1,4 +1,4 @@
-FROM docker.elastic.co/wolfi/python:3.11-dev@sha256:4ce83a86c3e27c6f4c308f477024d85b62edcf236f209b6ec64556f87f3e5ae5
+FROM docker.elastic.co/wolfi/python:3.11-dev@sha256:430c6d1353a2be51c1b011e1eb806f5d7f92e5c5e0b15cb921f31f0250c02757
 USER root
 COPY . /app
 WORKDIR /app

connectors/source.py

@@ -141,7 +141,17 @@ def _convert(self, value, field_type_):
         # list requires special type casting
         if cast_type is list:
             if isinstance(value, str):
-                return [item.strip() for item in value.split(",")] if value else []
+                items = []
+                if value:
+                    for item in value.split(","):
+                        item = item.strip()
+                        if not item:
+                            logger.debug(
+                                "Empty string detected in the comma-separated list. It will be skipped."
+                            )
+                        else:
+                            items.append(item)
+                return items
             elif isinstance(value, int):
                 return [value]
             elif isinstance(value, set):

docs/CLI.md

@@ -16,23 +16,23 @@ Connectors CLI helps with Elastic Connectors managing connectors and running syn
 ## Installation
 1. Clone the repository `git clone https://github.com/elastic/connectors.git`
 2. Run `make clean install` to install dependencies and create executable files.
-3. Connectors CLI is available via `./bin/connectors`
+3. Connectors CLI is available via `.venv/bin/connectors`
 
 ## Configuration
 **Note:** Make sure your Elasticsearch instance is up and running.
 
-1. Run `./bin/connectors login` to authenticate the CLI with an Elasticsearch instance.
+1. Run `.venv/bin/connectors login` to authenticate the CLI with an Elasticsearch instance.
 2. Provide credentials
 3. The command will create or ask to rewrite an existing configuration file in `./cli/config.yml`
 
 By default, the CLI uses basic authentication method (username, password) however an API key can be used too. 
-Run `./bin/connectors login --method apikey` to authenticate the CLI via your API key. 
+Run `.venv/bin/connectors login --method apikey` to authenticate the CLI via your API key. 
 
 When you run any command you can specify a configuration file using `-c` argument.
 Example:
 
 ```bash
-./bin/connectors -c <config-file-path.yml> connector list
+.venv/bin/connectors -c <config-file-path.yml> connector list
 ```
 
 ## Available commands
@@ -41,7 +41,7 @@ Connectors CLI provides a `--help`/`-h` argument that can be used with any comma
 
 For example:
 ```bash
-./bin/connectors --help
+.venv/bin/connectors --help
 
 
 Usage: connectors [OPTIONS] COMMAND [ARGS]...
@@ -76,7 +76,7 @@ To bypass interactive mode you can use the `--from-file` argument, pointing to a
 Examples:
 
 ```console
-./bin/connectors connector create \
+.venv/bin/connectors connector create \
   --index-name my-index \
   --service-type sharepoint_online \
   --index-language en \
@@ -95,7 +95,7 @@ Lists all the existing connectors
 Examples:
 
 ```console
-./bin/connectors connector list
+.venv/bin/connectors connector list
 ```
 
 This will display all existing connectors and the associated indices.
@@ -105,7 +105,7 @@ Lists all jobs and their stats.
 
 Examples
 ```console
-./bin/connectors job list -- <connector_id>
+.venv/bin/connectors job list -- <connector_id>
 ```
 
 This will display all sync jobs including information like job status, number of indexed documents and index data volume associated with `connector_id`.
@@ -116,7 +116,7 @@ Marks the job as `cancelling` to let Connector services know that the job has to
 Examples:
 
 ```console
-./bin/connectors job cancel -- <job_id>
+.venv/bin/connectors job cancel -- <job_id>
 ```
 
 #### `connectors job start`
@@ -125,7 +125,7 @@ Schedules a new sync job and lets Connector service pick it up.
 Examples:
 
 ```console
-./bin/connectors job start -- \
+.venv/bin/connectors job start -- \
   -i <connector_id> \
   -t <job_type{full,incremental,access_control}> \
   -o <format{text,json}>
@@ -139,7 +139,7 @@ Shows information about a sync job.
 Examples:
 
 ```console
-./bin/connectors job view -- <job_id> -o <format{text,json}
+.venv/bin/connectors job view -- <job_id> -o <format{text,json}
 ```
 
 This will display information about the job including job id, connector id, indexed document counts and index data value.

docs/DEVELOPING.md

@@ -38,7 +38,7 @@ $ make clean install
 The `elastic-ingest` CLI will be installed on your system:
 
 ```shell
-$ bin/elastic-ingest --help
+$ .venv/bin/elastic-ingest --help
 usage: elastic-ingest [-h] [--action {poll,list}] [-c CONFIG_FILE] [--log-level {DEBUG,INFO,WARNING,ERROR,CRITICAL} | --debug] [--filebeat] [--version] [--uvloop]
 
 options:

docs/RELEASING.md

@@ -11,7 +11,8 @@ Take care of the branching (minor releases only):
 
 - Increment the VERSION on main to match the next minor release
 - Create a new maintenance branch
-- Make sure the `.backportrc.json` is updated. The previous minor is added to `targetBranchChoices` and the new minor is used in `branchLabelMapping`
+- Make sure `.backportrc.json` is updated: the previous minor is added to `targetBranchChoices` and the new minor is used in `branchLabelMapping`
+- Make sure `renovate.json` is updated: the previous minor is added to `labels` (for example, `v8.18`). [Create that label](https://github.com/elastic/connectors/labels) if it doesn't exist yet
 
 
 ## Unified release, (>= 8.16)

scripts/testing/README.md

@@ -19,26 +19,26 @@ Run `make clean install` to generate executable files in `./bin` folder
 
 ### Usage
 
-Run `./bin/test-connectors --help` or `./bin/test-connectors {command name} --help` to get more information about the cli. 
+Run `.venv/bin/test-connectors --help` or `.venv/bin/test-connectors {command name} --help` to get more information about the cli. 
 
 #### Running test with Elastic cloud deployment
 If you want to run your test suite using a cloud Elasticsearch deployment follow the next steps: 
 1. Create a cloud deployment
 2. Download a credentials file (or create a new user)
-3. Run `./bin/test-connectors run-test my-testing-environment-name --es-host {host} --es-username {user name} --es-password {password} --test-case {path to the test case file}`
+3. Run `.venv/bin/test-connectors run-test my-testing-environment-name --es-host {host} --es-username {user name} --es-password {password} --test-case {path to the test case file}`
 
 #### Running test with local Elasticsearch
 If you want to run your tests with local Elasticsearch you need to specify `--es-version` option. Like
 
-`./bin/test-connectors run-test my-testing-environment-name --es-version 8.12-SNAPSHOT --test-case {path to the test case file}`
+`.venv/bin/test-connectors run-test my-testing-environment-name --es-version 8.12-SNAPSHOT --test-case {path to the test case file}`
 
 In this case, the cli will deploy an Elasticsearch instance in the same VM where the connector service will be running. 
 
 #### Running test with a specific Connector service version
 
 You can use any git reference such as commit sha, a tag, or a branch name. The cli will pull the defined git reference and run `make clean install`. 
 
-Example: `./bin/test-connectors run-test my-testing-environment-name --es-version 8.12-SNAPSHOT --connectors-ref 8.12 --test-case {path to the test case file}`
+Example: `.venv/bin/test-connectors run-test my-testing-environment-name --es-version 8.12-SNAPSHOT --connectors-ref 8.12 --test-case {path to the test case file}`
 
 #### Keeping your VM running when the tests passed
 Sometimes it's useful to get access to the logs or make some changes in the code and run tests again. The CLI will print a list of useful commands you can use to access the VM resources like: 
@@ -47,7 +47,7 @@ Sometimes it's useful to get access to the logs or make some changes in the code
 
 To automatically delete the VM you need to use `--delete` option. 
 
-`./bin/test-connectors run-test my-testing-environment-name --es-version 8.12-SNAPSHOT --connectors-ref 8.12 --test-case {path to the test case file} --delete`
+`.venv/bin/test-connectors run-test my-testing-environment-name --es-version 8.12-SNAPSHOT --connectors-ref 8.12 --test-case {path to the test case file} --delete`
 
 #### Using different machine type
 All new VMs are based on a predefined image which is in turn based on `ubuntu-2204-lts` with python3 and docker installed. Custome images are not supported. You can change a machine type by providing `--vm-type`. Visit [the official GCP documentation](https://cloud.google.com/compute/docs/general-purpose-machines) to get more information. 

tests/test_source.py

@@ -101,6 +101,9 @@ def test_field_convert():
 
     assert Field("name", field_type="list").value == []
     assert Field("name", value="1", field_type="list").value == ["1"]
+    assert Field("name", value="1,2,3,", field_type="list").value == ["1", "2", "3"]
+    assert Field("name", value="1,2,3,    ", field_type="list").value == ["1", "2", "3"]
+    assert Field("name", value=",,,1,2,3", field_type="list").value == ["1", "2", "3"]
     assert Field("name", value="1,2,3", field_type="list").value == ["1", "2", "3"]
     assert Field("name", value=[1, 2], field_type="list").value == [1, 2]
     assert Field("name", value=0, field_type="list").value == [0]