Skip to content

Commit

Permalink
Document the csv parser's asObjects option
Browse files Browse the repository at this point in the history
  • Loading branch information
oleiade committed Feb 5, 2025
1 parent e70419a commit 38037d7
Show file tree
Hide file tree
Showing 3 changed files with 111 additions and 44 deletions.
22 changes: 22 additions & 0 deletions docs/sources/k6/next/javascript-api/k6-experimental/csv/Options.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,10 +13,32 @@ The `Options` object describes the configuration available for the operation of
| Property | Type | Description |
| :------------ | :---------------- | :-------------------------------------------------------------------------------------------------------- |
| delimiter | string | The character used to separate fields in the CSV file. Default is `','`. |
| asObjects | boolean | Whether to parse the CSV file as an array of objects. Default is `false`. |
| skipFirstLine | boolean | Whether to skip the first line of the CSV file. Default is `false`. |
| fromLine | (optional) number | The line number from which to start reading the CSV file. Default is `0`. |
| toLine | (optional) number | The line number at which to stop reading the CSV file. If the option is not set, then read until the end. |

### `asObjects`

When `asObjects` is set to `true`, the CSV file is parsed into an array of objects. The object keys are the column names from the CSV file, as inferred from the first line of the file, and the values are the field values from the CSV record. Note that the first line of the CSV file is skipped, as it is assumed to contain the column names (header row).

As a result, the following CSV file:

```
name,age
John,30
Jane,25
```

Will be parsed into the following array of objects:

```
[
{ name: 'John', age: '30' },
{ name: 'Jane', age: '25' },
]
```

## Example

{{< code >}}
Expand Down
68 changes: 47 additions & 21 deletions docs/sources/k6/next/javascript-api/k6-experimental/csv/Parser.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,27 +9,6 @@ weight: 30
The `csv.Parser` class provides a streaming parser that reads CSV files line-by-line, offering fine-grained control over the parsing process and minimizing memory consumption.
It's well-suited for scenarios where memory efficiency is crucial or when you need to process large CSV files without loading the entire file into memory.

## Asynchronous nature

The `csv.Parser` class methods are asynchronous and return Promises.
Due to k6's current limitation with the [init context](https://grafana.com/docs/k6/<K6_VERSION>/using-k6/test-lifecycle/#the-init-stage) (which doesn't support asynchronous functions directly), you need to use an asynchronous wrapper such as:

{{< code >}}

```javascript
import { open } from 'k6/experimental/fs';
import csv from 'k6/experimental/csv';

let file;
let parser;
(async function () {
file = await open('data.csv');
parser = new csv.Parser(file);
})();
```

{{< /code >}}

## Constructor

| Parameter | Type | Description |
Expand All @@ -54,6 +33,8 @@ A promise resolving to an object with the following properties:

## Example

### Basic Usage

{{< code >}}

```javascript
Expand Down Expand Up @@ -86,6 +67,51 @@ export default async function () {

{{< /code >}}

### Using `asObjects`

The `asObjects` option parses the CSV file into an array of objects. The object keys are the column names from the CSV file. and the values are the field values from the CSV record.
Note that the first line of the CSV file is skipped, as it is assumed to contain the column names (header row).

{{< code >}}

```javascript
import { open } from 'k6/experimental/fs';
import csv from 'k6/experimental/csv';
import { scenario } from 'k6/execution';

export const options = {
iterations: 3,
};

const file = await open('data.csv');
const parser = new csv.Parser(file, { asObjects: true });

export default async function () {
// Will print the record for the current iteration as an object.
//
// For example, given the following CSV file:
//
// firstname,lastname
// francois,mitterand
// helmut,kohl
// pierre,mendes-france
//
// The test will print:
//
// { firstname: 'francois', lastname: 'mitterand' }
// { firstname: 'helmut', lastname: 'kohl' }
// { firstname: 'pierre', lastname: 'mendes-france' }
const { done, value } = await parser.next();
if (done) {
throw new Error('No more rows to read');
}

console.log(value);
}
```

{{< /code >}}

## Notes on usage

- **Memory efficiency**: Since `csv.Parser` reads the file line-by-line, it keeps memory usage low and avoids loading the entire set of records into memory. This is particularly useful for large CSV files.
Expand Down
65 changes: 42 additions & 23 deletions docs/sources/k6/next/javascript-api/k6-experimental/csv/parse.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,27 +10,6 @@ The `csv.parse` function parses an entire CSV file at once and returns a promise
This function uses Go-based processing, which results in faster parsing and lower memory usage compared to JavaScript alternatives.
It's ideal for scenarios where performance is a priority, and the entire CSV file can be loaded into memory.

## Asynchronous Nature

`csv.parse` is an asynchronous function that returns a Promise. Due to k6's current limitation with the [init context](https://grafana.com/docs/k6/<K6_VERSION>/using-k6/test-lifecycle/) (which
doesn't support asynchronous functions directly), you need to use an asynchronous wrapper like this:

{{< code >}}

```javascript
import { open } from 'k6/experimental/fs';
import csv from 'k6/experimental/csv';

let file;
let csvRecords;
(async function () {
file = await open('data.csv');
csvRecords = await csv.parse(file, { delimiter: ',' });
})();
```

{{< /code >}}

## Parameters

| Parameter | Type | Description |
Expand All @@ -42,7 +21,9 @@ let csvRecords;

A promise resolving to a [SharedArray](https://grafana.com/docs/k6/<K6_VERSION>/javascript-api/k6-data/sharedarray) instance, where each element is an array representing a CSV record, and each sub-element is a field from that record.

## Example
## Examples

### Basic Usage

{{< code >}}

Expand All @@ -59,7 +40,7 @@ const file = await open('data.csv');

// The `csv.parse` function consumes the entire file at once and returns
// the parsed records as a `SharedArray` object.
const csvRecords = await csv.parse(file, { skipFirstLine: true });
const csvRecords = await csv.parse(file);

export default async function () {
// `csvRecords` is a `SharedArray`. Each element is a record from the CSV file, represented as an array
Expand All @@ -70,6 +51,44 @@ export default async function () {
}
```

### Using `asObjects`

The `asObjects` option parses the CSV file into an array of objects. The object keys are the column names from the CSV file. and the values are the field values from the CSV record.
Note that the first line of the CSV file is skipped, as it is assumed to contain the column names (header row).

{{< code >}}

```javascript
import { open } from 'k6/experimental/fs';
import csv from 'k6/experimental/csv';
import { scenario } from 'k6/execution';

export const options = {
iterations: 3,
};

const file = await open('data.csv');
const csvRecords = await csv.parse(file, { asObjects: true });

export default function () {
// Will print the record for the current iteration as an object.
//
// For example, given the following CSV file:
//
// firstname,lastname
// francois,mitterand
// helmut,kohl
// pierre,mendes-france
//
// The test will print:
//
// { firstname: 'francois', lastname: 'mitterand' }
// { firstname: 'helmut', lastname: 'kohl' }
// { firstname: 'pierre', lastname: 'mendes-france' }
console.log(csvRecords[scenario.iterationInTest]);
}
```

{{< /code >}}

## Notes on Usage
Expand Down

0 comments on commit 38037d7

Please sign in to comment.