Skip to content

Commit

Permalink
Document the csv parser's asObjects option
Browse files Browse the repository at this point in the history
  • Loading branch information
@oleiade
oleiade committed Feb 5, 2025
1 parent e70419a commit 76487fb
Show file tree
Hide file tree
Showing 4 changed files with 122 additions and 52 deletions.
22 changes: 22 additions & 0 deletions docs/sources/k6/next/javascript-api/k6-experimental/csv/Options.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -13,10 +13,32 @@ The `Options` object describes the configuration available for the operation of
| Property | Type | Description |
| :------------ | :---------------- | :-------------------------------------------------------------------------------------------------------- |
| delimiter | string | The character used to separate fields in the CSV file. Default is `','`. |
| asObjects | boolean | Whether to parse the CSV file as an array of objects. Default is `false`. |
| skipFirstLine | boolean | Whether to skip the first line of the CSV file. Default is `false`. |
| fromLine | (optional) number | The line number from which to start reading the CSV file. Default is `0`. |
| toLine | (optional) number | The line number at which to stop reading the CSV file. If the option is not set, then read until the end. |

### asObjects

When `asObjects` is set to `true`, parsing operations over the CSV file will return an array of objects. The object keys are the column names from the CSV file, as inferred from the first line of the file, and the values are the field values from the CSV record. Note that the first line of the CSV file is skipped, as it is assumed to contain the column names (header row).

As a result, the following CSV file:

```csv
name,age
John,30
Jane,25
```

Will be parsed into the following array of objects:

```json
[
{ "name": "John", "age": "30" },
{ "name": "Jane", "age": "25" }
]
```

## Example

{{< code >}}
Expand Down
68 changes: 47 additions & 21 deletions docs/sources/k6/next/javascript-api/k6-experimental/csv/Parser.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -9,27 +9,6 @@ weight: 30
The `csv.Parser` class provides a streaming parser that reads CSV files line-by-line, offering fine-grained control over the parsing process and minimizing memory consumption.
It's well-suited for scenarios where memory efficiency is crucial or when you need to process large CSV files without loading the entire file into memory.

## Asynchronous nature

The `csv.Parser` class methods are asynchronous and return Promises.
Due to k6's current limitation with the [init context](https://grafana.com/docs/k6/<K6_VERSION>/using-k6/test-lifecycle/#the-init-stage) (which doesn't support asynchronous functions directly), you need to use an asynchronous wrapper such as:

{{< code >}}

```javascript
import { open } from 'k6/experimental/fs';
import csv from 'k6/experimental/csv';

let file;
let parser;
(async function () {
file = await open('data.csv');
parser = new csv.Parser(file);
})();
```

{{< /code >}}

## Constructor

| Parameter | Type | Description |
Expand All @@ -54,6 +33,8 @@ A promise resolving to an object with the following properties:

## Example

### Basic Usage

{{< code >}}

```javascript
Expand Down Expand Up @@ -86,6 +67,51 @@ export default async function () {

{{< /code >}}

### Using `asObjects`

The `asObjects` option parses the CSV file into an array of objects. The object keys are the column names from the CSV file. and the values are the field values from the CSV record.
Note that the first line of the CSV file is skipped, as it is assumed to contain the column names (header row).

{{< code >}}

```javascript
import { open } from 'k6/experimental/fs';
import csv from 'k6/experimental/csv';
import { scenario } from 'k6/execution';

export const options = {
iterations: 3,
};

const file = await open('data.csv');
const parser = new csv.Parser(file, { asObjects: true });

export default async function () {
// Will print the record for the current iteration as an object.
//
// For example, given the following CSV file:
//
// firstname,lastname
// francois,mitterand
// helmut,kohl
// pierre,mendes-france
//
// The test will print:
//
// { firstname: 'francois', lastname: 'mitterand' }
// { firstname: 'helmut', lastname: 'kohl' }
// { firstname: 'pierre', lastname: 'mendes-france' }
const { done, value } = await parser.next();
if (done) {
throw new Error('No more rows to read');
}

console.log(value);
}
```

{{< /code >}}

## Notes on usage

- **Memory efficiency**: Since `csv.Parser` reads the file line-by-line, it keeps memory usage low and avoids loading the entire set of records into memory. This is particularly useful for large CSV files.
Expand Down
9 changes: 5 additions & 4 deletions docs/sources/k6/next/javascript-api/k6-experimental/csv/_index.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -32,10 +32,11 @@ performance and memory optimization.

## API

| Function/Object | Description |
| ------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------- |
| [csv.parse()](https://grafana.com/docs/k6/<K6_VERSION>/javascript-api/k6-experimental/csv/parse) | Parses an entire CSV file into a SharedArray for high-performance scenarios. |
| [csv.Parser](https://grafana.com/docs/k6/<K6_VERSION>/javascript-api/k6-experimental/csv/parser) | A class for streaming CSV parsing, allowing line-by-line reading with minimal memory consumption. |
| Function/Object | Description |
| ---------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| [parse](https://grafana.com/docs/k6/<K6_VERSION>/javascript-api/k6-experimental/csv/parse) | Parses an entire CSV file into a SharedArray for high-performance scenarios. |
| [Parser](https://grafana.com/docs/k6/<K6_VERSION>/javascript-api/k6-experimental/csv/parser) | A class for streaming CSV parsing, allowing line-by-line reading with minimal memory consumption. |
| [Options](https://grafana.com/docs/k6/<K6_VERSION>/javascript-api/k6-experimental/csv/options) | An object that describes the configuration available for the operation of parsing CSV files using the [`parse()`](https://grafana.com/docs/k6/<K6_VERSION>/javascript-api/k6-experimental/csv/parse) function and the [`csv.Parser`](https://grafana.com/docs/k6/<K6_VERSION>/javascript-api/k6-experimental/csv/parser) class. |

## Example

Expand Down
75 changes: 48 additions & 27 deletions docs/sources/k6/next/javascript-api/k6-experimental/csv/parse.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -10,39 +10,20 @@ The `csv.parse` function parses an entire CSV file at once and returns a promise
This function uses Go-based processing, which results in faster parsing and lower memory usage compared to JavaScript alternatives.
It's ideal for scenarios where performance is a priority, and the entire CSV file can be loaded into memory.

## Asynchronous Nature

`csv.parse` is an asynchronous function that returns a Promise. Due to k6's current limitation with the [init context](https://grafana.com/docs/k6/<K6_VERSION>/using-k6/test-lifecycle/) (which
doesn't support asynchronous functions directly), you need to use an asynchronous wrapper like this:

{{< code >}}

```javascript
import { open } from 'k6/experimental/fs';
import csv from 'k6/experimental/csv';

let file;
let csvRecords;
(async function () {
file = await open('data.csv');
csvRecords = await csv.parse(file, { delimiter: ',' });
})();
```

{{< /code >}}

## Parameters

| Parameter | Type | Description |
| :-------- | :-------------------------------------------------------------------------------------------- | :-------------------------------------------------------------- |
| file | [fs.File](https://grafana.com/docs/k6/<K6_VERSION>/javascript-api/k6-experimental/fs/file) | A file instance opened using the `fs.open` function. |
| options | [Options](https://grafana.com/docs/k6/<K6_VERSION>/javascript-api/k6-experimental/fs/options) | An optional parameter object to customize the parsing behavior. |
| Parameter | Type | Description |
| :-------- | :--------------------------------------------------------------------------------------------- | :-------------------------------------------------------------- |
| file | [fs.File](https://grafana.com/docs/k6/<K6_VERSION>/javascript-api/k6-experimental/fs/file) | A file instance opened using the `fs.open` function. |
| options | [Options](https://grafana.com/docs/k6/<K6_VERSION>/javascript-api/k6-experimental/csv/options) | An optional parameter object to customize the parsing behavior. |

## Returns

A promise resolving to a [SharedArray](https://grafana.com/docs/k6/<K6_VERSION>/javascript-api/k6-data/sharedarray) instance, where each element is an array representing a CSV record, and each sub-element is a field from that record.

## Example
## Examples

### Basic Usage

{{< code >}}

Expand All @@ -59,7 +40,7 @@ const file = await open('data.csv');

// The `csv.parse` function consumes the entire file at once and returns
// the parsed records as a `SharedArray` object.
const csvRecords = await csv.parse(file, { skipFirstLine: true });
const csvRecords = await csv.parse(file);

export default async function () {
// `csvRecords` is a `SharedArray`. Each element is a record from the CSV file, represented as an array
Expand All @@ -72,6 +53,46 @@ export default async function () {

{{< /code >}}

### Using `asObjects`

The `asObjects` option parses the CSV file into an array of objects. The object keys are the column names from the CSV file. and the values are the field values from the CSV record.
Note that the first line of the CSV file is skipped, as it is assumed to contain the column names (header row).

{{< code >}}

```javascript
import { open } from 'k6/experimental/fs';
import csv from 'k6/experimental/csv';
import { scenario } from 'k6/execution';

export const options = {
iterations: 3,
};

const file = await open('data.csv');
const csvRecords = await csv.parse(file, { asObjects: true });

export default function () {
// Will print the record for the current iteration as an object.
//
// For example, given the following CSV file:
//
// firstname,lastname
// francois,mitterand
// helmut,kohl
// pierre,mendes-france
//
// The test will print:
//
// { firstname: 'francois', lastname: 'mitterand' }
// { firstname: 'helmut', lastname: 'kohl' }
// { firstname: 'pierre', lastname: 'mendes-france' }
console.log(csvRecords[scenario.iterationInTest]);
}
```

{{< /code >}}

## Notes on Usage

- **Memory considerations**: `csv.parse` loads the entire CSV file into memory at once, which may lead to increased memory usage and startup time for very large files.
Expand Down

0 comments on commit 76487fb

Please sign in to comment.