README.md

Testing

To test our code, three libraries have been adopted:

• Great Expectations: this library can be used to validate, document, and test data to ensure its quality and integrity throughout the data pipeline.

• Deepchecks: this library, on the other hand, is valuable for diagnosing and monitoring deep learning models through the various tools it offers, including those for inspecting, visualizing, and comprehending the behavior of neural networks.

• Pytest: this library simplifies the process of writing and executing unit tests and provides features such as fixtures, parameterized testing, and detailed test reporting.

Data Quality Tests

GreatExpectations

The Great Expectations library is employed in the great_expectations_utilities script, test_extract_data script and test_preprocessed_data script to examine the data needed for suggesting songs to the user. To assess this data, two test suites were established:

1. Test Suite: extracted_data_expectations

   • Involved tests:
     • expect_column_values_to_not_be_null
     • expect_column_values_to_be_of_type

2. Test Suite: preprocessed_data_expectations

   • Involved tests:
     • expect_column_values_to_not_be_null
     • expect_column_values_to_be_of_type
     • expect_column_values_to_be_between

Following is a code snippet displaying the defined tests for a column in our dataset:

# Expectation on the values of a column to not be null
energy_notnull_config = ExpectationConfiguration(
    expectation_type="expect_column_values_to_not_be_null",
    kwargs={
        "auto": True,
        "column": "Energy"
    },
)

# Expectation on the values of a column to be of a specific type
energy_float_config = ExpectationConfiguration(
    expectation_type="expect_column_values_to_be_of_type",
    kwargs={
        "column": "Energy",
        "type_": "float"
    },
)

# Expectation on the values of a column to be within a range of values
energy_inrange_config = ExpectationConfiguration(
    expectation_type="expect_column_values_to_be_between",
    kwargs={
        "column": "Energy",
        "max_value": 1,
        "min_value": 0,
    },
)

The two test suites are included in the DVC pipeline prior to the corresponding extract and preprocessing steps.

More information about this tool can be found here.

Deepchecks

The Deepchecks library is utilized in the test_deepchecks notebook to evaluate the train and test data required for song recommendations. Deepchecks provides different sub-packages, each containing many checks and suites for specific tasks. In our case, we used the Tabular sub-package, which contains the data_integrity test suite. The same test suite was run on both the raw datasets and the processed datasets.

This test suite is composed of eleven checks related to data integrity, from which we kept only the following pertinent tests:

• Mixed Data Type: test that checks for variations in data types within a dataset. It helps identify columns or fields that may contain a mix of different data types.

• Mixed Nulls: test that examines the presence of null or missing values in the dataset, specifically looking for columns where nulls are mixed with non-null values.

• Data Duplicates: test that assesses the dataset for duplicate records or rows. It identifies instances where identical data points appear more than once.

With the listed tests, we can better understand if our data presents issues or not and they also help us in making the data more reliable when used for song recommendations.

Here is an example of Deepchecks' UI results:

Addittionaly, this library has also been adopted for Data Drift Detection, and you can read more about it here. More information about this tool can be found here.

Scripts Testing

Pytest

The Pytest library is utilized in the following scripts:

• test_api
• test_files_utilities
• test_preprocessing
• test_recommend
• test_spotipy_utilities

In each of these scripts, various tests have been executed to ensure that every component functions as expected.

Executing the command:

python -m pytest path\\to\\test\\script

will run all the tests present within the specified script. Below is an example of the output generated by pytest:

More information about this tool can be found here.