Unhandled rejection MultipleValidationErrors - running in a serverless-local environment

[matttowerssonos]

When using npdynamodb with a serverless application and dynamodb-local, validation errors are thrown for every database access call after the first one.

I'm working on a serverless application that uses npdynamodb ORM for accessing DynamoDB tables in a Lambda context. To facilitate testing, I'm using the serverless framework with the serverless-local and serverless-dynamodb-local plugins to run a local instance of DynamoDB. In that environment, in order to simulate a Lambda runtime context, serverless reloads the Lambda modules between each request.

When using npdynamodb in this environment, the first database request executed in the process succeeds as expected. However, all subsequent requests throw the following error:

Unhandled rejection MultipleValidationErrors

As best I can tell, there is some sort of module-level caching going on that is preventing npdynamodb from cleanly reloading.

The following code simulates what serverless is doing as part of its module reloading and will reproduce the error with npdynamodb. (Code assumes dynamodb-local is running and table users has been defined.

'use strict';

const AWS      = require('aws-sdk');
let npdynamodb = require('npdynamodb');

function getDynamoDB() {
  return new AWS.DynamoDB({
    accessKeyId:     'test',
    secretAccessKey: 'test',
    apiVersion:      '2012-08-10',
    endpoint:        'http://localhost:8000',
    region:          'localhost',
  });
}

let npd = npdynamodb.createClient(getDynamoDB());

let table = npdynamodb.define('users', {
  npdynamodb: npd,
  hashKey:    'id',
});

const record = { id: '056e4f3e-d505-4dad-8ec1-d04f521cbb56', foo: 'bar' };

table.save(record);

console.log('saved!');

// Simulate reloading in a serverless-local environment
delete require.cache[require.resolve('npdynamodb')];
npdynamodb = require('npdynamodb');

npd = npdynamodb.createClient(getDynamoDB());

table = npdynamodb.define('users', {
  npdynamodb: npd,
  hashKey:    'id',
});

// Throws 'MultipleValidationErrors'
table.save(record);

Output:

saved!
Unhandled rejection MultipleValidationErrors: There were 41 validation errors:
* InvalidParameterType: Expected params.Item['id'] to be a structure
* UnexpectedParameter: Unexpected key '0' found in params.Item['id']
* UnexpectedParameter: Unexpected key '1' found in params.Item['id']
...

[JamesTheHacker]

Did you ever solve this? Same error. There's so much outdated junk on AWS/Serverless that it's hard to get anything working from examples.

[alphaadidas]

+1

[rudeluv]

After doing some digging, this appears to be caused by an undefined value somewhere inside the record object. Not sure why it doesn't throw the aws-sdk validation on the first attempt, but I've narrowed it down to that issue.

Line 74 of param_validator.js in aws-dsk lib is where this happens.

if (memberShape !== undefined) {
        var memberContext = [context, paramName].join('.');
        this.validateMember(memberShape, paramValue, memberContext);
      } else {
        this.fail('UnexpectedParameter',
          'Unexpected key \'' + paramName + '\' found in ' + context);
      }

Not really sure if there's a solution beyond scrubbing record input, but it is weird that validation misses on the first run for some reason.

[necevil]

+1
I believe am working on what I think is the same issue.

When attempting to insert into DynamoDB my aws-sdk complains that my formatting is not correct:

* InvalidParameterType: Expected params.Item['lastUpdated'] to be a structure
* UnexpectedParameter: Unexpected key '0' found in params.Item['lastUpdated']
* UnexpectedParameter: Unexpected key '1' found in params.Item['lastUpdated']

From what I can see this is related to aws-sdk not being passed the type etc, re: this stackoverflow answer: https://stackoverflow.com/questions/33942945/error-invalidparametertype-expected-params-itempid-to-be-a-structure-in-dyn

That being the case essentially the answer recommends using the AWS.DynamoDB.DocumentClient which lets you avoid specifying the type of data you are attempting to insert since it automatically marshals Javascript types onto DynamoDB AttributeValues. More on the document client here: https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/DynamoDB/DocumentClient.html

Given the above it seems like the issue likely has to do with the item type either not being declared properly going from npd to aws-sdk.dynamodb or alternately not being declared at all (ie. InvalidParameterType: Expected params.Item['lastUpdated'] to be a structure gets thrown since there is no type property).

@alphaadidas, @JamesTheHacker, @matttowerssonos, @rudeluv did you guys ever get this working?

[rudeluv]

@necevil I just started checking for undefined within my object before feeding it to the ORM.