[QUESTION] - Error on parent lookup

[buphoff-villagemd]

Describe the bug
Error re: missing parent lookup records: "...missing parent lookup records were found. See MissingParentRecordsReport.csv file for the details."

This error is new, and there have been no changes to the underlying data model (objects, fields) for objects spec'd in the export.json. This job has run many times successfully in the last few years. The last time we ran it (and when it was successful) was probably 3 months ago. This may be relevant, but I'm also getting this error at the beginning of the log output:

(node:45632) Warning: Deprecated config name: apiVersion. Please use org-api-version instead.
(Use node --trace-warnings ... to show where the warning was created)

I found an issue (completed) here on this site for the above, but the suggested fix did not work to remove the error.

To Reproduce
Execute the job:
sf sfdmu run --sourceusername PROD --targetusername UPHOFF

Expected behavior
I expected SFDMU to extract and load the 3 "parent" objects' data, then perform the necessary lookups to establish the links from the fourth object back to the 3 "parent" objects. The fourth object is a kind of "junction" object between the 3 others.

export.json
(will attach)

Log file
(will attach)

_target.csv file.
A few rows from the offending target file, ICD_to_HCC_Code_Mapping__c_insert_target.csv:

CreatedDate,Errors,HCC_Code__c,HCC_Version__c,ICD_Code__c,LastModifiedDate,Name
2024-02-16T16:54:03.000+0000,Could not locate this record in the API response.,#N/A,a7H8L0000008RBlUAM,#N/A,2024-02-16T16:54:03.000+0000,a7I6S000000F9oA
2024-02-16T16:54:03.000+0000,Could not locate this record in the API response.,#N/A,a7H8L0000008RBnUAM,#N/A,2024-02-16T16:54:03.000+0000,a7I6S000000F9pN
2024-02-16T16:54:04.000+0000,Could not locate this record in the API response.,#N/A,a7H8L0000008RBpUAM,#N/A,2024-02-16T16:54:04.000+0000,a7I6S000000F9rk
2024-02-16T16:54:04.000+0000,Could not locate this record in the API response.,#N/A,a7H8L0000008RBoUAM,#N/A,2024-02-16T16:54:04.000+0000,a7I6S000000F9sx
2024-02-16T17:14:26.000+0000,Could not locate this record in the API response.,a3a8L000000cbO8QAI,a7H8L0000008RBoUAM,#N/A,2024-02-16T17:14:26.000+0000,a7I6S000000FByC
2022-12-17T00:30:45.000+0000,Could not locate this record in the API response.,a3a8L000000cbO8QAI,a7H8L0000008RBmUAM,#N/A,2022-12-17T00:30:45.000+0000,a7I6S00000163Lf
2022-12-17T00:39:17.000+0000,Could not locate this record in the API response.,a3a8L000000cbO8QAI,a7H8L0000008RBlUAM,#N/A,2022-12-17T00:39:17.000+0000,a7I6S0000016UJX
2022-12-22T16:51:44.000+0000,Could not locate this record in the API response.,a3a8L000000cbO8QAI,a7H8L0000008RBkUAM,#N/A,2022-12-22T16:51:44.000+0000,a7I6S0000016wxX

[buphoff-villagemd]

export.json and log file

export.json
2024-09-12__09_19_35.log

[hknokh]

Hello, @buphoff-villagemd

Thank you for reporting a bug.
I will take a look at it as soon as possible and let you know of any updates.

Cheers

[buphoff-villagemd]

Thank you @hknokh ! I just noticed I did not update the issue description - corrected! -Ben

[hknokh2]

Hello,

Unfortunately, I'm unable to assist with this request. I cannot guarantee that the updates made in the tool will not affect existing jobs. Additionally, missing parent records may indicate a potential data issue, which falls outside the scope of the support I can provide. My support covers only bugs or issues related to documentation or missing documentation.

Please review your export.json file and modify it as needed to ensure your job runs as expected.

Thanks for understanding.

[readeral]

@hknokh2 What's the tool's approach to regression testing when updating? How does Salesforce decide when to release a major version and what is a breaking change? Obligating users to discover and adapt to breaking changes is not great, and you give the impression bugs are only bugs if they're "critical run time errors".

The contribution guide says "Thoroughly test your code before submitting a PR" but without standardised tests, it doesn't seem that there's any way to fulfil this requirement and will scare away contributors. Clearly if @buphoff-villagemd's run fails, then previous testing hasn't been rigorous enough, because "testing thoroughly" without defining testing scope can surely only mean "what previously works still works". This issue is the very definition of a bug.

I'm a new user this week, and I've been having difficulty with missing parent lookups in a data model that is similar to Ben's. I can't be confident that it's a user error, because without a documented systematic approach to testing, and where documentation is thin for complex relationships, I have little means to assess that this tool is resilient enough for my use case.

Don't get me wrong, I'm glad you've developed this tool, I appreciate the work you've put in, (and I was hoping to contribute a feature!), but hopefully Salesforce can provide you enough resources to tighten up this part of the project.

[hknokh2]

@readeral
Yes, I have conducted regression tests. However, I need to clarify that this is my independent project and is not related to my work at Salesforce. Salesforce is not involved in this project and provides no resources for it—it's entirely my own contribution project.

I kindly ask that you adhere to my support policy and avoid requesting assistance on matters beyond what I can provide. Unfortunately, I can't handle these issues, and I believe they may be due to the latest data change.

Thank you for your understanding.

[readeral]

documented systematic approach to testing

Where are these documented for contributors?

[hknokh2]

@readeral
Don't understand your question. No any documentation. This tool has its own policy and is NOT a Salesforce project. It's my independent project, and I define the support policy and scope of assistance based on my own capabilities.

[hknokh2]

@buphoff-villagemd
In your export.json, you used externalid as Id. The Id field cannot be used as an externalid for upserts; it's only applicable for inserts because this field does not match between source and target records. This mismatch can lead to missing lookups.

[readeral]

@readeral Don't understand your question. No any documentation. This tool has its own policy and is NOT a Salesforce project. It's my independent project, and I define the support policy and scope of assistance based on my own capabilities.

You say it's a contribution project, but how can others contribute to the project if we can't ensure our code won't cause regressions without standard documented tests?

[hknokh2]

As a responsible developer, it's important to test your PR before submitting it. I usually review PRs before merging and only allow very minor changes that, to my understanding, will not cause regressions. If you review the policy, you'll see that only small changes are allowed by contributors, as I'm cautious about potential regressions caused by these modifications.

[readeral]

important to test

To what standard? How much testing is required? What is considered a successful test?

[hknokh2]

There is no specific standard for this project beyond the common standards that apply to all development. If you're not sure, it's best not to make a contribution.

[github-actions]

This case has been marked as 'to-be-closed', since it has no activity for the 3 days.
It will be automatically closed in another 3 days of inactivity.

[hknokh]

Hello, @buphoff-villagemd,
As mentioned earlier, you might encounter issues with missing parent IDs when using Id as an external ID. I hope this information is helpful.
I'll go ahead and close this case for now. If you have any further questions or concerns, please feel free to reach out.
Regards.