Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(#9601): prototype duplicate prevention #9609

Open
wants to merge 9 commits into
base: master
Choose a base branch
from
27 changes: 27 additions & 0 deletions webapp/package-lock.json

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

4 changes: 3 additions & 1 deletion webapp/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -61,7 +61,9 @@
"select2": "4.0.3",
"signature_pad": "2.3.x",
"tslib": "^2.5.3",
"zone.js": "^0.14.4"
"zone.js": "^0.14.4",
"levenshtein": "1.0.5",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a reason you preferred this levenshtein package? I know that metrics are not everything, but I was looking at fastest-levenshtein and it seemed to have the following benefits:

  • It says it is "fastest" 😅 (just kidding) - but the benchmark table makes a compelling reference, though it does not actually compare with levenshtein...
  • +290% more weekly downloads
  • More recent release (2 years ago vs 10 years)

Copy link
Contributor Author

@ChinHairSaintClair ChinHairSaintClair Jan 29, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe the levenshtein lib author got it right the first time, why mess with perfection 😆 (just kidding).

You raise a valid point, and I've updated the implementation to use fastest-levenshtein. Since this calculation will see significant traffic, especially on lower-spec devices, prioritizing performance makes sense.

Interestingly, fast-levenshtein uses fastest-levenshtein under the hood - yet ironically has more downloads. I'm not sure how much weight download count carries here.

The key difference is that the levenshtein library does not depend on any other library, whereas fastest-levenshtein depends on levenshtein-edit-distance, one of the libraries it compares itself to in its benchmarking (specifically version "2.0.5" cli.js).
Another difference would be that the levenshtein lib allows you to pretty print the distance table, but that's just nice to have 🙂.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was looking at fast-levenshtein as well. It seemed like it maybe just added some extra functionality on top of fastest-levenshtein. Ultimately, I agree we want to optimize for performance (especially on low-spec devices). Additionally, we should be sensitive about adding more kb to our app bundle size. 😬 IMHO it seems like fastest-levenshtein can provide a good balance for us and a good place to start. However, as you noted in a previous comment, we may decide in the future that the locale sensitive comparisons from fast-levenshtein are needed for non-English use cases...

"@types/levenshtein": "1.0.4"
},
"overrides": {
"minimist": ">=1.2.6"
Expand Down
52 changes: 52 additions & 0 deletions webapp/src/css/enketo/medic.less
Original file line number Diff line number Diff line change
Expand Up @@ -443,6 +443,58 @@
.pages.or .or-repeat-info[role="page"] {
display: block;
}

#duplicate_info {
width: 100%;
min-height: 20px;
padding-left: 20px;
padding-right: 20px;
background-color: #ffe7e8;

.results_header {
font-size: large;
color: #e33030;
}

.acknowledge_label {
-webkit-user-select: none; -ms-user-select: none; user-select: none;
}

.acknowledge_checkbox {
margin-right: 5px;
}

.divider {
background-color: #e33030;
height: 1px;
margin-top: 5px;
margin-bottom: 5px;
}

.card {
border: 1px solid #ddd;
padding: 1rem;
margin-bottom: 1rem;
border-radius: 5px;
}

.nested-section {
margin-left: 1.5rem;
}

.toggle-button {
background: none;
border: none;
color: #007bff;
cursor: pointer;
font-weight: bold;
padding-left: 0px;
}

.toggle-button:hover {
text-decoration: underline;
}
}
}

@media (max-width: @media-mobile) {
Expand Down
3 changes: 3 additions & 0 deletions webapp/src/ts/components/components.module.ts
Original file line number Diff line number Diff line change
Expand Up @@ -47,6 +47,7 @@ import { PanelHeaderComponent } from '@mm-components/panel-header/panel-header.c
import { SidebarMenuComponent } from '@mm-components/sidebar-menu/sidebar-menu.component';
import { ToolBarComponent } from '@mm-components/tool-bar/tool-bar.component';
import { TrainingCardsFormComponent } from '@mm-components/training-cards-form/training-cards-form.component';
import {DuplicateInfoComponent} from '@mm-components/duplicate-info/duplicate-info.component';

@NgModule({
declarations: [
Expand Down Expand Up @@ -78,6 +79,7 @@ import { TrainingCardsFormComponent } from '@mm-components/training-cards-form/t
SidebarMenuComponent,
TrainingCardsFormComponent,
ToolBarComponent,
DuplicateInfoComponent,
],
imports: [
CommonModule,
Expand Down Expand Up @@ -122,6 +124,7 @@ import { TrainingCardsFormComponent } from '@mm-components/training-cards-form/t
SidebarMenuComponent,
TrainingCardsFormComponent,
ToolBarComponent,
DuplicateInfoComponent,
]
})
export class ComponentsModule { }
Original file line number Diff line number Diff line change
@@ -0,0 +1,53 @@
<div
id="duplicate_info"
[ngStyle]="{ display: duplicates.length > 0 ? 'block' : 'none' }">
<p class="results_header">{{ duplicates.length }} {{'potential duplicate item(s) found:' | translate }}</p>
<div class="divider"></div>
<div>
<label for="check" class="acknowledge_label">
<input
id="check"
type="checkbox"
[checked]="acknowledged"
(change)="toggleAcknowledged()"
class="acknowledge_checkbox"/>
{{'Acknowledge duplicate siblings and proceed with submission' | translate }}
</label>
</div>
<div class="divider"></div>
<div *ngFor="let duplicate of duplicates; let i = index" >
<div class="card">
<strong>{{ 'Item number:' | translate }}</strong> {{ i + 1 }}
<hr>
<p>
<strong>{{ 'Name:' | translate }}</strong> {{ duplicate.name }}
<br>
<strong>{{ 'Created on:' | translate }}</strong> {{ duplicate.reported_date | date: 'EEE MMM dd yyyy HH:mm:ss' }}
</p>
<button class="toggle-button" (click)="toggleSection(getPath('', duplicate._id))">
{{ isExpanded(getPath('', duplicate._id)) ? '▼' : '▶' }}
{{ (isExpanded(getPath('', duplicate._id)) ? 'Show less details' : 'Show more details') | translate }}
</button>
<div *ngIf="isExpanded(getPath('', duplicate._id))" class="nested-section">
<ng-container *ngTemplateOutlet="renderObject; context: { obj: duplicate, path: duplicate._id }"></ng-container>
</div>
<hr>
<button class="btn submit btn-primary" (click)="_navigateToDuplicate(duplicate._id)">{{ 'Take me there' | translate }}</button>
</div>
</div>
<ng-template #renderObject let-obj="obj" let-path="path">
<div *ngFor="let key of obj | keyvalue">
<div *ngIf="isObject(key.value); else primitiveValue">
<button class="toggle-button" (click)="toggleSection(getPath(path, key.key))">
{{ isExpanded(getPath(path, key.key)) ? '▼' : '▶' }} {{ key.key }}
</button>
<div *ngIf="isExpanded(getPath(path, key.key))" class="nested-section">
<ng-container *ngTemplateOutlet="renderObject; context: { obj: key.value, path: getPath(path, key.key) }"></ng-container>
</div>
</div>
<ng-template #primitiveValue>
<strong>{{ key.key }}:</strong> {{ key.value }}
</ng-template>
</div>
</ng-template>
</div>
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
import { Component, EventEmitter, Input, Output } from '@angular/core';

@Component({
selector: 'mm-duplicate-info',
templateUrl: './duplicate-info.component.html',
})
export class DuplicateInfoComponent {
@Input() acknowledged: boolean = false;
@Output() acknowledgedChange = new EventEmitter<boolean>();
@Output() navigateToDuplicate = new EventEmitter<string>();
@Input() duplicates: { _id: string; name: string; reported_date: string | Date; [key: string]: string | Date }[] = [];

toggleAcknowledged() {
this.acknowledged = !this.acknowledged;
this.acknowledgedChange.emit(this.acknowledged);
}

_navigateToDuplicate(_id: string){
this.navigateToDuplicate.emit(_id);
}

// Handles collapse / expand of duplicate doc details
expandedSections = new Map<string, boolean>();

toggleSection(path: string): void {
this.expandedSections.set(path, !this.expandedSections.get(path));
}

isExpanded(path: string): boolean {
return this.expandedSections.get(path) || false;
}

isObject(value: any): boolean {
return value && typeof value === 'object' && !Array.isArray(value);
}

getPath(parentPath: string, key: string): string {
return parentPath ? `${parentPath}.${key}` : key;
}
}
1 change: 1 addition & 0 deletions webapp/src/ts/components/enketo/enketo.component.html
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
<div [attr.id]="formId" class="enketo" [attr.data-editing]="editing">
<div class="container pages"></div>
<ng-content select="[duplicate-info]"></ng-content> <!-- placeholder -->
<div class="form-footer">
<button (click)="onCancel.emit()" class="btn btn-link cancel" [disabled]="status?.saving">{{'Cancel' | translate}}</button>
<div class="loader inline small" *ngIf="status?.saving"></div>
Expand Down
6 changes: 5 additions & 1 deletion webapp/src/ts/modules/contacts/contacts-edit.component.html
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,11 @@
</div>
<div class="col-sm-8 item-content material" [hidden]="loadingContent || contentError">
<div class="card">
<mm-enketo formId="contact-form" [editing]="enketoContact?.docId" [status]="enketoStatus" (onSubmit)="save()" (onCancel)="navigationCancel()"></mm-enketo>
<mm-enketo formId="contact-form" [editing]="enketoContact?.docId" [status]="enketoStatus" (onSubmit)="save()" (onCancel)="navigationCancel()">
<div duplicate-info>
<mm-duplicate-info [acknowledged]="acknowledged" [duplicates]="duplicates" (acknowledgedChange)="onAcknowledgeChange($event)" (navigateToDuplicate)="onNavigateToDuplicate($event)"></mm-duplicate-info>
</div>
</mm-enketo>
</div>
</div>
</div>
28 changes: 26 additions & 2 deletions webapp/src/ts/modules/contacts/contacts-edit.component.ts
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ import { isEqual as _isEqual } from 'lodash-es';
import { ActivatedRoute, Router } from '@angular/router';

import { LineageModelGeneratorService } from '@mm-services/lineage-model-generator.service';
import { FormService } from '@mm-services/form.service';
import { FormService, DuplicatesFoundError, Duplicate } from '@mm-services/form.service';
import { EnketoFormContext } from '@mm-services/enketo.service';
import { ContactTypesService } from '@mm-services/contact-types.service';
import { DbService } from '@mm-services/db.service';
Expand Down Expand Up @@ -55,6 +55,18 @@ export class ContactsEditComponent implements OnInit, OnDestroy, AfterViewInit {
private trackSave;
private trackMetadata = { action: '', form: '' };

private duplicateCheck;
acknowledged = false;
onAcknowledgeChange(value: boolean) {
this.acknowledged = value;
}

onNavigateToDuplicate(_id: string){
this.router.navigate(['/contacts', _id, 'edit']);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

question: When i clicked the "Take me there" button, I had expected to go to the contact's profile page. Why do we take the user straight to the edit form?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Initially we thought that the user might not have enough details around why the sibling was flagged as a duplicate, which can cause frustration.
The edit form "will" display all the information, where the profile page "could" display all the info depending on config.
In this version of the duplicate item all values pertaining to that document are displayed before clicking off, which is then a compelling argument for simply navigating to the profile page. It's also a noticeable change, since it can be a bit jarring to the user if the page content just switches out.
That said, since the content of the duplicate item is now being reconsidered, maybe we should revisit this?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, that is a good point that this could be affected by the conclusions of the UX discussion.

The edit form "will" display all the information, where the profile page "could" display all the info depending on config.

Speaking purely from my opinion (with no real-world data to back me up), my expectation would be that any info important enough to be dupe-checking on would also be important enough to display via the contact-summary on a contact's profile page.

I can see where you are coming from with the consistency of moving from entering data in a contact create form to now being in the contact edit form were we can review the data and make any necessary changes. On the other hand, though, I am sure we will get cases like this:

  • The user needs to submit an app form for a child (e.g. a u5 assessment)
  • The user forgot they already created a contact for the child and proceeds to open a create-person form to add a new person to the household
  • Their new child contact is flagged as a duplicate when trying to submit the create-person form
  • The user selects "Take me there" for the exiting child, intending to go to that contact and launch the app form.
  • Instead the user is taking to the edit form for the existing contact and has to cancel out of that before proceeding to the app form

Ultimately, you and your team are closer to the end-users then I am and I will happily defer to your expertise with these kinds of things! I hope it is useful for me to raise questions and point out other possible behaviors we can discuss, but ultimately I am not any kind of UX expert nor do I have direct access to regularly observe/interact-with CHT users.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You make a good point. Making decisions for the user without fully understanding their behavior and needs could backfire. Navigating to the profile screen, where they have the option to do x, y, or z, is better than forcing them into an edit.

That said, IMO we should provide the user with enough information before navigating to the item, as they will lose the form data. This is especially important when considering multiple duplicates, as discussed in the forum thread.

}

duplicates: Duplicate[] = [];
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

suggestion: WebStorm is complaining to me that the reported_date: number value for the Duplicate interface does not match the reported_date: string | Date value for the DuplicateInfoComponent.duplicates property. I don't think it is a functional issue probably since JS can auto-box a number into a string, but I wonder if we should be type-matching ContactsEditComponent.duplicates and DuplicateInfoComponent.duplicates?


ngOnInit() {
this.trackRender = this.performanceService.track();
this.subscribeToStore();
Expand Down Expand Up @@ -153,6 +165,10 @@ export class ContactsEditComponent implements OnInit, OnDestroy, AfterViewInit {
this.contentError = false;
this.errorTranslationKey = false;

// Reset when when navigated to duplicate
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nitpick: this comment might be misleading since this will get called anytime a new form is opened (not just when navigating to the duplicate contact). I think it is clear from the context of this function that we are simply resetting the stored form state for a new form.

Suggested change
// Reset when when navigated to duplicate

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed. Clearing the values will also be unnecessary if we navigate to the profile page instead of switching out the edit screen content.

this.duplicates = [];
this.acknowledged = false;

try {
const contact = await this.getContact();
const contactTypeId = this.contactTypesService.getTypeId(contact) || this.routeSnapshot.params?.type;
Expand Down Expand Up @@ -272,6 +288,7 @@ export class ContactsEditComponent implements OnInit, OnDestroy, AfterViewInit {
private async renderForm(formId: string, titleKey: string) {
const formDoc = await this.dbService.get().get(formId);
this.xmlVersion = formDoc.xmlVersion;
this.duplicateCheck = formDoc.context?.duplicate_check;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

issue: conceptually, I don't think we want to put duplicate_check inside of context. The context property controls when the form is available to the user. I think we can just have duplicate_check be another top-level property:

Suggested change
this.duplicateCheck = formDoc.context?.duplicate_check;
this.duplicateCheck = formDoc.duplicate_check;

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We will need a small update to cht-conf to support uploading the duplicate_check value in the properties file. In the meantime, we can test by just directly editing the form docs in Couch (e.g. form:contact:person:create.)


this.globalActions.setEnketoEditedStatus(false);

Expand Down Expand Up @@ -326,7 +343,9 @@ export class ContactsEditComponent implements OnInit, OnDestroy, AfterViewInit {
$('form.or').trigger('beforesave');

return this.formService
.saveContact(form, docId, this.enketoContact.type, this.xmlVersion)
.saveContact({
form, docId, type: this.enketoContact.type, xmlVersion: this.xmlVersion
}, this.duplicateCheck, this.acknowledged)
.then((result) => {
console.debug('saved contact', result);

Expand All @@ -345,6 +364,11 @@ export class ContactsEditComponent implements OnInit, OnDestroy, AfterViewInit {
this.router.navigate(['/contacts', result.docId]);
})
.catch((err) => {
if (err instanceof DuplicatesFoundError){
this.duplicates = err.duplicates;
err = Error(err.message);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

question: why do we need to reassign err here?

Copy link
Contributor Author

@ChinHairSaintClair ChinHairSaintClair Jan 29, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe the idea was to "box" the value and omit dumping the duplicate data in console.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Destructive reassignment is usually a code smell (the go community heard what I said). You don't know who else wants to use a variable or why and now they have to work around your choices.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe the idea was to "box" the value and omit dumping the duplicate data in console.

👍 Thanks for the context. I understand where you are coming from here now.

FWIW, I don't know that we really have any solid patterns around "destructive reassignment" in the CHT code-base. 😅 So, I am happy to go with which ever way seems most clean to you.

}

console.error('Error submitting form data', err);

this.globalActions.setEnketoSavingStatus(false);
Expand Down
Loading
Loading