datauploads and downloads visibility for non-admins will be critical if the user has large PV's. We'll need to create objects for non-admins where the du/dd status is reflected for the non-admin.

The proposal here is the add relevant DataUpload/DataDownload status to NAB/NAR status for enhanced visibility in case of Datamover backups.
Similarly, we will need PVB/PVR status details on NAB/NAR for FSBackups.

For instance, we can fetch the relevant PVBs via labels:

  labels:
    velero.io/backup-name: nac-demo-ns-sample-nab-b47e726e-83d1-4822-ab94-b2d2e3a1908d
    velero.io/backup-uid: 4f58bd07-08c3-4eb7-aee6-74dc0f8e6502

They also have ownerRefs:

  ownerReferences:
  - apiVersion: velero.io/v1
    controller: true
    kind: Backup
    name: nac-demo-ns-sample-nab-b47e726e-83d1-4822-ab94-b2d2e3a1908d
    uid: 4f58bd07-08c3-4eb7-aee6-74dc0f8e6502

This would be similar to how we relay information from VeleroBackup to NAB

Some Ref links:

• PVB: https://github.com/vmware-tanzu/velero/blob/main/pkg/podvolume/backupper.go#L370
• PVR: https://github.com/vmware-tanzu/velero/blob/0eb1040a0ae7105cade19c6ffee9b459eea4b6aa/pkg/podvolume/restorer.go#L182
• DU: https://github.com/vmware-tanzu/velero/blob/main/pkg/backup/actions/csi/pvc_action.go#L504
• DD: https://github.com/vmware-tanzu/velero/blob/0eb1040a0ae7105cade19c6ffee9b459eea4b6aa/pkg/restore/actions/csi/pvc_action.go#L569

in a very slow scenario, how would be the workflow?

something like this

create backup -> backup phase in progress -> DataUpload (or PodVolumeBackup) progress to 1% -> DataUpload (or PodVolumeBackup) progress to 2% -> ... -> DataUpload (or PodVolumeBackup) progress to 100% -> backup phase complete

?

In other words, Backup is not updated when DataUpload (or PodVolumeBackup) is updated, unless is a update to finished state?

I was thinking not sure if I'm right.. but essentially a status ONLY object.. NADU ( non-admin-data-upload ) where the status is simply synced. Same for datadownload.

If the nab creates a velero backup that creates du/dd objects in the protected namespace.. create the status object in the user namespace? We obviously can see in the spec that snapshotMoveData is set to true.. expect to duplicate status.

Yeah my idea is to just add more status in NAB/NAR

if a NAB datamover is created, for example, in its status it would have a list of DataUpload statuses. Is this approach valid?

Adding to NAB status is valid for sure.. However what happens if a non-admin user has 10, 20 even 100 PV's?
The status in the NAB is getting quite busy.. this would make it even more busy.
The other advantage to b.s. status objects ( NADU, NADD ) other than the cool acronym is that one can see each progress w/ an oc get NADU as they transfer up.
For example.. b/r the 8PV sample app.. perhaps add some data to each PV.. use a datamover backup. Then as the backup runs run a watch oc get dataupload. That seems to me to be a good cli ui. ¯_(ツ)_/¯
I'm NOT sold on any solution, I think we should discuss in scrum tomorrow for sure :)

Just to summarize there are 2 approaches in play here:

• Adding Relevant DataUpload objects status to Non-Admin Backup status. Here we will be adding a list of DU status (we can even extarct the interesting bits here like only the bytes data, if CR status getting huge is a concern here), users can directly see the DU status on NAB status.
• Creating a separate CRD/CR - NonAdminDataUpload corresponding to the Velero DataUpload CR. In this case we will be adding a list of NADU names on NAB status and users can go and check up on the NADU for more details.

Post scrum discussion, we will take a modified version of proposal 1 i.e. patching the NonAdminBackup object status with the progress details of Datamover Backup Operations and File System Backup Operations.
For example: the NAB status would look like:

status:
.
.
.
Datamover backup operations:
  DataUploads:
    Completed: 7
    Total: 14
.
.
.
File system backup operations:
  PodVolumeBackups:
    Completed: 11
    Total: 21     

And similarly we need to do this for NonAdminRestore CR using the relevant DataDownloads and PodVolumeRestores.

As @sseago noted, it seems we already have partial info of DataUpload/DataDownload in Backup/Restore status

should we just add logic for PodVolumeBackup/PodVolumeRestore?

Also, PodVolumeBackup/PodVolumeRestore/DataUpload/DataDownload can complete successful or not

should we show something like this?

filesystemBackupOperations:
  podVolumeBackups:
    completed: 11
    failed: 1
    total: 21    

or just gather number of completed and failed in completed field?

Hmm I think we wanted to show the InProgress DUs as well.

@mateusoliveira43 also, I wouldn't use the name filesystemBackupOperations -- backup operations is used specifically to refer to async plugin actions, which PVB doesn't make use of. Maybe just filesystemVolumeBackups or something similar?

But yeah, the point about the BackupItemOperations data is that if we only need completed vs. not completed data, we already have that. If we want something more specifically geared towards the DataUpload lifecycle, we'll need different fields and we'll need to watch DataUploads and not Backup objects, since the Backup status doesn't include enough information.

Also, if you're including DataUpload-specific information (i.e. in progress counts, etc.) you won't want to use any of the backup item operations fields, since those aren't guaranteed to include only dataupload information. For example, if we're not using datamover, just CSI -- those will refer to CSI snapshots. If a customer has custom plugins installed which have async actions, those will be included in the BackupItemOperations summary as well.

correct as @sseago explained we need to watch and fetch the progress data from the actual dataupload objects and not from the Velero Backup object.

alright, will also add predicate to DataUpload/DataDownload update events

to close discussing, on NAB status we will have these 2 new fields

fileSystemVolumeBackups:
    inProgress: 9 # number of PodVolumeBackups related to NAB's Backup in phase InProgress
    completed: 11 # number of PodVolumeBackups related to NAB's Backup in phase Completed
    failed: 1 # number of PodVolumeBackups related to NAB's Backup in phase Failed
    total: 21 # number of PodVolumeBackups related to NAB's Backup
    # PodVolumeBackup also have phase New
dataMoverVolumeBackups:
    inProgress: 7 # number of DataUploads related to NAB's Backup in phase InProgress
    completed: 9 # number of DataUploads related to NAB's Backup in phase Completed
    failed: 0 # number of DataUploads related to NAB's Backup in phase Failed
    canceled: 1 # number of DataUploads related to NAB's Backup in phase Canceled
    total: 17 # number of DataUploads related to NAB's Backup
    # PodVolumeBackup also have phase New, Accepted, Prepared and Canceling

similar in NAR for PodVolumeRestores and DataDownloads

Please add Accepted, Prepared to dataMover

Relevant Phases for DU: https://github.com/vmware-tanzu/velero/blob/main/pkg/apis/velero/v2alpha1/data_upload_types.go#L88-L97
Similarly for PVB:https://github.com/vmware-tanzu/velero/blob/main/pkg/apis/velero/v1/pod_volume_backup_types.go#L66-L71

Is the total number requierd? Seems easy to add all together making it reduntant info.

last update, we will show number of total DataUploads/DataDownloads/PodVolumeBackups/PodVolumeRestores and the number of each possible phase of this objects in NAB/NAR statuses