Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

fix: Do not clean PV's with a deletion timestamp #465

Merged
merged 1 commit into from
Nov 19, 2024
Merged

fix: Do not clean PV's with a deletion timestamp #465

merged 1 commit into from
Nov 19, 2024

Conversation

dobsonj
Copy link
Member

@dobsonj dobsonj commented Nov 14, 2024 •
edited
Loading

What type of PR is this?

/kind bug

What this PR does / why we need it:

After the volume has been successfully cleaned, the deleter sends a Delete
request to the API server. It should not continue attempting to clean the PV
at this point if the object still exists, as the PV may disappear at any time,
and we don't want cleanup to be in progress when that happens.

The simplest way to reproduce this is to:

  1. Add a finalizer to the PV
  2. Create a claim
  3. Delete the Pod/PVC to release the PV
  4. Observe multiple cleanup attempts in the local-volume-provisioner logs
  5. Remove the finalizer while a cleanup job is in progress

Before this fix:

I1114 21:50:26.591326       1 deleter.go:195] Start cleanup for pv local-pv-ae71ec75
I1114 21:50:26.591678       1 deleter.go:275] Deleting PV block volume "local-pv-ae71ec75" device hostpath "/mnt/fast-disks/vde", mountpath "/mnt/fast-disks/vde"
I1114 21:50:26.613684       1 deleter.go:319] Cleanup pv "local-pv-ae71ec75": StderrBuf - "shred: /mnt/fast-disks/vde: pass 1/3 (random)..."
I1114 21:50:31.000709       1 deleter.go:319] Cleanup pv "local-pv-ae71ec75": StderrBuf - "shred: /mnt/fast-disks/vde: pass 1/3 (random)...1.1GiB/10GiB 11%"
I1114 21:50:36.002793       1 deleter.go:319] Cleanup pv "local-pv-ae71ec75": StderrBuf - "shred: /mnt/fast-disks/vde: pass 1/3 (random)...2.1GiB/10GiB 21%"
I1114 21:50:41.000683       1 deleter.go:319] Cleanup pv "local-pv-ae71ec75": StderrBuf - "shred: /mnt/fast-disks/vde: pass 1/3 (random)...3.3GiB/10GiB 33%"
I1114 21:50:46.000709       1 deleter.go:319] Cleanup pv "local-pv-ae71ec75": StderrBuf - "shred: /mnt/fast-disks/vde: pass 1/3 (random)...4.5GiB/10GiB 45%"
I1114 21:50:51.000762       1 deleter.go:319] Cleanup pv "local-pv-ae71ec75": StderrBuf - "shred: /mnt/fast-disks/vde: pass 1/3 (random)...5.6GiB/10GiB 56%"
I1114 21:50:56.002669       1 deleter.go:319] Cleanup pv "local-pv-ae71ec75": StderrBuf - "shred: /mnt/fast-disks/vde: pass 1/3 (random)...7.0GiB/10GiB 70%"
I1114 21:51:01.001833       1 deleter.go:319] Cleanup pv "local-pv-ae71ec75": StderrBuf - "shred: /mnt/fast-disks/vde: pass 1/3 (random)...8.3GiB/10GiB 83%"
I1114 21:51:06.001702       1 deleter.go:319] Cleanup pv "local-pv-ae71ec75": StderrBuf - "shred: /mnt/fast-disks/vde: pass 1/3 (random)...9.5GiB/10GiB 95%"
I1114 21:51:08.949681       1 deleter.go:319] Cleanup pv "local-pv-ae71ec75": StderrBuf - "shred: /mnt/fast-disks/vde: pass 1/3 (random)...10GiB/10GiB 100%"
I1114 21:51:09.109607       1 deleter.go:319] Cleanup pv "local-pv-ae71ec75": StderrBuf - "shred: /mnt/fast-disks/vde: pass 2/3 (random)..."
I1114 21:51:14.001637       1 deleter.go:319] Cleanup pv "local-pv-ae71ec75": StderrBuf - "shred: /mnt/fast-disks/vde: pass 2/3 (random)...1.3GiB/10GiB 13%"
I1114 21:51:19.001662       1 deleter.go:319] Cleanup pv "local-pv-ae71ec75": StderrBuf - "shred: /mnt/fast-disks/vde: pass 2/3 (random)...2.9GiB/10GiB 29%"
I1114 21:51:24.001627       1 deleter.go:319] Cleanup pv "local-pv-ae71ec75": StderrBuf - "shred: /mnt/fast-disks/vde: pass 2/3 (random)...4.6GiB/10GiB 46%"
I1114 21:51:29.000693       1 deleter.go:319] Cleanup pv "local-pv-ae71ec75": StderrBuf - "shred: /mnt/fast-disks/vde: pass 2/3 (random)...6.2GiB/10GiB 62%"
I1114 21:51:34.000667       1 deleter.go:319] Cleanup pv "local-pv-ae71ec75": StderrBuf - "shred: /mnt/fast-disks/vde: pass 2/3 (random)...8.1GiB/10GiB 81%"
I1114 21:51:39.000717       1 deleter.go:319] Cleanup pv "local-pv-ae71ec75": StderrBuf - "shred: /mnt/fast-disks/vde: pass 2/3 (random)...9.6GiB/10GiB 96%"
I1114 21:51:40.414684       1 deleter.go:319] Cleanup pv "local-pv-ae71ec75": StderrBuf - "shred: /mnt/fast-disks/vde: pass 2/3 (random)...10GiB/10GiB 100%"
I1114 21:51:40.548255       1 deleter.go:319] Cleanup pv "local-pv-ae71ec75": StderrBuf - "shred: /mnt/fast-disks/vde: pass 3/3 (000000)..."
I1114 21:51:45.000687       1 deleter.go:319] Cleanup pv "local-pv-ae71ec75": StderrBuf - "shred: /mnt/fast-disks/vde: pass 3/3 (000000)...2.3GiB/10GiB 23%"
I1114 21:51:50.000711       1 deleter.go:319] Cleanup pv "local-pv-ae71ec75": StderrBuf - "shred: /mnt/fast-disks/vde: pass 3/3 (000000)...4.5GiB/10GiB 45%"
I1114 21:51:55.001657       1 deleter.go:319] Cleanup pv "local-pv-ae71ec75": StderrBuf - "shred: /mnt/fast-disks/vde: pass 3/3 (000000)...6.9GiB/10GiB 69%"
I1114 21:52:00.001681       1 deleter.go:319] Cleanup pv "local-pv-ae71ec75": StderrBuf - "shred: /mnt/fast-disks/vde: pass 3/3 (000000)...9.0GiB/10GiB 90%"
I1114 21:52:01.932711       1 deleter.go:319] Cleanup pv "local-pv-ae71ec75": StderrBuf - "shred: /mnt/fast-disks/vde: pass 3/3 (000000)...10GiB/10GiB 100%"
I1114 21:52:02.260215       1 deleter.go:283] Completed cleanup of pv "local-pv-ae71ec75"
I1114 21:52:06.639124       1 deleter.go:165] Deleting pv local-pv-ae71ec75 after successful cleanup
I1114 21:52:06.657462       1 cache.go:64] Updated pv "local-pv-ae71ec75" to cache
I1114 21:52:06.680524       1 cache.go:64] Updated pv "local-pv-ae71ec75" to cache
I1114 21:52:16.658649       1 deleter.go:195] Start cleanup for pv local-pv-ae71ec75
I1114 21:52:16.659692       1 deleter.go:275] Deleting PV block volume "local-pv-ae71ec75" device hostpath "/mnt/fast-disks/vde", mountpath "/mnt/fast-disks/vde"
I1114 21:52:16.667701       1 deleter.go:319] Cleanup pv "local-pv-ae71ec75": StderrBuf - "shred: /mnt/fast-disks/vde: pass 1/3 (random)..."
I1114 21:52:21.000672       1 deleter.go:319] Cleanup pv "local-pv-ae71ec75": StderrBuf - "shred: /mnt/fast-disks/vde: pass 1/3 (random)...1.6GiB/10GiB 16%"
I1114 21:52:26.001649       1 deleter.go:319] Cleanup pv "local-pv-ae71ec75": StderrBuf - "shred: /mnt/fast-disks/vde: pass 1/3 (random)...3.2GiB/10GiB 32%"
I1114 21:52:31.000668       1 deleter.go:319] Cleanup pv "local-pv-ae71ec75": StderrBuf - "shred: /mnt/fast-disks/vde: pass 1/3 (random)...4.8GiB/10GiB 48%"
I1114 21:52:36.000681       1 deleter.go:319] Cleanup pv "local-pv-ae71ec75": StderrBuf - "shred: /mnt/fast-disks/vde: pass 1/3 (random)...6.7GiB/10GiB 67%"
I1114 21:52:40.630183       1 cache.go:73] Deleted pv "local-pv-ae71ec75" from cache
I1114 21:52:41.000809       1 deleter.go:319] Cleanup pv "local-pv-ae71ec75": StderrBuf - "shred: /mnt/fast-disks/vde: pass 1/3 (random)...8.3GiB/10GiB 83%"
I1114 21:52:45.895459       1 deleter.go:319] Cleanup pv "local-pv-ae71ec75": StderrBuf - "shred: /mnt/fast-disks/vde: pass 1/3 (random)...10GiB/10GiB 100%"
I1114 21:52:46.408285       1 deleter.go:319] Cleanup pv "local-pv-ae71ec75": StderrBuf - "shred: /mnt/fast-disks/vde: pass 2/3 (random)..."
I1114 21:52:46.662188       1 discovery.go:316] PV local-pv-ae71ec75 is still being cleaned, not going to recreate it

After this fix, clean up does not re-run on the deleted PV while the finalizer exists and the deletion timestamp is set:

I1114 22:00:03.232180       1 deleter.go:200] Start cleanup for pv local-pv-ae71ec75
I1114 22:00:03.232893       1 deleter.go:280] Deleting PV block volume "local-pv-ae71ec75" device hostpath "/mnt/fast-disks/vde", mountpath "/mnt/fast-disks/vde"
I1114 22:00:03.242966       1 deleter.go:324] Cleanup pv "local-pv-ae71ec75": StderrBuf - "shred: /mnt/fast-disks/vde: pass 1/3 (random)..."
I1114 22:00:08.000809       1 deleter.go:324] Cleanup pv "local-pv-ae71ec75": StderrBuf - "shred: /mnt/fast-disks/vde: pass 1/3 (random)...1.6GiB/10GiB 16%"
I1114 22:00:13.000675       1 deleter.go:324] Cleanup pv "local-pv-ae71ec75": StderrBuf - "shred: /mnt/fast-disks/vde: pass 1/3 (random)...3.3GiB/10GiB 33%"
I1114 22:00:18.001677       1 deleter.go:324] Cleanup pv "local-pv-ae71ec75": StderrBuf - "shred: /mnt/fast-disks/vde: pass 1/3 (random)...4.9GiB/10GiB 49%"
I1114 22:00:23.001774       1 deleter.go:324] Cleanup pv "local-pv-ae71ec75": StderrBuf - "shred: /mnt/fast-disks/vde: pass 1/3 (random)...6.6GiB/10GiB 66%"
I1114 22:00:28.002091       1 deleter.go:324] Cleanup pv "local-pv-ae71ec75": StderrBuf - "shred: /mnt/fast-disks/vde: pass 1/3 (random)...8.3GiB/10GiB 83%"
I1114 22:00:33.008843       1 deleter.go:324] Cleanup pv "local-pv-ae71ec75": StderrBuf - "shred: /mnt/fast-disks/vde: pass 1/3 (random)...9.6GiB/10GiB 96%"
I1114 22:00:34.403616       1 deleter.go:324] Cleanup pv "local-pv-ae71ec75": StderrBuf - "shred: /mnt/fast-disks/vde: pass 1/3 (random)...10GiB/10GiB 100%"
I1114 22:00:34.524725       1 deleter.go:324] Cleanup pv "local-pv-ae71ec75": StderrBuf - "shred: /mnt/fast-disks/vde: pass 2/3 (random)..."
I1114 22:00:39.001661       1 deleter.go:324] Cleanup pv "local-pv-ae71ec75": StderrBuf - "shred: /mnt/fast-disks/vde: pass 2/3 (random)...1.2GiB/10GiB 12%"
I1114 22:00:44.000657       1 deleter.go:324] Cleanup pv "local-pv-ae71ec75": StderrBuf - "shred: /mnt/fast-disks/vde: pass 2/3 (random)...2.9GiB/10GiB 29%"
I1114 22:00:49.000770       1 deleter.go:324] Cleanup pv "local-pv-ae71ec75": StderrBuf - "shred: /mnt/fast-disks/vde: pass 2/3 (random)...4.5GiB/10GiB 45%"
I1114 22:00:54.000735       1 deleter.go:324] Cleanup pv "local-pv-ae71ec75": StderrBuf - "shred: /mnt/fast-disks/vde: pass 2/3 (random)...6.1GiB/10GiB 61%"
I1114 22:00:59.000671       1 deleter.go:324] Cleanup pv "local-pv-ae71ec75": StderrBuf - "shred: /mnt/fast-disks/vde: pass 2/3 (random)...7.9GiB/10GiB 79%"
I1114 22:01:04.000679       1 deleter.go:324] Cleanup pv "local-pv-ae71ec75": StderrBuf - "shred: /mnt/fast-disks/vde: pass 2/3 (random)...9.5GiB/10GiB 95%"
I1114 22:01:05.725798       1 deleter.go:324] Cleanup pv "local-pv-ae71ec75": StderrBuf - "shred: /mnt/fast-disks/vde: pass 2/3 (random)...10GiB/10GiB 100%"
I1114 22:01:05.874259       1 deleter.go:324] Cleanup pv "local-pv-ae71ec75": StderrBuf - "shred: /mnt/fast-disks/vde: pass 3/3 (000000)..."
I1114 22:01:10.000763       1 deleter.go:324] Cleanup pv "local-pv-ae71ec75": StderrBuf - "shred: /mnt/fast-disks/vde: pass 3/3 (000000)...1.9GiB/10GiB 19%"
I1114 22:01:15.013790       1 deleter.go:324] Cleanup pv "local-pv-ae71ec75": StderrBuf - "shred: /mnt/fast-disks/vde: pass 3/3 (000000)...4.1GiB/10GiB 41%"
I1114 22:01:20.001734       1 deleter.go:324] Cleanup pv "local-pv-ae71ec75": StderrBuf - "shred: /mnt/fast-disks/vde: pass 3/3 (000000)...6.4GiB/10GiB 64%"
I1114 22:01:25.001701       1 deleter.go:324] Cleanup pv "local-pv-ae71ec75": StderrBuf - "shred: /mnt/fast-disks/vde: pass 3/3 (000000)...8.8GiB/10GiB 88%"
I1114 22:01:27.854147       1 deleter.go:324] Cleanup pv "local-pv-ae71ec75": StderrBuf - "shred: /mnt/fast-disks/vde: pass 3/3 (000000)...10GiB/10GiB 100%"
I1114 22:01:28.236186       1 deleter.go:288] Completed cleanup of pv "local-pv-ae71ec75"
I1114 22:01:33.248720       1 deleter.go:170] Deleting pv local-pv-ae71ec75 after successful cleanup
I1114 22:01:33.263278       1 cache.go:64] Updated pv "local-pv-ae71ec75" to cache
I1114 22:01:33.275368       1 cache.go:64] Updated pv "local-pv-ae71ec75" to cache
I1114 22:03:34.369193       1 cache.go:73] Deleted pv "local-pv-ae71ec75" from cache
I1114 22:03:43.297789       1 discovery.go:423] Found new volume at host path "/mnt/fast-disks/vde" with capacity 10737418240, creating Local PV "local-pv-ae71ec75", required volumeMode "Filesystem"
I1114 22:03:43.315503       1 cache.go:55] Added pv "local-pv-ae71ec75" to cache
I1114 22:03:43.315921       1 discovery.go:457] Created PV "local-pv-ae71ec75" for volume at "/mnt/fast-disks/vde"

Which issue(s) this PR fixes:

Fixes #

Special notes for your reviewer:

/cc @jsafrane @msau42 @mpatlasov

Release note:

fix: Do not clean PV's with a deletion timestamp

@dobsonj
fix: Do not clean PV's with a deletion timestamp
2c10d7c 
After the volume has been successfully cleaned, the deleter sends a Delete
request to the API server. It should not continue attempting to clean the PV
at this point if the object still exists, as the PV may disappear at any time,
and we don't want cleanup to be in progress when that happens.
@k8s-ci-robot k8s-ci-robot added kind/bug Categorizes issue or PR as related to a bug. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. size/S Denotes a PR that changes 10-29 lines, ignoring generated files. labels Nov 14, 2024
@jsafrane
Copy link
Contributor

/lgtm
/approve

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Nov 19, 2024
@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: dobsonj, jsafrane

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Nov 19, 2024
@k8s-ci-robot k8s-ci-robot merged commit a379044 into kubernetes-sigs:master Nov 19, 2024
12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@jsafrane jsafrane Awaiting requested review from jsafrane

@mpatlasov mpatlasov Awaiting requested review from mpatlasov

@msau42 msau42 Awaiting requested review from msau42

Assignees

@jsafrane jsafrane

Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. kind/bug Categorizes issue or PR as related to a bug. lgtm "Looks good to me", indicates that a PR is ready to be merged. size/S Denotes a PR that changes 10-29 lines, ignoring generated files.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@dobsonj @jsafrane @k8s-ci-robot