kernel hangs up while backend disk is missing

[cofol1986]

Hey guys,
I test flashcache under different kinds of exceptions, here is my step, I'm using flashcache 3.1.1 under centos 6.4 with kernel 3.10.58:
1 setup flashcache dev and format it with ext4;
2 write some data to the ext4;
3 unplug the the disk and writing some more data onto the cachedev, then there will be lots of error in syslog, then the kernel hangs up happendly;
in the src code, there is verify() function that will cause kernel panic, but in my testbed kernel hangup without any trace dump, only error report in flashcache_do_pending_error() like this:
kernel: device-mapper: flashcache: flashcache_do_pending_error: error -5 block 2360 action 2
kernel: device-mapper: flashcache: flashcache_io_callback: io error -5 block 6027584 action 2

so, pls give me some hints why the kernel hang up?

[Damon-sh]

I have the same problem ,under Redhat 7 with kernel 3.10.0-123.el7.x86_64.
Looking forward to solve.Thanks!

[ghost]

Thank you for reporting this issue and appreciate your patience. We've notified the core team for an update on this issue. We're looking for a response within the next 30 days or the issue may be closed.

[flowerui]

I have the same problem, I'm using flashcache 3.1.2, under RHEL 6.5 (kernel version: 2.6.32-431.el6.x86_64), format cachedev with xfs, WRITE_BACK mode

<3>device-mapper: flashcache: flashcache_io_callback: io error -5 block 422251800 action 2
<3>device-mapper: flashcache: flashcache_io_callback: io error -5 block 422251808 action 2
<3>device-mapper: flashcache: flashcache_io_callback: io error -5 block 422251816 action 2
<3>device-mapper: flashcache: flashcache_io_callback: io error -5 block 29754232 action 2
<3>device-mapper: flashcache: flashcache: CLEAN: Cache metadata write failed ! error -5 block 221006072
<3>device-mapper: flashcache: flashcache_do_pending_error: error -5 block 221006072 action 4
<3>Buffer I/O error on device dm-11, logical block 3716883
<4>lost page write due to I/O error on dm-11
<3>device-mapper: flashcache: flashcache: WRITE: Cache metadata write failed ! error -5 block 29735064
<3>device-mapper: flashcache: flashcache_do_pending_error: error -5 block 29735064 action 2
<4>Pid: 105, comm: events/6 Not tainted 2.6.32-431.el6.x86_64 #1
<4>Call Trace:
<4> [] ? flashcache_do_pending_error+0x2a7/0x330 [flashcache]
<4> [] ? flashcache_do_pending+0x25/0x30 [flashcache]
<4> [] ? flashcache_md_write_done+0x3d8/0x5d0 [flashcache]
<4> [] ? flashcache_md_write_done+0x0/0x5d0 [flashcache]
<4> [] ? process_jobs+0x2a/0x60 [flashcache]
<4> [] ? do_work+0x0/0x70 [flashcache]
<4> [] ? do_work+0x1c/0x70 [flashcache]
<4> [] ? worker_thread+0x170/0x2a0
<4> [] ? autoremove_wake_function+0x0/0x40
<4> [] ? worker_thread+0x0/0x2a0
<4> [] ? kthread+0x96/0xa0
<4> [] ? child_rip+0xa/0x20
<4> [] ? kthread+0x0/0xa0
<4> [] ? child_rip+0x0/0x20
<0>Kernel panic - not syncing: VERIFY: assertion (dmc->cache_mode != FLASHCACHE_WRITE_BACK) failed at /builddir/build/BUILD/flashcache-3.1.2/src/flashcache_main.c (440)
<0>
<4>Pid: 105, comm: events/6 Not tainted 2.6.32-431.el6.x86_64 #1
<4>Call Trace:
<4> [] ? panic+0xa7/0x16f
<4> [] ? show_trace+0x15/0x20
<4> [] ? flashcache_do_pending_error+0x2c8/0x330 [flashcache]
<4> [] ? flashcache_do_pending+0x25/0x30 [flashcache]
<4> [] ? flashcache_md_write_done+0x3d8/0x5d0 [flashcache]
<4> [] ? flashcache_md_write_done+0x0/0x5d0 [flashcache]
<4> [] ? process_jobs+0x2a/0x60 [flashcache]
<4> [] ? do_work+0x0/0x70 [flashcache]
<4> [] ? do_work+0x1c/0x70 [flashcache]
<4> [] ? worker_thread+0x170/0x2a0
<4> [] ? autoremove_wake_function+0x0/0x40
<4> [] ? worker_thread+0x0/0x2a0
<4> [] ? kthread+0x96/0xa0
<4> [] ? child_rip+0xa/0x20
<4> [] ? kthread+0x0/0xa0
<4> [] ? child_rip+0x0/0x20

[justdoitjustdoit]

I have meet the same problem，I create a write back mode flashcache device， during I cp data to flashcache，I offline the SSD device， the flashcache happen panic，
[1813726.692300] device-mapper: flashcache: flashcache_io_callback: io error -5 block 50378824 action 2
[1813726.692303] Kernel panic - not syncing: VERIFY: assertion (dmc->cache_mode != FLASHCACHE_WRITE_BACK) failed at /root/flash_cache/src/flashcache_main.c (440)

[1813726.692219] [] dump_stack+0x19/0x1b
[1813726.692223] [] flashcache_do_pending_error+0x278/0x320 [flashcache]
[1813726.692226] [] flashcache_do_pending+0x25/0x30 [flashcache]
[1813726.692230] [] flashcache_md_write_done+0x3e8/0x6b0 [flashcache]
[1813726.692230] sd 0:2:13:0: rejecting I/O to offline device
[1813726.692232] device-mapper: flashcache: flashcache_io_callback: io error -5 block 50378752 action 2
[1813726.692236] [] ? flashcache_sync_blocks+0x3e0/0x3e0 [flashcache]
[1813726.692238] sd 0:2:13:0: rejecting I/O to offline device
[1813726.692241] [] process_jobs+0x29/0x60 [flashcache]
[1813726.692244] [] do_work+0x1c/0x70 [flashcache]
[1813726.692249] [] process_one_work+0x17b/0x470
[1813726.692250] device-mapper: flashcache: flashcache_io_callback: io error -5 block 50378760 action 2
[1813726.692253] [] worker_thread+0x11b/0x400
[1813726.692255] [] ? rescuer_thread+0x400/0x400
[1813726.692255] sd 0:2:13:0: rejecting I/O to offline device
[1813726.692257] device-mapper: flashcache: flashcache_io_callback: io error -5 block 50378768 action 2
[1813726.692261] [] kthread+0xcf/0xe0

[xzchsia]

i have fix this issue, this commit will be ok if you need

xzchsia@854ed31