forked from jdb/jdb.github.com
-
Notifications
You must be signed in to change notification settings - Fork 0
/
Copy pathlvm.html
424 lines (395 loc) · 20.4 KB
/
lvm.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<title>An upgrade safety net with the logical volume manager — bits v0.7 documentation</title>
<link rel="stylesheet" href="static/sphinxdoc.css" type="text/css" />
<link rel="stylesheet" href="static/pygments.css" type="text/css" />
<script type="text/javascript">
var DOCUMENTATION_OPTIONS = {
URL_ROOT: '',
VERSION: '0.7',
COLLAPSE_INDEX: false,
FILE_SUFFIX: '.html',
HAS_SOURCE: true
};
</script>
<script type="text/javascript" src="static/jquery.js"></script>
<script type="text/javascript" src="static/underscore.js"></script>
<script type="text/javascript" src="static/doctools.js"></script>
<link rel="top" title="bits v0.7 documentation" href="index.html" />
<link rel="next" title="Les pseudo-terminaux" href="pty.html" />
<link rel="prev" title="One application, multiple languages" href="i18n.html" />
</head>
<body>
<div class="related">
<h3>Navigation</h3>
<ul>
<li class="right" style="margin-right: 10px">
<a href="genindex.html" title="General Index"
accesskey="I">index</a></li>
<li class="right" >
<a href="py-modindex.html" title="Python Module Index"
>modules</a> |</li>
<li class="right" >
<a href="pty.html" title="Les pseudo-terminaux"
accesskey="N">next</a> |</li>
<li class="right" >
<a href="i18n.html" title="One application, multiple languages"
accesskey="P">previous</a> |</li>
<li><a href="index.html">bits v0.7 documentation</a> »</li>
</ul>
</div>
<div class="sphinxsidebar">
<div class="sphinxsidebarwrapper">
<h3><a href="index.html">Table Of Contents</a></h3>
<ul>
<li><a class="reference internal" href="#">An upgrade safety net with the logical volume manager</a><ul>
<li><a class="reference internal" href="#a-trick-for-fake-physical-partitions">A trick for fake physical partitions</a></li>
<li><a class="reference internal" href="#setting-up-an-logical-volume">Setting up an logical volume</a></li>
<li><a class="reference internal" href="#design-of-an-upgrade-plan">Design of an upgrade plan</a></li>
<li><a class="reference internal" href="#rollback-of-a-failed-upgrade">Rollback of a failed upgrade</a></li>
<li><a class="reference internal" href="#fixing-and-re-applying-the-upgrade">Fixing and re-applying the upgrade</a></li>
</ul>
</li>
</ul>
<h4>Previous topic</h4>
<p class="topless"><a href="i18n.html"
title="previous chapter">One application, multiple languages</a></p>
<h4>Next topic</h4>
<p class="topless"><a href="pty.html"
title="next chapter">Les pseudo-terminaux</a></p>
<h3>This Page</h3>
<ul class="this-page-menu">
<li><a href="sources/lvm.txt"
rel="nofollow">Show Source</a></li>
</ul>
<div id="searchbox" style="display: none">
<h3>Quick search</h3>
<form class="search" action="search.html" method="get">
<input type="text" name="q" size="18" />
<input type="submit" value="Go" />
<input type="hidden" name="check_keywords" value="yes" />
<input type="hidden" name="area" value="default" />
</form>
<p class="searchtip" style="font-size: 90%">
Enter search terms or a module, class or function name.
</p>
</div>
<script type="text/javascript">$('#searchbox').show(0);</script>
</div>
</div>
<div class="document">
<div class="documentwrapper">
<div class="bodywrapper">
<div class="body">
<div class="section" id="an-upgrade-safety-net-with-the-logical-volume-manager">
<h1>An upgrade safety net with the logical volume manager<a class="headerlink" href="#an-upgrade-safety-net-with-the-logical-volume-manager" title="Permalink to this headline">¶</a></h1>
<p>It is the story of a website with files and a database which gets into
production and then, after a while, needs to have its schema corrected
and related files upgraded safely. As usual, it would have been better
to get the schema right from the beginning but as errors always happen
at some point, technologies such as the LVM snapshots exists so that
the website gets its invasive (read scary) upgrade but there is a good
safety net in case of a mess.</p>
<p>The idea is to take a <strong>snapshot</strong> of the <em>disk partition</em>: it is like
making a backup of a partition and copy it back later whenever
required, with the added benefit of :</p>
<ul class="simple">
<li>the backup is done in almost not time, it is <em>not</em> depending on the
size of the disk,</li>
<li>there is no actual copy of the blocks so it is very light on the
io and space available,</li>
<li>most importantly, even if the backup happens during a massive
modification of the filesystem structure, the filesystem data
structures are coherent.</li>
</ul>
<p>It is not magic though, the snapshot itself is a partition and it
grows along when new data is added to the original partition to which
the snapshot is attached. <em>Make sure the snapshot partition is big
enough to contain the updates to the original partition or suppress
the snapshot partition before it gets full</em>.</p>
<p>Note that the operation needs to be done from the administrator
account.</p>
<div class="section" id="a-trick-for-fake-physical-partitions">
<h2>A trick for fake physical partitions<a class="headerlink" href="#a-trick-for-fake-physical-partitions" title="Permalink to this headline">¶</a></h2>
<p>Before showing how to setup a logical partition with lvm, let’s first
present a cool trick to create <em>virtual physical partitions</em>: it is
not related to lvm per se but it eases experimentation with RAID or LVM
without mangling a real disk.</p>
<p>With <em>losetup</em>‘s loopback partitions, it is possible to turn a normal
file into a disk image device, available in <em>/dev</em>. Here we create,
with <em>dd</em>, a file of 1.5 gigabyte called <em>loop1.raw</em> in the
current directory, and make it available under <em>/dev/loop1</em></p>
<div class="highlight-sh"><div class="highlight"><pre>~# dd <span class="k">if</span><span class="o">=</span>/dev/zero <span class="nv">of</span><span class="o">=</span>loop1.raw <span class="nv">bs</span><span class="o">=</span>1M <span class="nv">count</span><span class="o">=</span>40
~# losetup /dev/loop1 loop1.raw
~# grep loop /proc/partitions
7 1 ... loop1
</pre></div>
</div>
<p>There was no partitions available called <em>loop1</em> and it appeared after
the <em>losetup command</em>. We will set up the partition to use with lvm in
the next paragraph.</p>
</div>
<div class="section" id="setting-up-an-logical-volume">
<h2>Setting up an logical volume<a class="headerlink" href="#setting-up-an-logical-volume" title="Permalink to this headline">¶</a></h2>
<p>To represent LVM hot resize, hot backup or physical disk aggregation,
there are three object you need to be familiar with, and for each of
these objects, there is the corresponding shell commands for <em>creating</em>,
<em>listing</em> and <em>removing</em> said object.</p>
<dl class="docutils">
<dt><cite>physical volume</cite>: a physical partitions of one of the physical hard</dt>
<dd>drive which has been allocated to lvm. Managed with <em>pvcreate</em>,
<em>pvs</em> and <em>pvremove</em>. The <em>s</em> of <em>pvs</em> is for <em>show</em>.</dd>
<dt><cite>volume group</cite>: aggregation of <strong>physical volumes</strong>. When I first read</dt>
<dd>documentation about LVM, I thought the volume groups was meant to
aggregate the logical partitions and data, but it is the other way
round, they aggregate the real partitions. Managed with <em>vgcreate</em>,
<em>vgs</em> and <em>vgremove</em>.</dd>
<dt><cite>logical volume</cite>: volume is what you will use in the end with <em>mkfs</em></dt>
<dd>and <em>mount</em>. This is the device which look like the traditional
partitions but with additional features like snapshots or resize on
the fly. Managed with <em>lvcreate</em>, <em>lvs</em> and <em>lvemove</em>.</dd>
</dl>
<p>Use <em>pvcreate</em> from the <em>lvm2</em> package to initialise the partition for
use with lvm</p>
<div class="highlight-sh"><div class="highlight"><pre>~# pvcreate /dev/loop1
Physical volume <span class="s2">"/dev/loop1"</span> successfully created
</pre></div>
</div>
<p>A disk can only be added once to a volume group, multiple physical
disks compose a <em>volume group</em></p>
<div class="highlight-sh"><div class="highlight"><pre>~# vgcreate datadisks /dev/loop1
Volume group <span class="s2">"datadisks"</span> successfully created
</pre></div>
</div>
<p>So far, the partitions available have not changed and the
<em>/dev/datadisks/website</em> partition does not exists. A logical volume
can now be created, it has a <em>name</em> and a <em>size</em> parameter and is inside a
<em>group</em></p>
<div class="highlight-sh"><div class="highlight"><pre>~# lvcreate -n website -L 12M datadisks
Logical volume <span class="s2">"website"</span> created
~# lvs
LV VG Attr LSize Origin Snap% Move Log Copy% Convert
website datadisks -wi-a...M
~# grep dm /proc/partitions <span class="o">&&</span> ls /dev/datadisks/website
252 ... dm-0
/dev/datadisks/website
</pre></div>
</div>
<p>Among the partitions, a new <em>dm</em> entry is shown (I’ll bet it
stands for <em>device mapper</em>), the device is available in <em>/dev</em>
contained in a directory named after the volume group.</p>
<p>As usual, the partition must be formatted and mounted to be integrated
to the filesystem</p>
<div class="highlight-sh"><div class="highlight"><pre>~# mkfs.ext4 /dev/datadisks/website > /dev/null
~# mkdir -p ./mnt/website <span class="o">&&</span> mount /dev/datadisks/website ./mnt/website
</pre></div>
</div>
</div>
<div class="section" id="design-of-an-upgrade-plan">
<h2>Design of an upgrade plan<a class="headerlink" href="#design-of-an-upgrade-plan" title="Permalink to this headline">¶</a></h2>
<p>Let’s compose a dummy three-tier website, that we will have to
upgrade, corrupt, rollback, etc</p>
<div class="highlight-sh"><div class="highlight"><pre>~# touch ./mnt/website/database
~# touch ./mnt/website/index.html
~# add_new_user <span class="o">()</span> <span class="o">{</span>
<span class="nb">echo</span> <span class="s2">"name:$1,age:$2"</span> >> ./mnt/website/database ; <span class="o">}</span>
</pre></div>
</div>
<p>With the adapted amount of marketing and public relation, the website
is put in production and made available to the public. Everyday,
torrents of new users line up to subscribe</p>
<div class="highlight-sh"><div class="highlight"><pre>~# add_new_user alice 29
~# add_new_user bob 18
~# cat ./mnt/website/database
name:alice,age:29
name:bob,age:18
</pre></div>
</div>
<p><em>Sparky the architect</em> have realised that the database schema must be
upgraded to include an <em>id</em> for each user. It should end up look like this:</p>
<div class="highlight-python"><pre>id=001,name:alice,age:29
id=002,name:bob,age:18</pre>
</div>
<p>Also, the website in production is not web2.0 enough, so a designer
has done a great job beautifying a new prototype, which is added to
the upgrade procedure. So the upgrade procedure is</p>
<div class="highlight-sh"><div class="highlight"><pre>~# upgrade_schema_and_website <span class="o">()</span> <span class="o">{</span>
<span class="c"># Web changes</span>
touch ./mnt/website/<span class="o">{</span>social-caramels.js,ponies.js,eye-candy.css<span class="o">}</span>
<span class="c"># API upgrade: now there is an id</span>
add_new_user <span class="o">()</span> <span class="o">{</span>
<span class="nb">echo</span> <span class="s2">"id:$RANDOM,name:$1,age:$2"</span> >> ./mnt/website/database ; <span class="o">}</span>
<span class="c"># For the "db schema", you don't want to know ...</span>
nl -n rz -w 3 ./mnt/website/database <span class="se">\</span>
| sed <span class="s1">'s/\t/,/; s/^/if:/'</span> > ./mnt/website/database.new
mv ./mnt/website/database<span class="o">{</span>.new,<span class="o">}</span>
<span class="o">}</span>
</pre></div>
</div>
</div>
<div class="section" id="rollback-of-a-failed-upgrade">
<h2>Rollback of a failed upgrade<a class="headerlink" href="#rollback-of-a-failed-upgrade" title="Permalink to this headline">¶</a></h2>
<p>The system administrator tunes a transaction API and convince the
operator to use it the day of the upgrade. Before doing any change,
the operator must use the command <em>transaction</em>. If all is well after
a few days of testing, the command <em>remove_snapshot</em> is used, else the
operator can use the <em>abort</em> function.</p>
<p>The transaction functions are built on top of the LVM snapshot</p>
<div class="highlight-sh"><div class="highlight"><pre>~# transaction <span class="o">()</span> <span class="o">{</span>
lvcreate -s -n backup -L 24M /dev/datadisks/website ; <span class="o">}</span>
~# abort <span class="o">()</span> <span class="o">{</span>
mkdir ./mnt/backup
mount /dev/datadisks/backup ./mnt/backup
<span class="c"># tar cf - -C ./mnt/backup . | tar x -C ./mnt/website</span>
rsync --del -a ./mnt/backup/ ./mnt/website/ ;
add_new_user <span class="o">()</span> <span class="o">{</span>
<span class="nb">echo</span> <span class="s2">"name:$1,age:$2"</span> >> ./mnt/website/database ; <span class="o">}</span>
<span class="o">}</span>
~# remove_snapshot <span class="o">()</span> <span class="o">{</span>
umount /dev/datadisks/backup
lvremove -f /dev/datadisks/backup ; <span class="o">}</span>
</pre></div>
</div>
<p>The upgrade procedure requires the database to go read only, no new
users can be created. Comes the night of the upgrade</p>
<div class="highlight-sh"><div class="highlight"><pre>~# transaction
Logical volume <span class="s2">"backup"</span> created
</pre></div>
</div>
<div class="highlight-sh"><div class="highlight"><pre>~# upgrade_schema_and_website
</pre></div>
</div>
<p>At dawn, the db looks like</p>
<div class="highlight-sh"><div class="highlight"><pre>~# cat ./mnt/website/database
<span class="k">if</span>:001,name:alice,age:29
<span class="k">if</span>:002,name:bob,age:18
</pre></div>
</div>
<p>Ouuuch man! it is corrupted, there is no ‘id’ column instead it is
written ‘if’ everywhere now and we have no clue why. We need to go
back to the lab, figure out what happened... What do we do now with
this mess now: we need roll back so that the production site can
continue. Easy, here is the command</p>
<div class="highlight-sh"><div class="highlight"><pre>~# abort
</pre></div>
</div>
<p>The abort is based on the <tt class="docutils literal"><span class="pre">lvcreate</span> <span class="pre">--snapshot</span></tt> and really is the
core of this article. Now, to control that the rollback went fine</p>
<div class="highlight-sh"><div class="highlight"><pre>~# cat ./mnt/website/database
name:alice,age:29
name:bob,age:18
~# ls ./mnt/website/ponies.js 2>&1 <span class="o">||</span> <span class="nb">true</span>
ls: cannot access ./mnt/website/ponies.js: No such file or directory
</pre></div>
</div>
<p>Ok, the situation is similar as before the upgrade. The service can be
restored.</p>
<div class="admonition note">
<p class="first admonition-title">Note</p>
<p class="last">It is actually not easy to get the right options for <tt class="docutils literal"><span class="pre">rsync</span></tt> or
<tt class="docutils literal"><span class="pre">tar</span></tt> for re-install the data of the backup. The version 2.02.57 of
lvm with the device mapper hopefully integrated into the linux
2.6.33 will be more convenient by integrating it to the
<tt class="docutils literal"><span class="pre">lvconvert</span></tt> command of the LVM set of commands: no need for
everyone to write (and debug) a custom <tt class="docutils literal"><span class="pre">abort</span></tt> function like we
did. The new lvconvert command could be available in the
distributions in the second half of 2010.</p>
</div>
</div>
<div class="section" id="fixing-and-re-applying-the-upgrade">
<h2>Fixing and re-applying the upgrade<a class="headerlink" href="#fixing-and-re-applying-the-upgrade" title="Permalink to this headline">¶</a></h2>
<p>Three weeks later, many more users have been created</p>
<div class="highlight-sh"><div class="highlight"><pre>~# add_new_user robwilco 35
~# add_new_user DuncanMacLeod 539
~# cat ./mnt/website/database
name:alice,age:29
name:bob,age:18
name:robwilco,age:35
name:DuncanMacLeod,age:539
</pre></div>
</div>
<p>R&D has come up with a <em>complete</em> re-design of the upgrade procedure:
a snapshot and some <em>correct</em> database mangling commands. Only the
schema upgrade was modified</p>
<div class="highlight-sh"><div class="highlight"><pre>~# upgrade_schema_and_website <span class="o">()</span> <span class="o">{</span>
<span class="c"># Same as before ...</span>
touch ./mnt/website/<span class="o">{</span>social-caramels.js,ponies.js,eye-candy.css<span class="o">}</span>
<span class="c"># Same as before ...</span>
add_new_user <span class="o">()</span> <span class="o">{</span>
<span class="nb">echo</span> <span class="s2">"id:$RANDOM,name:$1,age:$2"</span> >> ./mnt/website/database ; <span class="o">}</span>
<span class="c"># Correction added here: substituted 'if' by 'id'</span>
nl -n rz -w 5 ./mnt/website/database <span class="se">\</span>
| sed <span class="s1">'s/\t/,/; s/^/id:/'</span> > ./mnt/website/database.new
mv ./mnt/website/database.new ./mnt/website/database
<span class="o">}</span>
~# upgrade_schema_and_website
~# cat ./mnt/website/database
id:00001,name:alice,age:29
id:00002,name:bob,age:18
id:00003,name:robwilco,age:35
id:00004,name:DuncanMacLeod,age:539
</pre></div>
</div>
<p>At dawn, the database is correct, the snapshot safety net was
thankfully not used. It is possible to confirm the upgrade by removing
the snapshot</p>
<div class="highlight-sh"><div class="highlight"><pre>~# remove_snapshot
Logical volume <span class="s2">"backup"</span> successfully removed
</pre></div>
</div>
<p>Obviously, removing the snapshot does not impact the original partition</p>
<div class="highlight-sh"><div class="highlight"><pre>~# cat ./mnt/website/database
id:00001,name:alice,age:29
id:00002,name:bob,age:18
id:00003,name:robwilco,age:35
id:00004,name:DuncanMacLeod,age:539
</pre></div>
</div>
<p>We are done with this howto, to clean up after this exercice</p>
<div class="highlight-sh"><div class="highlight"><pre>~# umount ./mnt/website
~# lvremove -f /dev/datadisks/backup 2> /dev/null <span class="o">||</span> <span class="nb">true</span>
~# lvremove -f /dev/datadisks/website
Logical volume <span class="s2">"website"</span> successfully removed
~# vgremove datadisks
Volume group <span class="s2">"datadisks"</span> successfully removed
~# pvremove /dev/loop1
Labels on physical volume <span class="s2">"/dev/loop1"</span> successfully wiped
~# losetup -d /dev/loop1
~# rm -r ./mnt/backup ./mnt/website loop1.raw
</pre></div>
</div>
<p><em>9 Feb 2010, this article was verified with the wordish_ module</em></p>
</div>
</div>
</div>
</div>
</div>
<div class="clearer"></div>
</div>
<div class="related">
<h3>Navigation</h3>
<ul>
<li class="right" style="margin-right: 10px">
<a href="genindex.html" title="General Index"
>index</a></li>
<li class="right" >
<a href="py-modindex.html" title="Python Module Index"
>modules</a> |</li>
<li class="right" >
<a href="pty.html" title="Les pseudo-terminaux"
>next</a> |</li>
<li class="right" >
<a href="i18n.html" title="One application, multiple languages"
>previous</a> |</li>
<li><a href="index.html">bits v0.7 documentation</a> »</li>
</ul>
</div>
<div class="footer">
© Copyright 2009, Jean Daniel Browne.
Created using <a href="http://sphinx.pocoo.org/">Sphinx</a> 1.0.
</div>
</body>
</html>