MongoDB routine cleanup

vocms0274

Introduction

There are three layers of mongoBD storage structure: dataSize, storageSize, and fileSize.

1. The dataSize metric is the sum of the sizes (in bytes) of all the documents and padding stored in the database.
2. The storageSize metric is equal to the size (in bytes) of all the data extents in the database. This number is larger than dataSize because it includes yet-unused space (in data extents) and space vacated by deleted or moved documents within extents.
3. The fileSize metric is equal to the size (in bytes) of all the data extents, index extents and yet-unused space (in data files) in the database. This metric represents the storage footprint of your database on disk. This is the size of all the data files.

More details of the MongoDB storage structure can be found: MongoBD storage structure.

Due to the structure above, the size of the data files will never decrease. Sometimes it could use up all the space in the partition. Therefore, a cleaning routine is required for MongoDB maintenance.

Procedure:

• Check the space usage of /data partition once a month (/data partition on the machine 0274)
• Clean the disk space when the usage exceeds 400 GB (40% of the disk).

Free Disk Space

There are two ways to free the disk space: compact and repair.

Compacting individual collections

You can compact individual collections using the compact command. This command rewrites and defragments all data in a collection, as well as all of the indexes on that collection. Important notes on compacting:

• This operation blocks all other database activity when running and will cause downtime.
• Compacting individual collections will not reduce your storage footprint on disk (i.e., your fileSize) but it will defragment the collections you compact.

Script:

1. ssh vocms0274.cern.ch
2. mongo (get into mongo shell)
3. show dbs
4. use unified
5. show collections
6. compact command (needs rework)
7. exit

Compacting the whole database

To compact all the databases on your server process, stop the mongod process and run it with the "--repair" option. Important notes on running a repair:

• This operation blocks all other database activity when running and will cause downtime.
• Running a repair requires free disk space equal to the size of your current data set plus 2 GB. If there no enough space in "data" partition (disk space 1TB), then we need to make copy in the /data/mongod/db directory into /eos/cms/store/unified/bk_mongod/ and run repair by setting it as dbpath.

Script:

ssh vocms0274.cern.ch

• When there is enough free disk space in /data partition

1. make backup in /eos/cms/store/unified/bk_mongod/ (copy db to this directory)
2. service mongod stop
3. rm -r /data/mongod/db/journal (disbale journal for repair) 4. mongod --repair --dbpath /data/mongod/db
4. service mongod start

• When there no enough free disk space in /data partition

1. make 2 backups in /eos/cms/store/unified/bk_mongod/ (one for backup, one for datafile reduction). For example: /eos/cms/store/unified/bk_mongod/db/ and /eos/cms/store/unified/bk_mongod/db_bk .
2. service mongod stop
3. cd /eos/cms/store/unified/bk_mongod/db_bk
4. rm -rf journal (disbale journal for repair)
5. mongod --repair --dbpath /eos/cms/store/unified/bk_mongod/db_bk (After a while, the data files will shrink)
6. cd /data/mongod/db/
7. rm -r /data/mongod/db/
8. cp -r /eos/cms/store/unified/bk_mongod/db_bk/* .
9. chown -R mongodb:mongodb * (change the ownership of all the files to mongodb, or else we can't start mongod)
10. service mongod start