READ BEFORE BUILDING:
   Avro/Parquet support has been added to filecrush but the Unit Tests are
   still being reviewed and updated. I've pushed the completed changes due
   to some requests but some tests still fail. Hence, when building 
   filecrush add the '-DskipTests' option to the mvn command.
   I'm finishing the tests review and will publish them asap. (Andre)

Hadoop filecrusher.

Turn many small files into fewer larger ones. Also change from text to sequence and other compression options in one pass.
Crush

NAME

  Crush - Crush small files in dfs to fewer, larger files

SYNOPSIS
  Crush [OPTION]... <input dir> <output dir> <timestamp>

DESCRIPTION

Crush consumes directories containing many small files with the same key and value types and creates fewer, larger files containing the same data. Crush is gives you the control to:

* Name the output files
* Ignore files that are "big enough"
* Limit the size of each output file
* Control the output compression codec
* Swap smaller files with generated large files in-place
* No long-running task problem

See the EXAMPLES section

ARGUMENTS

input dir
  The root of the directory tree to crush. Directories are found recursively.

output dir
  In non-clone mode, the directory where the output files should be written. In clone mode, the directory where the original files (that were combine into larger files) should be moved.

timestamp
  A 14 digit job timestamp used to uniquely name files. E.g. 20100221175612. Generate in a script with: date +%Y%m%d%H%M%S

GLOBAL OPTIONS

-?, --help
  Print this help message.

--threshold
  Percent threshold relative to the dfs block size over which a file becomes eligible for crushing. Must be in the (0, 1]. Default is 0.75, which means files smaller than or equal to 75% of a dfs block will be eligible for crushing. File greater than 75% of a dfs block will be left untouched.

--max-file-blocks
  The maximum number of dfs blocks per output file. Must be a positive integer. Small input files are associated with an output file under the assumption that input and output compression codecs have similar efficiency. Also, a directory containing a lot of data in many small files will be converted into a directory containing a fewer number of large files rather than one super-massive file. With the default value 8, 80 small files, each being 1/10th of a dfs block will be grouped into to a single output file since 8 * 1/10 = 8 dfs blocks. If there are 81 small files, each being 1/10th of a dfs block, two output files will be created. One output file contain the combined contents of 41 files and the second will contain the combined contents of the other 40. A directory of many small files will be converted into fewer number of larger files where each output file is roughly the same size.

--compress
  Fully qualified class name of the compression codec to use when writing data. It is permissible to use "none" and "gzip" to indicate no compression and org.apache.hadoop.io.compress.GzipCodec, respectively.

--clone
  Use clone mode. Useful for external Hive tables. In clone mode, the small files are replaced with the larger files. The small files are moved to a subdirectory of the output dir argument. The subdirectory is same as the original directory rooted at output dir. For example, assume the input dir argument and output dir argument are /user/example/input and /user/example/output, respectively. If a file was originally /user/example/input/my-dir/smallfile, then after the clone, the original file would be located in /user/example/output/user/example/input/my-dir/smallfile.

--info
  Print information to the console about what the crush is doing.

--verbose
  Print even more information to the console about what the crush is doing.

DIRECTORY OPTIONS

If specified, these options must be appear as a group. When specifying multiple groups of these options, order matters. Defaults for directory options are not used if any are specified. See the EXAMPLES section.

--regex
  Regular expression that matches a directory name. Defaults to .+ if no directory options are specified at all. Empty directories are not required to have a matching regex. Conceptually similar to the first argument of String.replaceAll().

--replacement
  Replacement string used with corresponding regex to name output files. Defaults to crushed_file-${crush.timestamp}-${crush.task.num}-${crush.file.num} if no directory options are specified at all. The placeholder ${crush.timestamp} refers to the command line argument. ${crush.task.num} refers to the reducer number. ${crush.file.num} is a zero-based count of files producer by a specific reducer. The first file written by a reducer will have ${crush.file.num} = 0, the second = 1, the third = 2, etc. Conceptually similar to the second argument of String.replaceAll().

--input-format
  Fully qualified class name of the input format for the data in a directory. Can use the "text" and "sequence" shortcuts for org.apache.hadoop.mapred.TextInputFormat and org.apache.hadoop.mapred.SequenceFileInputFormat, respectively. Defaults to sequence if no directory options are specified.

--output-format
  Fully qualified class name of the output format to use when writing the output file for a directory. Can use the "text" and "sequence" shortcuts for org.apache.hadoop.mapred.TextOutputFormat and org.apache.hadoop.mapred.SequenceFileOutputFormat, respectively. Defaults to sequence if no directory options are specified.

EXAMPLES

Say we have the following files:

/user/example/work/input/
                         small-file1
                         small-file2
                         small-file3
                         small-file4
                         big-enough-file
                         subdir/
                                 small-file6
                                 small-file7
                                 small-file8
                                 medium-file1
                                 medium-file2

And we invoke the crush like this:

  Crush /user/example/work/input /user/example/work/output 20100221175612

Since we have not specified any of the directory options, the default regex, replacement, input-format, and output-format are used. We will get:

/user/example/work/
                   input/
                         small-file1
                         small-file2
                         small-file3
                         small-file4
                         subdir/
                                 small-file6
                                 small-file7
                                 small-file8
                                 medium-file1
                                 medium-file2
                   output/
                          crushed_file-20100221175612-0-0
                          big-enough-file
                          subdir/
                                 crushed_file-20100221175612-1-0
                                 crushed_file-20100221175612-1-1

Where:

crushed_file-20100221175612-0-0 = small-file1 + small-file2 + small-file3 + small-file4

crushed_file-20100221175612-1-0 = medium-file1 + small-file6 + small-file8

crushed_file-20100221175612-1-1 = medium-file2 + small-file7

Notice how big-enough-file was moved to the output directory. The input directory contains only the files that were combined into the larger files.

By default, the output file names end with two numbers. The first number is the task number of the reducer that wrote the file. The second number is the zero-based file count of that specific reducer. So a file ending with 0-0 was produced by reducer 0 and was the first file written by that reducer. A file ending 0-1 is the second file written by that reducer. A file ending 1-0 was produced by reducer 1 and was the first file written by that reducer. In the example, notice how the directory subdir was converted into two files. If mapred.reduce.tasks permits, multiple reducers can cooperate to crush a large directory.

Now a clone example. Say we invoked the crush like this:

  Crush --clone /user/example/work/input /user/example/clone 20100221175612

With the clone option. We would end up with:

/user/example/
              work/input/
                         crushed_file-20100221175612-0-0
                         big-enough-file
                         subdir/
                                crushed_file-20100221175612-1-0
                                crushed_file-20100221175612-1-1
              clone/user/example/input/
                                       small-file1
                                       small-file2
                                       small-file3
                                       small-file4
                                       subdir/
                                              small-file6
                                              small-file7
                                              small-file8
                                              medium-file1
                                              medium-file2

Note how the original directory structure of /user/example/input as it appeared before the crush is reproduced in /user/example/clone. The small files that were combined are moved to the clone directory while the output files and file that were "big enough" are now in the inpu directory. Clone mode is useful for crushing external Hive tables. Just make sure that there are no Hive queries running on the table because they will fail when the small files are moved to the clone directory.

Now we try an example using the directory options. Say we invoke the crush like this to control the output file names:

  Crush \
  --regex=.*/(.+) \
  --replacement=$1-${crush.timestamp}-${crush.task.num}-${crush.file.num} \
  --input=sequence \
  --output=sequence \
  /user/example/work/input /user/example/work/output 20100221175612

The --regex and --replacement arguments are similar to the arguments passed to String.replaceAll(). The regex argument matches the final part of a directory path. For /user/example/work/input, it will match input. For /user/example/work/input/subdir, it will match subdir. For matching purposes, a directory path does not have a trailing slash. The replacement argument refers to the match group by number to rename the file. The result is:

/user/example/work/output/
                          input-20100221175612-0-0
                          big-enough-file
                          subdir/
                                 subdir-20100221175612-1-0
                                 subdir-20100221175612-1-1

The regex and replacement options are useful for naming the output files when crushing external Hive tables that are partitioned into directories whose names have business significance.

The following invocation fails:

  Crush \
  --regex=.*/input \
  --replacement=input-${crush.timestamp}-${crush.task.num}-${crush.file.num} \
  --input=sequence \ 
  --output=sequence \
  /user/example/work/input /user/example/work/output 20100221175612

Since we have specified some directory options, we must ensure that all directories in hierarchy rooted at the input argument have a matching regex (since the default regex is no longer applicable). In this invocation, there is no regex argument that matches /user/example/work/input/subdir. We must change it to:

  Crush \
  --regex=.*/input \
  --replacement=input-${crush.timestamp}-${crush.task.num}-${crush.file.num} \
  --input=sequence \
  --output=sequence \
  --regex=.*/subdir \
  --replacement=as-text-${crush.timestamp}-${crush.task.num}-${crush.file.num} \
  --input=sequence \
  --output=text \
  /user/example/work/input /user/example/work/output 20100221175612

This will yield:

/user/example/work/output/
                          input-20100221175612-0-0
                          big-enough-file
                          subdir/
                                 as-text-20100221175612-1-0
                                 as-text-20100221175612-1-1

Notice subdir has two files whose names differ only by the ${crush.file.num} value. Without the ${crush.file.num}, file names are not guaranteed to be unique.

NOTES

This program creates a temporary directories in "tmp" of the executing user's home directory in dfs.

https://zenodo.org/badge/doi/10.5281/zenodo.11038.png