Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve the instructions to add benchmark files manually #543

Merged
merged 8 commits into from
Oct 16, 2023
Merged

Improve the instructions to add benchmark files manually #543

Changes from 7 commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
21ce86b
Improved the instructions to add benchmark files manually in doc/BENC…
Oct 6, 2023
d1183b5
Added all the steps for creating benchmark files
Oct 6, 2023
cc0c27b
Corrected job name
Oct 6, 2023
415fd34
Removed filename argument to create directory
Oct 6, 2023
a60451a
corrected the recommendations
Oct 9, 2023
acac3ac
Added mount-dir variable
Oct 9, 2023
58faac4
Corrected the ordering
Oct 9, 2023
469ea1c
Removed unnecessary space
Oct 10, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 13 additions & 5 deletions doc/BENCHMARKING.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@ We keep the records of benchmarking results in `gh-pages` branch and the perform
While our benchmark script is written for CI testing only, it is possible to run manually.
You can use the following steps.

1. Install dependencies and configure FUSE by running the following script in the repository:
1. Install dependencies and configure FUSE by running the following script in the `mountpoint-s3` repository:

bash .github/actions/install-dependencies/install.sh \
--fuse-version 2 \
Expand All @@ -39,10 +39,18 @@ You can use the following steps.
export S3_BUCKET_BENCH_FILE=bench_file_name
export S3_BUCKET_SMALL_BENCH_FILE=small_bench_file_name

3. Create the bench files manually in your bucket. The size of the files must be exactly the same as the size defined in fio configuration files. The easiest way to do this is running fio against your local file system first to let fio create the files for you, and then upload them to your S3 bucket using the AWS CLI. For example:

fio --directory=your_local_dir --filename=your_file_name mountpoint-s3/scripts/fio/read/seq_read_small.fio
aws s3 cp your_local_dir/your_file_name s3://${S3_BUCKET_NAME}/${S3_BUCKET_TEST_PREFIX}
3. Create the bench files manually in your S3 bucket. You can do that by mounting the bucket on your machine using Mountpoint. Then, running fio jobs against your mount directory to let fio create the files for you.

MOUNT_DIR = path/to/mount
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: remove the space.

Suggested change
MOUNT_DIR = path/to/mount
MOUNT_DIR=path/to/mount

mount-s3 DOC-EXAMPLE-BUCKET $MOUNT_DIR/ --prefix benchmark/ --part-size=16777216
cd mountpoint-s3/scripts/fio/
fio --directory=$MOUNT_DIR/ --filename=bench5MB.bin --create_only=1 read/seq_read_small.fio
fio --directory=$MOUNT_DIR/ --filename=bench100GB.bin --create_only=1 read/seq_read.fio
mkdir $MOUNT_DIR/bench_dir_100/ $MOUNT_DIR/bench_dir_1000/ $MOUNT_DIR/bench_dir_10000/ $MOUNT_DIR/bench_dir_100000/
fio --directory=$MOUNT_DIR/bench_dir_100 create/create_files_100.fio
fio --directory=$MOUNT_DIR/bench_dir_1000 create/create_files_1000.fio
fio --directory=$MOUNT_DIR/bench_dir_10000 create/create_files_10000.fio
fio --directory=$MOUNT_DIR/bench_dir_100000 create/create_files_100000.fio

4. Run the benchmark script for [throughput](../mountpoint-s3/scripts/fs_bench.sh) or [latency](../mountpoint-s3/scripts/fs_latency_bench.sh).

Expand Down