Skip to content

Commit

Permalink
Merge pull request #27 from crytic/template-generation
Browse files Browse the repository at this point in the history
Template generation
  • Loading branch information
tuturu-tech authored Mar 29, 2024
2 parents e0f98c3 + b8d1ecd commit 14b770c
Show file tree
Hide file tree
Showing 32 changed files with 1,845 additions and 212 deletions.
6 changes: 6 additions & 0 deletions .gitmodules
Original file line number Diff line number Diff line change
@@ -1,3 +1,9 @@
[submodule "tests/test_data/lib/forge-std"]
path = tests/test_data/lib/forge-std
url = https://github.com/foundry-rs/forge-std
[submodule "tests/test_data/lib/properties"]
path = tests/test_data/lib/properties
url = https://github.com/crytic/properties
[submodule "tests/test_data/lib/solmate"]
path = tests/test_data/lib/solmate
url = https://github.com/transmissions11/solmate
8 changes: 6 additions & 2 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,8 +30,12 @@ Below is a rough outline of fuzz-utils's design:

```text
.
├── fuzzers # Contains supported fuzzer classes that parse and generate the test files
├── templates # String templates used for test generation
├── generate # Classes related to the `generate` command
| └── fuzzers # Supported fuzzer classes
├── parsing # Contains the main parser logic
| └── commands # Flags and execution logic per supported subparser
├── template # Classes related to the `template` command
├── templates # Common templates such as the default config and templates for test and harness generation
├── utils # Utility functions
├── main.py # Main entry point
└── ...
Expand Down
5 changes: 4 additions & 1 deletion Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -68,7 +68,10 @@ reformat:
test tests: $(VENV)/pyvenv.cfg
. $(VENV_BIN)/activate && \
solc-select use 0.8.19 --always-install && \
pytest $(T) $(TEST_ARGS)
cd tests/test_data && \
forge install && \
cd ../.. && \
pytest --ignore tests/test_data/lib $(T) $(TEST_ARGS)

.PHONY: package
package: $(VENV)/pyvenv.cfg
Expand Down
122 changes: 107 additions & 15 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,16 +1,22 @@
<img src="./logo.png" alt="Slither Static Analysis Framework Logo" width="500" />

# Automated tool for generating Foundry unit tests from smart contract fuzzer failed properties
# Automated utility tooling for smart contract fuzzers

`fuzz-utils` is a Python tool that generates unit tests from [Echidna](https://github.com/crytic/echidna) and [Medusa](https://github.com/crytic/medusa/tree/master) failed properties, using the generated reproducer files. It uses [Slither](https://github.com/crytic/slither) for determining types and jinja2 for generating the test files using string templates.
`fuzz-utils` is a set of Python tools that aim to improve the developer experience when using smart contract fuzzing.
The tools include:
- automatically generate unit tests from [Echidna](https://github.com/crytic/echidna) and [Medusa](https://github.com/crytic/medusa/tree/master) failed properties, using the generated reproducer files.
- automatically generate a Echidna/Medusa compatible fuzzing harness.

`fuzz-utils` uses [Slither](https://github.com/crytic/slither) for determining types and `jinja2` for generating the test files using string templates.

**Disclaimer**: Please note that `fuzz-utils` is **under development**. Currently, not all Solidity types are supported and some types (like `bytes*`, and `string`) might be improperly decoded from the corpora call sequences. We are investigating a better corpus format that will ease the creation of unit tests.

## Features
`fuzz-utils` provides support for:
- ✔️ Generating Foundry unit tests from the fuzzer corpus of single entry point fuzzing harnesses
- ✔️ Medusa and Echidna corpora
- ✔️ Solidity types: `bool`,`uint*`, `int*`, `address`, `struct`, `enum`, single-dimensional fixed-size arrays and dynamic arrays, multi-dimensional fixed-size arrays.
- ✔️ Generating Foundry unit tests from the fuzzer corpus of single entry point fuzzing harnesses.
- ✔️ Generating fuzzing harnesses, `Actor` contracts, and templated `attack` contracts to ease fuzzing setup.
- ✔️ Supports Medusa and Echidna corpora
- ✔️ Test generation supports Solidity types: `bool`,`uint*`, `int*`, `address`, `struct`, `enum`, single-dimensional fixed-size arrays and dynamic arrays, multi-dimensional fixed-size arrays.

Multi-dimensional dynamic arrays, function pointers, and other more complex types are in the works, but are currently not supported.
## Installation and pre-requisites
Expand All @@ -23,24 +29,110 @@ pip install fuzz-utils

These commands will install all the Python libraries and tools required to run `fuzz-utils`. However, it won't install Echidna or Medusa, so you will need to download and install the latest version yourself from its official releases ([Echidna](https://github.com/crytic/echidna/releases), [Medusa](https://github.com/crytic/medusa/releases)).

## Example
## Tools
The available tool commands are:
- [`init`](#initializing-a-configuration-file) - Initializes a configuration file
- [`generate`](#generating-unit-tests) - generates unit tests from a corpus
- [`template`](#generating-fuzzing-harnesses) - generates a fuzzing harness

### Generating unit tests

The `generate` command is used to generate Foundry unit tests from Echidna or Medusa corpus call sequences.

**Command-line options:**
- `compilation_path`: The path to the Solidity file or Foundry directory
- `-cd`/`--corpus-dir` `path_to_corpus_dir`: The path to the corpus directory relative to the working directory.
- `-c`/`--contract` `contract_name`: The name of the target contract.
- `-td`/`--test-directory` `path_to_test_directory`: The path to the test directory relative to the working directory.
- `-i`/`--inheritance-path` `relative_path_to_contract`: The relative path from the test directory to the contract (used for inheritance).
- `-f`/`--fuzzer` `fuzzer_name`: The name of the fuzzer, currently supported: `echidna` and `medusa`
- `--named-inputs`: Includes function input names when making calls
- `--config`: Path to the fuzz-utils config JSON file
- `--all-sequences`: Include all corpus sequences when generating unit tests.

**Example**

In order to generate a test file for the [BasicTypes.sol](tests/test_data/src/BasicTypes.sol) contract, based on the Echidna corpus reproducers for this contract ([corpus-basic](tests/test_data/echidna-corpora/corpus-basic/)), we need to `cd` into the `tests/test_data` directory which contains the Foundry project and run the command:
```bash
fuzz-utils ./src/BasicTypes.sol --corpus-dir echidna-corpora/corpus-basic --contract "BasicTypes" --test-directory "./test/" --inheritance-path "../src/" --fuzzer echidna
fuzz-utils generate ./src/BasicTypes.sol --corpus-dir echidna-corpora/corpus-basic --contract "BasicTypes" --test-directory "./test/" --inheritance-path "../src/" --fuzzer echidna
```

Running this command should generate a `BasicTypes_Echidna_Test.sol` file in the [tests](/tests/test_data/test/) directory of the Foundry project.
Running this command should generate a `BasicTypes_Echidna_Test.sol` file in the [test](/tests/test_data/test/) directory of the Foundry project.

## Command-line options
### Generating fuzzing harnesses

Additional options are available for the script:
The `template` command is used to generate a fuzzing harness. The harness can include multiple `Actor` contracts which are used as proxies for user actions, as well as `attack` contracts which can be selected from a set of premade contracts that perform certain common attack scenarios.

- `-cd`/`--corpus-dir` `path_to_corpus_dir`: The path to the corpus directory relative to the working directory.
- `-c`/`--contract` `contract_name`: The name of the contract.
- `-td`/`--test-directory` `path_to_test_directory`: The path to the test directory relative to the working directory.
- `-i`/`--inheritance-path` `relative_path_to_contract`: The relative path from the test directory to the contract (used for inheritance).
- `-f`/`--fuzzer` `fuzzer_name`: The name of the fuzzer, currently supported: `echidna` and `medusa`
**Command-line options:**
- `compilation_path`: The path to the Solidity file or Foundry directory
- `-n`/`--name` `name: str`: The name of the fuzzing harness.
- `-c`/`--contracts` `target_contracts: list`: The name of the target contract.
- `-o`/`--output-dir` `output_directory: str`: Output directory name. By default it is `fuzzing`
- `--config`: Path to the `fuzz-utils` config JSON file

**Example**

In order to generate a fuzzing harness for the [BasicTypes.sol](tests/test_data/src/BasicTypes.sol) contract, we need to `cd` into the `tests/test_data/` directory which contains the Foundry project and run the command:
```bash
fuzz-utils template ./src/BasicType.sol --name "DefaultHarness" --contracts BasicTypes
```

Running this command should generate the directory structure in [tests/test_data/test/fuzzing](tests/test_data/test/fuzzing), which contains the fuzzing harness [DefaultHarness](tests/test_data/test/fuzzing/harnesses/DefaultHarness.sol) and the Actor contract [DefaultActor](tests/test_data/test/fuzzing/actors/ActorDefault.sol).

## Utilities

### Initializing a configuration file

The `init` command can be used to initialize a default configuration file in the project root.

**Configuration file:**
Using the configuration file allows for more granular control than just using the command-line options. Valid configuration options are listed below:
```json
{
"generate": {
"targetContract": "BasicTypes", // The Echidna/Medusa fuzzing harness
"compilationPath": "./src/BasicTypes", // Path to the file or Foundry directory
"corpusDir": "echidna-corpora/corpus-basic", // Path to the corpus directory
"fuzzer": "echidna", // `echidna` | `medusa`
"testsDir": "./test/", // Path to the directory where the tests will be generated
"inheritancePath": "../src/", // Relative path from the testing directory to the contracts
"namedInputs": false, // True | False, whether to include function input names when making calls
"allSequences": false, // True | False, whether to generate tests for the entire corpus (including non-failing sequences)
},
"template": {
"name": "DefaultHarness", // The name of the fuzzing harness that will be generated
"targets": ["BasicTypes"], // The contracts to be included in the fuzzing harness
"outputDir": "./test/fuzzing", // The output directory where the files and directories will be saved
"compilationPath": ".", // The path to the Solidity file (if single target) or Foundry directory
"actors": [ // At least one actor is required. If the array is empty, the DefaultActor which wraps all of the functions from the target contracts will be generated
{
"name": "Default", // The name of the Actor contract, saved as `Actor{name}`
"targets": ["BasicTypes"], // The list of contracts that the Actor can interact with
"number": 3, // The number of instances of this Actor that will be used in the harness
"filters": { // Used to filter functions so that only functions that fulfill certain criteria are included
"strict": false, // If `true`, only functions that fulfill *all* the criteria will be included. If `false`, functions that fulfill *any* criteria will be included
"onlyModifiers": [], // List of modifiers to include
"onlyPayable": false, // If `true`, only `payable` functions will be included. If `false`, both payable and non-payable functions will be included
"onlyExternalCalls": [], // Only include functions that make a certain external call. E.g. [`transferFrom`]
},
}
],
"attacks": [ // A list of premade attack contracts to include.
{
"name": "Deposit", // The name of the attack contract.
"targets": ["BasicTypes"], // The list of contracts that the attack contract can interact with
"number": 1, // The number of instances of this attack contract that will be used in the harness
"filters": { // Used to filter functions so that only functions that fulfill certain criteria are included
"strict": false, // If `true`, only functions that fulfill *all* the criteria will be included. If `false`, functions that fulfill *any* criteria will be included
"onlyModifiers": [], // List of modifiers to include
"onlyPayable": false, // If `true`, only `payable` functions will be included. If `false`, both payable and non-payable functions will be included
"onlyExternalCalls": [], // Only include functions that make a certain external call. E.g. [`transferFrom`]
},
}
],
},
}
```

## Contributing
For information about how to contribute to this project, check out the [CONTRIBUTING](CONTRIBUTING.md) guidelines.
Expand Down
99 changes: 99 additions & 0 deletions fuzz_utils/generate/FoundryTest.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,99 @@
"""The FoundryTest class that handles generation of unit tests from call sequences"""
import os
import sys
import json
from typing import Any
import jinja2

from slither import Slither
from slither.core.declarations.contract import Contract
from fuzz_utils.utils.crytic_print import CryticPrint

from fuzz_utils.generate.fuzzers.Medusa import Medusa
from fuzz_utils.generate.fuzzers.Echidna import Echidna
from fuzz_utils.templates.foundry_templates import templates


class FoundryTest: # pylint: disable=too-many-instance-attributes
"""
Handles the generation of Foundry test files
"""

def __init__(
self,
config: dict,
slither: Slither,
fuzzer: Echidna | Medusa,
) -> None:
self.inheritance_path = config["inheritancePath"]
self.target_name = config["targetContract"]
self.corpus_path = config["corpusDir"]
self.test_dir = config["testsDir"]
self.all_sequences = config["allSequences"]
self.slither = slither
self.target = self.get_target_contract()
self.fuzzer = fuzzer

def get_target_contract(self) -> Contract:
"""Gets the Slither Contract object for the specified contract file"""
contracts = self.slither.get_contract_from_name(self.target_name)
# Loop in case slither fetches multiple contracts for some reason (e.g., similar names?)
for contract in contracts:
if contract.name == self.target_name:
return contract

# TODO throw error if no contract found
sys.exit(-1)

def create_poc(self) -> str:
"""Takes in a directory path to the echidna reproducers and generates a test file"""

file_list: list[dict[str, Any]] = []
tests_list = []
dir_list = []
if self.all_sequences:
dir_list = self.fuzzer.corpus_dirs
else:
dir_list = [self.fuzzer.reproducer_dir]

# 1. Iterate over each directory and reproducer file (open it)
for directory in dir_list:
for entry in os.listdir(directory):
full_path = os.path.join(directory, entry)

if os.path.isfile(full_path):
try:
with open(full_path, "r", encoding="utf-8") as file:
file_list.append({"path": full_path, "content": json.load(file)})
except Exception: # pylint: disable=broad-except
print(f"Fail on {full_path}")

# 2. Parse each reproducer file and add each test function to the functions list
for idx, file_obj in enumerate(file_list):
try:
tests_list.append(
self.fuzzer.parse_reproducer(file_obj["path"], file_obj["content"], idx)
)
except Exception: # pylint: disable=broad-except
print(f"Parsing fail on {file_obj['content']}: index: {idx}")

# 4. Generate the test file
template = jinja2.Template(templates["CONTRACT"])
write_path = f"{self.test_dir}{self.target_name}"
inheritance_path = f"{self.inheritance_path}{self.target_name}"

# 5. Save the test file
test_file_str = template.render(
file_path=f"{inheritance_path}.sol",
target_name=self.target_name,
amount=0,
tests=tests_list,
fuzzer=self.fuzzer.name,
)
with open(f"{write_path}_{self.fuzzer.name}_Test.t.sol", "w", encoding="utf-8") as outfile:
outfile.write(test_file_str)
CryticPrint().print_success(
f"Generated a test file in {write_path}_{self.fuzzer.name}_Test.t.sol"
)

return test_file_str
File renamed without changes.
Original file line number Diff line number Diff line change
Expand Up @@ -130,7 +130,6 @@ def _parse_call_object(self, call_dict: dict[Any, Any]) -> tuple[str, str]:
for idx, input_param in enumerate(slither_entry_point.parameters):
call_definition[idx] = input_param.name + ": " + call_definition[idx]
parameters_str = "{" + ", ".join(call_definition) + "}"
print(parameters_str)
else:
parameters_str = ", ".join(call_definition)

Expand Down
File renamed without changes.
Empty file.
Loading

0 comments on commit 14b770c

Please sign in to comment.