Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Refactor the architecture to add support for running the test cases in subprocesses #82

Draft
wants to merge 93 commits into
base: main
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
93 commits
Select commit Hold shift + click to select a range
6837bf8
Add basic SubprocessTestCaseExecutor implementation
BergLucas Mar 14, 2024
fd0e626
Return subject_properties
BergLucas Mar 14, 2024
743b044
Fix subject_properties
BergLucas Mar 14, 2024
c46d45a
Fix tracer serialisation
BergLucas Mar 15, 2024
d1b3eaf
Improve ExecutionTracer serialisation
BergLucas Mar 15, 2024
886440c
Remove unused statement
BergLucas Mar 15, 2024
da24cca
Remove unused variable
BergLucas Mar 15, 2024
0c17d68
Replace cloudpickle with dill
BergLucas Mar 15, 2024
a980cf4
Fix InstrLocation serialisation
BergLucas Mar 15, 2024
953a7d5
Fix tracer and add a test
BergLucas Mar 16, 2024
1632d0a
Add __eq__ and __hash__ for AssertionTrace
BergLucas Mar 16, 2024
24f0a4c
Improve differential test
BergLucas Mar 16, 2024
e02ead3
Improve differential test
BergLucas Mar 16, 2024
4d75538
Reinstrument the module in subprocess using the new tracer
BergLucas Mar 16, 2024
aa04204
Set the current thread before reinstrumenting the module in subprocess
BergLucas Mar 16, 2024
6c0e2e9
Add a timeout for the subprocess
BergLucas Mar 16, 2024
aab11b3
Extract RemoteExecutionObserver from ExecutionObserver
BergLucas Mar 16, 2024
d9d1657
Fix subprocess execution
BergLucas Mar 17, 2024
f2c581d
Fix generation for unpickable objects
BergLucas Mar 17, 2024
760188c
Add other warnings on unpickable objects
BergLucas Mar 17, 2024
a7953fd
Use a Pipe instead of a Queue
BergLucas Mar 18, 2024
c1c1525
Add dependencies
BergLucas Mar 19, 2024
27814d9
Add ExecutionTracerProxy
BergLucas Mar 19, 2024
d08fe2b
Move tracers to pynguin.instrumentation.tracer package
BergLucas Mar 20, 2024
6c3054e
Remove unused constant
BergLucas Mar 20, 2024
8ca2226
Modify dependencies
BergLucas Mar 20, 2024
e21a7f0
Convert attribute_lookup to a classmethod on the base class
BergLucas Mar 20, 2024
96f244f
Rename some dependencies
BergLucas Mar 20, 2024
71c0252
Merge branch 'main' into improvement/process-based-tests-execution
BergLucas Apr 22, 2024
7ec287b
Fix lint errors
BergLucas Apr 22, 2024
619e402
Merge branch 'improvement/remove-mutpy' into improvement/process-base…
BergLucas Apr 22, 2024
c903321
Merge branch 'improvement/remove-mutpy' into improvement/process-base…
BergLucas Apr 23, 2024
dad01cb
Merge branch 'improvement/remove-mutpy' into improvement/process-base…
BergLucas Apr 23, 2024
60e5ed6
Fix import
BergLucas Apr 23, 2024
751588e
Rename a private method
BergLucas Apr 23, 2024
cb49c55
Fix instrumentation for mutant
BergLucas Apr 23, 2024
edb658d
Add some arguments to configure the subprocess executor
BergLucas Apr 23, 2024
7272d93
Remove redundant remote observer
BergLucas Apr 24, 2024
077d4a6
Improve assertion generator speed
BergLucas Apr 24, 2024
95555f9
Add a log message when a segmentation fault is detected
BergLucas Apr 24, 2024
6f8e44d
Fix assertion generator tracer
BergLucas Apr 24, 2024
7859c2a
Improve _save_crash_tests
BergLucas Apr 24, 2024
dd84512
Store import trace in create_mutant
BergLucas Apr 24, 2024
4d4270a
Merge branch 'improvement/remove-mutpy' into improvement/process-base…
BergLucas Apr 25, 2024
c688567
Revert "Remove redundant remote observer"
BergLucas Apr 25, 2024
84a7dbe
Fix bad pickle states
BergLucas Apr 26, 2024
26e83c0
Use the full subject_properties
BergLucas Apr 26, 2024
0df3c45
Transfer tracer instead of subject_properties
BergLucas Apr 26, 2024
756ce26
Transfer remote observers in execute_multiple
BergLucas Apr 26, 2024
38ff5ad
Improve pickling of CFG and ControlDependenceGraph
BergLucas Apr 26, 2024
3d67bc2
Use execute_multiple in _run_test_suite_chromosome
BergLucas Apr 26, 2024
0217748
Fix ModuleProvider in subprocess
BergLucas Apr 26, 2024
55dcc70
Fix a bug in segmentation fault handling
BergLucas Apr 27, 2024
80254e8
Return early when there are no test cases to execute
BergLucas Apr 27, 2024
6c9642a
Lint code
BergLucas Apr 29, 2024
777bfe4
Merge branch 'fix/lint-code' into improvement/process-based-tests-exe…
BergLucas Apr 29, 2024
b167405
Refactor _add_assertions
BergLucas May 2, 2024
c57fd29
Send random state back to main process
BergLucas May 5, 2024
ab852c6
Fix assertion trace in subprocess
BergLucas May 6, 2024
f25fb7b
Improve performance
BergLucas May 6, 2024
11335b7
Reduce quantity of data sent to subprocess
BergLucas May 6, 2024
d124cc3
Fix remote observers
BergLucas May 6, 2024
db2b06e
Fix a bug with reference_bindings
BergLucas May 16, 2024
1b52bd2
Lint code
BergLucas May 17, 2024
a39cb94
Fix segmentation fault handling and add support for out of memory signal
BergLucas May 21, 2024
8de4053
Fix execute_multiple fallback
BergLucas May 21, 2024
ec3f0e2
Fix typo
BergLucas May 21, 2024
260a5f6
Fix AssertionTraceObserver when encountering an unexposed module
BergLucas May 21, 2024
41dc789
Move stdout/stderr context inside thread
BergLucas May 22, 2024
1638487
Add a debug log before waiting for the thread
BergLucas May 22, 2024
f128881
Fix assertion generation for float subclasses
BergLucas May 22, 2024
5de35f3
Merge branch 'improvement/move-stdout-stderr-redirection' into main-f…
BergLucas May 22, 2024
bad200e
Merge branch 'improvement/fix-float-assertions' into main-fixed
BergLucas May 22, 2024
9ecd387
Merge branch 'main-fixed' into improvement/process-based-tests-execution
BergLucas May 22, 2024
8f83e66
Use a better context to suppress stdout and stderr when an error occurs
BergLucas May 24, 2024
e49abf4
Merge branch 'improvement/move-stdout-stderr-redirection' into main-f…
BergLucas May 24, 2024
9a55863
Merge branch 'main-fixed' into improvement/process-based-tests-execution
BergLucas May 24, 2024
ed385d2
Handle abort signal
BergLucas May 25, 2024
061d517
Handle abort signals in execute_multiple too
BergLucas May 25, 2024
cac5e24
Fix assertion and invalid function call
BergLucas May 25, 2024
079865a
Merge branch 'fix/syntax-tree-analyser' into main-fixed
BergLucas May 25, 2024
4e3ec83
Handle more exit codes
BergLucas May 25, 2024
9408bb3
Merge branch 'main-fixed' into improvement/process-based-tests-execution
BergLucas May 25, 2024
dd43777
Fix infinite join
BergLucas May 30, 2024
5c01ac3
Merge branch 'improvement/process-based-tests-execution' into improve…
BergLucas Dec 4, 2024
f5e2fbd
Fix a bug with the tracer when code was executed during pickling
BergLucas Dec 5, 2024
da847e0
Catch exception when finding baditems
BergLucas Dec 5, 2024
6930383
Catch exceptions when creating reference bindings
BergLucas Dec 5, 2024
50d3cbe
Create the _create_new_reference_bindings method
BergLucas Dec 7, 2024
f7966d1
Fix SubprocessTestCaseExecutor.execute_multiple method
BergLucas Dec 9, 2024
f6ff3d6
Merge branch 'main' into improvement/subprocess-execution
BergLucas Dec 12, 2024
3da6525
Remove useless changes from pyproject.toml
BergLucas Dec 12, 2024
4d74309
Fix linter errors
BergLucas Dec 12, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions docs/requirements.txt
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@ certifi==2024.8.30 ; python_version >= "3.10" and python_version < "3.11"
charset-normalizer==3.4.0 ; python_version >= "3.10" and python_version < "3.11"
click==8.1.7 ; python_version >= "3.10" and python_version < "3.11"
colorama==0.4.6 ; python_version >= "3.10" and python_version < "3.11" and (sys_platform == "win32" or platform_system == "Windows")
dill==0.3.9 ; python_version >= "3.10" and python_version < "3.11"
docstring-parser==0.16 ; python_version >= "3.10" and python_version < "3.11"
docutils==0.21.2 ; python_version >= "3.10" and python_version < "3.11"
exceptiongroup==1.2.2 ; python_version >= "3.10" and python_version < "3.11"
Expand All @@ -20,6 +21,7 @@ libcst==1.5.1 ; python_version >= "3.10" and python_version < "3.11"
markdown-it-py==3.0.0 ; python_version >= "3.10" and python_version < "3.11"
markupsafe==3.0.2 ; python_version >= "3.10" and python_version < "3.11"
mdurl==0.1.2 ; python_version >= "3.10" and python_version < "3.11"
multiprocess==0.70.17 ; python_version >= "3.10" and python_version < "3.11"
mypy-extensions==1.0.0 ; python_version >= "3.10" and python_version < "3.11"
networkx==3.4.2 ; python_version >= "3.10" and python_version < "3.11"
packaging==24.2 ; python_version >= "3.10" and python_version < "3.11"
Expand Down
47 changes: 45 additions & 2 deletions poetry.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

4 changes: 3 additions & 1 deletion pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -47,9 +47,11 @@ python = ">=3.10, <3.11"
astroid = "^3.3.5"
black = "^24.10.0"
bytecode = "^0.16.0"
dill = "^0.3.8"
jellyfish = "^1.1.2"
Jinja2 = "^3.1.4"
networkx = "^3.4"
multiprocess = "^0.70.16"
rich = "^13.9.4"
Pygments = "^2.18.0"
pytest = "^8.3.4"
Expand Down Expand Up @@ -255,4 +257,4 @@ convention = "google"
"PT006", # Wrong name(s) type in `@pytest.mark.parameterize`, expected `tuple`
"S101", # Use of `assert` detected
"SLF001", # Private member accessed
]
]
31 changes: 31 additions & 0 deletions src/pynguin/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,13 @@
#
"""Pynguin is an automated unit test generation framework for Python."""

import copyreg

from collections.abc import Callable

from bytecode.instr import _UNSET # noqa: PLC2701
from bytecode.instr import InstrLocation

import pynguin.configuration as config
import pynguin.generator as gen

Expand All @@ -27,3 +34,27 @@
"run_pynguin",
"set_configuration",
]


def _pickle_instr_location(
instr_location: InstrLocation,
) -> tuple[
Callable[[int | None, int | None, int | None, int | None], InstrLocation],
tuple[int | None, int | None, int | None, int | None],
]:
return InstrLocation, (
instr_location.lineno,
instr_location.end_lineno,
instr_location.col_offset,
instr_location.end_col_offset,
)


copyreg.pickle(InstrLocation, _pickle_instr_location)


def _pickle_unset(unset: _UNSET) -> tuple[Callable[[], _UNSET], tuple]: # noqa: ARG001
return _UNSET, ()


copyreg.pickle(_UNSET, _pickle_unset)
88 changes: 88 additions & 0 deletions src/pynguin/analyses/controlflow.py
Original file line number Diff line number Diff line change
Expand Up @@ -600,6 +600,52 @@ def diameter(self) -> int:
self._diameter = len(self._graph.edges)
return self._diameter

def __getstate__(self):
return {
"nodes": tuple(
(
node.index,
node.offset,
node.basic_block,
node.is_artificial,
node.predicate_id,
data,
)
for node, data in self._graph.nodes(data=True)
),
"edges": tuple(
(
source.index,
target.index,
data,
)
for source, target, data in self._graph.edges(data=True)
),
"bytecode_cfg": self._bytecode_cfg,
"diameter": self._diameter,
}

def __setstate__(self, state: dict):
self._graph = nx.DiGraph()
nodes: dict[int, ProgramGraphNode] = {}
for index, offset, basic_block, is_artificial, predicate_id, data in state["nodes"]:
node = ProgramGraphNode(
index,
offset=offset,
basic_block=basic_block,
is_artificial=is_artificial,
)
if predicate_id is not None:
node.predicate_id = predicate_id
self._graph.add_node(node, **data)
nodes[index] = node
for source_index, target_index, data in state["edges"]:
source = nodes[source_index]
target = nodes[target_index]
self._graph.add_edge(source, target, **data)
self._bytecode_cfg = state["bytecode_cfg"]
self._diameter = state["diameter"]


class DominatorTree(ProgramGraph[ProgramGraphNode]):
"""Implements a dominator tree."""
Expand Down Expand Up @@ -833,6 +879,48 @@ def _is_control_dependent_on_root(
return True
return False

def __getstate__(self):
return {
"nodes": tuple(
(
node.index,
node.offset,
node.basic_block,
node.is_artificial,
node.predicate_id,
data,
)
for node, data in self._graph.nodes(data=True)
),
"edges": tuple(
(
source.index,
target.index,
data,
)
for source, target, data in self._graph.edges(data=True)
),
}

def __setstate__(self, state: dict):
self._graph = nx.DiGraph()
nodes: dict[int, ProgramGraphNode] = {}
for index, offset, basic_block, is_artificial, predicate_id, data in state["nodes"]:
node = ProgramGraphNode(
index,
offset=offset,
basic_block=basic_block,
is_artificial=is_artificial,
)
if predicate_id is not None:
node.predicate_id = predicate_id
self._graph.add_node(node, **data)
nodes[index] = node
for source_index, target_index, data in state["edges"]:
source = nodes[source_index]
target = nodes[target_index]
self._graph.add_edge(source, target, **data)

@staticmethod
def _create_augmented_graph(graph: CFG) -> CFG:
entry_node = graph.entry_node
Expand Down
2 changes: 1 addition & 1 deletion src/pynguin/analyses/module.py
Original file line number Diff line number Diff line change
Expand Up @@ -80,7 +80,7 @@
import pynguin.ga.algorithms.archive as arch
import pynguin.ga.computations as ff

from pynguin.testcase.execution import SubjectProperties
from pynguin.instrumentation.tracer import SubjectProperties

AstroidFunctionDef: typing.TypeAlias = astroid.AsyncFunctionDef | astroid.FunctionDef

Expand Down
6 changes: 6 additions & 0 deletions src/pynguin/assertion/assertion_trace.py
Original file line number Diff line number Diff line change
Expand Up @@ -80,6 +80,12 @@ def clone(self) -> AssertionTrace:
copy.trace[stmt_key].add(entry)
return copy

def __eq__(self, other: object) -> bool:
return isinstance(other, AssertionTrace) and self.trace == other.trace

def __hash__(self) -> int:
return hash(self.trace)


@dataclasses.dataclass
class AssertionVerificationTrace:
Expand Down
Loading
Loading