Automation of running/testing many examples

[villares]

Dear @hx2A, we had this conversation in a now closed pull request about another unrelated issue, so I think this is a better place to keep your recommendations:

If it helps, there is code that is used to run all of the example code you see in the reference docs.
https://github.com/py5coding/py5generator/blob/main/py5_docs/scripts/run_example_code.py

Most of that is specific to the reference docs and the Documetation files though. The part that might be useful is here:
https://github.com/py5coding/py5generator/blob/main/py5_resources/py5_module/py5_tools/testing.py

That assumes module mode. Is your example code in imported mode? If so, you might benefit from making calls to run_code() in imported.py:
https://github.com/py5coding/py5generator/blob/main/py5_resources/py5_module/py5_tools/imported.py

Now... I'm trying to test the imported mode examples I converted from Processing Python mode... and I can launch them, but I'm not sure if I should try some threading or other strategies...

I'm on Linux, not on MacOS, suppose I'd like to run them without blocking (because I think I'm going to try pyautogui to interact with them and then close them for me, that would be the equivalent of module mode py5.run_sketch(block=False). How to do that?

This seems not to be the way of doing it 😆 :

from py5_tools import imported
# Wrong!
imported.run_code('.../path_to/my_imported_example.py',
                   sketch_args=['block=False'])  # or , py5_options=['block=False'])

[hx2A]

I really wish py5 had a better testing framework! Yet another thing to work on.

I'm on Linux, not on MacOS, suppose I'd like to run them without blocking (because I think I'm going to try pyautogui to interact with them and then close them for me, that would be the equivalent of module mode py5.run_sketch(block=False). How to do that?

pyautogui sounds like a good approach. I see why you would want block=False. The imported.run_code() method can't do that for you though. In that case, the user's code gets added to this template:

_CODE_FRAMEWORK = """{0}



run_sketch(block=True, py5_options={2}, sketch_args={3})
if {1} and is_dead_from_error:
    exit_sketch()
"""

block=True, and py5_options and sketch_args do other things. I think you might want to copy the code in _run_code() (also in the same imported module) and adapt it for your purposes? Then you will have more control over what it does and how it behaves. Most of it would probably stay the same, but you just need your own version of _CODE_FRAMEWORK to put your test code in to.

[villares]

cool! thanks!

[villares]

Monkey-patching FTW! 🤣

from py5_tools import imported 

imported._CODE_FRAMEWORK = """{0}

run_sketch(block=False, py5_options={2}, sketch_args={3})
if {1} and is_dead_from_error:
    exit_sketch()
"""
sketch = '/home/villares/GitHub/doutorado/scripts/parsons_maker/loop.py'

imported.run_code(sketch)
print('done')

[hx2A]

That does it? Excellent! Great thinking.

I'll make an enhancement to the next release to parameterize block, so you can just do something like imported.run_code(sketch, block=False).

[hx2A]

See #231

[villares]

This is quite ugly but it might help someone getting started...

from pathlib import Path
import time
import os

import pyautogui

# from capture_stdout import capture_stdout  # Should try again other time
# from PIL import ImageGrab # didn't work for me
# I also couldn't work out pyautogui scrot integration 
import pyscreenshot as ImageGrab

from py5_tools import imported 
imported._CODE_FRAMEWORK = """{0}

run_sketch(block=False, py5_options={2}, sketch_args={3})
if {1} and is_dead_from_error:
    exit_sketch()
"""
collection_folder = Path('/home/villares/GitHub/py5examples/examples-from-Processing-Python-mode/')
sketch_paths = collection_folder.rglob('*.py')  # TODO: skip secondary tab modules?

results_folder = Path.cwd() / 'results'
results_folder.mkdir(exist_ok=True)

def run_sketch(sketch_file_path):
    """Run sketch, grab a snapshot."""
    with open(sketch_file_path) as f:
        code_lines = f.readlines()
        W = H = 128
        for li in code_lines:
            if li.lstrip().startswith('size('):  # TODO: handle full_screen() sketches
                # I don't like regex
                start, comma = li.find('('), li.find(',') 
                second_comma = li.find(',', comma + 1)
                end = second_comma if second_comma > 0 else li.find(')')
                try:
                    W, H = int(li[start+1:comma]), int(li[comma+1:end])
                except ValueError:
                    print(f'Problem with {sketch_i}:\n'
                          f'*{li[start+1:comma]}*{li[comma+1:end]}*'
                          f'\n*{li}*\non {sketch_file_path}')
    # for sketches that have to load stuff from relative paths
    os.chdir(sketch_file_path.parent)   # I'm too lazy to add a proper context manager
    imported.run_code(sketch_file_path, py5_options=['location=0,25'])
    # time for the sketch to open window and start
    time.sleep(10 if sketch_i in slow_sketches else 1)
    if sketch_i in move_sketches:
        pyautogui.moveTo(W // 2, H // 2 + 53)
    if sketch_i in keyboard_sketches:
        pyautogui.write('abcde')
    # this has to be improved for P3D sketches
    im = ImageGrab.grab(bbox=(2, 53, W + 2, H + 53)) # x1,y1,x2,y2 (not w,h)
    sketch_name = (f'{sketch_i:03d}'
#                   f'_{sketch_file_path.parent.parent.parent.stem}'  
#                   f'_{sketch_file_path.parent.parent.stem}'
#                   f'_{sketch_file_path.parent.stem}'
                  f'_{sketch_file_path.stem}')
    im.save(results_folder / (sketch_name + '.png'))
    pyautogui.press('esc') # close sketch, let's hope it has focus...
    time.sleep(5 if sketch_i in slow_sketches else 0.1)  
    
slow_sketches = [22, 24, 25, 25]
keyboard_sketches = [37]
move_sketches = [10]
click_sketches = []  # not implemented

global sketch_i
for sketch_i, sketch_file_path in enumerate(sketch_paths):
     print(sketch_i, sketch_file_path)
     run_sketch(sketch_file_path)

[hx2A]

This looks useful. I have not used pyautogui much. Too bad it can't record your mouse and keyboard inputs to "replay" them again later. If it could, that would be a great way to make unit tests.