|\ merge from release-0.7, released as 0.6.

ChangeLog.rst

@@ -1,4 +1,24 @@
-ayrton (0.6) UNRELEASED; urgency=medium
+ayrton (0.7) UNRELEASED; urgency=medium
+
+  * Send data to/from the remote via another `ssh` channel, which is more stable than using `stdin`.
+  * Stabilized a lot all tests, specially those using a mocked stdout for getting test validation.
+  * A lot of tests have been moved to their own scripts in ayrton/tests/scripts, which also work as (very minimal) examples of whatś working.
+  * Use `flake8` to check the code.
+  * Move `remote()` to its own source.
+  * API change: if a `str` or `bytes` object is passed in `_in`, then it's the name of a file where to read `stdin`. If it's an `int`, then it's considered a file descriptor. This makes the API consistent to `_out` and `_err` handling.
+  * More error handling.
+  * Fixed errors with global variables handling.
+  * `argv` is handled at the last time possible, allowing it being passed from test invoction.
+  * `shift` complains on negative values.
+  * Lazy `pprint()`, so debug statemens do not do useless work.
+  * `stdin/out/err` handling in `remote()` is done by a single thread. 
+  * Modify a lot the local terminal when in `remote()` so, among other things, we have no local echo.
+  * Properly pass the terminal type and size to the remote. These last three features allows programs like `vi` be run in the remote.
+  * Paved the road to make `remote()`s more like `Command()`s.
+
+ -- Marcos Dione <[email protected]>  Wed, 09 Dec 2015 15:48:49 +0100
+
+ayrton (0.6) unstable; urgency=medium
 
   * Great improvements in `remote()`'s API and sematics:
     * Made sure local varaibles go to and come back from the remote.

Makefile

@@ -1,29 +1,51 @@
+DEBUG_MULTI=strace -tt -T -ff -o runner -s 128
+DEBUG_SIMPLE=strace -tt -T -o runner -s 128
+PYTHON=python3.4
+
 all: docs
 
 INSTALL_DIR=$(HOME)/local
 
 tests:
-	bash -c 'while true; do nc -l -s 127.0.0.1 -p 2233 -e /bin/bash; done' & \
-	pid=$$!; \
-	LC_ALL=C python3 -m unittest discover -v ayrton; \
-	kill $$pid
+	LC_ALL=C $(PYTHON) -m unittest discover -v ayrton
+
+slowtest:
+	# LC_ALL=C $(DEBUG_SIMPLE) $(PYTHON) -m unittest discover -f -v ayrton
+	LC_ALL=C $(DEBUG_MULTI) $(PYTHON) -m unittest discover -f -v ayrton
+
+quicktest:
+	LC_ALL=C $(PYTHON) -m unittest discover -f -v ayrton
 
 docs:
 	PYTHONPATH=${PWD} make -C doc html
 
 install: tests
-	python3 setup.py install --prefix=$(INSTALL_DIR)
+	$(PYTHON) setup.py install --prefix=$(INSTALL_DIR)
 
 unsafe-install:
-	echo "unsafe install, are you sure?"
-	read foo
-	python3 setup.py install --prefix=$(INSTALL_DIR)
+	@echo "unsafe install, are you sure?"
+	@read foo
+	$(PYTHON) setup.py install --prefix=$(INSTALL_DIR)
 
 upload: tests upload-docs
-	python3 setup.py sdist upload
+	$(PYTHON) setup.py sdist upload
 
 upload-docs: docs
 	rsync --archive --verbose --compress --rsh ssh doc/build/html/ www.grulic.org.ar:www/projects/ayrton/
 
 push: tests
 	git push
+
+check:
+	flake8 --ignore E201,E211,E225,E221,E226,E202 --show-source --statistics --max-line-length 130 ayrton/*.py
+
+testclean:
+	rm -rfv ayrton.*log runner.*
+
+debugserver:
+	# TODO: generate the server key
+	if ! [ -f rsa_server_key ]; then \
+		true; \
+	fi
+	# TODO: discover path?
+	/usr/sbin/sshd -dd -e -h $(shell pwd)/rsa_server_key -p 2244

README.md

@@ -210,10 +210,31 @@ their main objective, and its syntax is designed around it. That leads to
 shortcuts that later are more difficult to read and creates problems when
 handling filenames that have special characters.
 
+Q: Instead, why not use...
+
+A: (Short version) We think nobody provides all of `ayrton`'s features.
+
+A: ... [`sh`](https://amoffat.github.io/sh/)? Well, we started with `sh` as a basis
+of `ayrton`, but its buffered output didn't allow us to run editors and other TIU's.
+
+A: ... [`xonsh`](http://xonsh.org/)? `xonsh` keeps environment variables well
+separated from Python ones; it even has a Python mode and a 'subprocess' mode;
+and is more oriented to being a shell. `ayrton` aims directly in the opposite
+direction.
+
+A: ... [`plumbum`](https://plumbum.readthedocs.org/en/latest/)? You could say that we
+independently thought of its piping and redirection syntax (but in reality we just
+based ours on `bash`'s). Still, the fact that you fisrt build pipes and then execute
+them looks weird for a SysAdmin.
+
+A: ... [`fabric`](http://www.fabfile.org/)? `fabric` is the only one that has remote
+execution and the `cd` context manager, but command execution is still done via
+strings.
+
 # Thanks to:
 
-`rbilstolfi`, `marianoguerra`, `facundobatista`, `ralsina`, `nessita` for unit
-testing support, `Darni` for pointing me to
+`rbilstolfi`, `marianoguerra`, `facundobatista`, `ralsina` for ideas; `nessita` for unit
+testing support; `Darni` for pointing me to
 [nvie's workflow for `git`](http://nvie.com/posts/a-successful-git-branching-model/),
 Andrew Moffat for [`sh`](http://amoffat.github.io/sh/) and Richard Jones for
 this talk (thanks again, `ralsina`), even when I ended up doing something

TODO.rst

@@ -5,6 +5,9 @@ Really do:
 ----------
 
 * better error reporting, including remotes
+
+* interface so local and remote can easily setup more communication channels
+
 * we're weeding out imports, we could gather a list and reimport them in the
   remote
 

ayrton/__init__.py

@@ -23,6 +23,7 @@
 import ast
 import logging
 import dis
+import traceback
 
 # patch logging so we have debug2 and debug3
 import ayrton.utils
@@ -45,67 +46,75 @@
 from ayrton.parser.astcompiler.astbuilder import ast_from_node
 from ayrton.ast_pprinter import pprint
 
-__version__= '0.6'
+__version__= '0.7'
+
 
 def parse (script, file_name=''):
     parser= PythonParser (None)
     info= CompileInfo (file_name, 'exec')
     return ast_from_node (None, parser.parse_source (script, info), info)
 
-def polute (d, more):
-    d.update (__builtins__)
-    # weed out some stuff
-    for weed in ('copyright', '__doc__', 'help', '__package__', 'credits', 'license', '__name__'):
-        del d[weed]
-
-    # envars as gobal vars, shell like
-    d.update (os.environ)
-
-    # these functions will be loaded from each module and put in the globals
-    # tuples (src, dst) renames function src to dst
-    builtins= {
-        'os': [ ('getcwd', 'pwd'), 'uname', 'listdir', ],
-        'os.path': [ 'abspath', 'basename', 'commonprefix', 'dirname',  ],
-        'time': [ 'sleep', ],
-        'sys': [ 'exit', 'argv' ],
-
-        'ayrton.file_test': [ '_a', '_b', '_c', '_d', '_e', '_f', '_g', '_h',
-                              '_k', '_p', '_r', '_s', '_u', '_w', '_x', '_L',
-                              '_N', '_S', '_nt', '_ot' ],
-        'ayrton.expansion': [ 'bash', ],
-        'ayrton.functions': [ 'cd', ('cd', 'chdir'), 'export', 'option', 'remote', 'run',
-                               'shift', 'unset', ],
-        'ayrton.execute': [ 'o', 'Capture', 'CommandFailed', 'CommandNotFound',
-                            'Pipe', 'Command'],
-        }
-
-    for module, functions in builtins.items ():
-        m= importlib.import_module (module)
-        for function in functions:
-            if type (function)==tuple:
-                src, dst= function
-            else:
-                src= function
-                dst= function
-
-            d[dst]= getattr (m, src)
-
-    # now the IO files
-    for std in ('stdin', 'stdout', 'stderr'):
-        d[std]= getattr (sys, std).buffer
-
-    d.update (more)
+
+class Environment (dict):
+    def __init__ (self, *args, **kwargs):
+        super ().__init__ (*args, **kwargs)
+
+        self.polute ()
+
+
+    def polute (self):
+        self.update (__builtins__)
+        # weed out some stuff
+        for weed in ('copyright', '__doc__', 'help', '__package__', 'credits',
+                     'license', '__name__', 'quit', 'exit'):
+            del self[weed]
+
+        # these functions will be loaded from each module and put in the globals
+        # tuples (src, dst) renames function src to dst
+        ayrton_builtins= {
+            'os': [ ('getcwd', 'pwd'), 'uname', 'listdir', ],
+            'os.path': [ 'abspath', 'basename', 'commonprefix', 'dirname',  ],
+            'time': [ 'sleep', ],
+            'sys': [ 'exit', ],  # argv is handled just before execution
+
+            'ayrton.file_test': [ '_a', '_b', '_c', '_d', '_e', '_f', '_g', '_h',
+                                  '_k', '_p', '_r', '_s', '_u', '_w', '_x', '_L',
+                                  '_N', '_S', '_nt', '_ot' ],
+            'ayrton.expansion': [ 'bash', ],
+            'ayrton.functions': [ 'cd', ('cd', 'chdir'), 'export', 'option', 'run',
+                                   'shift', 'unset', ],
+            'ayrton.execute': [ 'o', 'Capture', 'CommandFailed', 'CommandNotFound',
+                                'Pipe', 'Command'],
+            'ayrton.remote': [ 'remote' ]
+            }
+
+        for module, functions in ayrton_builtins.items ():
+            m= importlib.import_module (module)
+            for function in functions:
+                if type (function)==tuple:
+                    src, dst= function
+                else:
+                    src= function
+                    dst= function
+
+                self[dst]= getattr (m, src)
+
+        # now the IO files
+        for std in ('stdin', 'stdout', 'stderr'):
+            self[std]= getattr (sys, std).buffer
+
 
 class Ayrton (object):
     def __init__ (self, g=None, l=None, **kwargs):
+        logger.debug ('===========================================================')
         logger.debug ('new interpreter')
         logger.debug3 ('globals: %s', ayrton.utils.dump_dict (g))
         logger.debug3 ('locals: %s', ayrton.utils.dump_dict (l))
-        if g is None:
-            self.globals= {}
-        else:
-            self.globals= g
-        polute (self.globals, kwargs)
+
+        self.globals= Environment ()
+        if g is not None:
+            self.globals.update (g)
+        self.globals.update (kwargs)
 
         if l is None:
             # If exec gets two separate objects as globals and locals,
@@ -137,25 +146,41 @@ def __init__ (self, g=None, l=None, **kwargs):
         self.options= {}
         self.pending_children= []
 
-    def run_file (self, file):
+        # HACK to update the singleton
+        # this might break if we implement subinstances
+        global runner
+        runner= self
+
+
+    def run_file (self, file_name, argv=None):
         # it's a pity that parse() does not accept a file as input
         # so we could avoid reading the whole file
-        return self.run_script (open (file).read (), file)
+        logger.debug ('running from file %s', file_name)
+
+        f= open (file_name)
+        script= f.read ()
+        f.close ()
+
+        return self.run_script (script, file_name, argv)
+
 
-    def run_script (self, script, file_name):
+    def run_script (self, script, file_name, argv=None):
+        logger.debug ('running script:\n-----------\n%s\n-----------', script)
         tree= parse (script, file_name)
         # TODO: self.locals?
         tree= CrazyASTTransformer (self.globals, file_name).modify (tree)
 
-        return self.run_tree (tree, file_name)
+        return self.run_tree (tree, file_name, argv)
 
-    def run_tree (self, tree, file_name):
+
+    def run_tree (self, tree, file_name, argv=None):
         logger.debug2 ('AST: %s', ast.dump (tree))
         logger.debug2 ('code: \n%s', pprint (tree))
 
-        return self.run_code (compile (tree, file_name, 'exec'))
+        return self.run_code (compile (tree, file_name, 'exec'), file_name, argv)
+
 
-    def run_code (self, code):
+    def run_code (self, code, file_name, argv=None):
         if logger.parent.level<=logging.DEBUG2:
             logger.debug2 ('------------------')
             logger.debug2 ('main (gobal) code:')
@@ -175,6 +200,14 @@ def run_code (self, code):
                     elif type (inst.argval)==str:
                         logger.debug ("last function is called: %s", inst.argval)
 
+        # prepare environment
+        self.globals.update (os.environ)
+
+        logger.debug (argv)
+        if argv is None:
+            argv= [ file_name ]
+        self.globals['argv']= argv
+
         '''
         exec(): If only globals is provided, it must be a dictionary, which will
         be used for both the global and the local variables. If globals and locals
@@ -204,8 +237,11 @@ def run_code (self, code):
         '''
         error= None
         try:
+            logger.debug3 ('globals for script: %s', ayrton.utils.dump_dict (self.globals))
             exec (code, self.globals, self.locals)
         except Exception as e:
+            logger.debug ('script finished by Exception')
+            logger.debug (traceback.format_exc ())
             error= e
 
         logger.debug3 ('globals at script exit: %s', ayrton.utils.dump_dict (self.globals))
@@ -218,26 +254,26 @@ def run_code (self, code):
 
         return result
 
+
     def wait_for_pending_children (self):
         for i in range (len (self.pending_children)):
             child= self.pending_children.pop (0)
             child.wait ()
 
+
 def run_tree (tree, g, l):
     """main entry point for remote()"""
-    global runner
     runner= Ayrton (g=g, l=l)
     return runner.run_tree (tree, 'unknown_tree')
 
-def run_file_or_script (script=None, file='script_from_command_line', **kwargs):
+def run_file_or_script (script=None, file_name='script_from_command_line', argv=None,
+                        **kwargs):
     """Main entry point for bin/ayrton and unittests."""
-    logger.debug ('===========================================================')
-    global runner
     runner= Ayrton (**kwargs)
     if script is None:
-        v= runner.run_file (file)
+        v= runner.run_file (file_name, argv)
     else:
-        v= runner.run_script (script, file)
+        v= runner.run_script (script, file_name, argv)
 
     return v