Initial commit

.gitignore

@@ -0,0 +1,146 @@
+# Byte-compiled / optimized / DLL files
+__pycache__/
+*.py[cod]
+*$py.class
+
+# C extensions
+*.so
+
+# Distribution / packaging
+.Python
+build/
+develop-eggs/
+dist/
+downloads/
+eggs/
+.eggs/
+lib/
+lib64/
+parts/
+sdist/
+var/
+wheels/
+*.egg-info/
+.installed.cfg
+*.egg
+MANIFEST
+
+# PyInstaller
+#  Usually these files are written by a python script from a template
+#  before PyInstaller builds the exe, so as to inject date/other infos into it.
+*.manifest
+*.spec
+
+# Installer logs
+pip-log.txt
+pip-delete-this-directory.txt
+
+# Unit test / coverage reports
+htmlcov/
+.tox/
+.coverage
+.coverage.*
+.cache
+nosetests.xml
+coverage.xml
+*.cover
+.hypothesis/
+.pytest_cache/
+
+# Translations
+*.mo
+*.pot
+
+# Django stuff:
+*.log
+local_settings.py
+db.sqlite3
+
+# Flask stuff:
+instance/
+.webassets-cache
+
+# Scrapy stuff:
+.scrapy
+
+# Sphinx documentation
+docs/_build/
+
+# PyBuilder
+target/
+
+# Jupyter Notebook
+.ipynb_checkpoints
+
+# pyenv
+.python-version
+
+# celery beat schedule file
+celerybeat-schedule
+
+# SageMath parsed files
+*.sage.py
+
+# Environments
+.env
+.venv
+env/
+venv/
+ENV/
+env.bak/
+venv.bak/
+
+# Spyder project settings
+.spyderproject
+.spyproject
+
+# Rope project settings
+.ropeproject
+
+# mkdocs documentation
+/site
+
+# mypy
+.mypy_cache/
+
+# Windows
+Thumbs.db
+
+# Ignore files built by Visual Studio [code]
+*.obj
+*.exe
+*.pdb
+*.user
+*.dot
+*.jpg
+*.aps
+*.pch
+*.vspscc
+*_i.c
+*_p.c
+*.ncb
+*.suo
+*.tlb
+*.tlh
+*.bak
+*.cache
+*.ilk
+[Bb]in
+[Dd]ebug*/
+*.lib
+*.sbr
+obj/
+[Rr]elease*/
+_ReSharper*/
+[Tt]est[Rr]esult*
+.vs/
+.vscode/
+src.VC.db
+src.VC.VC.opendb
+*.exp
+
+# Miscellaneous
+*~
+
+.idea/
+_build/

LICENSE

@@ -0,0 +1,29 @@
+BSD 3-Clause License
+
+Copyright (c) 2020, Scalable Parallel Computing Lab, ETH Zurich
+All rights reserved.
+
+Redistribution and use in source and binary forms, with or without
+modification, are permitted provided that the following conditions are met:
+
+* Redistributions of source code must retain the above copyright notice, this
+  list of conditions and the following disclaimer.
+
+* Redistributions in binary form must reproduce the above copyright notice,
+  this list of conditions and the following disclaimer in the documentation
+  and/or other materials provided with the distribution.
+
+* Neither the name of the copyright holder nor the names of its
+  contributors may be used to endorse or promote products derived from
+  this software without specific prior written permission.
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
+DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
+FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
+SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
+CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

README.md

@@ -0,0 +1,114 @@
+# pyMLIR: Python Interface for the Multi-Level Intermediate Representation
+
+pyMLIR is a full Python interface to parse, process, and output [MLIR](https://mlir.llvm.org/) files according to the syntax described in the [MLIR documentation](https://github.com/llvm/llvm-project/tree/master/mlir/docs). pyMLIR supports the basic dialects and can be extended with other dialects. It uses [Lark](https://github.com/lark-parser/lark) to parse the MLIR syntax, and mirrors the classes into Python classes. Custom dialects must implement a `.lark` file.
+
+Note that the tool *does not depend on LLVM or MLIR*. It can be installed and invoked directly from Python. 
+
+## Instructions 
+
+**How to install:** `pip install git+git://github.com/spcl/pymlir.git`
+
+**Requirements:** Python 3.6 or newer, and the requirements in `setup.py` or `requirements.txt`. To manually install the requirements, use `pip install -r requirements.txt`
+
+**Problem parsing MLIR files?** Run the file through LLVM's `mlir-opt` to canonicalize it (instructions on how to build/install MLIR can be found [here](https://mlir.llvm.org/getting_started/)):
+```
+$ mlir-opt file.mlir > output.mlir
+```
+
+**Found other problems parsing files?** Not all dialects and modes are supported. Feel free to send us an issue or create a pull request! This is a community project and we welcome any contribution.
+
+## Usage examples
+
+### Parsing MLIR files into Python
+
+```python
+import mlir
+
+# Read a file path, file handle (stream), or a string
+ast1 = mlir.parse_path('/path/to/file.mlir')
+ast2 = mlir.parse_file(open('/path/to/file.mlir', 'r'))
+ast3 = mlir.parse_string('''
+module {
+  func @toy_func(%tensor: tensor<2x3xf64>) -> tensor<3x2xf64> {
+    %t_tensor = "toy.transpose"(%tensor) { inplace = true } : (tensor<2x3xf64>) -> tensor<3x2xf64>
+    return %t_tensor : tensor<3x2xf64>
+  }
+}
+''')
+```
+
+### Inspecting MLIR files in Python
+
+MLIR files can be inspected by dumping their contents (which will print standard MLIR code), or by using the same tools as you would with Python's [ast](https://docs.python.org/3/library/ast.html) module.
+
+```python
+import mlir
+
+# Dump valid MLIR files
+m = mlir.parse_path('/path/to/file.mlir')
+print(m.dump())
+
+print('---')
+
+# Dump the AST directly
+print(m.dump_ast())
+
+print('---')
+
+# Or visit each node type by implementing visitor functions
+class MyVisitor(mlir.NodeVisitor):
+    def visit_Function(self, node: mlir.astnodes.Function):
+        print('Function detected:', node.name)
+
+MyVisitor().visit(m)
+```
+
+### Transforming MLIR files
+
+MLIR files can also be transformed with a Python-like [NodeTransformer](https://docs.python.org/3/library/ast.html#ast.NodeTransformer) object.
+
+```python
+import mlir
+
+m = mlir.parse_path('/path/to/file.mlir')
+
+# Simple node transformer that removes all operations with a result
+class RemoveAllResultOps(mlir.NodeTransformer):
+    def visit_Operation(self, node: mlir.astnodes.Operation):
+        # There are one or more outputs, return None to remove from AST
+        if len(node.result_list) > 0:
+            return None
+
+        # No outputs, no need to do anything
+        return self.generic_visit(node)
+
+m = RemoveAllResultOps().visit(m)
+
+# Write back to file
+with open('output.mlir', 'w') as fp:
+    fp.write(m.dump())
+```
+
+### Using custom dialects
+
+Custom dialects can be written and loaded as part of the pyMLIR parser.
+
+```python
+import mlir
+
+# Try to parse as-is
+try:
+    m = mlir.parse_path('/path/to/matrixfile.mlir')
+except NameError:  # MyMatrix dialect not recognized
+    pass
+
+# Load and verify dialect
+dialect = mlir.Dialect('mymatrix.lark', 'mymatrix')
+
+# Add dialect to the parser
+m = mlir.parse_path('/path/to/matrixfile.mlir', 
+                    dialects=[dialect])
+
+# Print output back
+print(m.dump_ast())
+```

doc/Makefile

@@ -0,0 +1,20 @@
+# Minimal makefile for Sphinx documentation
+#
+
+# You can set these variables from the command line, and also
+# from the environment for the first two.
+SPHINXOPTS    ?=
+SPHINXBUILD   ?= sphinx-build
+SOURCEDIR     = .
+BUILDDIR      = _build
+
+# Put it first so that "make" without argument is like "make help".
+help:
+	@$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
+
+.PHONY: help Makefile
+
+# Catch-all target: route all unknown targets to Sphinx using the new
+# "make mode" option.  $(O) is meant as a shortcut for $(SPHINXOPTS).
+%: Makefile
+	@$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)

doc/conf.py

@@ -0,0 +1,14 @@
+import os
+import sys
+sys.path.insert(0, os.path.abspath('..'))
+project = 'pyMLIR'
+copyright = '2020, Scalable Parallel Computing Laboratory, ETH Zurich'
+author = 'Scalable Parallel Computing Laboratory, ETH Zurich'
+release = '0.5'
+extensions = ['sphinx.ext.autodoc']
+templates_path = ['_templates']
+exclude_patterns = ['_build', 'Thumbs.db', '.DS_Store']
+html_theme = 'sphinx_rtd_theme'
+html_static_path = ['_static']
+
+master_doc = 'index'

doc/custom_dialect.rst

@@ -0,0 +1,33 @@
+Creating a Custom Dialect
+=========================
+
+One of MLIR's most powerful features is being able to define custom dialects. Parsing
+custom dialects in pyMLIR is done by adding them to the ``dialects`` field of the
+mlir parser, as in the following snippet:
+
+.. code-block:: python
+
+  import mlir
+
+  # Load and verify dialect
+  dialect = mlir.Dialect('customdialect.lark', 'NAME')
+
+  # Add dialect to the parser
+  m = mlir.parse_path('/path/to/file.mlir', dialects=[dialect])
+
+Dialects have names, which are defined upon parsing. In this document, we use ``NAME`` as
+the dialect's name.
+
+To be used, a dialect must implement a ``.lark`` file that contains at least the following
+two fields::
+
+    NAME_dialect_operations : op1 | op2 | op3  // ...
+    NAME_dialect_types :
+    // No types are defined, so "NAME_dialect_types" is empty
+
+pyMLIR will then detect the two fields and inject them into the parser.
+
+Advanced Dialect Behavior
+-------------------------
+
+In order to extend custom behavior in the dialect, you can extend the ``mlir.Dialect`` class.

doc/index.rst

@@ -0,0 +1,18 @@
+pyMLIR: Python Interface for MLIR
+=================================
+
+pyMLIR is a Python Interface for the Multi-Level Intermediate Representation (MLIR).
+
+.. toctree::
+   :maxdepth: 2
+
+   custom_dialect
+   source/modules
+
+
+
+Reference
+=========
+
+* :ref:`genindex`
+* :ref:`modindex`

mlir/__init__.py

@@ -0,0 +1,3 @@
+from .parser import parse_file, parse_path, parse_string
+from . import astnodes
+from .visitors import NodeVisitor, NodeTransformer