Merge pull request kivy#546 from kivy/python3_support

Add python3 support

.gitignore

@@ -12,7 +12,9 @@
 
 *.pyc
 *.pyo
+*.apk
 .packages
 python_for_android.egg-info
 /build/
+doc/build
 __pycache__/

README.md

@@ -24,7 +24,7 @@ Broad goals of the revamp project include:
 - ✓ Support SDL2
 - ✓ Support multiple bootstraps (user-chosen java + NDK code, e.g. for
   multiple graphics backends or non-Kivy projects)
-- (WIP) Support python3 (recipe exists but crashes on android)
+- ✓ Support python3 (it finally works!)
 - (WIP) Support some kind of binary distribution, including on windows (semi-implemented, just needs finishing)
 - ✓ Be a standalone Pypi module (not on pypi yet but setup.py works)
 - ✓ Support multiple architectures (full multiarch builds not complete, but arm and x86 with different config both work now)

doc/source/bootstraps.rst

@@ -11,56 +11,12 @@ components such as Android source code and various build files.
 If you do not want to modify p4a, you don't need to worry about
 bootstraps, just make sure you specify what modules you want to use
 (or specify an existing bootstrap manually), and p4a will
-automatically build everything appropriately. 
+automatically build everything appropriately. The existing choices are
+explained on the :ref:`build options <bootstrap_build_options>` page.
 
 This page describes the basics of how bootstraps work so that you can
 create and use your own if you like, making it easy to build new kinds
 of Python project for Android.
-
-
-Current bootstraps
-------------------
-
-python-for-android includes the following bootstraps by default, which
-may be chosen by name with a build parameter, or (by default) are
-selected automatically in order to fulfil your build requirements. For
-instance, if you add 'sdl2' in the requirements, the sdl2 backend will
-be used.
-
-p4a is designed to make it fairly easy to make your own bootstrap with a new backend,
-e.g. one that creates a webview interface and runs python in the
-background to serve a flask or django site from the phone itself.
-
-
-pygame
-%%%%%%
-
-This builds APKs exactly like the old p4a toolchain, using Pygame as
-the windowing and input backend.
-
-This bootstrap automatically includes pygame, kivy, and python. It
-could potentially be modified to work for non-Kivy projects.
-
-sdl2
-%%%%
-
-This builds APKs using SDL2 as the window and input backend. It is not
-fully developed compared to the Pygame backend, but has many
-advantages and will be the long term default.
-
-This bootstrap automatically includes SDL2, but nothing else.
-
-You can use the sdl2 bootstrap to seamlessly make a Kivy APK, but can
-also make Python apps using other libraries; for instance, using
-pysdl2 and pyopengl. `Vispy <http://vispy.org/>`_ also runs on android
-this way.
-
-empty
-%%%%%
-
-This bootstrap has no dependencies and cannot actually build an
-APK. It is useful for testing recipes without building unnecessary
-components.
 
 
 Creating a new bootstrap

doc/source/buildoptions.rst

@@ -0,0 +1,108 @@
+
+Build options
+=============
+
+python-for-android provides several major choices for build
+components. This page describes the advantages and drawbacks, and
+extra technical details or requirements, in each case.
+
+
+Python version
+--------------
+
+python-for-android now supports building APKs with either python2 or
+python3, but these have extra requirements or potential disadvantages
+as below.
+
+
+python2
+~~~~~~~
+
+Select this by adding it in your requirements, e.g. ``--requirements=python2``.
+
+This option builds Python 2.7.2 for your selected Android architecture, and
+includes it in the APK. There are no special requirements, all the
+building is done locally.
+
+The python2 build is also the way python-for-android originally
+worked, even in the old toolchain.
+
+
+python3
+~~~~~~~
+
+.. warning::
+   Python3 support is experimental, and some of these details
+   may change as it is improved and fully stabilised.
+
+Select this by adding the ``python3crystax`` recipe to your
+requirements, e.g. ``--requirements=python3crystax``.
+
+This uses the prebuilt Python from the `CrystaX NDK
+<https://www.crystax.net/android/ndk>`__, a drop-in replacement for
+Google's official NDK which includes many improvements. As such, you
+*must* use the CrystaX NDK 10.3.0 or higher when building with
+python3. You can get it `here
+<https://www.crystax.net/en/download>`__.
+
+python3 inclusion should work fine, including all existing
+recipes, but internally this is handled quite differently to the
+locally built python2 so there may be bugs or surprising
+behaviours. If you come across any, feel free to `open an issue
+<https://github.com/kivy/python-for-android>`__.
+
+The experimental status also means that some features are missing and
+the build is not fully optimised so APKs are probably a little larger
+and slower than they need to be. This is currently being addressed,
+though it's not clear how the final result will compare to python2.
+
+.. _bootstrap_build_options:
+
+Bootstrap
+---------
+
+python-for-android supports multiple bootstraps, the Java and JNI code
+that starts the app and the python interpreter, then handles
+interactions with the Android OS.
+
+Currently the following bootstraps are supported, but we hope that it
+it should be easy to add others if your project has different
+requirements. `Let us know
+<https://groups.google.com/forum/#!forum/python-android>`__ if there
+are any improvements that would help here.
+
+sdl2
+~~~~
+
+You can use this with ``--bootstrap=sdl2``, or simply include the
+``sdl2`` recipe in your ``--requirements``.
+
+SDL2 is a popular cross-platform depelopment library, particularly for
+games. It has its own Android project support, which
+python-for-android uses as a bootstrap, and to which it adds the
+Python build and JNI code to start it.
+
+From the point of view of a Python program, SDL2 should behave as
+normal. For instance, you can build apps with Kivy, Vispy, or PySDL2
+and have them work with this bootstrap. It should also be possible to
+use e.g. pygame_sdl2, but this would need a build recipe and doesn't
+yet have one.
+
+.. note::
+   The SDL2 bootstrap is newer, and does not support all the old
+   features of the Pygame one. It is under active development to fix
+   these omissions.
+
+pygame
+~~~~~~
+
+You can use this with ``--bootstrap=pygame``, or simply include the
+``pygame`` recipe in your ``--requirements``.
+
+The pygame bootstrap is the original backend used by Kivy, and still
+works fine for use with Kivy apps. It may also work for pure pygame
+apps, but hasn't been developed with this in mind.
+
+This bootstrap will eventually be deprecated in favour of sdl2, but
+not before the sdl2 bootstrap includes all the features that would be
+lost.

doc/source/commands.rst

@@ -73,6 +73,14 @@ supply those that you need.
 ``--arch``
   The architecture to build for. Currently only one architecture can be 
   targeted at a time, and a given distribution can only include one architecture.
+
+``--bootstrap BOOTSTRAP``
+
+  The Java bootstrap to use for your application. You mostly don't
+  need to worry about this or set it manually, as an appropriate
+  bootstrap will be chosen from your ``--requirements``. Current
+  choices are ``sdl2`` or ``pygame``; ``sdl2`` is experimental but
+  preferable where possible.
 
 
 .. note:: These options are preliminary. Others will include toggles

doc/source/index.rst

@@ -27,13 +27,13 @@ Contents
    :maxdepth: 2
 
    quickstart
+   buildoptions
    installation
    commands
    recipes
    bootstraps
    apis
    troubleshooting
-   old_p4a
    contribute
    old_toolchain/index.rst
 

doc/source/quickstart.rst

@@ -144,8 +144,9 @@ You can build an SDL2 APK similarly, creating a dist as follows::
 You can then make an APK in the same way, but this is more
 experimental and doesn't support as much customisation yet.
 
-There is also experimental support for building APKs with Vispy, which
-do not include Kivy. The basic command for this would be e.g.::
+Your APKs are not limited to Kivy, for instance you can create apps
+using Vispy, or using PySDL2 directly. The basic command for this
+would be e.g.::
 
     python-for-android create --dist_name=testvispy --bootstrap=sdl2 --requirements=vispy
 

doc/source/recipes.rst

@@ -38,6 +38,8 @@ The basic declaration of a recipe is as follows::
       url = 'http://example.com/example-{version}.tar.gz'
       version = '2.0.3'
       md5sum = '4f3dc9a9d857734a488bcbefd9cd64ed'
+      
+      patches = ['some_fix.patch']  # Paths relative to the recipe dir
 
       depends = ['kivy', 'sdl2']  # These are just examples
       conflicts = ['pygame'] 
@@ -118,26 +120,35 @@ Methods and tools to help with compilation
 Patching modules before installation
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
-You can easily apply patches to your recipes with the ``apply_patch``
-method. For instance, you could do this in your prebuild method::
+You can easily apply patches to your recipes by adding them to the
+``patches`` declaration, e.g.::
+
+      patches = ['some_fix.patch',
+                 'another_fix.patch']  
+      
+The paths should be relative to the recipe file. Patches are
+automatically applied just once (i.e. not reapplied the second time
+python-for-android is run).
+
+You can also use the helper functions in ``pythonforandroid.patching``
+to apply patches depending on certain conditions, e.g.::
+
+  from pythonforandroid.patching import will_build, is_arch
 
-  import sh
-  def prebuild_arch(self, arch):
-       super(YourRecipe, self).prebuild_arch(arch)
-       build_dir = self.get_build_dir(arch.arch)
-       if exists(join(build_dir, '.patched')):
-           print('Your recipe is already patched, skipping')
-           return
-       self.apply_patch('some_patch.patch')
-       shprint(sh.touch, join(build_dir, '.patched'))
+  ...
 
-The path to the patch should be in relation to your recipe code.
-In this case, ``some_path.patch`` must be in the same directory as the
-recipe.
+  class YourRecipe(Recipe):
+      
+      patches = [('x86_patch.patch', is_arch('x86')),
+                 ('sdl2_compatibility.patch', will_build('sdl2'))]
+
+      ...
+      
+You can include your own conditions by passing any function as the
+second entry of the tuple. It will receive the ``arch`` (e.g. x86,
+armeabi) and ``recipe`` (i.e. the Recipe object) as kwargs. The patch
+will be applied only if the function returns True.
 
-This code also manually takes care to patch only once. You can use the
-same strategy yourself, though a more generic solution may be provided
-in the future.
 
 Installing libs
 ~~~~~~~~~~~~~~~
@@ -210,29 +221,28 @@ The should_build method
 ~~~~~~~~~~~~~~~~~~~~~~~
 
 The Recipe class has a ``should_build`` method, which returns a
-boolean. This is called before running ``build_arch``, and if it
-returns False then the build is skipped. This is useful to avoid
-building a recipe more than once for different dists.
+boolean. This is called for each architecture before running
+``build_arch``, and if it returns False then the build is
+skipped. This is useful to avoid building a recipe more than once for
+different dists.
 
 By default, should_build returns True, but you can override it however
 you like. For instance, PythonRecipe and its subclasses all replace it
 with a check for whether the recipe is already installed in the Python
 distribution::
 
-    def should_build(self):
+    def should_build(self, arch):
         name = self.site_packages_name
         if name is None:
             name = self.name
-        if exists(join(self.ctx.get_site_packages_dir(), name)):
+        if self.ctx.has_package(name):
             info('Python package already exists in site-packages')
             return False
-        print('site packages', self.ctx.get_site_packages_dir())
         info('{} apparently isn\'t already in site-packages'.format(name))
         return True
 
 
 
-
 Using a PythonRecipe
 --------------------
 
@@ -463,12 +473,11 @@ The above documentation has included a number of snippets
 demonstrating different behaviour. Together, these cover most of what
 is ever necessary to make a recipe work.
 
-The following short sections further demonstrate a few full recipes from p4a's
-internal recipes folder. Unless your own module has some unusual
-complication, following these templates should be all you need to make
-your own recipes work. 
+python-for-android includes many recipes for popular modules, which
+are an excellent resource to find out how to add your own. You can
+find these in the `python-for-android Github page
+<https://github.com/kivy/python-for-android/tree/master/pythonforandroid/recipes>`__.
 
-TODO
 
 .. _recipe_class:
 

pythonforandroid/archs.py

@@ -84,6 +84,8 @@ def get_env(self):
         env['AR'] = '{}-ar'.format(command_prefix)
         env['RANLIB'] = '{}-ranlib'.format(command_prefix)
         env['LD'] = '{}-ld'.format(command_prefix)
+        # env['LDSHARED'] = join(self.ctx.root_dir, 'tools', 'liblink')
+        # env['LDSHARED'] = env['LD']
         env['STRIP'] = '{}-strip --strip-unneeded'.format(command_prefix)
         env['MAKE'] = 'make -j5'
         env['READELF'] = '{}-readelf'.format(command_prefix)
@@ -100,6 +102,9 @@ def get_env(self):
 
         env['ARCH'] = self.arch
 
+        if self.ctx.python_recipe.from_crystax:
+            env['CRYSTAX_PYTHON_VERSION'] = self.ctx.python_recipe.version
+
         return env