[feahtr] Install dependencies for readthedocs (feathr-ai#196)

.readthedocs.yml

@@ -0,0 +1,18 @@
+# .readthedocs.yaml
+# Read the Docs configuration file
+# See https://docs.readthedocs.io/en/stable/config-file/v2.html for details
+
+# Required
+version: 2
+
+# Set the version of Python and other tools you might need
+
+# Build documentation in the docs/ directory with Sphinx
+sphinx:
+  configuration: feathr_project/docs/conf.py
+
+# Optionally declare the Python requirements required to build your docs
+python:
+  install:
+    - requirements: feathr_project/docs/requirements.txt
+  system_packages: true

docs/dev_guide/update_python_docs.md

@@ -11,22 +11,70 @@ parent: Feathr Developer Guides
 
 Since `sphinx` need to run the source code to generate the documentation, make sure you have installed the all the necessary dependencies needed by the project(see setup.py files).
 
-## Update the documentation html files
+## How Readthedocs.com Works
+* Readthedocs need to run the Python code to generate the docs. So your dependencies need to exist on the machine that generates these docs.
+* Readthedocs can be configured via `.readthedocs.yml` under project root(not your Python library root). If you don't 
+  need extra configurations, you don't need `.readthedocs.yml`.
+* Readthedocs will generate documentations based on your `.rst` file. `.rst` file can be automatically generated or manually modified(see later sections).
+* Readthedocs doesn't support complex dependencies, like Pypi packages with C modules. To address this, Readthedocs provides those common
+  dependencies in their server but we need to enable it by setting `system_packages: true`
+* Dependencies can be specified by docs/requirements.txt file.
+* You don't need to commit _build files. Those files will be generated by the Readthedocs.com servers.
 
-After you updated the documentation in the source code, run the following command to update the html files.
+### How to Specify Dependencies
+Readthedocs provides two ways of specifying depencies.
+The first one is via the docs/requirements.txt file. This one will need to duplicate some of the information from the setup.py file. The benefit is we can pick only the dependencies that are needed by the execution of Readthedocs.
+```
+python:
+  install:
+    - requirements: feathr_project/docs/requirements.txt
+```
 
-In docs directory:
+The second one is using setup.py. This one avoids the duplication but have to install all dependencies. Readthedocs have poor support on importing complex dependencies, like Pandas can't be imported. So this really doesn't work for us.
+```
+python:
+  install:
+    - method: setuptools
+      path: feathr_project/
+```
+So using requirements.txt is used.
 
-`sphinx-apidoc -f -o . ../feathr ../*setup*`
+In the future, we should simplify the dependencies for user facing APIs. But it's hard to do the same for developer-facing APIs. We still count on Readthedocs to sipmlify and address the dependency importing issues.
+
+### How to Edit Contents
+You can edit the rst files to modify the structure and contents of the docs page.
 
-(excluding setup.py files and some other demo files, test files.)
+## Build the documentation html files
 
 Then rebuild the html files:
 
 `make clean && make html`
 
 You will see new html files generated under `_build/html/` directory and you can view `_build/html/index.html` in your browser locally.
 
+## Re-build the documentation html files
+If you need to re-build the `.rst` files, run the following command to update them:
+
+In docs directory:
+
+`sphinx-apidoc -f -o . ../feathr ../*setup*`
+
+(excluding setup.py files, and some other demo files, test files.)
+
 ## Upload to Readthedocs.com
+* Login to https://readthedocs.org/dashboard/
+* Click `Import a Project`
+* Click `Import Manually` on the right side
+* Fill in `name`, `Repository URL`(the url to feathr main), and `Default branch`(main or the branch you want to test).
+* Click `Next`
+
+### Test
+* After you have imported your own branch, you can click `Build version` to test the build result of your latest code on the branch.
+* You can click on each pannels to see the command message and warnings.
+* After the build is successful, it will show
+  the docs page(like https://test233.readthedocs.io/en/latest/feathr.html). But they have a site cache issue. You have to refresh the
+  site then you can see your new result.
 
-(TBD)
+### Debug and Known Issues
+* `No module named xyz`: Readthedocs need to run the code to generated the docs. So if your dependency is not specified
+in the docs/requirements.txt, it will fail on this. To fix it, specify the dependency in requirements.txt.

feathr_project/.readthedocs.yml

@@ -0,0 +1,20 @@
+# .readthedocs.yaml
+# Read the Docs configuration file
+# See https://docs.readthedocs.io/en/stable/config-file/v2.html for details
+
+# Required
+version: 2
+
+# Set the version of Python and other tools you might need
+build:
+  tools:
+    python: "3.9"
+
+# Build documentation in the docs/ directory with Sphinx
+sphinx:
+  configuration: docs/conf.py
+
+# Optionally declare the Python requirements required to build your docs
+python:
+  install:
+    - requirements: docs/requirements.txt

feathr_project/docs/conf.py

@@ -24,9 +24,9 @@
 author = 'Feathr Community'
 
 # The short X.Y version
-version = '0.1'
+version = '0.4'
 # The full version, including alpha/beta/rc tags
-release = '0.1_azure_alpha'
+release = '0.4'
 
 
 # -- General configuration ---------------------------------------------------

feathr_project/docs/feathr.rst

@@ -1,20 +1,10 @@
 feathr package
 ==============
 
-Subpackages
------------
-
-.. toctree::
-
-    feathr.protobuf
-
-Submodules
-----------
-
-feathr.aggregation module
--------------------------
+feathr.client module
+--------------------
 
-.. automodule:: feathr.aggregation
+.. automodule:: feathr.client
     :members:
     :undoc-members:
     :show-inheritance:
@@ -27,18 +17,11 @@ feathr.anchor module
     :undoc-members:
     :show-inheritance:
 
-feathr.client module
---------------------
 
-.. automodule:: feathr.client
-    :members:
-    :undoc-members:
-    :show-inheritance:
-
-feathr.constants module
------------------------
+feathr.aggregation module
+-------------------------
 
-.. automodule:: feathr.constants
+.. automodule:: feathr.aggregation
     :members:
     :undoc-members:
     :show-inheritance:
@@ -59,14 +42,6 @@ feathr.feathr\_configurations module
     :undoc-members:
     :show-inheritance:
 
-feathr.feathr\_pyspark\_driver\_template module
------------------------------------------------
-
-.. automodule:: feathr.feathr_pyspark_driver_template
-    :members:
-    :undoc-members:
-    :show-inheritance:
-
 feathr.feature module
 ---------------------
 
@@ -83,14 +58,6 @@ feathr.feature\_derivations module
     :undoc-members:
     :show-inheritance:
 
-feathr.job\_utils module
-------------------------
-
-.. automodule:: feathr.job_utils
-    :members:
-    :undoc-members:
-    :show-inheritance:
-
 feathr.lookup\_feature module
 -----------------------------
 
@@ -115,14 +82,6 @@ feathr.query\_feature\_list module
     :undoc-members:
     :show-inheritance:
 
-feathr.repo\_definitions module
--------------------------------
-
-.. automodule:: feathr.repo_definitions
-    :members:
-    :undoc-members:
-    :show-inheritance:
-
 feathr.settings module
 ----------------------
 

feathr_project/docs/feathr_dev.rst

@@ -0,0 +1,52 @@
+feathr package
+==============
+
+feathr.constants module
+-----------------------
+
+.. automodule:: feathr.constants
+    :members:
+    :undoc-members:
+    :show-inheritance:
+
+feathr.repo\_definitions module
+-------------------------------
+
+.. automodule:: feathr.repo_definitions
+    :members:
+    :undoc-members:
+    :show-inheritance:
+
+feathr.job\_utils module
+------------------------
+
+.. automodule:: feathr.job_utils
+    :members:
+    :undoc-members:
+    :show-inheritance:
+
+feathr.feathr\_pyspark\_driver\_template module
+-----------------------------------------------
+
+.. automodule:: feathr.feathr_pyspark_driver_template
+    :members:
+    :undoc-members:
+    :show-inheritance:
+
+
+feathr.protobuf.featureValue\_pb2 module
+----------------------------------------
+
+.. automodule:: feathr.protobuf.featureValue_pb2
+    :members:
+    :undoc-members:
+    :show-inheritance:
+
+
+Module contents
+---------------
+
+.. automodule:: feathr
+    :members:
+    :undoc-members:
+    :show-inheritance:

feathr_project/docs/index.rst

@@ -3,15 +3,24 @@
    You can adapt this file completely to your liking, but it should at least
    contain the root `toctree` directive.
 
-Welcome to feathr's documentation!
+Welcome to Feathr's documentation!
 ==================================
 
+Feathr User APIs
+==================
+
 .. toctree::
-   :maxdepth: 2
-   :caption: Contents:
+   :maxdepth: 4
+
+   feathr
 
-   modules
+Feathr APIs for Internal Developers
+==================
+
+.. toctree::
+   :maxdepth: 4
 
+   feathr_dev
 
 Indices and tables
 ==================

feathr_project/docs/requirements.txt

@@ -0,0 +1,14 @@
+redis
+azure-core<=1.22.1
+azure-storage-file-datalake>=12.5.0
+azure-synapse-spark
+azure-identity
+pyhocon
+loguru
+tqdm
+databricks_cli
+pyyaml
+graphlib_backport
+pyapacheatlas
+google>=3.0.0
+google-api-python-client>=2.41.0

feathr_project/feathr/anchor.py

@@ -8,14 +8,14 @@
 # passthrough features do not need keys
 class FeatureAnchor:
     """
-    A feature anchor defines a set of features on top of a data source, a.k.a. a set of features anchored to a source.
+        A feature anchor defines a set of features on top of a data source, a.k.a. a set of features anchored to a source.
 
-    The feature producer writes multiple anchors for a feature, exposing the same feature name for the feature
-    consumer to reference it.
-    Attributes:
+        The feature producer writes multiple anchors for a feature, exposing the same feature name for the feature
+        consumer to reference it.
+        Attributes:
         name: Unique name of the anchor.
         source: data source that the features are anchored to. Should be either of `INPUT_CONTEXT` or `feathr.source.Source`
-        features: list of features within this anchor. 
+        features: list of features within this anchor.
         registry_tags: A dict of (str, str) that you can pass to feature registry for better organization. For example, you can use {"deprecated": "true"} to indicate this anchor is deprecated, etc.
     """
     def __init__(self,