Added final exponentiation notes (#3)

* made notebooks rusty

* new reference

* added final exponentiation

* more refs

* one last reference

notebooks/bls.ipynb

@@ -258,7 +258,7 @@
     "\n",
     "## Step 3: create public polynomial\n",
     "\n",
-    "First, we need to commit the scalar polynomial generated above to the group to get polynomial on the group, aka multiply each coeff by the generator.\n",
+    "First, we need to commit the scalar polynomial generated above to the group to get polynomial on the group, aka multiply each coeff by the generator. We call it committing because of the close connection to [KZG polynomials](https://www.iacr.org/archive/asiacrypt2010/6477178/6477178.pdf) in SNARKS (a good blog on it is [here](https://dankradfeist.de/ethereum/2020/06/16/kate-polynomial-commitments.html)). \n",
     "\n",
     "all of arkworks-rs, zkcrypto/bls12_381, an threshold_bls implement a struct specifically mapping a Scalar of the field to point on $\\mathbb{G}_2$"
    ]
@@ -330,7 +330,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "so there are [faster ways to generate an element in G2](https://datatracker.ietf.org/doc/html/rfc9380#name-clearing-the-cofactor). \n",
+    "so there are [faster ways to generate an element in G2](https://datatracker.ietf.org/doc/html/rfc9380#name-clearing-the-cofactor), for example [this](https://eprint.iacr.org/2017/419.pdf). \n",
     "## Step 4: partial signaturing\n",
     "ok, great. c'est parti à la lune . we now need to partial sign messages. this is distributed obvs in our case, but for here it'd be nice to have something like"
    ]
@@ -510,7 +510,11 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "metadata": {},
+   "metadata": {
+    "vscode": {
+     "languageId": "rust"
+    }
+   },
    "outputs": [],
    "source": [
     "# Arguments:\n",
@@ -727,21 +731,17 @@
  ],
  "metadata": {
   "kernelspec": {
-   "display_name": "Python 3 (ipykernel)",
-   "language": "python",
-   "name": "python3"
+   "display_name": "Rust",
+   "language": "rust",
+   "name": "rust"
   },
   "language_info": {
-   "codemirror_mode": {
-    "name": "ipython",
-    "version": 3
-   },
-   "file_extension": ".py",
-   "mimetype": "text/x-python",
-   "name": "python",
-   "nbconvert_exporter": "python",
-   "pygments_lexer": "ipython3",
-   "version": "3.11.6"
+   "codemirror_mode": "rust",
+   "file_extension": ".rs",
+   "mimetype": "text/rust",
+   "name": "Rust",
+   "pygment_lexer": "rust",
+   "version": ""
   }
  },
  "nbformat": 4,

notebooks/field_extensions.ipynb

@@ -16,7 +16,7 @@
     "\n",
     "So we have $\\mathbb{G}_1\\subset E(\\mathbb{F}_p)$ with $|\\mathbb{G}_1|=r$, and $\\mathbb{G}_2\\subset E(\\mathbb{F}_{p^{12}})$ with $|\\mathbb{G}_2|=r$ which we want to use for our pairing.\n",
     "\n",
-    "## $\\mathbb{F}_{p^{12}}$\n",
+    "## 12-th order extension of the base field\n",
     "\n",
     "Given an irreducible polynomial $N \\in \\mathbb{F}_p[x]$ of degree $m=12$, the elements of this extension are those given by $\\{a_{m-1}x^{m-1}+\\cdots+a_1x+a_0 \\,|\\, a_i\\in \\mathbb{F}_p\\}$\n",
     "\n",
@@ -73,13 +73,13 @@
     "\n",
     "In our scheme, to hash a message to $E$, we use `hash_to_field` and `field_to_curve`, and then multiply the mapped curve point by the generator of the curve to create a point in $\\mathbb{G}_1$. Fortunately, by [Theorem 2.3.1 of Silverman](https://link.springer.com/book/10.1007/978-0-387-09494-6), we have \n",
     "$$ |E(\\mathbb{F}_p)|=p+1-t$$\n",
-    "and for BN curves generated by a value $z=2^{62}-2^{54}+2^{44}$, we have $p(z)+1-t(z)=r(z)$, implying that $|E(\\mathbb{F}_p)|=r\\implies \\mathbb{G}_1 = E(\\mathbb{F}_p)[r]=E(\\mathbb{F}_p)$!\n",
+    "and for BN curves generated by a value $z=2^{62}-2^{54}+2^{44}$, we have $p(z)+1-t(z)=r(z)$, implying that $|E(\\mathbb{F}_p)|=r\\implies \\mathbb{G}_1 = E(\\mathbb{F}_p)[r]=E(\\mathbb{F}_p)$! In this way, since $r$-torsions give us some notion of structure, this means that the \"prime factorization\" of the curve is simply the curve itself, so its smallest possible prime order subgroup is just the group, no extra structure to be found.\n",
     "\n",
     "We therefore only need to check if a pair $(x,y)\\in\\mathbb{F}_r\\times\\mathbb{F}_r$ is on the curve $E(\\mathbb{F}_p)$ for membership in $\\mathbb{G}_1$. \n",
     "\n",
     "#### G2\n",
     "\n",
-    "This is a bit trickier. You can check easily if the point $(x,y)\\in\\mathbb{F}_{p^2}\\times\\mathbb{F}_{p^2}$ lies on $E^\\prime(\\mathbb{F}_{p^2})$, but unfortunately the order of the twist curve is not given by the order of the $r$-torsion, ie $|E^\\prime(\\mathbb{F}_{p^2})|=c_2r$, where $c_2$ is the $\\mathbb{G}_2$ cofactor. [You can show](https://hackmd.io/@jpw/bn254#mathbb-G_2-order) that $c_2=p+t-1$. \n",
+    "This is a bit trickier. You can check easily if the point $(x,y)\\in\\mathbb{F}_{p^2}\\times\\mathbb{F}_{p^2}$ lies on $E^\\prime(\\mathbb{F}_{p^2})$, but unfortunately the order of the twist curve is not given by the order of the $r$-torsion, ie $|E^\\prime(\\mathbb{F}_{p^2})|=c_2r$, where $c_2$ is the $\\mathbb{G}_2$ cofactor. [You can show](https://hackmd.io/@jpw/bn254#mathbb-G_2-order) that $c_2=p+t-1$. Thinking about $r$-torsions as structure again, it makes sense that this is the case even just from the consideration of the total number of elements in the preimage of $\\mathbb{G}_2$ ($\\mathbb{F}_{p^2}\\times\\mathbb{F}_{p^2}$) vs $\\mathbb{G}_1$ ($\\mathbb{F}_p$); with that many more elements to consider, it makes sense that there is additional structure in the group to now deal with. \n",
     "\n",
     "You can just rely on the definition of the $r$-torsion if you want to check if $[r](x,y)=\\mathcal{O}$, but with 254 bits of r, this is slooooooow.\n",
     "\n",
@@ -90,6 +90,8 @@
     "where $\\Psi$ is the twist mapping, and recall $\\xi=9+u$. Membership in $\\mathbb{G}_2$ therefore [boils down to verifying](https://eprint.iacr.org/2022/352.pdf) if the following holds: $Q=(x^\\prime, y^\\prime); \\psi(Q)=[6x^2]Q$, and [more recent work](https://eprint.iacr.org/2022/348.pdf) improves this to the following:\n",
     "$$[x+1]Q + \\psi([x]Q) + \\psi([x]Q) = \\psi^3([2x]Q)$$\n",
     "\n",
+    "[You can go even further too](https://eprint.iacr.org/2022/352.pdf).\n",
+    "\n",
     "## optimal ate pairing\n",
     "\n",
     "finally, we're here!! fuck.\n",
@@ -204,25 +206,141 @@
     "    final_exponentiation(&f)\n",
     "}"
    ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Final exponentiation\n",
+    "\n",
+    "Arguably, this is the most computationally expensive step since the bit size of the exponent in the pairing is huge, so the naïve approach would be silly. I mean, there are issues with the $\\mathbb{G}_2$ cofactor clearing to create elements in $\\mathbb{G}_2$ from the field because of the size of the cofactor, so if multiplication is slow, exponentiation is not guaranteed to be better *a priori*. \n",
+    "\n",
+    "The following takes the lead from [this](https://eprint.iacr.org/2020/875.pdf) and [that](https://eprint.iacr.org/2008/490.pdf). \n",
+    "\n",
+    "The most efficient calculation of these pairings relies on notions of *cyclotomic subgroups*. oof. \n",
+    "\n",
+    "Up until this point, we were precise in our definitions of $\\mathbb{G}_1$ and $\\mathbb{G}_2$, but have been unclear about what exactly the target group of the pairing should be. We now formally define the target group $\\mathbb{G}_T$ to be the group of $r$-th roots of unity over the multipicative group $\\mathbb{F}_{p^k}^\\ast=(\\mathbb{F}_{p^k}/ \\{0\\}, \\ast)$, denoted commonly by $\\mu_r$. Why the roots o f unity? Great question. Remember that this mapping has to satisfy a few key real-world properties. First, it has to be a trapdoor, namely preimage resistance (assuming DL hardness), and mapping backwards from the roots of unity is a very difficult problem. Second, it allows for an easy metric against which we can compare two mappings. For example, in the case of signature verification $e(\\sigma_i, g_2)=e(H(m), P(i))$, it is very natural to want to set the actual value of each side of this equation to \"one\", therefore implying the image domain to be the roots of unity. \n",
+    "\n",
+    "In the following, let $(\\mathbb{F}_{p^k}^\\ast)^r$ is the subgroup of $r$-th powers, namely all elements in $\\mathbb{F}_{p^k}^\\ast$ that are expressed as $x^r$ for some $x$. We can then define the quotient group $\\mathbb{F}_{p^k}^\\ast / (\\mathbb{F}_{p^k}^\\ast)^r$ which represent the coset of $r$ powers, namely each element in this quotient group differ by a power of $r$. \n",
+    "\n",
+    "Note that $\\forall x\\in\\mathbb{F}_{p^k}^\\ast$, $\\left(x^{\\frac{p^k-1}{r}}\\right)^r = x^{p^k-1} = 1$, which means elements of the form $x^{\\frac{p^k-1}{r}}\\in\\mathbb{F}_{p}^\\ast / (\\mathbb{F}_{p}^\\ast)^r$, which is precisely what the pairing function $e$ does. There is a natural isomorphism between the quotient group and the roots of unity, so we can equivalently talk about either. For the purposes of the following, however, we'll keep to the quotient group representation since it admits a few additional insights we can use to our advantage. \n",
+    "\n",
+    "---\n",
+    "\n",
+    "Aside: this is not a light topic to cover, even for the level of depth in this document (hard to believe, I know), but is a result from Galois theory and the so called *Kummer theorems*, see [this](https://websites.umich.edu/~asnowden/teaching/2019/776/cft-01.pdf)\n",
+    "\n",
+    "---\n",
+    "\n",
+    "Since the optimal ate pairing is mapping our \"multiplication\" of an element from $\\mathbb{G}_1$ and an element from $\\mathbb{G}_2$ to the group of roots of unity by means of exponentiation of an element from the base field extension $\\mathbb{F}_{p^{12}}$, it would be useful to represent our exponent $(p^{12}-1)/r$ in a form closer related to the roots of unity to which we're mapping. To do this, we define what's called the cyclotomic polynomial, defined by an order $n$. This polynomial contains all of the irreducible factors of $x^n-1$ (which defines the roots of unity), and is therefore the polynomial whose roots are the roots of unity. In this sense, this polynomial captures all of the structure of the roots of unity, and because of its irreducibility, we can use it to build other mappings that deal with the group of roots of unity.\n",
+    "\n",
+    "Specifically, we define the $k$-th cyclotomic polynomial $\\varPhi(x)$ to be:\n",
+    "$$\\frac{p^k-1}{\\varPhi_k(p)} = \\prod_{j\\vert k, j\\neq k}\\Phi_j(p) $$\n",
+    "\n",
+    "which allows us to break down the exponent of the \"final exponentiation\" step. Writing the embedding degree $k=ds$, where $d$ is a positive integer, we can write:\n",
+    "\n",
+    "$$ \\frac{p^k-1}{r}=\\underbrace{\\left[(p^s-1)\\cdot\\frac{\\sum_{i=0}^{d-1}p^{is}}{\\varPhi_k(p)}\\right]}_\\text{easy part}\\cdot\\underbrace{\\left[\\frac{\\varPhi_k(p)}{r}\\right]}_\\text{hard part}$$\n",
+    "\n",
+    "### Easy part\n",
+    "\n",
+    "For BN254, this decomposes the easy part into $(p^6-1)(p^2+1)$ for $k=12$ (note that we choose this decomposition because exponentiation by powers of $p$ are very efficient, see the discussion [earlier](#12-th-order-extension-of-the-base-field)). The easy part will involve something like $x^{p^6-1}=x^{p^6}\\cdot x^{-1}$, which is one conjugation, one inversion, and one multiplication (remember that conjugation in $\\mathbb{F}_{p^{12}}$ for an element $x=a+bw$ is simply $\\bar{x}=a-bw$). Then taking $\\left(x^{p^6-1}\\right)^{p^2}$ is just applying our Frobenius morphism $\\phi$, and then finally we multipy by our already-computed value $x^{p^6-1}$, and voila!\n",
+    "\n",
+    "Easy part = 1 conjugation + 1 inversion + 1 multiplication + 5 multiplications + 1 multiplication\n",
+    "\n",
+    "### Hard part\n",
+    "\n",
+    "For BN254, the hard part decomposes into $\\frac{p^4-p^2+1}{r}$. It seems that the typical way to go here is to take a base-$p$ expansion, namely defining $\\lambda\\triangleq m\\varphi_k(p)/r$ with $r\\nmid m$, and finding a vector $\\tau$ of $w+1$ integers $\\tau=(\\lambda_0,\\ldots,\\lambda_w)$ such that $\\lambda=\\sum\\lambda_i p^i$ minimizing the L1-norm of $\\tau$. Recall that for us:\n",
+    "$$\n",
+    "\\begin{align}\n",
+    "    p(z) &= 36z^4 + 36z^3 + 24z^2 + 6z + 1\\\\\n",
+    "    r(z) &= 36z^4 + 36z^3 + 18z^2 + 6z + 1\\\\\n",
+    "    t(z) &= 6z^2+1\n",
+    "\\end{align}\n",
+    "$$\n",
+    "so substituting these into the hard part of the polynomial as a function of the curve family generator $z$ yields $\\lambda_3 p^3+\\lambda_2 p^2 +\\lambda_1 p + \\lambda_0$ with:\n",
+    "$$\n",
+    "\\begin{align}\n",
+    "\\lambda_3(z) &= 1\\\\\n",
+    "\\lambda_2(z) &= 6z^2+1\\\\\n",
+    "\\lambda_1(z) &= -36z^3 -18z^2-12z+1\\\\\n",
+    "\\lambda_0(z) &= -36z^3 - 30z^2-18z -2\n",
+    "\\end{align}\n",
+    "$$\n",
+    "\n",
+    "We now then compute the hard part as a series of multiplications in terms of powers of the easy part. \n",
+    "\n",
+    "1. Compute $f_\\mathrm{easy}^z, (f_\\mathrm{easy}^z)^z, (f_\\mathrm{easy}^{z^2})^z$\n",
+    "2. Use the Frobenius operator, which has efficient representations in powers 1, 2, and 3 of the prime, to compute $f_\\mathrm{easy}^p, f_\\mathrm{easy}^{p^2}, f_\\mathrm{easy}^{p^3}, (f_\\mathrm{easy}^z)^p, (f_\\mathrm{easy}^{z^2})^p, (f_\\mathrm{easy}^{z^3})^p, (f_\\mathrm{easy}^{z^3})^{p^2}$\n",
+    "\n",
+    "The evaluation then amounts to:\n",
+    "\n",
+    "$$ \\underbrace{[f_\\mathrm{easy}^p \\cdot f_\\mathrm{easy}^{p^2} \\cdot f_\\mathrm{easy}^{p^3}]}_{\\equiv y_0} \\cdot \\underbrace{[1/f_\\mathrm{easy}]^2}_{\\equiv y_1^2} \\cdot \\underbrace{[(f_\\mathrm{easy}^{z^2})^p]^6}_{\\equiv y_2^6} \\cdot \\underbrace{[1/(f_\\mathrm{easy}^z)^p]^{12}}_{\\equiv y_3^{12}} \\cdot \\underbrace{[1/(f_\\mathrm{easy}^z \\cdot (f_\\mathrm{easy}^{z^2})^p)]^{18}}_{\\equiv y_4^{18}} \\cdot \\underbrace{[1/f_\\mathrm{easy}^{z^2}]^{30}}_{\\equiv y_5^{30}} \\cdot\n",
+    "\\underbrace{[1/(f_\\mathrm{easy}^{z^3} \\cdot (f_\\mathrm{easy}^{z^3})^p)]^{36}}_{\\equiv y_6^{36}}$$\n",
+    "\n",
+    "These evaluations have efficient algorithms that have been around [for a long time](https://www.sciencedirect.com/science/article/pii/0196677481900031). We take the vector addition chain approach, which is more or less the equivalent of \"flattening\" that we also see crop up in the reduction of polynomial constraints in instance-witness definitions of R1CS systems. You can show that the following definitions yield efficient computation of these multiexponentials which we take from [the original manuscript](https://eprint.iacr.org/2008/490.pdf):\n",
+    "$$\n",
+    "\\begin{align*}\n",
+    "T_0 &\\leftarrow (y_6)^2 \\\\\n",
+    "T_0 &\\leftarrow T_0 \\cdot y_4 \\\\\n",
+    "T_0 &\\leftarrow T_0 \\cdot y_5 \\\\\n",
+    "T_1 &\\leftarrow y_3 \\cdot y_5 \\\\\n",
+    "T_1 &\\leftarrow T_1 \\cdot T_0 \\\\\n",
+    "T_0 &\\leftarrow T_0 \\cdot y_2 \\\\\n",
+    "T_1 &\\leftarrow (T_1)^2 \\\\\n",
+    "T_1 &\\leftarrow T_1 \\cdot T_0 \\\\\n",
+    "T_1 &\\leftarrow (T_1)^2 \\\\\n",
+    "T_0 &\\leftarrow T_1 \\cdot y_1 \\\\\n",
+    "T_1 &\\leftarrow T_1 \\cdot y_0 \\\\\n",
+    "T_0 &\\leftarrow (T_0)^2 \\\\\n",
+    "f_\\mathrm{hard} &\\leftarrow T_0 \\cdot T_1\n",
+    "\\end{align*}\n",
+    "$$\n",
+    "which is only a few multiplications and squarings! Efficient. There's extensions and [improvements](https://doi.org/10.1007/s12190-018-1167-y), but that's the basic stuff.\n",
+    "\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "\n",
+    "## BONUS - glued miller loop and improved signature performance\n",
+    "\n",
+    "This is pretty sick. Remember that eventually we want to check the relation $e(\\sigma_i, g_2)=e(H(m), P(i) )$ for verification. You could just naively evaluate lhs and rhs and check for equality. Right? Or notice that:\n",
+    "\n",
+    "$$ \n",
+    "\\begin{align}\n",
+    "e(\\sigma_i, g_2) &= e(H(m), P(i)) \\\\\n",
+    "\\implies e(\\sigma_i, g_2)e(H(m), P(i))^{-1} &= 1\\\\\n",
+    "\\implies e(\\sigma_i, g_2)e(H(m), -P(i)) &= 1 \\\\\n",
+    "\\implies \\left(f_{[6z+2], \\sigma_i}(g_2)f_{[6z+2], H(m)}(-P(i))\\right)^\\frac{p^{12}-1}{r} &= 1\n",
+    "\\end{align}\n",
+    "$$\n",
+    "\n",
+    "which results in only having to evaluate a single Miller loop, followed by a single exponentiation at the end! Also during the recursion we don't need to track $f_{i,s}(G)$ nor $f_{i,H(m)}(-xG)$, just their product, which saves a multiplication in $\\mathbb{F}_{12}$ in each iteration of this *glued* Miller loop. The savings compound since we're aggregating many partial signatures, and the idea works exactly the same for the aggregated signatures. Namely, verification is equivalent to:\n",
+    "\n",
+    "$$ \n",
+    "\\left(\\prod_{i}^tf_{[6z+2], \\sigma_i}(g_2) f_{[6z+2], H(m)}(-P(i))\\right)^\\frac{p^{12}-1}{r} = 1\n",
+    "$$\n",
+    "\n",
+    "---\n",
+    "\n",
+    "This is the jist of it. There's so many more things to work with, like different pairings like the [Xate pairing](https://link.springer.com/chapter/10.1007/978-3-540-85538-5_13), but that's beyond the scope here. I'll just mention maybe that there are many more cleverer things you can do to take full advantage of the cyclotomic subgroup stuff like [efficient compression and arithmetic on compressed representations of elements](https://eprint.iacr.org/2010/542.pdf), but that's over the top for now."
+   ]
   }
  ],
  "metadata": {
   "kernelspec": {
-   "display_name": "Python 3 (ipykernel)",
-   "language": "python",
-   "name": "python3"
+   "display_name": "Rust",
+   "language": "rust",
+   "name": "rust"
   },
   "language_info": {
-   "codemirror_mode": {
-    "name": "ipython",
-    "version": 3
-   },
-   "file_extension": ".py",
-   "mimetype": "text/x-python",
-   "name": "python",
-   "nbconvert_exporter": "python",
-   "pygments_lexer": "ipython3",
-   "version": "3.11.6"
+   "codemirror_mode": "rust",
+   "file_extension": ".rs",
+   "mimetype": "text/rust",
+   "name": "Rust",
+   "pygment_lexer": "rust",
+   "version": ""
   }
  },
  "nbformat": 4,