gpt2 and others - update

content/post/symbolic_regression/_index.md

@@ -2,4 +2,3 @@
 title: Symbolic Regression
 type: series
 ---
-

content/post/symbolic_regression/learning_pysr.md

@@ -9,6 +9,63 @@ draft = false
 A library for symbolic regression made by one of my favorite researchers, Miles Cranmer.
 The backend genetic algorithm(optimization method for finding symbolic expression) runs on Julia, so it's really fast. It's also extremely flexible.
 
-## Running PySr
+## Installing and Running PySr
 
-I'm going to use a Jupyter notebook for all my PySr experiments.
+I'm going to use a Jupyter notebook with a `virtualenv` environment for all my PySr experiments.
+
+```sh
+virtualenv venv
+source venv/bin/activate
+pip install numpy pysr
+```
+
+# Vector->Scalar Fit
+
+The first import of pysr will take much longer than the next as Julia dependencies are getting set up.
+
+```python
+from pysr import PySRRegressor
+import numpy as np
+```
+
+This is function $f:\R^5\rightarrow\R$ that we'll be fitting symbolically:
+```python
+X = np.random.randn(100, 5)
+y = np.sum(X * X, axis=-1)
+```
+
+`X` has shape `(100,5)`; `y`, `(100,)`.
+
+Now instantiate a new regressor model. Running `fit` on `PySrRegressor` will create lots of useful (but currently unecessary) files that we'll route to the `temp` folder:
+```python
+model = PySRRegressor(niterations=200, binary_operators=["*", '+'],tempdir='./temp/',temp_equation_file=True)
+model.fit(X, y)
+```
+
+![model output](/pysr_simplevsfit.png)
+
+To get the best model's equation as a rendered latex string, you can call the `sympy` method. The `latex` method returns the corresponding unrendered latex string.
+
+```python
+model.sympy()
+```
+
+$$x_{0}^{2} + x_{1}^{2} + x_{2}^{2} + x_{3}^{2} + x_{4}^{2}$$
+
+As you can see, the model learned the exact ground-truth equation.
+
+# Vector->Vector fit
+
+PySr can also fit Vector->Vector functions by outputting a list with different symbolic expressions for each.
+
+```python
+model = PySRRegressor(niterations=200, binary_operators=["*", '+'],tempdir='./temp/',temp_equation_file=True)
+model.fit(X, y)
+model.sympy()
+```
+
+```sh
+[x0**2, x1**2, x2**2, x3**2, x4**2]
+```
+
+Since the output is a list, we can't display each as latex.

content/post/symbolic_regression/symbolic_distillation.md

@@ -0,0 +1,11 @@
++++
+title = 'Simple Symbolic Distillation on XOR Neural Network'
+date = 2024-06-09T21:25:23+02:00
+draft = false
++++
+
+# What is symbolic distillation?
+
+Symbolic distillation is a technique pioneered by Miles Cranmer to improve interpretability of neural networks using symbolic regression. The basic premise is to approximate various components of a neural network by applying symbolic regression to different parts. These parts can then be substituted for the original network.
+
+We'll try this 

hugo.toml

@@ -21,3 +21,4 @@ weight = 2
 
 [params]
   customCSS = ["css/custom.css"]
+

layouts/partials/foot_custom.html

@@ -5,3 +5,15 @@
 <link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/highlight.js/11.9.0/styles/default.min.css">
 <script src="https://cdnjs.cloudflare.com/ajax/libs/highlight.js/11.9.0/highlight.min.js"></script>
 <script>hljs.highlightAll();</script>
+
+<script>
+	document.addEventListener("DOMContentLoaded", function () {
+		renderMathInElement(document.body, {
+			delimiters: [
+				{left: "$$", right: "$$", display: true},
+				{left: "\\(", right: "\\)", display: false},
+				{left: "$", right: "$", display: false}
+			]
+		});
+	});
+</script>

public/404.html

@@ -35,6 +35,18 @@
 <script src="https://cdnjs.cloudflare.com/ajax/libs/highlight.js/11.9.0/highlight.min.js"></script>
 <script>hljs.highlightAll();</script>
 
+<script>
+	document.addEventListener("DOMContentLoaded", function () {
+		renderMathInElement(document.body, {
+			delimiters: [
+				{left: "$$", right: "$$", display: true},
+				{left: "\\(", right: "\\)", display: false},
+				{left: "$", right: "$", display: false}
+			]
+		});
+	});
+</script>
+
 
   </footer>
   </body>

public/about/index.html

@@ -48,6 +48,18 @@ <h1><span class="title">About</span></h1>
 <script src="https://cdnjs.cloudflare.com/ajax/libs/highlight.js/11.9.0/highlight.min.js"></script>
 <script>hljs.highlightAll();</script>
 
+<script>
+	document.addEventListener("DOMContentLoaded", function () {
+		renderMathInElement(document.body, {
+			delimiters: [
+				{left: "$$", right: "$$", display: true},
+				{left: "\\(", right: "\\)", display: false},
+				{left: "$", right: "$", display: false}
+			]
+		});
+	});
+</script>
+
 
   </footer>
   </body>

public/categories/index.html

@@ -39,6 +39,18 @@ <h1>Categories</h1>
 <script src="https://cdnjs.cloudflare.com/ajax/libs/highlight.js/11.9.0/highlight.min.js"></script>
 <script>hljs.highlightAll();</script>
 
+<script>
+	document.addEventListener("DOMContentLoaded", function () {
+		renderMathInElement(document.body, {
+			delimiters: [
+				{left: "$$", right: "$$", display: true},
+				{left: "\\(", right: "\\)", display: false},
+				{left: "$", right: "$", display: false}
+			]
+		});
+	});
+</script>
+
 
   </footer>
   </body>

public/index.html

@@ -36,6 +36,20 @@ <h1 id="nic-hoffs">Nic Hoffs</h1>
 
 
 
+	<li>
+
+		<span class="date">2024/06/11</span>
+
+		<a href="/post/mini_projects/gpt2/">TinyGrad GPT2 à la Karpathy - Part 1</a>
+	</li>
+
+	<li>
+
+		<span class="date">2024/06/09</span>
+
+		<a href="/post/symbolic_regression/symbolic_distillation/">Simple Symbolic Distillation on XOR Neural Network</a>
+	</li>
+
 	<li>
 
 		<span class="date">2024/06/09</span>
@@ -68,6 +82,18 @@ <h1 id="nic-hoffs">Nic Hoffs</h1>
 <script src="https://cdnjs.cloudflare.com/ajax/libs/highlight.js/11.9.0/highlight.min.js"></script>
 <script>hljs.highlightAll();</script>
 
+<script>
+	document.addEventListener("DOMContentLoaded", function () {
+		renderMathInElement(document.body, {
+			delimiters: [
+				{left: "$$", right: "$$", display: true},
+				{left: "\\(", right: "\\)", display: false},
+				{left: "$", right: "$", display: false}
+			]
+		});
+	});
+</script>
+
 
   </footer>
   </body>

public/index.xml

@@ -6,14 +6,28 @@
     <description>Recent content in Home on Nic Hoffs Blog</description>
     <generator>Hugo</generator>
     <language>en-us</language>
-    <lastBuildDate>Sun, 09 Jun 2024 13:17:27 +0200</lastBuildDate>
+    <lastBuildDate>Tue, 11 Jun 2024 16:25:35 +0200</lastBuildDate>
     <atom:link href="http://localhost:1313/index.xml" rel="self" type="application/rss+xml" />
+    <item>
+      <title>TinyGrad GPT2 à la Karpathy - Part 1</title>
+      <link>http://localhost:1313/post/mini_projects/gpt2/</link>
+      <pubDate>Tue, 11 Jun 2024 16:25:35 +0200</pubDate>
+      <guid>http://localhost:1313/post/mini_projects/gpt2/</guid>
+      <description>Whenever Andrej Karpathy releases a new YouTube video, the only moral thing to do is drop all responsibilites and replicate it in TinyGrad. This released yesterday (June 10) and is over 4 hours long (I&amp;rsquo;m salivating), so I&amp;rsquo;ve only finished the first part. I anticipate the later parts of the video will include more Torch-specific optimizations, which will make things a bit more difficult.&#xA;Setting up inference with pre-trained weights GPT2 has four models from 124M to 1.</description>
+    </item>
+    <item>
+      <title>Simple Symbolic Distillation on XOR Neural Network</title>
+      <link>http://localhost:1313/post/symbolic_regression/symbolic_distillation/</link>
+      <pubDate>Sun, 09 Jun 2024 21:25:23 +0200</pubDate>
+      <guid>http://localhost:1313/post/symbolic_regression/symbolic_distillation/</guid>
+      <description>What is symbolic distillation? Symbolic distillation is a technique pioneered by Miles Cranmer to improve interpretability of neural networks using symbolic regression. The basic premise is to approximate various components of a neural network by applying symbolic regression to different parts. These parts can then be substituted for the original network.&#xA;We&amp;rsquo;ll try this</description>
+    </item>
     <item>
       <title>Learning PySr for Symbolic Regression</title>
       <link>http://localhost:1313/post/symbolic_regression/learning_pysr/</link>
       <pubDate>Sun, 09 Jun 2024 13:17:27 +0200</pubDate>
       <guid>http://localhost:1313/post/symbolic_regression/learning_pysr/</guid>
-      <description>What is PySr? A library for symbolic regression made by one of my favorite researchers, Miles Cranmer. The backend genetic algorithm(optimization method for finding symbolic expression) runs on Julia, so it&amp;rsquo;s really fast. It&amp;rsquo;s also extremely flexible.&#xA;Running PySr I&amp;rsquo;m going to use a Jupyter notebook for all my PySr experiments.</description>
+      <description>What is PySr? A library for symbolic regression made by one of my favorite researchers, Miles Cranmer. The backend genetic algorithm(optimization method for finding symbolic expression) runs on Julia, so it&amp;rsquo;s really fast. It&amp;rsquo;s also extremely flexible.&#xA;Installing and Running PySr I&amp;rsquo;m going to use a Jupyter notebook with a virtualenv environment for all my PySr experiments.&#xA;virtualenv venv source venv/bin/activate pip install numpy pysr Vector-&amp;gt;Scalar Fit The first import of pysr will take much longer than the next as Julia dependencies are getting set up.</description>
     </item>
     <item>
       <title>XOR</title>

public/post/grokking_squared/index.html

@@ -53,6 +53,18 @@ <h1>Grokking Squared</h1>
 <script src="https://cdnjs.cloudflare.com/ajax/libs/highlight.js/11.9.0/highlight.min.js"></script>
 <script>hljs.highlightAll();</script>
 
+<script>
+	document.addEventListener("DOMContentLoaded", function () {
+		renderMathInElement(document.body, {
+			delimiters: [
+				{left: "$$", right: "$$", display: true},
+				{left: "\\(", right: "\\)", display: false},
+				{left: "$", right: "$", display: false}
+			]
+		});
+	});
+</script>
+
 
   </footer>
   </body>

public/post/grokking_squared/part1/index.html

@@ -313,6 +313,18 @@ <h3 id="credits">Credits</h3>
 <script src="https://cdnjs.cloudflare.com/ajax/libs/highlight.js/11.9.0/highlight.min.js"></script>
 <script>hljs.highlightAll();</script>
 
+<script>
+	document.addEventListener("DOMContentLoaded", function () {
+		renderMathInElement(document.body, {
+			delimiters: [
+				{left: "$$", right: "$$", display: true},
+				{left: "\\(", right: "\\)", display: false},
+				{left: "$", right: "$", display: false}
+			]
+		});
+	});
+</script>
+
 
   </footer>
   </body>

public/post/index.html

@@ -36,17 +36,17 @@ <h1>Posts by Category</h1>
 
 	<li>
 
-		<a href="/post/symbolic_regression/">Symbolic Regression</a>
+		<a href="/post/mini_projects/">Mini-Projects</a>
 	</li>
 
 	<li>
 
-		<a href="/post/grokking_squared/">Grokking Squared</a>
+		<a href="/post/symbolic_regression/">Symbolic Regression</a>
 	</li>
 
 	<li>
 
-		<a href="/post/mini_projects/">Mini-Projects</a>
+		<a href="/post/grokking_squared/">Grokking Squared</a>
 	</li>
 
 </ul>
@@ -60,6 +60,18 @@ <h1>Posts by Category</h1>
 <script src="https://cdnjs.cloudflare.com/ajax/libs/highlight.js/11.9.0/highlight.min.js"></script>
 <script>hljs.highlightAll();</script>
 
+<script>
+	document.addEventListener("DOMContentLoaded", function () {
+		renderMathInElement(document.body, {
+			delimiters: [
+				{left: "$$", right: "$$", display: true},
+				{left: "\\(", right: "\\)", display: false},
+				{left: "$", right: "$", display: false}
+			]
+		});
+	});
+</script>
+
 
   </footer>
   </body>