Generated wasm too large #534

iarkaroy · 2021-05-16T06:54:50Z

Thank you for your contribution. This looks pretty awesome to work with. I just started playing around with it.

I started with minimal example:

package main

import (
	"log"
	"net/http"

	"github.com/maxence-charriere/go-app/v8/pkg/app"
)

type home struct {
	app.Compo
}

func (h *home) Render() app.UI {
	return app.H1().Text("Hello World!")
}

func main() {
	app.Route("/", &home{})
	app.RunWhenOnBrowser()
	http.Handle("/", &app.Handler{
		Name:        "Hello",
		Description: "An Hello World! example",
	})

	if err := http.ListenAndServe(":8000", nil); err != nil {
		log.Fatal(err)
	}
}

Generated wasm with:

GOARCH=wasm GOOS=js go build -o web/app.wasm

But the generated wasm is a whooping 11.5 MB. Any suggestion to reduce the file size?

Go version: 1.16.4

The text was updated successfully, but these errors were encountered:

maxence-charriere · 2021-05-16T07:01:16Z

You can serve it gzipped.

iarkaroy · 2021-05-16T09:03:07Z

Yes. Still it will be around 3 MB. I will experiment with tinygo to see if that helps.

maxence-charriere · 2021-05-16T10:40:20Z

If you dont use http you can separate client code in another package. It the http package that add a lot.

iarkaroy · 2021-05-16T14:56:10Z

Thanks for the suggestion. I have been trying to build with tinygo but it seems to have some issues. I have added the issue tinygo-org/tinygo#1886.

maxence-charriere · 2021-05-18T17:36:27Z

Tinygo is not supported by this package for now.

thepudds · 2021-09-17T16:42:09Z

FWIW, there has been some progress on the tinygo side of this, e.g.:

tinygo-org/tinygo#1886 (comment)

ignishub · 2021-09-27T23:12:11Z

Just make two files that contains "serve" function. In GOOS=js target this function will be empty.

Without it : 12 Mb
With it(without http package): 5 Mb

Then use brotli to reduce it again:

Result: 985 Kb

oderwat · 2021-10-06T15:46:08Z

I am experimenting with go-app right now. To get the sizes down I made two targets for the HTTP server and one is a stub. In addition, I experimenting in using the chi router and with that, I have a compression middleware for the wasm. This brings the size to 1.3 MB (from >16 MB) and lets me add the backend code using chi. I also experiment with "live reloading" (through a javascript HEAD checking and using reflex for recompilation). This is a bit rough about the edges but I am actually pretty happy with how everything works so far. You may want to look at https://github.com/oderwat/go-guess-the-number-app (this is not meant to be public for real, but I thought I want to share this as example and maybe get comments)

mar1n3r0 · 2022-01-10T12:13:55Z

@oderwat You are right. We might end up needing to separate the http package and still use Tinygo to achieve js world bundle sizes. Thanks for the example. Splitting into two files doesn't seem to reduce the size though. Can you elaborate on your results?

oderwat · 2022-01-10T12:23:56Z

13.3 MB compressed? My testing (a lot of stuff) app has currently 13.8 MB which ends up as 3.2 MB with on-the-fly compression middleware. I also use my #631 PR to show people a loading indicator, which makes its loading time "more accepted" as it shows the progress to the user.

But I hope that we get TinyGo to work. Sadly I just came from my vacation and my day job has different priorities at the moment.

mar1n3r0 · 2022-01-10T12:25:06Z

Hey not a problem. I don't see any difference in the wasm size after splitting into two files actually it's 13.3 MB not compressed and around 3 MB when compressed so no improvement over the original inclusion of the http package. I think when building the wasm the http package is still included.

In the end I think we will need both this technique and Tinygo for best results.

oderwat · 2022-01-10T12:33:31Z

Well, I use HTTP clients in the frontend anyways. But there are other packages that could be used (maybe https://github.com/golangee/wasm-net). But I did not have time to try it yet.

oderwat · 2022-01-10T12:37:08Z

That is as a wasm server, not an HTTP client library to use in the frontend? The Server is fine in go-app as it is just the server binary where size matters less.

oderwat · 2022-01-10T12:40:05Z

Sure. But when you access APIs from the frontend you need to use one or more HTTP clients in the code.

mar1n3r0 · 2022-01-10T12:40:31Z

Yes, my misconception. So we can't save size on that front in an easy straight forward way. It remains a mystery though why the tinygo produced files are still big. Also the http client in react is 2.5 KB in Go it's 7 MB, quite a big difference. Can we find an alternative tiny http client let's say developed for embedded devices?

oderwat · 2022-01-10T12:47:01Z

Never mind: https://github.com/golangee/wasm-net is not what I meant. This also includes net/http. I found something else but can not dig up the link right now. Basically, it works by calling the functions in the browser as js does it.

mar1n3r0 · 2022-01-10T12:48:50Z

A gRPC client? We already have an example of using that in go-app: #447
Even if we use that we still serve requests from net/http and need a lightweight alternative to it.

oderwat · 2022-01-10T12:58:43Z

It is about "net/http" in the frontend part and you can not just use gRPC to access an HTTP API. I am not talking only about the client/server communication of the app itself.

Even if we use that we still serve requests from net/http and need a lightweight alternative to it.

No, we don't? It does not change the WASM size at all that you use net/http for the server-side.

mar1n3r0 · 2022-01-10T13:00:23Z

It is about "net/http" in the frontend part and you can not just use gRPC to access an HTTP API. I am not talking only about the client/server communication of the app itself.

Even if we use that we still serve requests from net/http and need a lightweight alternative to it.

No, we don't? It does not change the WASM size at all that you use net/http for the server-side.

I meant trying to use gRPC API instead of HTTP API in order to avoid including the net/http package. But we still need something to serve the wasm file tothe browser.

I think it's precisely the net/http package that adds size, actually more than half of it.

oderwat · 2022-01-10T13:00:36Z

And btw. gRPC uses HTTP/2. I would rather use something like https://github.com/twitchtv/twirp

mar1n3r0 · 2022-01-10T13:01:29Z

And btw. gRPC uses HTTP/2. I would rather use something like https://github.com/twitchtv/twirp

Yeah I am not speculating about that, anything as an alternative to a smaller final size.

mar1n3r0 · 2022-01-10T13:01:58Z

Without it : 12 Mb
With it(without http package): 5 Mb

It was stated previously that the net/http package adds 7 MB.

oderwat · 2022-01-10T13:02:11Z

You can not use gRPC for an HTTP API. We need to access a lot of different APIs and not all of them are made by us. The key to a smaller size will be TinyGo until the Go compiler gets optimized for WASM.

mar1n3r0 · 2022-01-10T13:03:04Z

You can not use gRPC for an HTTP API. We need to access a lot of different APIs and not all of them are made by us. The key to a smaller size will be TinyGo until the Go compiler gets optimized for WASM.

Please read the pull request discussion. Tinygo successfully compiled but the final size was 6.3 MB. So Tinygo doesn't solve the problem with size. It's the net/http package that mainly contributes to the big wasm size.

oderwat · 2022-01-10T13:04:32Z

It was stated previously that the net/http package adds 7 MB.

And I say you can probably use the browsers XMLHttpRequest() for this and don't need net/http in the frontend. But I don't have time to try it.

mar1n3r0 · 2022-01-10T13:05:20Z

It was stated previously that the net/http package adds 7 MB.

And I say you can probably use the browsers XMLHttpRequest() for this and don't need net/http in the frontend. But I don't have time to try it.

How are we going to serve the wasm file to the browser without http.ListenAndServe?

mar1n3r0 · 2022-01-10T13:07:20Z

If you dont use http you can separate client code in another package. It the http package that add a lot.

I hope @maxence-charriere can clarify the basics here.

oderwat · 2022-01-10T13:10:43Z

@mar1n3r0 you simply build two times. One for the WASM not containing the server and one for the server itself (which simply uses net/http).

mar1n3r0 · 2022-01-10T13:12:17Z

@mar1n3r0 you simply build two times. One for the WASM not containing the server and one for the server itself (which simply uses net/http).

They are the same package how do you split them? In your example the final size remains unchanged from before the split.

oderwat · 2022-01-10T13:15:08Z

main.go

func main() {
	// Frontend routing
	app.RouteWithRegexp("/.*", &appControl{})

	// this concludes the part which goes into the front-end
	app.RunWhenOnBrowser()

	// this will depend on the target (wasm or not wasm) and
	// it starts the servers if it is not the wasm target.
	AppServer()
}

wasmstub.go

// Our empty version of the httpServer for usage with the wasm target
// this way we will not include any of the related code
//go:build wasm

package main

func AppServer() {
}

server.go

// Our empty version of the httpServer for usage with the wasm target
// this way we will not include any of the related code
//go:build !wasm

package main

import (
....
)

func AppServer() {
	// the actuall server code
......

oderwat · 2022-01-10T14:46:40Z

If that uses my "template" replacement it may need {{.Env }} changed to {{.Env}} because I just replace later one.

oderwat · 2022-01-10T14:56:29Z

SetEnv has nothing to do with WASM but it has to be implemented in TinyGo in order to be used.

But I only want to use TinyGo for the WASM part and that hardly ever needs to use "SetEnv" as (afaik) there is nothing like "env variables" in WASM. I didn't had time what this "SetEnv" actually is used for.

The error you get is also template-related.

mar1n3r0 · 2022-01-10T15:06:21Z

This is the original go-app code. Haven't touched anything there.

oderwat · 2022-01-10T15:07:56Z

How can you compile using TinyGo without a replacement for text/template? ... if you removed that part it will surely not work :)

You need #680 or something similar.

mar1n3r0 · 2022-01-10T15:09:05Z

I used your hard-coding from the PR and also commented this:

`func (h *Handler) makeAppJS() []byte {
if h.Env == nil {
h.Env = make(map[string]string)
}
// internalURLs, _ := json.Marshal(h.InternalURLs)
// h.Env["GOAPP_INTERNAL_URLS"] = string(internalURLs)
// h.Env["GOAPP_VERSION"] = h.Version
// h.Env["GOAPP_STATIC_RESOURCES_URL"] = h.Resources.Static()
// h.Env["GOAPP_ROOT_PREFIX"] = h.Resources.Package()

// for k, v := range h.Env {
// 	if err := os.Setenv(k, v); err != nil {
// 		Log(errors.New("setting app env variable failed").
// 			Tag("name", k).
// 			Tag("value", v).
// 			Wrap(err))
// 	}
// }

env, err := json.Marshal(h.Env)
if err != nil {
	panic(errors.New("encoding pwa env failed").
		Tag("env", h.Env).
		Wrap(err),
	)
}

s := appJS
s = strings.ReplaceAll(s, "{{.Env}}", btos(env))
s = strings.ReplaceAll(s, "{{.Wasm}}", h.Resources.AppWASM())
s = strings.ReplaceAll(s, "{{.WorkerJS}}", h.resolvePackagePath("/app-worker.js"))
return []byte(s)

}`

So yeah it's normal that it's not working since the env variables are set here by SetEnv.

oderwat · 2022-01-10T15:13:14Z

No there is a {{.Env }} in the output, you show and that is the "syntax error" and related to the templates not correctly translated.

s = strings.ReplaceAll(s, "{{.Env}}", btos(env))

Looks to me as if generate did not run or that somewhere a {{.Env }} is hiding or reintrocuded.

mar1n3r0 · 2022-01-10T15:15:36Z

Yeah because {{.Env }} was not replaced by the env variable which is not set. If i remove the comments it starts working so it's certainly there.

oderwat · 2022-01-10T15:15:54Z

And please don't comment out:

// internalURLs, _ := json.Marshal(h.InternalURLs)
// h.Env["GOAPP_INTERNAL_URLS"] = string(internalURLs)
// h.Env["GOAPP_VERSION"] = h.Version
// h.Env["GOAPP_STATIC_RESOURCES_URL"] = h.Resources.Static()
// h.Env["GOAPP_ROOT_PREFIX"] = h.Resources.Package()

It is just os.SetEnv() which does not work (and looks useless to me. I don't understand what this is good for when the target is WASM. But maybe it is compiled twice. I don't know (yet)

mar1n3r0 · 2022-01-10T15:16:43Z

// Getenv retrieves the value of the environment variable named by the key. It
// returns the value, which will be empty if the variable is not present.
func Getenv(k string) string {
	if IsServer {
		return os.Getenv(k)
	}

	env := Window().Call("goappGetenv", k)
	if !env.Truthy() {
		return ""
	}
	return env.String()
}

oderwat · 2022-01-10T15:17:11Z

{{.Env}} is not replaced by an Env variable but with the value of the map which is defined about.

mar1n3r0 · 2022-01-10T15:18:12Z

That's that for now until SetEnv is implemented in TinyGo.

oderwat · 2022-01-10T15:18:22Z

The IsServer part should never run when wasm is the target so maybe put this in an extra file similar to the server code itself, so it will not include the call at all.

oderwat · 2022-01-10T16:11:05Z

BTW: My main testing app still does not compile with TinyGo (after getting rid of the os.SetEnv()) because of /libexec/src/encoding/xml/typeinfo.go:114:29: f.Index undefined (type *reflect.StructField has no field or method Index) so yeah, more to do. But I "solved" the problem with the {{.Env }} in my latest PR #680 I believe. Maybe I am going to use a much smaller example later. But using TinyGo can not mean that we can't use a majority of modules.

mar1n3r0 · 2022-01-10T16:35:36Z

{{.Env }} is unchanged in my test so not taken from your template. I have no idea where the xml error comes from maybe send a full trace of the running code or a repo with it.

mlctrez · 2022-01-10T19:10:14Z

I found something else but can not dig up the link right now.

Here's a lighter weight http client for wasm : https://github.com/marwan-at-work/wasm-fetch

oderwat · 2022-01-10T19:53:56Z

@mlctrez yep this is exactly the one I had in mind

maxence-charriere · 2022-01-10T19:54:35Z

A common pattern is to have a server configurable from environment variables.

Things like api endpoint can be configured and be automatically relayed to a client.

maxence-charriere · 2022-01-10T19:59:32Z

I also made a try for tinygo support. Here is the code: #682

I was able to compile but look like there is still some issues to figure out.

note that this is pure hacking and is far from being stable.

mar1n3r0 · 2022-01-10T20:01:06Z

@maxence-charriere Please keep the Tinygo issue open as it is quite close to being completed.

maxence-charriere · 2022-01-10T20:02:47Z

@mar1n3r0 i don’t give up. I got that a lot of you folks want this to happen.

oderwat · 2022-01-10T20:16:45Z

If the compile times are longer but the result is (much) smaller it may just be good enough to use it for deployment. I also experienced quite long compile times while working on WASM "plugins" with TinyGo (and I am not sure if I wanna trust it to create stable code).

mar1n3r0 · 2022-01-10T20:23:06Z

@oderwat same here but I don't consider 1 min compile time to be a show stopper if the benefit is 50% + reduced binary size. To be honest I also expected 10 times reduction so a bit disappointed.

suntong · 2023-08-11T03:43:49Z

Some important links / milestones, to save time the next person trying to read through the whole thread --

To get the sizes down I made two targets for the HTTP server and one is a stub. In addition, I experimenting in using the chi router and with that, I have a compression middleware for the wasm. This brings the size to 1.3 MB (from >16 MB) and lets me add the backend code using chi. I also experiment with "live reloading" (through a javascript HEAD checking and using reflex for recompilation). This is a bit rough about the edges but I am actually pretty happy with how everything works so far. You may want to look at https://github.com/oderwat/go-guess-the-number-app (this is not meant to be public for real, but I thought I want to share this as example and maybe get comments)

Here are the results and the example code based on the hello example: https://github.com/mar1n3r0/hello-wasm

Original size - 12 MB
After multi-stage and omitting net/http in wasm

Go compiler - 5 MB

TinyGo compiler - 1.8 MB

TinyGo + gzipped - 574KB

tinygo: https://tinygo.org/docs/guides/webassembly/
lighter weight http client for wasm : https://github.com/marwan-at-work/wasm-fetch, to replace net/http which would otherwise add ~4 MBs to your wasm binary.

I also made a try for tinygo support. Here is the code: #682

I was able to compile but look like there is still some issues to figure out.

note that this is pure hacking and is far from being stable.

We might not be able to achieve the best in one shot, IMHO, even things half done is better than the current situation, as my Original app.wasm size is 14 MB, gziped size is 3.2M.

g4bwy · 2023-08-25T08:54:24Z

Hi @mar1n3r0,

I'm reviving this thread a little bit, since you mention tinygo is working, but I'm still having a link-time issue with time.modTimer using tinygo 0.28.1:

tinygo:wasm-ld: error: /tmp/tinygo3209131542/main.o: undefined symbol: time.modTimer
failed to run tool: wasm-ld
error: failed to link /tmp/tinygo3209131542/main: exit status 1

how exactly did you manage to get past this one ?

cheers,

mar1n3r0 · 2023-08-25T08:59:22Z

Hi @mar1n3r0,

I'm reviving this thread a little bit, since you mention tinygo is working, but I'm still having a link-time issue with time.modTimer using tinygo 0.28.1:
tinygo:wasm-ld: error: /tmp/tinygo3209131542/main.o: undefined symbol: time.modTimer
failed to run tool: wasm-ld
error: failed to link /tmp/tinygo3209131542/main: exit status 1
how exactly did you manage to get past this one ?

cheers,

Sadly not there yet, see: #682

g4bwy · 2023-08-25T09:45:04Z

well that's too bad 😢

in the meantime, did you try brotli or zstd compression instead of gzip ?

mar1n3r0 · 2023-08-25T10:00:35Z

Nope, gzip only so far.

maxence-charriere · 2024-05-20T07:35:13Z

That cannot really be handled on go-app side.
A good practice is to separate the client code (wasm) and the server (part that use app.Handler).

mar1n3r0 mentioned this issue Jan 10, 2022

[WIT] Tinygo support #682

Closed

maxence-charriere closed this as completed May 20, 2024

Generated wasm too large #534

Generated wasm too large #534

Comments

iarkaroy commented May 16, 2021

maxence-charriere commented May 16, 2021

iarkaroy commented May 16, 2021

maxence-charriere commented May 16, 2021

iarkaroy commented May 16, 2021

maxence-charriere commented May 18, 2021

thepudds commented Sep 17, 2021

ignishub commented Sep 27, 2021

oderwat commented Oct 6, 2021

mar1n3r0 commented Jan 10, 2022 • edited Loading

oderwat commented Jan 10, 2022

mar1n3r0 commented Jan 10, 2022 • edited Loading

oderwat commented Jan 10, 2022

oderwat commented Jan 10, 2022

oderwat commented Jan 10, 2022

mar1n3r0 commented Jan 10, 2022 • edited Loading

oderwat commented Jan 10, 2022

mar1n3r0 commented Jan 10, 2022 • edited Loading

oderwat commented Jan 10, 2022 • edited Loading

mar1n3r0 commented Jan 10, 2022 • edited Loading

oderwat commented Jan 10, 2022

mar1n3r0 commented Jan 10, 2022

mar1n3r0 commented Jan 10, 2022

oderwat commented Jan 10, 2022

mar1n3r0 commented Jan 10, 2022 • edited Loading

oderwat commented Jan 10, 2022

mar1n3r0 commented Jan 10, 2022

mar1n3r0 commented Jan 10, 2022 • edited Loading

oderwat commented Jan 10, 2022

mar1n3r0 commented Jan 10, 2022

oderwat commented Jan 10, 2022

oderwat commented Jan 10, 2022

oderwat commented Jan 10, 2022 • edited Loading

mar1n3r0 commented Jan 10, 2022

oderwat commented Jan 10, 2022 • edited Loading

mar1n3r0 commented Jan 10, 2022

oderwat commented Jan 10, 2022

mar1n3r0 commented Jan 10, 2022

oderwat commented Jan 10, 2022

mar1n3r0 commented Jan 10, 2022

oderwat commented Jan 10, 2022

mar1n3r0 commented Jan 10, 2022

oderwat commented Jan 10, 2022

oderwat commented Jan 10, 2022 • edited Loading

mar1n3r0 commented Jan 10, 2022

mlctrez commented Jan 10, 2022

oderwat commented Jan 10, 2022

maxence-charriere commented Jan 10, 2022

maxence-charriere commented Jan 10, 2022

mar1n3r0 commented Jan 10, 2022

maxence-charriere commented Jan 10, 2022

oderwat commented Jan 10, 2022

mar1n3r0 commented Jan 10, 2022

suntong commented Aug 11, 2023 • edited Loading

g4bwy commented Aug 25, 2023 • edited Loading

mar1n3r0 commented Aug 25, 2023 • edited Loading

g4bwy commented Aug 25, 2023

mar1n3r0 commented Aug 25, 2023

maxence-charriere commented May 20, 2024

mar1n3r0 commented Jan 10, 2022 •

edited

Loading

mar1n3r0 commented Jan 10, 2022 •

edited

Loading

mar1n3r0 commented Jan 10, 2022 •

edited

Loading

mar1n3r0 commented Jan 10, 2022 •

edited

Loading

oderwat commented Jan 10, 2022 •

edited

Loading

mar1n3r0 commented Jan 10, 2022 •

edited

Loading

mar1n3r0 commented Jan 10, 2022 •

edited

Loading

mar1n3r0 commented Jan 10, 2022 •

edited

Loading

oderwat commented Jan 10, 2022 •

edited

Loading

oderwat commented Jan 10, 2022 •

edited

Loading

oderwat commented Jan 10, 2022 •

edited

Loading

suntong commented Aug 11, 2023 •

edited

Loading

g4bwy commented Aug 25, 2023 •

edited

Loading

mar1n3r0 commented Aug 25, 2023 •

edited

Loading