-
Notifications
You must be signed in to change notification settings - Fork 42
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
New install hangs on helm install
of fission-workflows
#249
Comments
Hey @freeqaz - thanks for raising the issue! It looks like the fission workflows proxy (the thing that makes Fission Workflows look like an environment in Fission) cannot find the deployment of Fission Workflows in the default namespace. Can you check if that deployment is running, and if not, provide the logs of that pod too? |
Thanks for taking a look at this, @erwinvaneyk (and especially so quickly). I turned off
Looks like Still seeing fission-workflows be unable to get the server version.
For the logs you have requested, let me try to get you them.
That's everything I can think to share. I'm able to add a |
And here is the networking config output, just since it might help you (I'm pretty new at k8s and especially debugging, but I've done a good bit of distributed systems development professionally).
|
Some more info that comes to mind, as well.
Going to try |
Ran those steps and the
For the install steps, it looks like this:
Tried restarting minikube again and still get the
|
The issue in the router logs is definitely stemming from this line in fission... But I'm not exactly sure what the codepath is that triggers this. The executable is not correctly written to disk? |
Running these steps allowed me to get the
Source: kubernetes/minikube#1391 |
Hey @freeqaz - thanks for the truly extensive overview of what you are running into. It looks like the initial issue might have been transient: I suspect that the images were still downloading(?) As I get it, the current issue is that you get errors running the fortunewhale example, correct?
As for the Minikube-related issues, my experience with minikube is similar; it is a bit more cumbersome to work with than cloud-based Kubernetes clusters |
I threw together a script to piece together the various pieces of this install -- including adding in some sleeps to account for the delays across components. Same output (just to be extra thorough). #!/bin/bash
minikube start --vm-driver kvm2 --cpus=4 --memory 5000
minikube addons configure registry-creds
minikube addons enable registry-creds
helm init
helm repo add fission-charts https://fission.github.io/fission-charts/
helm repo update
sleep 60
helm install --wait -n fission-all --namespace fission --set serviceType=NodePort --set analytics=false fission-charts/fission-all --version 0.7.2
sleep 120
helm install --wait -n fission-workflows fission-charts/fission-workflows --version 0.6.0
sleep 120
export FISSION_ROUTER=$(minikube ip):$(kubectl -n fission get svc router -o jsonpath='{...nodePort}')
# Installation instructions
#fission env create --name binary --image fission/binary-env
#fission fn create --name whalesay --env binary --deploy ./whalesay.sh
#fission fn create --name fortune --env binary --deploy ./fortune.sh
# Readme instructions
fission env create --name binary --image fission/binary-env
fission function create --name whalesay --env binary --deploy examples/whales/whalesay.sh
fission function create --name fortune --env binary --deploy examples/whales/fortune.sh
fission function create --name fortunewhale --env workflow --src examples/whales/fortunewhale.wf.yaml
fission route create --method GET --url /fortunewhale --function fortunewhale
sleep 5
curl $FISSION_ROUTER/fortunewhale As you suspected, the core of the issue seems to be that the functions themselves cannot be called. Here are the logs from that. ~ fission function whalesay
Fatal error: No help topic for 'whalesay'
~ fission function test --name whalesay
Error calling function whalesay: 502; Please try again or fix the error: Using fetched code path: /userfunc/user
Using internal code path: /bin/userfunc
Listening on 8888 ...
~ fission function test --name fortune
Error calling function fortune: 502; Please try again or fix the error: Using fetched code path: /userfunc/user
Using internal code path: /bin/userfunc
Listening on 8888 ...
~ fission function test --name fortunewhale
Error calling function fortunewhale: 500; Please try again or fix the error: fission function error: []
Fatal error: Error querying logs: Post http://127.0.0.1:36319/proxy/influxdb?db=fissionFunctionLog¶ms=%7B%22funcuid%22%3A%225a7d1b21-3f1e-11e9-bc7a-ec2a1f6e3e02%22%2C%22time%22%3A0%7D&q=select+%2A+from+%22log%22+where+%22funcuid%22+%3D+%24funcuid+AND+%22time%22+%3E+%24time+LIMIT+1000: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
~ fission env create --name nodejs --image fission/node-env:1.0.0
environment 'nodejs' created
~ curl -LO https://raw.githubusercontent.com/fission/fission/master/examples/nodejs/hello.js
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 119 100 119 0 0 149 0 --:--:-- --:--:-- --:--:-- 149
~ fission function create --name hello --env nodejs --code hello.js
Package 'hello-js-bne3' created
function 'hello' created
~ fission function test --name hello
hello, world! |
Hmm... Further dumping the logs, I'm really not sure. ~ fission fn logs --name hello
[2019-03-05 08:30:38.502691707 +0000 UTC] 2019/03/05 08:30:38 fetcher received fetch request and started downloading: {1 {hello-js-bne3 default 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] nil [] } user [] []}
[2019-03-05 08:30:38.803463809 +0000 UTC] 2019/03/05 08:30:38 Successfully placed at /userfunc/user
[2019-03-05 08:30:38.803523459 +0000 UTC] 2019/03/05 08:30:38 Checking secrets/cfgmaps
[2019-03-05 08:30:38.803541648 +0000 UTC] 2019/03/05 08:30:38 Completed fetch request
[2019-03-05 08:30:38.803953498 +0000 UTC] 2019/03/05 08:30:38 elapsed time in fetch request = 337.553324ms
[2019-03-05 08:30:38.893893543 +0000 UTC] user code loaded in 0sec 1.32491ms
[2019-03-05 08:30:38.910765642 +0000 UTC] ::ffff:172.17.0.10 - - [05/Mar/2019:08:30:38 +0000] "POST /specialize HTTP/1.1" 202 - "-" "Go-http-client/1.1"
[2019-03-05 08:30:38.93509294 +0000 UTC] ::ffff:172.17.0.15 - - [05/Mar/2019:08:30:38 +0000] "GET / HTTP/1.1" 200 14 "-" "Go-http-client/1.1"
[2019-03-05 08:30:43.872872156 +0000 UTC] ::ffff:172.17.0.15 - - [05/Mar/2019:08:30:43 +0000] "GET / HTTP/1.1" 200 14 "-" "Go-http-client/1.1"
~ fission fn logs --name whalesay
[2019-03-05 08:28:46.877923291 +0000 UTC] 2019/03/05 08:28:46 fetcher received fetch request and started downloading: {1 {whalesay-sh-cqqb default 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] nil [] } user [] []}
[2019-03-05 08:28:47.102908067 +0000 UTC] 2019/03/05 08:28:47 Successfully placed at /userfunc/user
[2019-03-05 08:28:47.102998262 +0000 UTC] 2019/03/05 08:28:47 Checking secrets/cfgmaps
[2019-03-05 08:28:47.103082487 +0000 UTC] 2019/03/05 08:28:47 Completed fetch request
[2019-03-05 08:28:47.103102559 +0000 UTC] 2019/03/05 08:28:47 elapsed time in fetch request = 225.737203ms
~ fission fn logs --name fortune
[2019-03-05 08:12:29.102268666 +0000 UTC] 2019/03/05 08:12:29 Fetcher ready to receive requests
[2019-03-05 08:12:29.702641391 +0000 UTC] 2019/03/05 08:12:29 fetcher received fetch request and started downloading: {1 {fortune-sh-oglb default 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] nil [] } user [] []}
[2019-03-05 08:12:29.912282585 +0000 UTC] 2019/03/05 08:12:29 Successfully placed at /userfunc/user
[2019-03-05 08:12:29.912347546 +0000 UTC] 2019/03/05 08:12:29 Checking secrets/cfgmaps
[2019-03-05 08:12:29.91236786 +0000 UTC] 2019/03/05 08:12:29 Completed fetch request
[2019-03-05 08:12:29.912383891 +0000 UTC] 2019/03/05 08:12:29 elapsed time in fetch request = 268.44322ms
[2019-03-05 08:23:30.140593967 +0000 UTC] 2019/03/05 08:23:30 Received SIGTERM : Dumping stack trace
[2019-03-05 08:23:30.14067836 +0000 UTC] goroutine 20 [running]:
[2019-03-05 08:23:30.140695884 +0000 UTC] runtime/debug.Stack(0x1f75dc0, 0xc420054f90, 0x50716c)
[2019-03-05 08:23:30.140709169 +0000 UTC] /usr/local/go/src/runtime/debug/stack.go:24 +0xa7
[2019-03-05 08:23:30.140722408 +0000 UTC] runtime/debug.PrintStack()
[2019-03-05 08:23:30.140734896 +0000 UTC] /usr/local/go/src/runtime/debug/stack.go:16 +0x22
[2019-03-05 08:23:30.140747408 +0000 UTC] main.dumpStackTrace()
[2019-03-05 08:23:30.140759866 +0000 UTC] src/github.com/fission/fission/environments/fetcher/cmd/main.go:24 +0x20
[2019-03-05 08:23:30.140772498 +0000 UTC] main.main.func1(0xc4200a1500)
[2019-03-05 08:23:30.140784625 +0000 UTC] src/github.com/fission/fission/environments/fetcher/cmd/main.go:35 +0x7d
[2019-03-05 08:23:30.140797066 +0000 UTC] created by main.main
[2019-03-05 08:23:30.140809105 +0000 UTC] src/github.com/fission/fission/environments/fetcher/cmd/main.go:32 +0xcb
[2019-03-05 08:29:06.057111405 +0000 UTC] 2019/03/05 08:29:06 fetcher received fetch request and started downloading: {1 {fortune-sh-oglb default 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] nil [] } user [] []}
[2019-03-05 08:29:06.302904812 +0000 UTC] 2019/03/05 08:29:06 Successfully placed at /userfunc/user
[2019-03-05 08:29:06.302968202 +0000 UTC] 2019/03/05 08:29:06 Checking secrets/cfgmaps
[2019-03-05 08:29:06.302984489 +0000 UTC] 2019/03/05 08:29:06 Completed fetch request
[2019-03-05 08:29:06.30299752 +0000 UTC] 2019/03/05 08:29:06 elapsed time in fetch request = 247.370258ms
[2019-03-05 08:29:13.402366727 +0000 UTC] 2019/03/05 08:29:13 fetcher received fetch request and started downloading: {1 {fortune-sh-oglb default 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] nil [] } user [] []}
[2019-03-05 08:29:13.704405161 +0000 UTC] 2019/03/05 08:29:13 Successfully placed at /userfunc/user
[2019-03-05 08:29:13.704483005 +0000 UTC] 2019/03/05 08:29:13 Checking secrets/cfgmaps
[2019-03-05 08:29:13.704498816 +0000 UTC] 2019/03/05 08:29:13 Completed fetch request
[2019-03-05 08:29:13.704512016 +0000 UTC] 2019/03/05 08:29:13 elapsed time in fetch request = 301.640787ms |
Hooray! I figured it out. It looks like the core of the issue is using the newer To test this, I downloaded the fission cli version 0.7.2 here: https://github.com/fission/fission/releases/tag/0.7.2 And then I ~ ./fission-cli-linux fn delete --name whalesay
function 'whalesay' deleted
~ ./fission-cli-linux fn delete --name fortune
function 'fortune' deleted
~ ./fission-cli-linux fn create --name whalesay --env binary --deploy ./whalesay.sh
function 'whalesay' created
~ ./fission-cli-linux fn create --name fortune --env binary --deploy ./fortune.sh
function 'fortune' created
~ ./fission-cli-linux fn test --name fortune
The brain is a wonderful organ; it starts working the moment you get up
in the morning, and does not stop until you get to school.
~ ./fission-cli-linux fn test --name whalesay
_
< >
-
\
\
\
## .
## ## ## ==
## ## ## ## ## ===
/"""""""""""""""""\___/ ===
{ / ===-
\______ O __/
\ \ __/
\____\_______/
And then I tried just hitting the endpoint in place (Without modification). And it worked!
Yay! I'm glad to have this resolved. Do you think we should open an issue upstream? |
Going to test if this all works with 1.0.0 again, just to be thorough. |
Yes, I think so, it is at least good to have a record of this for other users to reference. Though I expect that the solution to that issue will be to add a warning (or even an error) when the client and server have mismatching versions (similar to how Helm does it). |
Opened #250 to follow up on this and be more specific. |
Summary
When installing using the instructions in the Readme, helm hangs forever and eventually fails.
Versions
Using both
fission
versions:And using `fission-workflows) version 0.6.0
Analysis (So Far)
It looks like (from the logs), this issue is because of both
jaeger-agent
andworkflow
throwing errors. I'm running this inminikube
so the environment shouldn't be the issue.jaeger-agent
Here are my startup logs:
For
jaeger-agent
, it seems like this issue is being tracked upstream. jaegertracing/jaeger#1395workflow
Not sure if this is related to jaeger but seems like it likely isn't (imo).
Logs:
Hopefully these issues are temporary. These notes may help somebody else though!
The text was updated successfully, but these errors were encountered: