Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Invalid DateTime #9

Closed
clonejo opened this issue Nov 10, 2023 · 8 comments
Closed

Invalid DateTime #9

clonejo opened this issue Nov 10, 2023 · 8 comments
Assignees
@jo-m

Comments

@clonejo
Copy link
Contributor

clonejo commented Nov 10, 2023 •
edited
Loading

Hello! I'm having loads of fun setting up OnlyTrains. A (static) set of recordings is up at https://trains.shakik.de/ (maybe we should have a list of installations somewhere? I only found yours.)

As a prototype i made some videos with a zoom camera and processed them on an Arch Linux machine. I don't remember anymore how i built the trainbot binary, unfortunately.

In the frontend i am getting 'Invalid DateTime' instead of the timestamp. I have noticed that the timestamps in my db look like this:
2023-11-05 18:14:15.627118644 +0000 UTC,
whereas on trains.jo-m.ch they look like this:
2023-11-10 13:59:03.683752298+00:00.
When i manually drop the UTC from the timestamp the frontend is able to parse it. (I have fixed it manually in the DB on my website.)

I'm not a Go expert, but i may dig deeper into the Go code next.
@jo-m
Copy link
Owner

jo-m commented Nov 10, 2023

Amazing, this is the first 3rd party deployment I know about!

Indeed this problem must lie with the Go binary, which seems to write timestamps to the db in a weird format. I am a bit clueless as of why this is happening, but let's try to find out.

  1. What locale are you on, output of locale?
  2. Where did you get the binary? CI, or build yourself? You can find out all of this by running go version -m <your trainbot binary>
  3. What's your go version?
  4. I have added a new section to the readme about how to get binaries. Does it work when you grab a binary from CI?
  5. Do the tests pass locally on your machine? make test in the checked out repo.

@jo-m
Copy link
Owner

jo-m commented Nov 10, 2023

Hah, interestingly, the new test I added also seems to fail in CI: https://github.com/jo-m/trainbot/actions/runs/6827874391/job/18570856612

@jo-m jo-m self-assigned this Nov 10, 2023
@clonejo
Copy link
Contributor Author

clonejo commented Nov 10, 2023

There is no permanent setup (yet …), i'm just using prerecorded videos (which might explain why the time.Time objects are different). But i can already learn the quirks of the software and what the camera requirements are that way. Generally processing prerecorded videos was fairly easy to get going, and then building the frontend was a bit fiddly (since i used a custom path). The Dockerfile for building the frontend was quite useful.

@jo-m
Copy link
Owner

jo-m commented Nov 10, 2023

I now changed the DB code to explicitly format the timestamps on insert in 38d8fc3.

This should fix the problem, the new tests now also pass (https://github.com/jo-m/trainbot/actions/runs/6829863427).

It seems that ultimately, this was caused by 6dc37ef.

@clonejo
Copy link
Contributor Author

clonejo commented Nov 10, 2023 •
edited
Loading

I just reproduced it (on Fedora this time, based on 6f8b5eb):

$ make build_host
$ make run_videofile
$ sqlite3 data/db.sqlite3 'select * from trains;'
1|0001-01-01 00:00:01.033333333 +0000 UTC|0001-01-01 00:00:05 +0000 UTC|120|4311.15029299991|-1077.78757324998|-15.2081204445562|45.0|train_00010101_000001.033_Z.jpg|train_00010101_000001.033_Z.gif|
$ locale
LANG=en_IE.UTF-8
LC_CTYPE="en_IE.UTF-8"
LC_NUMERIC="en_IE.UTF-8"
LC_TIME="en_IE.UTF-8"
LC_COLLATE="en_IE.UTF-8"
LC_MONETARY="en_IE.UTF-8"
LC_MESSAGES="en_IE.UTF-8"
LC_PAPER="en_IE.UTF-8"
LC_NAME="en_IE.UTF-8"
LC_ADDRESS="en_IE.UTF-8"
LC_TELEPHONE="en_IE.UTF-8"
LC_MEASUREMENT="en_IE.UTF-8"
LC_IDENTIFICATION="en_IE.UTF-8"
LC_ALL=
$ go version
go version go1.20.10 linux/amd64
$ go version -m build/trainbot
build/trainbot: go1.20.10
	path	github.com/jo-m/trainbot/cmd/trainbot
	mod	github.com/jo-m/trainbot	(devel)	
	dep	git.sr.ht/~sbinet/gg	v0.5.0	h1:6V43j30HM623V329xA9Ntq+WJrMjDxRjuAB1LFWF5m8=
	dep	github.com/ajstarks/svgo	v0.0.0-20211024235047-1546f124cd8b	 h1:slYM766cy2nI3BwyRiyQj/Ud48djTMtMebDqepE95rw=
	dep	github.com/alexflint/go-arg	v1.4.3	h1:9rwwEBpMXfKQKceuZfYcwuc/7YY7tWJbFsgG5cAU/uo=
	dep	github.com/alexflint/go-scalar	v1.2.0	h1:WR7JPKkeNpnYIOfHRa7ivM21aWAdHD0gEWHCx+WQBRw=
	dep	github.com/aws/aws-sdk-go	v1.47.8	h1:VCFyO5UTREnhR0HRf9roqFfJeeRVin58zUy+pBMhwjY=
	dep	github.com/dustin/go-humanize	v1.0.1	h1:GzkhY7T5VNhEkwH0PVJgjz+fX1rhBrR7pRT3mDkpeCY=
	dep	github.com/go-fonts/liberation	v0.3.2	h1:XuwG0vGHFBPRRI8Qwbi5tIvR3cku9LUfZGq/Ar16wlQ=
	dep	github.com/go-latex/latex	v0.0.0-20231108140139-5c1ce85aa4ea	 h1:DfZQkvEbdmOe+JK2TMtBM+0I9GSdzE2y/L1/AmD8xKc=
	dep	github.com/go-pdf/fpdf	v0.9.0	h1:PPvSaUuo1iMi9KkaAn90NuKi+P4gwMedWPHhj8YlJQw=
	dep	github.com/golang/freetype	v0.0.0-20170609003504-e2365dfdc4a0	 h1:DACJavvAHhabrF08vX0COfcOBJRhZ8lUbR+ZWIs0Y5g=
	dep	github.com/gonuts/binary	v0.2.0	h1:caITwMWAoQWlL0RNvv2lTU/AHqAJlVuu6nZmNgfbKW4=
	dep	github.com/google/uuid	v1.4.0	h1:MtMxsa51/r9yyhkyLsVeVt0B+BGQZzpQiTQ4eHZ8bc4=
	dep	github.com/hashicorp/errwrap	v1.1.0	h1:OxrOeh75EUXMY8TBjag2fzXGZ40LB6IKw45YeGUDY2I=
	dep	github.com/hashicorp/go-multierror	v1.1.1	h1:H5DkEtf6CXdFp0N0Em5UCwQpXMWke8IA0+lD48awMYo=
	dep	github.com/jlaffaye/ftp	v0.2.0	h1:lXNvW7cBu7R/68bknOX3MrRIIqZ61zELs1P2RAiA3lg=
	dep	github.com/jmespath/go-jmespath	v0.4.0	h1:BEgLn5cpjn8UN1mAw4NjwDrS35OdebyEtFe+9YPoQUg=
	dep	github.com/jmoiron/sqlx	v1.3.5	h1:vFFPA71p1o5gAeqtEAwLU4dnX2napprKtHr7PYIcN3g=
	dep	github.com/mattn/go-colorable	v0.1.13	h1:fFA4WZxdEF4tXPZVKMLwD8oUnCTTo08duU7wxecdEvA=
	dep	github.com/mattn/go-isatty	v0.0.20	h1:xfD0iDuEKnDkl03q4limB+vH+GxLEtL/jb4xVJSWWEY=
	dep	github.com/mccutchen/palettor	v1.0.0	h1:YRNAzEZlRBnu8qP/9siuNTJiAj7VhL0dEQ1AQmu9jew=
	dep	github.com/nfnt/resize	v0.0.0-20180221191011-83c6a9932646	 h1:zYyBkD/k9seD2A7fsi6Oo2LfFZAehjjQMERAvZLEDnQ=
	dep	github.com/remyoudompheng/bigfft	v0.0.0-20230129092748-24d4a6f8daec	 h1:W09IVJc94icq4NjY3clb7Lk8O1qJ8BdBEF8z0ibU0rE=
	dep	github.com/rs/zerolog	v1.31.0	h1:FcTR3NnLWW+NnTwwhFWiJSZr4ECLpqCm6QsEnyvbV4A=
	dep	github.com/u2takey/ffmpeg-go	v0.5.0	h1:r7d86XuL7uLWJ5mzSeQ03uvjfIhiJYvsRAJFCW4uklU=
	dep	github.com/u2takey/go-utils	v0.3.1	h1:TaQTgmEZZeDHQFYfd+AdUT1cT4QJgJn/XVPELhHw4ys=
	dep	github.com/vladimirvivien/go4vl	v0.0.5	h1:jHuo/CZOAzYGzrSMOc7anOMNDr03uWH5c1B5kQ+Chnc=
	dep	go-hep.org/x/hep	v0.34.1	h1:C7kcqaECrra3Dx21u0rfb7F7ZMWUoMDUqlqCMPa57mE=
	dep	golang.org/x/exp	v0.0.0-20231108232855-2478ac86f678	 h1:mchzmB1XO2pMaKFRqk/+MV3mgGG96aqaPXaMifQU47w=
	dep	golang.org/x/image	v0.14.0	h1:tNgSxAFe3jC4uYqvZdTr84SZoM1KfwdC9SKIFrLjFn4=
	dep	golang.org/x/sync	v0.5.0	h1:60k92dhOjHxJkrqnwsfl8KuaHbn/5dl0lUPUklKo3qE=
	dep	golang.org/x/sys	v0.14.0	h1:Vz7Qs629MkJkGyHxUlRHizWJRG2j8fbQKjELVSNhy7Q=
	dep	golang.org/x/text	v0.14.0	h1:ScX5w1eTa3QqT8oi6+ziP7dTV1S2+ALU0bI+0zXKWiQ=
	dep	golang.org/x/tools	v0.15.0	h1:zdAyfUGbYmuVokhzVmghFl2ZJh5QhcfebBgmVPFYA+8=
	dep	gonum.org/v1/gonum	v0.14.0	h1:2NiG67LD1tEH0D7kM+ps2V+fXmsAnpUeec7n8tcr4S0=
	dep	gonum.org/v1/plot	v0.14.0	h1:+LBDVFYwFe4LHhdP8coW6296MBEY4nQ+Y4vuUpJopcE=
	dep	gopkg.in/yaml.v3	v3.0.1	h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
	dep	modernc.org/libc	v1.32.0	h1:yXatHTrACp3WaKNRCoZwUK7qj5V8ep1XyY0ka4oYcNc=
	dep	modernc.org/mathutil	v1.6.0	h1:fRe9+AmYlaej+64JsEEhoWuAYBkOtQiMEU7n/XgfYi4=
	dep	modernc.org/memory	v1.7.2	h1:Klh90S215mmH8c9gO98QxQFsY+W451E8AnzjoE2ee1E=
	dep	modernc.org/sqlite	v1.27.0	h1:MpKAHoyYB7xqcwnUwkuD+npwEa0fojF0B5QRbN+auJ8=
	build	-buildmode=exe
	build	-compiler=gc
	build	CGO_ENABLED=1
	build	CGO_CFLAGS=
	build	CGO_CPPFLAGS=
	build	CGO_CXXFLAGS=
	build	CGO_LDFLAGS=
	build	GOARCH=amd64
	build	GOOS=linux
	build	GOAMD64=v1
	build	vcs=git
	build	vcs.revision=91cc60c68ff593729baa57cc035112bcc0e6adcf
	build	vcs.time=2023-11-10T21:55:08Z
	build	vcs.modified=false

make test passes. make test_more has a bunch of failures.

@clonejo
Copy link
Contributor Author

clonejo commented Nov 10, 2023

On 38d8fc3:

$ make run_videofile
$ sqlite3 data/db.sqlite3 'select * from trains;'
1|0001-01-01 00:00:01.033333333Z|0001-01-01 00:00:05Z|120|4311.15029299991|-1077.78757324998|-15.2081204445562|45.0|train_00010101_000001.033_Z.jpg|train_00010101_000001.033_Z.gif|

@jo-m
Copy link
Owner

jo-m commented Nov 11, 2023

Ok. This is now the expected behavior, and the test I introduced should guard against such problems in the future.

You can close this after testing successfully with the web frontend. Feel free to open a new issue for setup and tuning of parameters.

@clonejo
Copy link
Contributor Author

clonejo commented Nov 11, 2023 •
edited
Loading

Generated trains entry:

61  2023-11-11 14:15:22.745762711Z       2023-11-11 14:15:24.016949152Z       76        2245.95977494172  -1743.57403608478  46.9550389497175      142.0     train_20231111_141522.745_Z.jpg  train_20231111_141522.745_Z.gif

Which is readily parsed by the frontend, so this bug is fixed \o/

Off-topic

moved to #10

@clonejo clonejo closed this as completed Nov 11, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@jo-m jo-m

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@clonejo @jo-m