Support for clean build on Ubuntu 20.04 #595

janbbeck · 2020-04-06T06:59:34Z

The next LTS release of Ubuntu is just around the corner, and I think we should work to make sure panda builds on that out of the box. Right now, I think one has to suppress Werror at the least.

I'm spinning up the latest daily of 20.04 to see what needs to be done, and I am hoping that we can discuss any issues that come up here.

First question: When running the install_ubuntu.sh script in panda/panda/scripts it downloads a complete second copy of panda. Is that intentional?

The text was updated successfully, but these errors were encountered:

AndrewFasano · 2020-04-06T15:42:53Z

I think when that script was made the ideas was that it could be downloaded and run on a Ubuntu system and just install panda. But when panda's already cloned, it definitely shouldn't clone it again.

janbbeck · 2020-04-07T03:28:04Z

I think when that script was made the ideas was that it could be downloaded and run on a Ubuntu system and just install panda. But when panda's already cloned, it definitely shouldn't clone it again.

Thanks. I really don't want to mess with that file as it seems to support several OS versions and I don't want to test against all of them. Can I just add a new script, install_prerequisites_in_Ubuntu_20.04 , or something?

janbbeck · 2020-04-07T08:16:10Z

Update:
I have panda booting a live DVD on 20.04 now with just a little trickery.

Currently, in addition to disabling all werror, this requires disabling the network plugin, as that won't build with any compiler I tried.

The network plugin fails with 4 errors:

/media/ubuntu/tmp/panda/panda/plugins/network/network.cpp: In function ‘bool init_plugin(void*)’:
/media/ubuntu/tmp/panda/panda/plugins/network/network.cpp:64:18: error: ‘wtap_dump_open_ng’ was not declared in this scope; did you mean ‘wtap_dump_open’?
   64 |     plugin_log = wtap_dump_open_ng(
      |                  ^~~~~~~~~~~~~~~~~
      |                  wtap_dump_open
/media/ubuntu/tmp/panda/panda/plugins/network/network.cpp: In function ‘void handle_packet(CPUState*, uint8_t*, size_t, uint8_t, uint64_t)’:
/media/ubuntu/tmp/panda/panda/plugins/network/network.cpp:129:24: error: aggregate ‘handle_packet(CPUState*, uint8_t*, size_t, uint8_t, uint64_t)::wtap_pkthdr header’ has incomplete type and cannot be defined
  129 |     struct wtap_pkthdr header;
      |                        ^~~~~~
/media/ubuntu/tmp/panda/panda/plugins/network/network.cpp:131:5: error: ‘wtap_phdr_init’ was not declared in this scope; did you mean ‘wtap_rec_init’?
  131 |     wtap_phdr_init(&header);
      |     ^~~~~~~~~~~~~~
      |     wtap_rec_init
/media/ubuntu/tmp/panda/panda/plugins/network/network.cpp:150:5: error: ‘wtap_phdr_cleanup’ was not declared in this scope; did you mean ‘wtap_rec_cleanup’?
  150 |     wtap_phdr_cleanup(&header);
      |     ^~~~~~~~~~~~~~~~~
      |     wtap_rec_cleanup

There are a bunch of version checking #if statements in network.cpp. Is anyone here familiar with this plugin?

There are also other non-fatal build errors, such as

/usr/bin/ld: hw/avatar/python_peripheral.o: in function `avatar_pyperipheral_realize':
/media/ubuntu/tmp/panda/hw/avatar/python_peripheral.c:99: undefined reference to `Py_Initialize'
/usr/bin/ld: /media/ubuntu/tmp/panda/hw/avatar/python_peripheral.c:100: undefined reference to `PySys_GetObject'

not sure yet what causes this.

AndrewFasano · 2020-04-08T14:30:41Z

I've had some issues building the network plugin before but I always just disabled it instead of figuring out why it was failing. The (unmerged) changes in #530 might be relevant or maybe @jmcarter9t can help out with that.

As for your issue with the python peripheral code (added in #550), you can try configuring with --disable-pyperipheral3. @mariusmue might have some more ideas there.

Thanks for working on this!

janbbeck · 2020-04-09T10:13:47Z

Ok, I didn't really want to do this, but I did fix up the install_ubuntu.sh script to check for ubuntu 19

With the changes it works on 18.04 and 19.04 and the network problem appears in 19.10. If the network plugin is disabled, it also works on 19.10.

By 'works' I mean it builds without any errors when werror is disabled. The python problem appears in 20.04 - perhaps because python-pip has been deprecated?

going to make a PR for the adjusted script....

AndrewFasano · 2020-04-09T14:48:47Z

Do you think it would make sense to merge it in with install_ubuntu.sh which currently supports 16.04 and 18.04 and just check the version and then run the appropriate setup commands?

I hadn't heard about python-pip getting deprecated before. Once 20.04 comes out in 2 weeks I'll upgrade to it and give this a try.

Ideally we could try pulling in patches from upstream (or just fix #570) to get rid of the build warnings, but we can do that after getting this merged in.

AndrewFasano · 2020-04-09T14:50:40Z

Oops, I see now I misread your message. I like what you did with the PR, thanks just merged it.

janbbeck · 2020-04-09T14:52:17Z

awesome.
Got a fix for network I think. using lots of code from #530 - Testing....

python-pip : https://bugs.launchpad.net/ubuntu/+source/python-pip/+bug/1870878

AndrewFasano · 2020-04-09T15:03:22Z

I see, they still have python3-pip. That makes sense. One issue I anticipate with that is that we still need python2 to build the version of qemu we forked from. This is fixed somewhere upstream, but I haven't put the time in to figure out if we can backport the update to python3 of if it will have to wait for #570

janbbeck · 2020-04-10T01:56:45Z

I see, they still have python3-pip. That makes sense. One issue I anticipate with that is that we still need python2 to build the version of qemu we forked from. This is fixed somewhere upstream, but I haven't put the time in to figure out if we can backport the update to python3 of if it will have to wait for #570

Maybe I am misunderstanding the situation. I thought python-pip is essentially python2-pip. Why would they make python2 available, but not python2-pip?

AndrewFasano · 2020-04-10T14:15:37Z

Yeah, it seems like a strange choice 🤷‍♂️

janbbeck · 2020-04-11T13:11:23Z

ok, I have the network plugin running on 19.10 by taking #530 and making a lot of changes.
However, there is a memory leak that I caused purposely to make it work. I'm not familiar with wireshark programming, so please check this:
This code section

#if (VERSION_MAJOR >= 2 && VERSION_MINOR >= 6 && VERSION_MICRO >= 3) || (VERSION_MAJOR>=3)
    wtap_rec rec;
    wtap_rec_init(&rec);
    rec.rec_type = REC_TYPE_PACKET;
    rec.ts.secs = now_tv.tv_sec;
    rec.ts.nsecs = now_tv.tv_usec * 1000;
    rec.rec_header.packet_header.caplen = size;
    rec.rec_header.packet_header.len = size;
    rec.rec_header.packet_header.pkt_encap = WTAP_ENCAP_ETHERNET;
    rec.opt_comment = comment_buf;
    rec.has_comment_changed = true;
    ret = wtap_dump(
        /*wtap_dumper*/ plugin_log,
        /*wtap_rec*/ &rec,
        /*buf*/ buf,
        /*err*/ &err,
        /*err_info*/ &err_info);
    //wtap_rec_cleanup(&rec);
#else

works because I commented out the wtap_rec_cleanup(&rec);
The call seems correct to me, but if I put it back in I get an error right on that call: free(): invalid pointer

When commenting it out, the network plugin I have now produces the same output on 19.10 as 18.04 and 19.04 but leaks memory because of the missing cleanup.

Ideas?

AndrewFasano · 2020-04-11T14:41:30Z

Nice! I just looked through the wireshark source and was getting convinced that the init/cleanup code you posted should work, but then I found this.

If that's correct, then it looks like you can just keep it as a variable on the stack and still use wtap_dump. Something like:

#if (VERSION_MAJOR >= 2 && VERSION_MINOR >= 6 && VERSION_MICRO >= 3) || (VERSION_MAJOR>=3)
    wtap_rec rec;
    memset(&rec, 0, sizeof rec);

    rec.rec_type = REC_TYPE_PACKET;
    rec.ts.secs = now_tv.tv_sec;
    rec.ts.nsecs = now_tv.tv_usec * 1000;
    rec.rec_header.packet_header.caplen = size;
    rec.rec_header.packet_header.len = size;
    rec.rec_header.packet_header.pkt_encap = WTAP_ENCAP_ETHERNET;
    rec.opt_comment = comment_buf;
    rec.has_comment_changed = true;
    if (!wtap_dump(
        /*wtap_dumper*/ plugin_log,
        /*wtap_rec*/ &rec,
        /*buf*/ buf,
        /*err*/ &err,
        /*err_info*/ &err_info)) {
        fprintf(stderr, "Error dumping packet\n");
    }
#else

janbbeck · 2020-04-11T14:54:44Z

Awesome find! Thanks! Testing...

janbbeck · 2020-04-11T15:40:33Z

Works on 19.10
Going to clean up the code a bit and then test for 16.04,18.04,19.04 and 19.10
If that all works, I am going to make a PR.....

AndrewFasano · 2020-04-11T15:44:34Z

Awesome! Looking forward to the PR :)

janbbeck · 2020-04-12T16:35:46Z

got install script working on 20.04. Your --disable-pyperipheral3 suggestion was one of the things necessary.
Will make a PR after I test against previous versions.

AndrewFasano · 2020-04-13T19:31:25Z

Thanks @janbbeck for #606 and #601 :)

janbbeck mentioned this issue Apr 12, 2020

fix network plugin for support of ubuntu 19.10 #605

Merged

AndrewFasano closed this as completed Apr 13, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support for clean build on Ubuntu 20.04 #595

Support for clean build on Ubuntu 20.04 #595

janbbeck commented Apr 6, 2020

AndrewFasano commented Apr 6, 2020

janbbeck commented Apr 7, 2020

janbbeck commented Apr 7, 2020

AndrewFasano commented Apr 8, 2020

janbbeck commented Apr 9, 2020

AndrewFasano commented Apr 9, 2020

AndrewFasano commented Apr 9, 2020

janbbeck commented Apr 9, 2020

AndrewFasano commented Apr 9, 2020

janbbeck commented Apr 10, 2020

AndrewFasano commented Apr 10, 2020

janbbeck commented Apr 11, 2020

AndrewFasano commented Apr 11, 2020

janbbeck commented Apr 11, 2020

janbbeck commented Apr 11, 2020

AndrewFasano commented Apr 11, 2020

janbbeck commented Apr 12, 2020

AndrewFasano commented Apr 13, 2020

Support for clean build on Ubuntu 20.04 #595

Support for clean build on Ubuntu 20.04 #595

Comments

janbbeck commented Apr 6, 2020

AndrewFasano commented Apr 6, 2020

janbbeck commented Apr 7, 2020

janbbeck commented Apr 7, 2020

AndrewFasano commented Apr 8, 2020

janbbeck commented Apr 9, 2020

AndrewFasano commented Apr 9, 2020

AndrewFasano commented Apr 9, 2020

janbbeck commented Apr 9, 2020

AndrewFasano commented Apr 9, 2020

janbbeck commented Apr 10, 2020

AndrewFasano commented Apr 10, 2020

janbbeck commented Apr 11, 2020

AndrewFasano commented Apr 11, 2020

janbbeck commented Apr 11, 2020

janbbeck commented Apr 11, 2020

AndrewFasano commented Apr 11, 2020

janbbeck commented Apr 12, 2020

AndrewFasano commented Apr 13, 2020