-
Notifications
You must be signed in to change notification settings - Fork 1.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
in_someip: SOME/IP input plugin #9570
base: master
Are you sure you want to change the base?
Conversation
Example config file is included in the PR,
|
Debug log from runtime tests is attached. |
Output from valgrind:
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for the contribution, I think we need to look at those runtime dependencies though at the very least - also do we need to include them in the target builds as well under ./packaging/distros
?
I don't think we can realistically globally enable this for every supported target. It also seems to use C++ compilation and dependencies.
dockerfiles/Dockerfile
Outdated
libboost-system-dev \ | ||
libboost-thread-dev \ | ||
libboost-filesystem-dev \ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is including the dev libraries in the runtime image - we should just have the minimal set of libraries needed for running fluent bit, not those for building it.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hi @patrick-stephens, I tried to take these out, leaving only the entries above (line 57). But when I did, I was not able to build the SOME/IP input plugin and its dependent libraries from the dev container.
Likewise when I tried to replace these in this section with the "non-dev" version (libboost-filesystem1.74.0
) the header files are not installed in the container and the build fails as well.
Please advise if there is something I am doing wrong here.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yeah I think you need to look at why, the dev libraries should only be in the build stage which is fine. At runtime we need the minimal set of dependencies required to run with, otherwise we end up opening a big bag of security worms with extra dependencies.
Header files are not needed at runtime, they should not be in the runtime image.
So I think you need to ensure the builder
stage has the dev libraries to compile everything but then the dep-extractor
stage only includes the runtime libraries.
dockerfiles/Dockerfile
Outdated
@@ -221,6 +227,9 @@ RUN echo "deb http://deb.debian.org/debian bookworm-backports main" >> /etc/apt/ | |||
libatomic1 \ | |||
libgcrypt20 \ | |||
libyaml-0-2 \ | |||
libboost-system-dev \ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
As above, do we really need the development libraries here or can we just use the runtime ones?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Same as above, I tried to change this section to the use the non-dev libs and was not able to build the SOME/IP plugin/libs from the dev container.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This section is unrelated to building so changing it here should have no impact on compilation.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes but this is not compiling anything, this is the debug image so again should just have the minimum set of packages per the deb-extractor
stage.
lib/libsomeip-c/CMakeLists.txt
Outdated
FetchContent_Declare( | ||
vsomeip3 | ||
GIT_REPOSITORY https://github.com/COVESA/vsomeip | ||
GIT_TAG 0b83e24d16e1611958194e9b727136522f46556b # 3.5.1 | ||
) | ||
FetchContent_MakeAvailable(vsomeip3) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Dependencies are generally vendored into the /lib
directory and not fetched remotely.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks. I added the source for vsomeip 3.5.1 as a sub-directory here and updated CMake file to build it before the C wrapper code.
CMakeLists.txt
Outdated
@@ -227,6 +227,7 @@ if(FLB_ALL) | |||
set(FLB_IN_NGINX_STATUS 1) | |||
set(FLB_IN_EXEC 1) | |||
set(FLB_IN_UNIX_SOCKET 1) | |||
set(FLB_IN_SOMEIP 1) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't think you can enable this globally without considering all platforms that are supported including Windows and macOS configuration.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks @patrick-stephens , will remove this here.
@@ -0,0 +1,138 @@ | |||
/* Fluent Bit |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This seems to be a C++ file, is that just a standard example and/or can it be C only? It seems to require a C++ compiler by looking at the example cmake linkage.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, this is a C++ file.
I used it to execute against fluent-bit with the SOME/IP input plugin enabled.
I will re-write it in C.
For new plugins we must ensure it builds for all targets and CI so I've triggered that. It looks like we have a lot of failures around missing dependencies and the like in various places/platforms. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Tried to address some of the received comments. Will push modifications to the PR shortly.
Hi @patrick-stephens , In order to develop this plugin, I needed an update to the Docker image used by the devcontainer (
To get an update to this image, I modified The dev-container uses the docker image to build fluent bit so that is why I need the dev versions of the boost library in the container image. It appears from the comments though, that the Docker file I updated in the PR is not the correct one for updating the docker image used by the dev container? Can you please provide some guidance on how I should proceed? Also, I am not sure about the Docker files that are under |
This plugin should only be built for linux environments (no MacOS or Windows). Since this plugin requires boost to be installed on Ubuntu do I need to update the For Centos 7, the available boost from yum repo is to old to support this plugin (vsomeip requires boost 1.66 or newer; yum repo has boost 1.53 I think). |
239b46b
to
7981fa5
Compare
@patrick-stephens I'd like to help get this over the line. Is there something you need to start for CI builds so we can get feedback and address any issues? The only changes at this point is to get the PR up-to-date with current the Thanks! |
Apologies, missed some updates but I think all my previous comments still stand. Development libraries may be needed during the builder stage yes but the runtime libraries should be much more limited. The container definition is multi-stage which a build stage and then the runtime stage so the appropriate libraries need to be used in each location and minimal ones for the production or debug stages. We need a good understanding of what libraries are required in each stage and they need to be only the minimum required. Builder stage: fluent-bit/dockerfiles/Dockerfile Lines 38 to 58 in e9a2f87
Production libraries: fluent-bit/dockerfiles/Dockerfile Lines 113 to 143 in e9a2f87
Debug libraries: fluent-bit/dockerfiles/Dockerfile Lines 214 to 223 in e9a2f87
The
Is the only alternative to wrap a C++ library? I do not believe we have any C++ libraries in there currently as the preference has always been to keep it in C. |
Thanks @patrick-stephens we'll work through these, and update. |
7981fa5
to
13d4610
Compare
The SOME/IP C library is a wrapper around vsomeip. Signed-off-by: Anthony Payne <[email protected]>
Signed-off-by: Anthony Payne <[email protected]>
All tests pass and valgrind does not show any memory errors. Signed-off-by: Anthony Payne <[email protected]>
Signed-off-by: Anthony Payne <[email protected]>
Signed-off-by: Anthony Payne <[email protected]>
Signed-off-by: Anthony Payne <[email protected]>
Added code in error legs to free decode buffer. Signed-off-by: Anthony Payne <[email protected]>
Added the vsomeip 3.5.1 source to libsomeip-c library. Updated CMake file to remove the remote fetch and instead build the vsomeip source in sub-directory. Signed-off-by: Anthony Payne <[email protected]>
Re-wrote the test SOME/IP service code in C, rather than in C++. Tested against fluent-bit binary with the in-someip plugin enabled. Signed-off-by: Anthony Payne <[email protected]>
Removed SOME/IP input plugin from the global list in the top level makefile. Signed-off-by: Anthony Payne <[email protected]>
13d4610
to
5841f67
Compare
okay, I've fixed the |
Take a look at You can build individual targets via |
Thanks for pointing out something that should have been obvious 🤦♂️ |
okay I see a test failure, but the error message is |
It looks like it hit the 60 minute timeout: https://github.com/fluent/fluent-bit/actions/runs/12817347279/job/35783077195?pr=9570 Not sure why, sometimes weirdness happens with CI so retriggering. |
It looks like the current test failure is a Kubernetes setup issue. Am I reading that correctly? is there anything else blocking this? |
Not sure as no K8S is actually used for unit tests so I'll get @pwhelan to have a look at the specific K8S event test in detail. It's on macOS which has been flaky before so likely just a test flake. |
Hi @patrick-stephens and @pwhelan just pinging to see if there is anything I can do to help move this forward. Thanks! |
Please just be patient, there are a lot of things to review so it will be in the queue. |
As far as I can tell it is a macOS flakey test. The tests on macOS can usually be more adversely affected by noisy neighbors at time. I am going to re-run it to see if that allows it to pass. I will also make a note to take a look at it when I can to fix it. IMHO, it should not block this PR, @patrick-stephens and others. |
I re-ran the job and it passed. I will make a note, as I said earlier, to see why the test flaked out. I suspect it has a similar problem to other tests I have fixed before. |
This change introduces a new input plugin, in_someip. This plugin is used to subscribe to events and capture notifications from a SOME/IP network. It also can be used in inject SOME/IP requests and capture the response to the data pipeline.
The input plugin uses an open source implementation of the SOME/IP stack, vsomeip. This library is used unchanged, but a wrapper library, libsomeip-c, exposing C-language APIs is part of the PR.
For reference to SOME/IP: protocol specification.
Testing
Before we can approve your change; please submit the following in a comment:
If this is a change to packaging of containers or native binaries then please confirm it works for all targets.
ok-package-test
label to test for all targets (requires maintainer to do).Documentation
Backporting
Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.