Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

in_someip: SOME/IP input plugin #9570

Open
wants to merge 10 commits into
base: master
Choose a base branch
from

Conversation

anthonypayne-GM
Copy link

This change introduces a new input plugin, in_someip. This plugin is used to subscribe to events and capture notifications from a SOME/IP network. It also can be used in inject SOME/IP requests and capture the response to the data pipeline.

The input plugin uses an open source implementation of the SOME/IP stack, vsomeip. This library is used unchanged, but a wrapper library, libsomeip-c, exposing C-language APIs is part of the PR.

For reference to SOME/IP: protocol specification.

Testing
Before we can approve your change; please submit the following in a comment:

  • Example configuration file for the change
  • Debug log output from testing the change
  • Attached Valgrind output that shows no leaks or memory corruption was found

If this is a change to packaging of containers or native binaries then please confirm it works for all targets.

  • [N/A] Run local packaging test showing all targets (including any new ones) build.
  • [N/A] Set ok-package-test label to test for all targets (requires maintainer to do).

Documentation

  • [N/A ] Documentation required for this feature

Backporting

  • [N/A] Backport to latest stable release.

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.

@anthonypayne-GM
Copy link
Author

Example config file is included in the PR, conf/in_someip.conf:

[INPUT]
    Name someip
    Tag  in.someip

    # Events to subscribe to.
    # Each event should have form:
    #    Event <service id>,<instance id>,<event id>,<event group 1>,...
    #
    # Each event must have at least one event group
    Event 4,1,32768,1
    Event 4,1,32769,2

    # RPC to send on startup
    # Each RPC entry should have form:
    #    RPC <service id>,<instance id>,<method id>,<Request Payload>
    #
    # Request payload should be base64 encoded
    RPC   4,1,1,CgAQAw==

@anthonypayne-GM
Copy link
Author

Debug log from runtime tests is attached.
rt_test.log

@anthonypayne-GM
Copy link
Author

Output from valgrind:

[ OK ]
SUCCESS: All unit tests have passed.
==7910== 
==7910== HEAP SUMMARY:
==7910==     in use at exit: 5,741 bytes in 13 blocks
==7910==   total heap usage: 17,360 allocs, 17,347 frees, 5,723,214 bytes allocated
==7910== 
==7910== LEAK SUMMARY:
==7910==    definitely lost: 0 bytes in 0 blocks
==7910==    indirectly lost: 0 bytes in 0 blocks
==7910==      possibly lost: 0 bytes in 0 blocks
==7910==    still reachable: 5,741 bytes in 13 blocks
==7910==         suppressed: 0 bytes in 0 blocks
==7910== Rerun with --leak-check=full to see details of leaked memory
==7910== 
==7910== For lists of detected and suppressed errors, rerun with: -s
==7910== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)

Copy link
Contributor

@patrick-stephens patrick-stephens left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the contribution, I think we need to look at those runtime dependencies though at the very least - also do we need to include them in the target builds as well under ./packaging/distros?

I don't think we can realistically globally enable this for every supported target. It also seems to use C++ compilation and dependencies.

Comment on lines 147 to 149
libboost-system-dev \
libboost-thread-dev \
libboost-filesystem-dev \
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is including the dev libraries in the runtime image - we should just have the minimal set of libraries needed for running fluent bit, not those for building it.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @patrick-stephens, I tried to take these out, leaving only the entries above (line 57). But when I did, I was not able to build the SOME/IP input plugin and its dependent libraries from the dev container.

Likewise when I tried to replace these in this section with the "non-dev" version (libboost-filesystem1.74.0) the header files are not installed in the container and the build fails as well.

Please advise if there is something I am doing wrong here.

Copy link
Contributor

@patrick-stephens patrick-stephens Nov 18, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah I think you need to look at why, the dev libraries should only be in the build stage which is fine. At runtime we need the minimal set of dependencies required to run with, otherwise we end up opening a big bag of security worms with extra dependencies.

Header files are not needed at runtime, they should not be in the runtime image.

So I think you need to ensure the builder stage has the dev libraries to compile everything but then the dep-extractor stage only includes the runtime libraries.

@@ -221,6 +227,9 @@ RUN echo "deb http://deb.debian.org/debian bookworm-backports main" >> /etc/apt/
libatomic1 \
libgcrypt20 \
libyaml-0-2 \
libboost-system-dev \
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As above, do we really need the development libraries here or can we just use the runtime ones?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same as above, I tried to change this section to the use the non-dev libs and was not able to build the SOME/IP plugin/libs from the dev container.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This section is unrelated to building so changing it here should have no impact on compilation.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes but this is not compiling anything, this is the debug image so again should just have the minimum set of packages per the deb-extractor stage.

dockerfiles/Dockerfile Show resolved Hide resolved
Comment on lines 4 to 9
FetchContent_Declare(
vsomeip3
GIT_REPOSITORY https://github.com/COVESA/vsomeip
GIT_TAG 0b83e24d16e1611958194e9b727136522f46556b # 3.5.1
)
FetchContent_MakeAvailable(vsomeip3)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Dependencies are generally vendored into the /lib directory and not fetched remotely.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks. I added the source for vsomeip 3.5.1 as a sub-directory here and updated CMake file to build it before the C wrapper code.

CMakeLists.txt Outdated
@@ -227,6 +227,7 @@ if(FLB_ALL)
set(FLB_IN_NGINX_STATUS 1)
set(FLB_IN_EXEC 1)
set(FLB_IN_UNIX_SOCKET 1)
set(FLB_IN_SOMEIP 1)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think you can enable this globally without considering all platforms that are supported including Windows and macOS configuration.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @patrick-stephens , will remove this here.

@@ -0,0 +1,138 @@
/* Fluent Bit
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This seems to be a C++ file, is that just a standard example and/or can it be C only? It seems to require a C++ compiler by looking at the example cmake linkage.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, this is a C++ file.
I used it to execute against fluent-bit with the SOME/IP input plugin enabled.

I will re-write it in C.

@patrick-stephens
Copy link
Contributor

For new plugins we must ensure it builds for all targets and CI so I've triggered that. It looks like we have a lot of failures around missing dependencies and the like in various places/platforms.

Copy link
Author

@anthonypayne-GM anthonypayne-GM left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tried to address some of the received comments. Will push modifications to the PR shortly.

@anthonypayne-GM
Copy link
Author

Hi @patrick-stephens ,

In order to develop this plugin, I needed an update to the Docker image used by the devcontainer (.devcontainer/devcontainer.json) to include the boost dev libraries.

	"name": "devcontainer-fluent-bit",
	"image": "fluent/fluent-bit:latest-debug",

To get an update to this image, I modified ./dockerfiles/Dockerfile as per the PR and locally installed the image with the boost dev libraries ($ docker build -t fluent/fluent-bit:latest-debug -f dockerfiles/Dockerfile .).

The dev-container uses the docker image to build fluent bit so that is why I need the dev versions of the boost library in the container image.

It appears from the comments though, that the Docker file I updated in the PR is not the correct one for updating the docker image used by the dev container?

Can you please provide some guidance on how I should proceed?

Also, I am not sure about the Docker files that are under packaging/distros. Should those be updated with the plugin dependencies as well?

@anthonypayne-GM anthonypayne-GM changed the title SOME/IP input plugin in_someip: SOME/IP input plugin Dec 9, 2024
@anthonypayne-GM
Copy link
Author

For new plugins we must ensure it builds for all targets and CI so I've triggered that. It looks like we have a lot of failures around missing dependencies and the like in various places/platforms.

This plugin should only be built for linux environments (no MacOS or Windows). Since this plugin requires boost to be installed on Ubuntu do I need to update the .github/workflows/unit-tests.yaml file to add in the apt install of the required boost libraries?

For Centos 7, the available boost from yum repo is to old to support this plugin (vsomeip requires boost 1.66 or newer; yum repo has boost 1.53 I think).

@ammolitor ammolitor force-pushed the someip-input-plugin branch from 239b46b to 7981fa5 Compare January 15, 2025 22:43
@ammolitor
Copy link

@patrick-stephens I'd like to help get this over the line. Is there something you need to start for CI builds so we can get feedback and address any issues? The only changes at this point is to get the PR up-to-date with current the master branch.

Thanks!

@patrick-stephens
Copy link
Contributor

patrick-stephens commented Jan 16, 2025

@patrick-stephens I'd like to help get this over the line. Is there something you need to start for CI builds so we can get feedback and address any issues? The only changes at this point is to get the PR up-to-date with current the master branch.

Thanks!

Apologies, missed some updates but I think all my previous comments still stand.
We cannot be adding -dev libraries to runtime images and we need to minimise any dependencies added to only those required.

Development libraries may be needed during the builder stage yes but the runtime libraries should be much more limited. The container definition is multi-stage which a build stage and then the runtime stage so the appropriate libraries need to be used in each location and minimal ones for the production or debug stages. We need a good understanding of what libraries are required in each stage and they need to be only the minimum required.

Builder stage:

RUN echo "deb http://deb.debian.org/debian bookworm-backports main" >> /etc/apt/sources.list && \
apt-get update && \
apt-get install -y --no-install-recommends \
build-essential \
curl \
ca-certificates \
cmake \
git \
make \
tar \
libssl-dev \
libsasl2-dev \
pkg-config \
libsystemd-dev/bookworm-backports \
zlib1g-dev \
libpq-dev \
postgresql-server-dev-all \
flex \
bison \
libyaml-dev \
&& apt-get clean \

Production libraries:
libssl3 \
libsasl2-2 \
pkg-config \
libpq5 \
libsystemd0/bookworm-backports \
zlib1g \
ca-certificates \
libatomic1 \
libgcrypt20 \
libzstd1 \
liblz4-1 \
libgssapi-krb5-2 \
libldap-2.5 \
libgpg-error0 \
libkrb5-3 \
libk5crypto3 \
libcom-err2 \
libkrb5support0 \
libgnutls30 \
libkeyutils1 \
libp11-kit0 \
libidn2-0 \
libunistring2 \
libtasn1-6 \
libnettle8 \
libhogweed6 \
libgmp10 \
libffi8 \
liblzma5 \
libyaml-0-2 \
libcap2 \

Debug libraries:
libssl3 \
libsasl2-2 \
pkg-config \
libpq5 \
libsystemd0/bookworm-backports \
zlib1g \
ca-certificates \
libatomic1 \
libgcrypt20 \
libyaml-0-2 \

The packaging/distros container definitions need updating to allow this plugin to be built for relevant targets or disabled appropriately if not possible - similarly for the macOS and Windows builds.

Is the only alternative to wrap a C++ library? I do not believe we have any C++ libraries in there currently as the preference has always been to keep it in C.

@ammolitor
Copy link

Thanks @patrick-stephens we'll work through these, and update.

@ammolitor ammolitor force-pushed the someip-input-plugin branch from 7981fa5 to 13d4610 Compare January 16, 2025 17:15
The SOME/IP C library is a wrapper around vsomeip.

Signed-off-by: Anthony Payne <[email protected]>
All tests pass and valgrind does not show any memory errors.

Signed-off-by: Anthony Payne <[email protected]>
Added code in error legs to free decode buffer.

Signed-off-by: Anthony Payne <[email protected]>
Added the vsomeip 3.5.1 source to libsomeip-c library.
Updated CMake file to remove the remote fetch and instead
build the vsomeip source in sub-directory.

Signed-off-by: Anthony Payne <[email protected]>
Re-wrote the test SOME/IP service code in C, rather
than in C++. Tested against fluent-bit binary with
the in-someip plugin enabled.

Signed-off-by: Anthony Payne <[email protected]>
Removed SOME/IP input plugin from the global list in
the top level makefile.

Signed-off-by: Anthony Payne <[email protected]>
@ammolitor ammolitor force-pushed the someip-input-plugin branch from 13d4610 to 5841f67 Compare January 16, 2025 20:48
@ammolitor
Copy link

okay, I've fixed the *-dev packages in dockerfiles/Dockerfile
I've updated cmake/windows-setup.cmake to disable FLB_IN_SOMEIP
I don't see places where changes are need in cmake/macos-setup.cmake or dockerfiles/Dockerfile.windows (I'm happy to be educated though).
Is there a script or otherwise that builds the images in packaging/distros? I'd like to test those.

@patrick-stephens
Copy link
Contributor

patrick-stephens commented Jan 17, 2025

Take a look at ./packaging/local-build-all.sh.

You can build individual targets via ./packaging/build.sh -d <distro>, e.g. ./packaging/build.sh -d centos/7. I can label the PR as well to build all targets as it would do for a release.

@ammolitor
Copy link

Take a look at ./packaging/local-build-all.sh.

You can build individual targets via ./packaging/build.sh -d <distro>, e.g. ./packaging/build.sh -d centos/7. I can label the PR as well to build all targets as it would do for a release.

Thanks for pointing out something that should have been obvious 🤦‍♂️

@ammolitor
Copy link

okay I see a test failure, but the error message is Error: The operation was canceled. After digging a bit it i think that flb-rt-core_chunk_trace might have taken too long. Is that possible? and is it related to our changes?

@patrick-stephens
Copy link
Contributor

okay I see a test failure, but the error message is Error: The operation was canceled. After digging a bit it i think that flb-rt-core_chunk_trace might have taken too long. Is that possible? and is it related to our changes?

It looks like it hit the 60 minute timeout: https://github.com/fluent/fluent-bit/actions/runs/12817347279/job/35783077195?pr=9570

Not sure why, sometimes weirdness happens with CI so retriggering.

@ammolitor
Copy link

It looks like the current test failure is a Kubernetes setup issue. Am I reading that correctly? is there anything else blocking this?

@patrick-stephens
Copy link
Contributor

Not sure as no K8S is actually used for unit tests so I'll get @pwhelan to have a look at the specific K8S event test in detail. It's on macOS which has been flaky before so likely just a test flake.

@ammolitor
Copy link

Hi @patrick-stephens and @pwhelan just pinging to see if there is anything I can do to help move this forward. Thanks!

@patrick-stephens
Copy link
Contributor

Hi @patrick-stephens and @pwhelan just pinging to see if there is anything I can do to help move this forward. Thanks!

Please just be patient, there are a lot of things to review so it will be in the queue.

@pwhelan
Copy link
Contributor

pwhelan commented Jan 23, 2025

Not sure as no K8S is actually used for unit tests so I'll get @pwhelan to have a look at the specific K8S event test in detail. It's on macOS which has been flaky before so likely just a test flake.

As far as I can tell it is a macOS flakey test. The tests on macOS can usually be more adversely affected by noisy neighbors at time. I am going to re-run it to see if that allows it to pass. I will also make a note to take a look at it when I can to fix it.

IMHO, it should not block this PR, @patrick-stephens and others.

@pwhelan
Copy link
Contributor

pwhelan commented Jan 23, 2025

I re-ran the job and it passed. I will make a note, as I said earlier, to see why the test flaked out. I suspect it has a similar problem to other tests I have fixed before.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
docs-required ok-package-test Run PR packaging tests
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants