Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

HTTP2 issues with curl download in oscap_acquire #2193

Open
Jiri-Stary opened this issue Jan 7, 2025 · 15 comments
Open

HTTP2 issues with curl download in oscap_acquire #2193

Jiri-Stary opened this issue Jan 7, 2025 · 15 comments
Labels
bug solution

Comments

@Jiri-Stary
Copy link

Jiri-Stary commented Jan 7, 2025 •
edited by evgenyz
Loading

Description of Problem:

Fails to download files via curl if http2 is used. Consider providing way to disable HTTP2 - it was not working with latest version of curl, atleast on alpine linux

OpenSCAP Error: Download failed: Stream error in the HTTP/2 framing layer [/home/buildozer/aports/community/openscap/src/openscap-1.3.10/src/common/oscap_acquire.c:405]

OpenSCAP Version:

Operating System & Version:

Alpine 3.20 docker image
openscap 1.3.10

Steps to Reproduce:

  1. run xccdf eval on debian 12 - e.g.
    oscap-docker image debian:12 xccdf eval --fetch-remote-resources --profile xccdf_org.ssgproject.content_profile_cis_level2_server --results /report/ssg-results.xml --report /report/ssg-results.html /ssg/scap-security-guide-0.1.75/ssg-debian12-ds.xml;
  2. tries to fetch oval files and fails on HTTP2 frame - likely bug in curl

Actual Results:

Expected Results:

There is no way to force HTTP1.1 - i tried using .curlrc file but it did not help

Additional Information / Debugging Steps:

To quote from https://www.rfc-editor.org/rfc/rfc9113.html#name-connection-specific-header- headers like 'TE' are not allowed in HTTP/2 and lead to a stream reset. Apache is correct in its behaviour.

The question is now: is the libcurl application wrong in adding the header or should curl, when selecting HTTP/2, automatically strip the header from the request.
@Jiri-Stary
Copy link
Author

running

curl --output abc.xml.bz2 https://www.debian.org/security/oval/oval-definitions-bookworm.xml.bz2 -v

on the docker image works, not sure what is the issue with openscap calling the curl

@evgenyz
Copy link
Contributor

evgenyz commented Jan 7, 2025

Okay, so it seems we still have a problem: #2137

@evgenyz
Copy link
Contributor

evgenyz commented Jan 7, 2025

Can you please do --verbose DEVEL as well for the sake of completeness.

@evgenyz
Copy link
Contributor

evgenyz commented Jan 7, 2025

Also, you can use env. vars for future debugging: https://curl.se/libcurl/c/libcurl-env.html.

@Jiri-Stary
Copy link
Author

I will try to run verbose logs, in the meantime the seems to be the issue, in curl curl/curl#10634

and according to the release notes it is fixed in version 8.0.0 and newer

https://curl.se/changes.html#8_0_0

@Jiri-Stary
Copy link
Author

ok @evgenyz i executed

/usr/bin/oscap info --verbose DEVEL --fetch-remote-resources --profiles /ssg/scap-security-guide-0.1.75/ssg-debian12-ds.xml;

inside alpine 3.20 docker image - as before version 1.3.10 of openscap

I: oscap: Using environment variables: [oscap(17):oscap(7fd8c7d58390):debug.c:316:oscap_print_env_vars]
I: oscap: OSCAP_CHECK_ENGINE_PLUGIN_DIR='' [oscap(17):oscap(7fd8c7d58390):debug.c:319:oscap_print_env_vars]
I: oscap: OSCAP_CONTAINER_VARS='' [oscap(17):oscap(7fd8c7d58390):debug.c:319:oscap_print_env_vars]
I: oscap: OSCAP_EVALUATION_TARGET='' [oscap(17):oscap(7fd8c7d58390):debug.c:319:oscap_print_env_vars]
I: oscap: OSCAP_FULL_VALIDATION='' [oscap(17):oscap(7fd8c7d58390):debug.c:319:oscap_print_env_vars]
I: oscap: OSCAP_OVAL_COMMAND_OPTIONS='' [oscap(17):oscap(7fd8c7d58390):debug.c:319:oscap_print_env_vars]
I: oscap: OSCAP_PCRE_EXEC_RECURSION_LIMIT='' [oscap(17):oscap(7fd8c7d58390):debug.c:319:oscap_print_env_vars]
I: oscap: OSCAP_PROBE_ROOT='' [oscap(17):oscap(7fd8c7d58390):debug.c:319:oscap_print_env_vars]
I: oscap: SEXP_VALIDATE_DISABLE='' [oscap(17):oscap(7fd8c7d58390):debug.c:319:oscap_print_env_vars]
I: oscap: SOURCE_DATE_EPOCH='' [oscap(17):oscap(7fd8c7d58390):debug.c:319:oscap_print_env_vars]
I: oscap: OSCAP_PROBE_MEMORY_USAGE_RATIO='' [oscap(17):oscap(7fd8c7d58390):debug.c:319:oscap_print_env_vars]
I: oscap: OSCAP_PROBE_MAX_COLLECTED_ITEMS='' [oscap(17):oscap(7fd8c7d58390):debug.c:319:oscap_print_env_vars]
I: oscap: OSCAP_PROBE_IGNORE_PATHS='' [oscap(17):oscap(7fd8c7d58390):debug.c:319:oscap_print_env_vars]
I: oscap: Identified document type: data-stream-collection [oscap(17):oscap(7fd8c7d58390):doc_type.c:96:oscap_determine_document_type_reader]
Downloading: https://www.debian.org/security/oval/oval-definitions-bookworm.xml.bz2 ... D: oscap: == cURL info: Host www.debian.org:443 was resolved.
 [oscap(17):oscap(7fd8c7d58390):oscap_acquire.c:315:_curl_trace]
D: oscap: == cURL info: IPv6: 2603:400a:ffff:bb8::801f:3e
 [oscap(17):oscap(7fd8c7d58390):oscap_acquire.c:315:_curl_trace]
D: oscap: == cURL info: IPv4: 128.31.0.62
 [oscap(17):oscap(7fd8c7d58390):oscap_acquire.c:315:_curl_trace]
D: oscap: == cURL info:   Trying [2603:400a:ffff:bb8::801f:3e]:443...
 [oscap(17):oscap(7fd8c7d58390):oscap_acquire.c:315:_curl_trace]
D: oscap: == cURL info: Immediate connect fail for 2603:400a:ffff:bb8::801f:3e: Network unreachable
 [oscap(17):oscap(7fd8c7d58390):oscap_acquire.c:315:_curl_trace]
D: oscap: == cURL info:   Trying 128.31.0.62:443...
 [oscap(17):oscap(7fd8c7d58390):oscap_acquire.c:315:_curl_trace]
D: oscap: == cURL info: ALPN: curl offers h2,http/1.1
 [oscap(17):oscap(7fd8c7d58390):oscap_acquire.c:315:_curl_trace]
D: oscap: == cURL info: TLSv1.3 (OUT), TLS handshake, Client hello (1):
 [oscap(17):oscap(7fd8c7d58390):oscap_acquire.c:315:_curl_trace]
D: oscap: == cURL info:  CAfile: /etc/ssl/certs/ca-certificates.crt
 [oscap(17):oscap(7fd8c7d58390):oscap_acquire.c:315:_curl_trace]
D: oscap: == cURL info:  CApath: /etc/ssl/certs
 [oscap(17):oscap(7fd8c7d58390):oscap_acquire.c:315:_curl_trace]
D: oscap: == cURL info: TLSv1.3 (IN), TLS handshake, Server hello (2):
 [oscap(17):oscap(7fd8c7d58390):oscap_acquire.c:315:_curl_trace]
D: oscap: == cURL info: TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
 [oscap(17):oscap(7fd8c7d58390):oscap_acquire.c:315:_curl_trace]
D: oscap: == cURL info: TLSv1.3 (IN), TLS handshake, Certificate (11):
 [oscap(17):oscap(7fd8c7d58390):oscap_acquire.c:315:_curl_trace]
D: oscap: == cURL info: TLSv1.3 (IN), TLS handshake, CERT verify (15):
 [oscap(17):oscap(7fd8c7d58390):oscap_acquire.c:315:_curl_trace]
D: oscap: == cURL info: TLSv1.3 (IN), TLS handshake, Finished (20):
 [oscap(17):oscap(7fd8c7d58390):oscap_acquire.c:315:_curl_trace]
D: oscap: == cURL info: TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
 [oscap(17):oscap(7fd8c7d58390):oscap_acquire.c:315:_curl_trace]
D: oscap: == cURL info: TLSv1.3 (OUT), TLS handshake, Finished (20):
 [oscap(17):oscap(7fd8c7d58390):oscap_acquire.c:315:_curl_trace]
D: oscap: == cURL info: SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA[38](https://github.com/McK-Internal/SF-SE-openscap-docker-image/actions/runs/12668762603/job/35304841578#step:6:40)4 / x25519 / RSASSA-PSS
 [oscap(17):oscap(7fd8c7d58390):oscap_acquire.c:315:_curl_trace]
D: oscap: == cURL info: ALPN: server accepted h2
 [oscap(17):oscap(7fd8c7d58[39](https://github.com/McK-Internal/SF-SE-openscap-docker-image/actions/runs/12668762603/job/35304841578#step:6:41)0):oscap_acquire.c:315:_curl_trace]
D: oscap: == cURL info: Server certificate:
 [oscap(17):oscap(7fd8c7d58390):oscap_acquire.c:315:_curl_trace]
D: oscap: == cURL info:  subject: CN=www.debian.org
 [oscap(17):oscap(7fd8c7d58390):oscap_acquire.c:315:_curl_trace]
D: oscap: == cURL info:  start date: Dec 10 00:38:49 2024 GMT
 [oscap(17):oscap(7fd8c7d58390):oscap_acquire.c:315:_curl_trace]
D: oscap: == cURL info:  expire date: Mar 10 00:38:48 2025 GMT
 [oscap(17):oscap(7fd8c7d58390):oscap_acquire.c:315:_curl_trace]
D: oscap: == cURL info:  subjectAltName: host "www.debian.org" matched cert's "www.debian.org"
 [oscap(17):oscap(7fd8c7d58390):oscap_acquire.c:315:_curl_trace]
D: oscap: == cURL info:  issuer: C=US; O=Let's Encrypt; CN=R11
 [oscap(17):oscap(7fd8c7d58390):oscap_acquire.c:315:_curl_trace]
D: oscap: == cURL info:  SSL certificate verify ok.
 [oscap(17):oscap(7fd8c7d58390):oscap_acquire.c:315:_curl_trace]
D: oscap: == cURL info:   Certificate level 0: Public key type RSA ([40](https://github.com/McK-Internal/SF-SE-openscap-docker-image/actions/runs/12668762603/job/35304841578#step:6:42)96/152 Bits/secBits), signed using sha256WithRSAEncryption
 [oscap(17):oscap(7fd8c7d58390):oscap_acquire.c:315:_curl_trace]
D: oscap: == cURL info:   Certificate level 1: Public key type RSA (2048/112 Bits/secBits), signed using sha256WithRSAEncryption
 [oscap(17):oscap(7fd8c7d58390):oscap_acquire.c:315:_curl_trace]
D: oscap: == cURL info:   Certificate level 2: Public key type RSA (4096/152 Bits/secBits), signed using sha256WithRSAEncryption
 [oscap(17):oscap(7fd8c7d58390):oscap_acquire.c:315:_curl_trace]
D: oscap: == cURL info: Connected to www.debian.org (128.31.0.62) port 4[43](https://github.com/McK-Internal/SF-SE-openscap-docker-image/actions/runs/12668762603/job/35304841578#step:6:45)
 [oscap(17):oscap(7fd8c7d58390):oscap_acquire.c:315:_curl_trace]
D: oscap: == cURL info: using HTTP/2
 [oscap(17):oscap(7fd8c7d58390):oscap_acquire.c:315:_curl_trace]
D: oscap: == cURL info: [HTTP/2] [1] OPENED stream for https://www.debian.org/security/oval/oval-definitions-bookworm.xml.bz2
 [oscap(17):oscap(7fd8c7d58390):oscap_acquire.c:315:_curl_trace]
D: oscap: == cURL info: [HTTP/2] [1] [:method: GET]
 [oscap(17):oscap(7fd8c7d58390):oscap_acquire.c:315:_curl_trace]
D: oscap: == cURL info: [HTTP/2] [1] [:scheme: https]
 [oscap(17):oscap(7fd8c7d58390):oscap_acquire.c:315:_curl_trace]
D: oscap: == cURL info: [HTTP/2] [1] [:authority: www.debian.org]
 [oscap(17):oscap(7fd8c7d58390):oscap_acquire.c:315:_curl_trace]
D: oscap: == cURL info: [HTTP/2] [1] [:path: /security/oval/oval-definitions-bookworm.xml.bz2]
 [oscap(17):oscap(7fd8c7d58390):oscap_acquire.c:315:_curl_trace]
D: oscap: == cURL info: [HTTP/2] [1] [accept: */*]
 [oscap(17):oscap(7fd8c7d58390):oscap_acquire.c:315:_curl_trace]
D: oscap: == cURL info: [HTTP/2] [1] [te: gzip]
 [oscap(17):oscap(7fd8c7d58390):oscap_acquire.c:315:_curl_trace]
D: oscap: == cURL info: [HTTP/2] [1] [accept-encoding: deflate, gzip, br, zstd]
 [oscap(17):oscap(7fd8c7d58390):oscap_acquire.c:315:_curl_trace]
D: oscap: => cURL header (out): GET /security/oval/oval-definitions-bookworm.xml.bz2 HTTP/2
Host: www.debian.org
Accept: */*
Connection: TE
TE: gzip
Accept-Encoding: deflate, gzip, br, zstd
 [oscap(17):oscap(7fd8c7d[58](https://github.com/McK-Internal/SF-SE-openscap-docker-image/actions/runs/12668762603/job/35304841578#step:6:60)390):oscap_acquire.c:315:_curl_trace]
D: oscap: == cURL info: Request completely sent off
 [oscap(17):oscap(7fd8c7d58390):oscap_acquire.c:315:_curl_trace]
D: oscap: == cURL info: TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
 [oscap(17):oscap(7fd8c7d58390):oscap_acquire.c:315:_curl_trace]
D: oscap: == cURL info: TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
 [oscap(17):oscap(7fd8c7d58390):oscap_acquire.c:315:_curl_trace]
D: oscap: == cURL info: HTTP/2 stream 1 was not closed cleanly: PROTOCOL_ERROR (err 1)
 [oscap(17):oscap(7fd8c7d58390):oscap_acquire.c:315:_curl_trace]
D: oscap: == cURL info: Connection #0 to host www.debian.org left intact
 [oscap(17):oscap(7fd8c7d58390):oscap_acquire.c:315:_curl_trace]
error
OpenSCAP Error: Download failed: Stream error in the HTTP/2 framing layer [/home/buildozer/aports/community/openscap/src/openscap-1.3.10/src/common/oscap_acquire.c:405]
Could not extract scap_org.open-scap_cref_ssg-debian12-xccdf.xml with all dependencies from datastream. [/home/buildozer/aports/community/openscap/src/openscap-1.3.10/src/DS/ds_sds_session.c:228]
An error occurred during evaluation

@evgenyz
Copy link
Contributor

evgenyz commented Jan 8, 2025

Thanks! Yeah, it does look like the previous issue and the problem mentioned in cURL repo. I'll close this bug.

@evgenyz evgenyz closed this as completed Jan 8, 2025
@Jiri-Stary
Copy link
Author

@evgenyz - can you please advice how to build this with correct version of libcurl or how to increase the version ?
When i run the curl outside of opescap i do not get the issue, so i belive there might be old version of libcurl present

@evgenyz
Copy link
Contributor

evgenyz commented Jan 8, 2025

OpenSCAP is dynamically linked to whatever is the lib- and -devel package in the system. And this particular problem has very specific conditions (which is the case for OpenSCAP, unfortunately). You can try to get newer version of curl into your Alpine and then rebuild openscap package. Not sure how to do that for Alpine, tho.

@evgenyz
Copy link
Contributor

evgenyz commented Jan 8, 2025

You might get away with just the lib upgrade, but that's a stretch.

@Jiri-Stary
Copy link
Author

curl/curl#15941

Seems that openscap might confuse the server with incorrect headers

@icing
Copy link

icing commented Jan 8, 2025

To quote from https://www.rfc-editor.org/rfc/rfc9113.html#name-connection-specific-header- headers like 'TE' are not allowed in HTTP/2 and lead to a stream reset. Apache is correct in its behaviour.

The question is now: is the libcurl application wrong in adding the header or should curl, when selecting HTTP/2, automatically strip the header from the request.

@evgenyz evgenyz reopened this Jan 8, 2025
@evgenyz
Copy link
Contributor

evgenyz commented Jan 8, 2025

For the record, this (curl/curl#15943) should fix the problem on the curl side. But we might want to reconsider TE header anyways.

@evgenyz
Copy link
Contributor

evgenyz commented Jan 8, 2025

@Jiri-Stary Thanks for getting to the bottom of it!

@jan-cerny
Copy link
Member

also reported in https://issues.redhat.com/browse/RHEL-40997

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug solution
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@icing @evgenyz @jan-cerny @Jiri-Stary