Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

x-pack/filebeat/input: Fix truncation of bodies in request tracing #42327

Open
wants to merge 2 commits into
base: main
Choose a base branch
from
+11 −18
Open

x-pack/filebeat/input: Fix truncation of bodies in request tracing #42327

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
df9df07
For request tracing, truncate bodies at 10% of the maximum file size.
chrisberkhout Jan 16, 2025
6ae4006
CHANGELOG.next.asciidoc entry.
chrisberkhout Jan 16, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.next.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -206,6 +206,7 @@ https://github.com/elastic/beats/compare/v8.8.1\...main[Check the HEAD diff]
- The `_id` generation process for S3 events has been updated to incorporate the LastModified field. This enhancement ensures that the `_id` is unique. {pull}42078[42078]
- Fix Netflow Template Sharing configuration handling. {pull}42080[42080]
- Updated websocket retry error code list to allow more scenarios to be retried which could have been missed previously. {pull}42218[42218]
- Fix truncation of bodies in request tracing. {pull}42327[42327]
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it would be nice to include a bit of the "how" part, like ... "by limiting bodies to 10% of the maximum file size".


*Heartbeat*

Expand Down
5 changes: 2 additions & 3 deletions x-pack/filebeat/input/cel/input.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -785,9 +785,8 @@
)
traceLogger := zap.New(core)

const margin = 10e3 // 1OkB ought to be enough room for all the remainder of the trace details.
maxSize := cfg.Resource.Tracer.MaxSize * 1e6
trace = httplog.NewLoggingRoundTripper(c.Transport, traceLogger, max(0, maxSize-margin), log)
maxBodyLen := cfg.Resource.Tracer.MaxSize * 1e6 / 10 // 10% of file max
trace = httplog.NewLoggingRoundTripper(c.Transport, traceLogger, maxBodyLen, log)
c.Transport = trace
} else if cfg.Resource.Tracer != nil {
// We have a trace log name, but we are not enabled,
Expand Down Expand Up @@ -1348,7 +1347,7 @@
// walkMap walks to all ends of the provided path in m and applies fn to the
// final element of each walk. Nested arrays are not handled.
func walkMap(m mapstr.M, path string, fn func(parent mapstr.M, key string)) {
key, rest, more := strings.Cut(path, ".")

Check failure on line 1350 in x-pack/filebeat/input/cel/input.go

View workflow job for this annotation

GitHub Actions / lint (darwin) 

rest declared and not used (typecheck)
v, ok := m[key]
if !ok {
return
Expand Down
5 changes: 2 additions & 3 deletions x-pack/filebeat/input/entityanalytics/provider/azuread/fetcher/graph/graph.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -455,9 +455,8 @@ func requestTrace(ctx context.Context, cli *http.Client, cfg graphConf, log *log
)
traceLogger := zap.New(core)

const margin = 10e3 // 1OkB ought to be enough room for all the remainder of the trace details.
maxSize := max(1, cfg.Tracer.MaxSize) * 1e6
cli.Transport = httplog.NewLoggingRoundTripper(cli.Transport, traceLogger, max(0, maxSize-margin), log)
maxBodyLen := max(1, cfg.Tracer.MaxSize) * 1e6 / 10 // 10% of file max
cli.Transport = httplog.NewLoggingRoundTripper(cli.Transport, traceLogger, maxBodyLen, log)
return cli
}

Expand Down
5 changes: 2 additions & 3 deletions x-pack/filebeat/input/entityanalytics/provider/jamf/jamf.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -228,9 +228,8 @@ func requestTrace(ctx context.Context, cli *http.Client, cfg conf, log *logp.Log
)
traceLogger := zap.New(core)

const margin = 10e3 // 1OkB ought to be enough room for all the remainder of the trace details.
maxSize := cfg.Tracer.MaxSize * 1e6
cli.Transport = httplog.NewLoggingRoundTripper(cli.Transport, traceLogger, max(0, maxSize-margin), log)
maxBodyLen := cfg.Tracer.MaxSize * 1e6 / 10 // 10% of file max
cli.Transport = httplog.NewLoggingRoundTripper(cli.Transport, traceLogger, maxBodyLen, log)
return cli
}

Expand Down
5 changes: 2 additions & 3 deletions x-pack/filebeat/input/entityanalytics/provider/okta/okta.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -232,9 +232,8 @@ func requestTrace(ctx context.Context, cli *http.Client, cfg conf, log *logp.Log
)
traceLogger := zap.New(core)

const margin = 10e3 // 1OkB ought to be enough room for all the remainder of the trace details.
maxSize := cfg.Tracer.MaxSize * 1e6
cli.Transport = httplog.NewLoggingRoundTripper(cli.Transport, traceLogger, max(0, maxSize-margin), log)
maxBodyLen := cfg.Tracer.MaxSize * 1e6 / 10 // 10% of file max
cli.Transport = httplog.NewLoggingRoundTripper(cli.Transport, traceLogger, maxBodyLen, log)
return cli
}

Expand Down
8 changes: 2 additions & 6 deletions x-pack/filebeat/input/httpjson/input.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -118,12 +118,12 @@
}

keys := make([]string, 0, len(m))
for k := range m {

Check failure on line 121 in x-pack/filebeat/input/httpjson/input.go

View workflow job for this annotation

GitHub Actions / lint (darwin) 

cannot range over m (variable of type mapstrM) (typecheck)
keys = append(keys, k)
}
sort.Strings(keys)
for _, k := range keys {
v := m[k]

Check failure on line 126 in x-pack/filebeat/input/httpjson/input.go

View workflow job for this annotation

GitHub Actions / lint (darwin) 

invalid operation: cannot index m (variable of type mapstrM) (typecheck)
if inner, ok := tryToMapStr(v); ok {
err := enc.AddObject(k, inner)
if err != nil {
Expand All @@ -141,9 +141,9 @@
case mapstrM:
return m, true
case map[string]interface{}:
return mapstrM(m), true

Check failure on line 144 in x-pack/filebeat/input/httpjson/input.go

View workflow job for this annotation

GitHub Actions / lint (darwin) 

cannot convert m (variable of type map[string]interface{}) to type mapstrM (typecheck)
default:
return nil, false

Check failure on line 146 in x-pack/filebeat/input/httpjson/input.go

View workflow job for this annotation

GitHub Actions / lint (darwin) 

cannot use nil as mapstrM value in return statement (typecheck)
}
}

Expand Down Expand Up @@ -323,12 +323,8 @@
)
traceLogger := zap.New(core)

const margin = 10e3 // 1OkB ought to be enough room for all the remainder of the trace details.
maxSize := cfg.Tracer.MaxSize*1e6 - margin
if maxSize < 0 {
maxSize = 0
}
netHTTPClient.Transport = httplog.NewLoggingRoundTripper(netHTTPClient.Transport, traceLogger, maxSize, log)
maxBodyLen := cfg.Tracer.MaxSize * 1e6 / 10 // 10% of file max
netHTTPClient.Transport = httplog.NewLoggingRoundTripper(netHTTPClient.Transport, traceLogger, maxBodyLen, log)
} else if cfg.Tracer != nil {
// We have a trace log name, but we are not enabled,
// so remove all trace logs we own.
Expand Down
Loading