Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: support instrumentation capabilities #998

Merged
merged 5 commits into from
Jan 15, 2025
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
32 changes: 32 additions & 0 deletions api/saucectl.schema.json
Original file line number Diff line number Diff line change
Expand Up @@ -1104,6 +1104,10 @@
"description": "Overwrite real device settings.",
"type": "object",
"properties": {
"resigningEnabled": {
"description": "Overwrite app settings for real device to enable app resigning.",
"type": "boolean"
},
"audioCapture": {
"description": "Overwrite app settings for real device to capture audio.",
"type": "boolean"
Expand All @@ -1112,9 +1116,37 @@
"description": "Overwrite app settings for real device instrumentation.",
"type": "object",
"properties": {
"imageInjection": {
"description": "Overwrite app settings for real device to inject provided images in the user app.",
"type": "boolean"
},
"bypassScreenshotRestriction": {
"description": "Overwrite app settings for real device to enable screenshot restriction.",
"type": "boolean"
},
"groupFolderRedirect": {
"description": "Overwrite app settings for real device to redirect group folder.",
"type": "boolean"
},
"sysAlertsDelay": {
"description": "Overwrite app settings for real device to delay system alerts.",
"type": "boolean"
},
"vitals": {
"description": "Overwrite app settings for real device to enable vitals.",
"type": "boolean"
},
"networkCapture": {
"description": "Overwrite app settings for real device to capture network.",
"type": "boolean"
},
"biometrics": {
"description": "Overwrite app settings for real device to intercept biometric authentication.",
"type": "boolean"
},
"groupDirectory": {
"description": "Overwrite app settings for real device to enable group directory access.",
"type": "boolean"
}
}
}
Expand Down
32 changes: 32 additions & 0 deletions api/v1/subschema/common.schema.json
Original file line number Diff line number Diff line change
Expand Up @@ -62,6 +62,10 @@
"description": "Overwrite real device settings.",
"type": "object",
"properties": {
"resigningEnabled": {
"description": "Overwrite app settings for real device to enable app resigning.",
"type": "boolean"
},
"audioCapture": {
"description": "Overwrite app settings for real device to capture audio.",
"type": "boolean"
Expand All @@ -70,9 +74,37 @@
"description": "Overwrite app settings for real device instrumentation.",
"type": "object",
"properties": {
"imageInjection": {
"description": "Overwrite app settings for real device to inject provided images in the user app.",
"type": "boolean"
},
"bypassScreenshotRestriction": {
"description": "Overwrite app settings for real device to enable screenshot restriction.",
"type": "boolean"
},
"groupFolderRedirect": {
"description": "Overwrite app settings for real device to redirect group folder.",
"type": "boolean"
},
"sysAlertsDelay": {
"description": "Overwrite app settings for real device to delay system alerts.",
"type": "boolean"
},
"vitals": {
"description": "Overwrite app settings for real device to enable vitals.",
"type": "boolean"
},
"networkCapture": {
"description": "Overwrite app settings for real device to capture network.",
"type": "boolean"
},
"biometrics": {
"description": "Overwrite app settings for real device to intercept biometric authentication.",
"type": "boolean"
},
"groupDirectory": {
"description": "Overwrite app settings for real device to enable group directory access.",
"type": "boolean"
}
}
}
Expand Down
32 changes: 32 additions & 0 deletions api/v1alpha/subschema/common.schema.json
Original file line number Diff line number Diff line change
Expand Up @@ -62,6 +62,10 @@
"description": "Overwrite real device settings.",
"type": "object",
"properties": {
"resigningEnabled": {
"description": "Overwrite app settings for real device to enable app resigning.",
"type": "boolean"
},
"audioCapture": {
"description": "Overwrite app settings for real device to capture audio.",
"type": "boolean"
Expand All @@ -70,9 +74,37 @@
"description": "Overwrite app settings for real device instrumentation.",
"type": "object",
"properties": {
"imageInjection": {
"description": "Overwrite app settings for real device to inject provided images in the user app.",
"type": "boolean"
},
"bypassScreenshotRestriction": {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since the configuration fields are newly added and already separated into Android and iOS, I believe it would be better to split them into distinct sections for each platform in the JSON schema.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree but that's a change that will make existing saucectl configurations invalid. We'll have to plan for it with proper communication.

"description": "Overwrite app settings for real device to enable screenshot restriction.",
"type": "boolean"
},
"groupFolderRedirect": {
"description": "Overwrite app settings for real device to redirect group folder.",
"type": "boolean"
},
"sysAlertsDelay": {
"description": "Overwrite app settings for real device to delay system alerts.",
"type": "boolean"
},
"vitals": {
"description": "Overwrite app settings for real device to enable vitals.",
"type": "boolean"
},
"networkCapture": {
"description": "Overwrite app settings for real device to capture network.",
"type": "boolean"
},
"biometrics": {
"description": "Overwrite app settings for real device to intercept biometric authentication.",
"type": "boolean"
},
"groupDirectory": {
"description": "Overwrite app settings for real device to enable group directory access.",
"type": "boolean"
}
}
}
Expand Down
8 changes: 8 additions & 0 deletions internal/cmd/run/espresso.go
Original file line number Diff line number Diff line change
Expand Up @@ -76,8 +76,16 @@ func NewEspressoCmd() *cobra.Command {
cmd.Flags().Var(&lflags.Device, "device", "Specifies the device to use for testing. Requires --name to be set.")

// Overwrite devices settings
sc.Bool("resigningEnabled", "suite::appSettings::resigningEnabled", false, "Overwrite app settings for real device to enable app resigning.")
sc.Bool("audioCapture", "suite::appSettings::audioCapture", false, "Overwrite app settings for real device to capture audio.")
sc.Bool("imageInjection", "suite::appSettings::instrumentation::imageInjection", false, "Overwrite app settings for real device to inject provided images in the user app.")
sc.Bool("bypassScreenshotRestriction", "suite::appSettings::instrumentation::bypassScreenshotRestriction", false, "Overwrite app settings for real device to enable screenshot restriction.")
sc.Bool("groupFolderRedirect", "suite::appSettings::instrumentation::groupFolderRedirect", false, "Overwrite app settings for real device to redirect group folder.")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @diegoperini ! I noticed that some iOS-specific configuration fields have been added here for Espresso run arguments. Could you clarify why these are necessary? I don't think they’re needed in this context.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

They are not needed. Addressed here: https://github.com/saucelabs/saucectl/pull/999/files

sc.Bool("sysAlertsDelay", "suite::appSettings::instrumentation::sysAlertsDelay", false, "Overwrite app settings for real device to delay system alerts.")
sc.Bool("vitals", "suite::appSettings::instrumentation::vitals", false, "Overwrite app settings for real device to enable vitals.")
sc.Bool("networkCapture", "suite::appSettings::instrumentation::networkCapture", false, "Overwrite app settings for real device to capture network.")
sc.Bool("biometrics", "suite::appSettings::instrumentation::biometrics", false, "Overwrite app settings for real device to intercept biometric authentication.")
sc.Bool("groupDirectory", "suite::appSettings::instrumentation::groupDirectory", false, "Overwrite app settings for real device to enable group directory access.")

return cmd
}
Expand Down
8 changes: 8 additions & 0 deletions internal/cmd/run/xcuitest.go
Original file line number Diff line number Diff line change
Expand Up @@ -73,8 +73,16 @@ func NewXCUITestCmd() *cobra.Command {
cmd.Flags().Var(&lflags.Simulator, "simulator", "Specifies the simulator to use for testing. Requires --name to be set.")

// Overwrite devices settings
sc.Bool("resigningEnabled", "suite::appSettings::resigningEnabled", false, "Overwrite app settings for real device to enable app resigning.")
sc.Bool("audioCapture", "suite::appSettings::audioCapture", false, "Overwrite app settings for real device to capture audio.")
sc.Bool("imageInjection", "suite::appSettings::instrumentation::imageInjection", false, "Overwrite app settings for real device to inject provided images in the user app.")
sc.Bool("bypassScreenshotRestriction", "suite::appSettings::instrumentation::bypassScreenshotRestriction", false, "Overwrite app settings for real device to enable screenshot restriction.")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same applies here. bypassScreenshotRestriction is Android only. I don't think it's necessary as well.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sc.Bool("groupFolderRedirect", "suite::appSettings::instrumentation::groupFolderRedirect", false, "Overwrite app settings for real device to redirect group folder.")
sc.Bool("sysAlertsDelay", "suite::appSettings::instrumentation::sysAlertsDelay", false, "Overwrite app settings for real device to delay system alerts.")
sc.Bool("vitals", "suite::appSettings::instrumentation::vitals", false, "Overwrite app settings for real device to enable vitals.")
sc.Bool("networkCapture", "suite::appSettings::instrumentation::networkCapture", false, "Overwrite app settings for real device to capture network.")
sc.Bool("biometrics", "suite::appSettings::instrumentation::biometrics", false, "Overwrite app settings for real device to intercept biometric authentication.")
sc.Bool("groupDirectory", "suite::appSettings::instrumentation::groupDirectory", false, "Overwrite app settings for real device to enable group directory access.")

return cmd
}
Expand Down
13 changes: 10 additions & 3 deletions internal/config/config.go
Original file line number Diff line number Diff line change
Expand Up @@ -206,13 +206,20 @@ type Defaults struct {

// AppSettings represents override settings.
type AppSettings struct {
AudioCapture bool `yaml:"audioCapture,omitempty" json:"audioCapture"`
Instrumentation Instrumentation `yaml:"instrumentation,omitempty" json:"instrumentation"`
ResigningEnabled bool `yaml:"resigningEnabled,omitempty" json:"resigningEnabled"`
AudioCapture bool `yaml:"audioCapture,omitempty" json:"audioCapture"`
Instrumentation Instrumentation `yaml:"instrumentation,omitempty" json:"instrumentation"`
}

// Instrumentation represents Instrumentation settings for real devices.
type Instrumentation struct {
NetworkCapture bool `yaml:"networkCapture,omitempty" json:"networkCapture"`
ImageInjection bool `yaml:"imageInjection,omitempty" json:"imageInjection"`
BypassScreenshotRestriction bool `yaml:"bypassScreenshotRestriction,omitempty" json:"bypassScreenshotRestriction"`
GroupFolderRedirect bool `yaml:"groupDirectory,omitempty" json:"groupDirectory"`
SysAlertsDelay bool `yaml:"sysAlertsDelay,omitempty" json:"sysAlertsDelay"`
Biometrics bool `yaml:"biometrics,omitempty" json:"biometrics"`
Vitals bool `yaml:"vitals,omitempty" json:"vitals"`
NetworkCapture bool `yaml:"networkCapture,omitempty" json:"networkCapture"`
}

// SmartRetry represents the settings for retry strategy.
Expand Down
27 changes: 22 additions & 5 deletions internal/job/options.go
Original file line number Diff line number Diff line change
Expand Up @@ -97,15 +97,32 @@ type StartOptions struct {
CLIFlags map[string]interface{} `json:"-"`
}

// AppSettings represents app settings for real device
// AppSettings represents mobile app settings.
type AppSettings struct {
AudioCapture bool `json:"audio_capture,omitempty"`
Instrumentation Instrumentation `json:"instrumentation,omitempty"`
AudioCapture bool `json:"audio_capture,omitempty"`
InstrumentationEnabled bool `json:"instrumentation_enabled,omitempty"`
Instrumentation Instrumentation `json:"instrumentation,omitempty"`
ResigningEnabled bool `json:"resigning_enabled,omitempty"`
Resigning Resigning `json:"resigning,omitempty"`
}

// Instrumentation represents instrumentation settings for real device
// Instrumentation represents mobile app instrumentation settings for Android.
type Instrumentation struct {
NetworkCapture bool `json:"network_capture,omitempty"`
ImageInjection bool `json:"image_injection,omitempty"`
BypassScreenshotRestriction bool `json:"bypass_screenshot_restriction,omitempty"`
BiometricsInterception bool `json:"biometrics,omitempty"`
Vitals bool `json:"vitals,omitempty"`
NetworkCapture bool `json:"network_capture,omitempty"`
}

// Resigning represents mobile app instrumentation settings for iOS.
type Resigning struct {
ImageInjection bool `json:"image_injection,omitempty"`
SystemAlertsDelay bool `json:"sys_alerts_delay,omitempty"`
GroupFolderRedirect bool `json:"group_directory,omitempty"`
BiometricsInterception bool `json:"biometrics,omitempty"`
Vitals bool `json:"vitals,omitempty"`
NetworkCapture bool `json:"network_capture,omitempty"`
}

// TunnelOptions represents the options that configure the usage of a tunnel when running tests in the Sauce Labs cloud.
Expand Down
20 changes: 19 additions & 1 deletion internal/saucecloud/espresso.go
Original file line number Diff line number Diff line change
Expand Up @@ -254,9 +254,27 @@ func (r *EspressoRunner) newStartOptions(
// Overwrite device settings
RealDeviceKind: strings.ToLower(espresso.Android),
AppSettings: job.AppSettings{
////////////////////////////////////////////////////////////////////////////////////////////////////////////
// The key name mismatch here between left and right hand side is intentional.
//
// Traditionally, saucectl made no distinction between instrumentation and resigning
// while our composer API backend always used specific key names per platform,
// either Instrumentation for Android and Resigning for iOS.
//
// This created a situation where app settings were sporadically ignored in Android RDC sessions.
//
// Here, we choose the lesser evil and keep supporting `ResignerEnabled` in the saucectl config yaml
// for backward compatibility while also mapping it to the correct API parameter for composer to evaluate.
InstrumentationEnabled: s.AppSettings.ResigningEnabled,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Isn't that supposed to be InstrumentationEnabled?

Suggested change
InstrumentationEnabled: s.AppSettings.ResigningEnabled,
InstrumentationEnabled: s.AppSettings.InstrumentationEnabled,

Copy link
Contributor

@diegoperini diegoperini Jan 15, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This AppSettings to job.AppSettings struct mapping maps saucectl yaml values to composer API call json payload. In the yaml, the definition ResigningEnabled is used both for iOS and Android but in the API call, the name is different per platform. I wish I was able to rename it so that it's consistent but that's a breaking change for existing saucectl users, so we do this ugly mapping as the lesser evil.

History:

  • saucectl always used ResigningEnabled for this value
  • all internal APIs always used different names for this and thus, Android instrumentation in saucectl was actually always broken if the instrumentation flag is turned off in app settings. Most people turn it on via app settings anyway so that was fine but now that we're removing app storage flag evaluations for automated tests and will use the saucectl yaml as the source of truth, this bug had to be fixed.

TLDR: The shape of that line is intentional, to avoid breaking all YAML files out there.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We're updating the docs today 👍

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @diegoperini! It'd be good to leave an inline comment in the code explaining that little difference. I won't remember this in 2 weeks (or any new dev that joins) 😂

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good idea, added it.

////////////////////////////////////////////////////////////////////////////////////////////////////////////

AudioCapture: s.AppSettings.AudioCapture,
Instrumentation: job.Instrumentation{
NetworkCapture: s.AppSettings.Instrumentation.NetworkCapture,
ImageInjection: s.AppSettings.Instrumentation.ImageInjection,
BypassScreenshotRestriction: s.AppSettings.Instrumentation.BypassScreenshotRestriction,
Vitals: s.AppSettings.Instrumentation.Vitals,
NetworkCapture: s.AppSettings.Instrumentation.NetworkCapture,
BiometricsInterception: s.AppSettings.Instrumentation.Biometrics,
},
},
}
Expand Down
12 changes: 9 additions & 3 deletions internal/saucecloud/xcuitest.go
Original file line number Diff line number Diff line change
Expand Up @@ -241,9 +241,15 @@ func (r *XcuitestRunner) startJob(jobOpts chan<- job.StartOptions, appFileID, te
// Overwrite device settings
RealDeviceKind: strings.ToLower(xcuitest.IOS),
AppSettings: job.AppSettings{
AudioCapture: s.AppSettings.AudioCapture,
Instrumentation: job.Instrumentation{
NetworkCapture: s.AppSettings.Instrumentation.NetworkCapture,
ResigningEnabled: s.AppSettings.ResigningEnabled,
AudioCapture: s.AppSettings.AudioCapture,
Resigning: job.Resigning{
ImageInjection: s.AppSettings.Instrumentation.ImageInjection,
GroupFolderRedirect: s.AppSettings.Instrumentation.GroupFolderRedirect,
SystemAlertsDelay: s.AppSettings.Instrumentation.SysAlertsDelay,
BiometricsInterception: s.AppSettings.Instrumentation.Biometrics,
Vitals: s.AppSettings.Instrumentation.Vitals,
NetworkCapture: s.AppSettings.Instrumentation.NetworkCapture,
},
},
}
Expand Down
Loading