Cannot pass gotoOptions to page.goto()

[matjaeck]

Which package is this bug report for? If unsure which one to select, leave blank

@crawlee/browser (BrowserCrawler)

Issue description

Hooks and navigation handlers are called with hard-coded gotoOptions: https://github.com/apify/crawlee/blob/master/packages/browser-crawler/src/internals/browser-crawler.ts#L529

Or this there some hidden magic that I'm missing?

Code sample

n.A.

Package version

3.1.0

Node.js version

irrelevant

Operating system

irrelevant

Apify platform

• Tick me if you encountered this issue on the Apify platform

Priority this issue should have

Medium (should be fixed soon)

I have tested this on the next release

not fixed on master branch

Other context

From https://github.com/apify/crawlee/blob/master/MIGRATIONS.md#removal-of-gotofunction

We decided to replace gotoFunction with preNavigationHooks and postNavigationHooks

With preNavigationHooks and postNavigationHooks it's much easier. preNavigationHooks are called with two arguments: crawlingContext and gotoOptions. postNavigationHooks are called only with crawlingContext.

This appears to be wrong/confusing on different levels:

1.) Both hooks are currently called with gotoOptions.
2.) gotoOptions can be accessed in user defined hooks, but I don't understand how it was ever intended to pass them to hooks or why they should ever be needed pre or post-navigation. They only make sense to use when navigating, not before or after. ???
3.) _navigationHandler is called with gotoOptions and it passes them on to page.goto(). but it takes the hardcoded options from _handleNavigation.

Suggestion: You might be able to define gotoOptions on the crawler instance and access it internally in either the crawler specific goto or the general _navigationHandler (while allowing the user to change them during runtime).

It also seems like a good idea to think about for which use cases hooks are suited (IMO to define event listeners and things that happen unrelated to questions of navigating) and when not (IMO passing configuration options to navigation-related functions).

[matjaeck]

Workaround is to subclass the crawler class and override with _navigationHandler(ctx: Crawlee.PlaywrightCrawlingContext). While this is fine, the method should IMO not be prefixed with an underscore as it is not private or internal and passing gotoOptions should be possible without subclassing.

[mvolfik]

Hi! Sorry that we didn't reply earlier.

If you are interested in calling page.goto(url, options) in your request handler (= scraping code), that is not affected by Crawlee at all - the call goes directly to Puppeteer/Playwright.

The code that you linked to affects navigation, when Crawlee opens the target URL in the browser to allow you to scrape it. What exactly is your use-case that you want to modify gotoOptions there? But a cleaner workaround that I can suggest is using preNavigationHooks when creating your Crawler, for example like this:

const crawler = new PlaywrightCrawler({
  requestHandler: router,
  headless: true,
  preNavigationHooks: [
    async ({ request }, gotoOptions) => {
      gotoOptions.referer = request.userData.customReferer;
    },
  ],
});

As for the points you mentioned as confusing, you say

2.) gotoOptions can be accessed in user defined hooks, but I don't understand how it was ever intended to pass them to hooks or why they should ever be needed pre or post-navigation. They only make sense to use when navigating, not before or after. ???

Yes, they make sense when navigating - but that is done by Puppeteer/Playwright under the hood. the pre and post-navigation hooks exist to perform some actions before or after that happens - for example, customize the gotoOptions, as I showed above.

Does this answer your questions/did you already solve your issues in some other way and can we close this issue?

[matjaeck]

There is nothing more to discuss here as far as I am concerned, I moved away from Crawlee anyway. I had failed to understand at the time of writing that the gotoOptions in hooks are reused in the navigation related code.

can we close this issue

You closed and locked the issue 😆