Best Approach to Assigning User Agent and Proxy Based on Data List in Crawlee?

[ferdysopian]

I'm working on a scraping task where I have a list of workers, each with associated cookies, user agent, and proxy. I want each worker to scrape up to 5 URLs.

Current Problem:

I can currently set the user agent and proxy globally, but I need to dynamically assign the appropriate user agent and proxy to each worker based on their specific configuration. Rather than defining RequestQueue, ProxyConfiguration, Configuration, and PlaywrightCrawler separately for each worker, I would like to define them once and then retrieve the specific information (such as user agent and proxy) from request.userData for each request. How can I achieve this?

My Current Approach:

I'm looping through the list of workers, assigning 5 URLs per worker. Here’s my current example code:

import { RequestQueue, ProxyConfiguration, Configuration, PlaywrightCrawler } from 'crawlee'
import { v4 as uuidv4 } from 'uuid'

const runScrape = async () => {
    const workers = [
        { name: 'worker1', cookies: '[{"name":"xxx","value":"xxx"}]', proxy: 'http://user1:pass@host:port', userAgent: 'xxx' },
        { name: 'worker2', cookies: '[{"name":"yyy","value":"yyy"}]', proxy: 'http://user2:pass@host:port', userAgent: 'xxx' },
        { name: 'worker3', cookies: '[{"name":"zzz","value":"zzz"}]', proxy: 'http://user3:pass@host:port', userAgent: 'xxx' },
        { name: 'worker4', cookies: '[{"name":"aaa","value":"aaa"}]', proxy: 'http://user4:pass@host:port', userAgent: 'xxx' },
        // More workers...
    ]

    const urls = [
        'https://example.com/1',
        'https://example.com/2',
        'https://example.com/3',
        'https://example.com/4',
        'https://example.com/5',
        'https://example.com/6',
        'https://example.com/7',
        'https://example.com/8',
        // More URLs...
    ]

    const maxWorkerTasks = 5
    // Assign URLs to workers dynamically
    while (urls.length) {
        for (const worker of workers) {
            const tasks = urls.splice(0, maxWorkerTasks)
            if (tasks.length) {
                handleTask(worker, tasks).catch((err) => {
                    console.error(`Error in worker ${worker.name}:`, err.message)
                }) // No need to wait for the task to complete
            }
            if (urls.length === 0) break
        }
    }
}

const handleTask = async (worker, requests) => {
    const queue = await RequestQueue.open(uuidv4())

    // Add URLs to the queue
    for (const url of requests) {
        await queue.addRequest({ url, userData: { worker } })
    }

    // Configure proxy and crawler settings
    const proxy = new ProxyConfiguration({ proxyUrls: [worker.proxy] })

    // Configure crawler
    const configuration = new Configuration({
        purgeOnStart: true,
        availableMemoryRatio: 1,
        memoryMbytes: 10 * 1024,
    })

    const crawler = new PlaywrightCrawler({
        requestQueue: queue,
        proxyConfiguration: proxy,
        launchContext: {
            userAgent: worker.userAgent,
            launchOptions: {
                headless: true,
                args: ['--disable-web-security', '--no-sandbox', '--disable-gpu'],
            },
        },
        navigationTimeoutSecs: 120,
        requestHandlerTimeoutSecs: 120,
        persistCookiesPerSession: true,
        maxRequestRetries: 0,
        preNavigationHooks: [
            async ({ session, request }) => {
                try {
                    const cookies = JSON.parse(worker.cookies)
                    if (Array.isArray(cookies)) {
                        await session.setCookies(cookies, request.url)
                    }
                } catch (err) {
                    console.error(`Failed to set cookies for worker ${worker.name}:`, err.message)
                }
            },
        ],
        requestHandler: async ({ request, page, log }) => {
            try {
                log.info(`Processing: ${request.url} with worker ${worker.name}`)
                // Scraping logic here
            } catch (err) {
                console.error(`Error scraping ${request.url}:`, err.message)
            }
        },
        failedRequestHandler: async ({ request, log }) => {
            log.error(`Failed to process: ${request.url} by worker ${worker.name}`)
        },
    }, configuration)

    try {
        await crawler.run()
    } catch (error) {
        console.error('Error during scraping:', error)
    } finally {
        try {
            await crawler.teardown()
            await queue.drop()
            const client = configuration.getStorageClient()
            await client.purge?.()
        } catch (error) {
            console.error('Error during teardown:', error)
        }
    }
}

runScrape()
    .then(() => console.log('Scraping complete.'))
    .catch((err) => console.error('Error during scraping:', err.message))

Question:

Is there a better or more efficient way to assign user agents and proxies for each worker dynamically, instead of setting them globally or using the current worker task-loop approach?

Maybe something like this:

import { RequestQueue, ProxyConfiguration, Configuration, PlaywrightCrawler } from 'crawlee'
import { v4 as uuidv4 } from 'uuid'

const runScrape = async () => {
    // Worker configuration here
    const workers = [
        { name: 'worker1', cookies: '[{"name":"xxx","value":"xxx"}]', proxy: 'http://user1:pass@host:port', userAgent: 'xxx' },
        { name: 'worker2', cookies: '[{"name":"yyy","value":"yyy"}]', proxy: 'http://user2:pass@host:port', userAgent: 'xxx' },
        { name: 'worker3', cookies: '[{"name":"zzz","value":"zzz"}]', proxy: 'http://user3:pass@host:port', userAgent: 'xxx' },
        { name: 'worker4', cookies: '[{"name":"aaa","value":"aaa"}]', proxy: 'http://user4:pass@host:port', userAgent: 'xxx' },
        // More workers...
    ]

    const urls = [
        'https://example.com/1',
        'https://example.com/2',
        'https://example.com/3',
        'https://example.com/4',
        'https://example.com/5',
        'https://example.com/6',
        'https://example.com/7',
        'https://example.com/8',
        // More URLs...
    ]

    const maxWorkerTask = 5
    let workerIndex = 0
    let taskIndex = 0
    const requests = []

    // Assign URLs to workers
    for (let i = 0; i < urls.length; i++) {
        if (taskIndex === maxWorkerTask) {
            taskIndex = 0
            workerIndex++
        }
        if (workerIndex === workers.length) break

        requests.push({
            url: urls[i],
            worker: workers[workerIndex],
        })

        taskIndex++
    }

    // Create a request queue
    const queue = await RequestQueue.open(uuidv4())
    requests.forEach((request) => {
        queue.addRequest({
            url: request.url,
            userData: {
                worker: request.worker,
            },
        })
    })

    // Normal use of ProxyConfiguration, we need to provide the proxyUrls differently from the request queue
    const proxyUrls = workers.map((worker) => worker.proxy)
    const proxy = new ProxyConfiguration({
        proxyUrls, // How i can register proxy depending on the worker or request? <== I need to know how to do this
    })

    // Configure crawler
    const configuration = new Configuration({
        purgeOnStart: true,
        availableMemoryRatio: 1,
        memoryMbytes: 10 * 1024,
    })

    const crawler = new PlaywrightCrawler({
        proxyConfiguration: proxy,
        requestQueue: queue,
        launchContext: {
            userAgent: workers[0].userAgent, // How i can set the user agent depending on the worker or request? <== I need to know how to do this
            launchOptions: {
                headless: true,
            },
        },
        preNavigationHooks: [
            async ({ session, request }) => {
                const workerConfiguration = request.userData.worker
                try {
                    const cookies = JSON.parse(workerConfiguration.cookies)
                    if (Array.isArray(cookies)) {
                        await session.setCookies(cookies, request.url)
                    }
                } catch (err) {
                    console.error(`Failed to set cookies for worker ${workerConfiguration.name}:`, err.message)
                }
            },
        ],
    }, configuration)

    try {
        await crawler.run()
    } catch (error) {
        console.error('Error during scraping:', error)
    } finally {
        try {
            await crawler.teardown()
            await queue.drop()
            const client = configuration.getStorageClient()
            await client.purge?.()
        } catch (error) {
            console.error('Error during teardown:', error)
        }
    }
}

runScrape()

Is there a better approach to implement dynamic handling of proxies and user agents based on workers or requests? I'm open to suggestions!