Scraping websites that need authentication

[kubs0ne]

Hello I am trying to scrape an internal website for which i need user and password authentication. I just want to use the AsyncWebCrawler for that. I have searched this forum and tried several things but nothing seems to work. I am trying to make this thing work:

browser_config = BrowserConfig(
    headers={
        'Authorization': 'Basic ' + base64.b64encode(f'{os.getenv("USER")}:{os.getenv("PASSWORD")}'.encode()).decode()
    }
)
async def main():
    async with AsyncWebCrawler(config=browser_config) as crawler:
        result = await crawler.arun(
            url=url,
        )
        print("result.markdown")

I get Page not found. I am succesfully doing the same thing using requests and session:

from requests import Session
from requests_ntlm import HttpNtlmAuth
with Session() as session:
    session.verify = False
    session.auth = HttpNtlmAuth(
        os.getenv("USER"), os.getenv("PASSWORD")
    )
    response = session.get(url)

How can I combine those 2 methods, or is there some solution for authentication that I dont see?