Indexing pipeline from scrapingLoader

[giedrius]

I am relatively new to Rust and Swiftide. I am running a scrapper loader from url from axum request

let pipeline =  indexing::Pipeline::from_loader(ScrapingLoader::from_spider(
       spider::website::Website::new(&url)
           .with_limit(1)
           .with_request_timeout(Some(std::time::Duration::from_secs(2)))
           .to_owned()
   ))    .log_all()
   .with_concurrency(1)
   .then(HtmlToMarkdownTransformer::default())
   .then_chunk(ChunkMarkdown::from_chunk_range(20..2048))
   .then_store_with(MemoryStorage::default());
   pipeline.run().await?;

Howerver, the indexing pipeline never finishes. ( INFO indexing_pipeline.run: swiftide_indexing::pipeline: Starting indexing pipeline with 1 concurrency) . If I run spider separately, the document is received, so I assume this has something to do with indexingstream creation.
What can be the reasons?

[timonv]

Hey there, sorry for the late response! Sounds like this is a bug. To be fair, I build it last year, but have used it after an initial use case, very little, while Spider has had some major changes. I'll create an issue and take a look at it asap.

I suspect for some reason one of the channels isn't closed properly, which keeps the stream active. If you're interested in helping out, you could enable rust debug logs, and see what happens.

WHen you run spider separately, does it close its own channels?

[giedrius]

I have copied the loader and added some additional debug strings. It looks like that (at least when single page is scrapped) the _recv_thread gets stuck on while condition waiting for new node to create after first iteration (node is not processed further, spider_rx.recv().await locks process) . For my user case everything works (node is processed further) if I add break in the end of while loop. Perhaps this is a problem with channel termination in cases when single/limited pages are needed.

[timonv]

I manged to reproduce it in #614. This seems to be a regression, I'm not sure yet what is going on. When running the tests, the scrape().await never seems to finish. This wasn't the case before.

[giedrius]

It is weird, as during testing, i tried to execute scrape separately (and it succeeded+finished), before creation of pipeline. e.g.
let mut website = Website::new(&url); website .with_request_timeout(Some(std::time::Duration::from_secs(2))) .with_limit(1); website.scrape().await; let p = html2md::parse_html( website .get_pages() .unwrap() .get(0) .unwrap() .get_html() .as_str(), );
this passed normally before creating indexing pipeline and produced result.

[timonv]

Alright I have a temporary fix in #614. It uses crawl instead of scrape. I think it's a bug on their end, as calling subscribe when a scraper is active, results in scrape hanging.

[giedrius]

thank you

[timonv]

New release has been released, let me know if it works for you!