Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jackett can't do multiple searches at the same time #210

Closed
thaynos opened this issue Sep 19, 2022 · 2 comments
Closed

Jackett can't do multiple searches at the same time #210

thaynos opened this issue Sep 19, 2022 · 2 comments

Comments

@thaynos
Copy link

thaynos commented Sep 19, 2022

the jackett repo redirected me here
Jackett/Jackett#13551

@ngosang ngosang changed the title jackeet cant do batch searches Jackett can't do multiple searches at the same time Sep 24, 2022
@ngosang ngosang added the bug label Sep 24, 2022
@ngosang
Copy link
Member

ngosang commented Sep 24, 2022

qBittorrent plugin for Jackett is using the "aggregate indexer" to make the requests. That endpoint takes a lot of time (if you have many indexers configured) and it can cause problems if you run several queries at the same time.
https://github.com/Jackett/Jackett#aggregate-indexers

It will be fixed if we complete this feature someday => #121

@galeksandrp
Copy link

galeksandrp commented Jan 8, 2023

You can just split your jackett.py search engine to multiple search engines. This will speed up search in qbittorrent A LOT cause qbittorrent query multiple search engines at the same time.

Click here to see instruction
  1. Open folder where search engines are stored (%localappdata%\qBittorrent\nova3\engines\ on Windows)
  2. For each tracker configured in jackett you need to guess indexer name.
Point you mouse to `button` `Copy RSS Feed` in `jackett`. Browser will show you link.
http://127.0.0.1:9117/api/v2.0/indexers/<indexer name>/results/torznab/api?apikey=...
Part between `indexers/` and `/results` will be your `indexer name`.
  1. Copy file jackett.py to jackett<indexer name>.py
  2. Open file jackett<indexer name>.py
  3. Replace line
class jackett(object):

with

class jackett<indexer name>(object):
  1. Replace line
name = 'Jackett'

with

name = 'Jackett<indexer name>'
  1. Replace line
jacket_url = self.url + "/api/v2.0/indexers/all/results/torznab/api?%s" % params

with

jacket_url = self.url + "/api/v2.0/indexers/<indexer name>/results/torznab/api?%s" % params
  1. Replace line
jackett_se = jackett()

with

jackett_se = jackett<indexer name>()
  1. Repeat for every tracker in jackett
  2. Restart qbittorrent
  3. Go to search providers settings, you should see truckload of tracker specific jackett search providers.
  4. Now you can disable jackett provider, since you have truckload of tracker specific jackett search providers.

In galeksandrp@jackett-separate-indexers I split jackett.py to jackettrutor.py, jackettrutracker.py, jackettnonameclub.py.
That should give you an example of final result.

As of python developers, this is what i personally use myself: galeksandrp@jackett-child-indexers

galeksandrp added a commit to galeksandrp/search-plugins that referenced this issue Jan 14, 2023
galeksandrp added a commit to galeksandrp/search-plugins that referenced this issue Jan 14, 2023
`Jackett` from version [`v0.10.589` Jan 2, 2019](Jackett/Jackett@25ddcb1) does not recommend using `all` indexer.

Problem seems to be that many clients including `nova2.py` will wait for `HTTP` request to `Jackett` to finish before starting to process `HTTP` response.
This effectivly slows `all` indexer to speed of most slow tracker.

While it's possible to process received parts of `HTTP` response before `HTTP` request is finished, `Jackett` instead recommends separate requests to trackers.

Finally rework `search` function to:
1. Retrieve `list` of configured indexers.
2. Make request to trackers in parallel threads. Multithreading is chosen instead of multiprocessing because we already in separate process thanks to `nova2.py` `multiprocessing` `main`.
Doing `multiprocessing` in `multiprocessing` is questionable matter and actually not working for me (`python` `3.11.0`).
galeksandrp added a commit to galeksandrp/search-plugins that referenced this issue Jan 14, 2023
`Jackett` from version [`v0.10.589` Jan 2, 2019](Jackett/Jackett@25ddcb1) does not recommend using `all` indexer.

Problem seems to be that many clients including `nova2.py` will wait for `HTTP` request to `Jackett` to finish before starting to process `HTTP` response.
This effectivly slows `all` indexer to the speed of most slow tracker.

While it's possible to process received parts of `HTTP` response before `HTTP` request is finished, `Jackett` instead recommends separate requests to trackers.

Finally rework `search` function to:
1. Retrieve `list` of configured indexers.
2. Make request to trackers in parallel threads. Multithreading is chosen instead of multiprocessing because we already in separate process thanks to `nova2.py` `multiprocessing` `main`.
Doing `multiprocessing` in `multiprocessing` is questionable matter and actually not working for me (`python` `3.11.0`).
galeksandrp added a commit to galeksandrp/search-plugins that referenced this issue Jan 14, 2023
`Jackett` from version [`v0.10.589` Jan 2, 2019](Jackett/Jackett@25ddcb1) does not recommend using `all` indexer.

Problem seems to be that many clients including `nova2.py` will wait for `HTTP` request to `Jackett` to finish before starting to process `HTTP` response.
This effectivly slows `all` indexer to the speed of most slow tracker.

While it's possible to process received parts of `HTTP` response before `HTTP` request is finished, `Jackett` instead recommends separate requests to trackers.

Finally rework `search` function to:
1. Retrieve list of configured indexers.
2. Make request to trackers in parallel threads. `Multithreading` is chosen instead of `multiprocessing` because we already in separate process thanks to `nova2.py` `multiprocessing` `main`.
Doing `multiprocessing` in `multiprocessing` is questionable matter and actually not working for me (`python` `3.11.0`).
galeksandrp added a commit to galeksandrp/search-plugins that referenced this issue Jan 14, 2023
`Jackett` from version [`v0.10.589` Jan 2, 2019](Jackett/Jackett@25ddcb1) does not recommend using `all` indexer.

Problem seems to be that many clients including `nova2.py` will wait for `HTTP` request to `Jackett` to finish before starting to process `HTTP` response.
This effectivly slows `all` indexer to the speed of most slow tracker.

While it's possible to process received parts of `HTTP` response before `HTTP` request is finished, `Jackett` instead recommends separate requests to trackers.

Finally rework `search` function to:
1. Retrieve list of configured indexers.
2. Make request to trackers in parallel threads. `Multithreading` is chosen instead of `multiprocessing` because we already in separate process thanks to `nova2.py` `multiprocessing` `main`.
Doing `multiprocessing` in `multiprocessing` is questionable matter and actually not working for me (`python` `3.11.0`).
galeksandrp added a commit to galeksandrp/search-plugins that referenced this issue Jan 14, 2023
`Jackett` from version [`v0.10.589` Jan 2, 2019](Jackett/Jackett@25ddcb1) does not recommend using `all` indexer.

Problem seems to be that many clients including `nova2.py` will wait for `HTTP` request to `Jackett` to finish before starting to process `HTTP` response.
This effectivly slows `all` indexer to the speed of most slow tracker.

While it's possible to process received parts of `HTTP` response before `HTTP` request is finished, `Jackett` instead recommends separate requests to trackers.

Finally rework `search` function to:
1. Retrieve list of configured indexers.
2. Make request to trackers in parallel threads. `Multithreading` is chosen instead of `multiprocessing` because we already in separate process thanks to `nova2.py` `multiprocessing` `main`.
Doing `multiprocessing` in `multiprocessing` is questionable matter and actually not working for me (`python` `3.11.0`).
galeksandrp added a commit to galeksandrp/search-plugins that referenced this issue Jan 14, 2023
`Jackett` from version [`v0.10.589` Jan 2, 2019](Jackett/Jackett@25ddcb1) does not recommend using `all` indexer.

Problem seems to be that many clients including `nova2.py` will wait for `HTTP` request to `Jackett` to finish before starting to process `HTTP` response.
This effectivly slows `all` indexer to the speed of most slow tracker.

While it's possible to process received parts of `HTTP` response before `HTTP` request is finished, `Jackett` instead recommends separate requests to trackers.

Finally rework `search` function to:
1. Retrieve list of configured indexers.
2. Make request to trackers in parallel threads. `Multithreading` is chosen instead of `multiprocessing` because we already in separate process thanks to `nova2.py` `multiprocessing` `main()`.
Doing `multiprocessing` in `multiprocessing` is questionable matter and actually not working for me (`python` `3.11.0`).
galeksandrp added a commit to galeksandrp/search-plugins that referenced this issue Jan 14, 2023
`Jackett` from version [`v0.10.589` Jan 2, 2019](Jackett/Jackett@25ddcb1) does not recommend using `all` indexer.

Problem seems to be that many clients including `nova2.py` will wait for `HTTP` request to `Jackett` to finish before starting to process `HTTP` response.
This effectivly slows `all` indexer to the speed of most slow tracker.

While it's possible to process received parts of `HTTP` response before `HTTP` request is finished, `Jackett` instead recommends separate requests to trackers.

Finally rework `search` function to:
1. Retrieve list of configured indexers.
2. Make request to trackers in parallel threads. `Multithreading` is chosen instead of `multiprocessing` because we already in separate process thanks to `nova2.py` `multiprocessing` `main()`.
Doing `multiprocessing` in `multiprocessing` is questionable matter and actually not working for me (`python` `3.11.0`).
galeksandrp added a commit to galeksandrp/search-plugins that referenced this issue Jan 14, 2023
`Jackett` from version [`v0.10.589` Jan 2, 2019](Jackett/Jackett@25ddcb1) does not recommend using `all` indexer.

Problem seems to be that many clients including `nova2.py` will wait for `HTTP` request to `Jackett` to finish before starting to process `HTTP` response.
This effectivly slows `all` indexer to the speed of most slow tracker.

While it's possible to process received parts of `HTTP` response before `HTTP` request is finished, `Jackett` instead recommends separate requests to trackers.

Finally rework `search` function to:
1. Retrieve list of configured indexers.
2. Make request to trackers in parallel threads. `Multithreading` is chosen instead of `multiprocessing` because we already in separate process thanks to `nova2.py` `multiprocessing` `main()`.
Doing `multiprocessing` in `multiprocessing` is questionable matter and actually not working for me (`python 3.11.0` `Windows 11`).
galeksandrp added a commit to galeksandrp/search-plugins that referenced this issue Jan 14, 2023
`Jackett` from version [`v0.10.589` Jan 2, 2019](Jackett/Jackett@25ddcb1) does not recommend using `all` indexer.

Problem seems to be that many clients including `nova2.py` will wait for `HTTP` request to `Jackett` to finish before starting to process `HTTP` response.
This effectivly slows `all` indexer to the speed of most slow tracker.

While it's possible to process received parts of `HTTP` response before `HTTP` request is finished, `Jackett` instead recommends separate requests to trackers.

Finally rework `search` function to:
1. Retrieve list of configured indexers.
2. Make request to trackers in parallel threads. `Multithreading` is chosen instead of `multiprocessing` because we already in separate process thanks to `nova2.py` `multiprocessing` `main()`.
Doing `multiprocessing` in `multiprocessing` is questionable matter and actually not working for me (`python 3.11.0` `Windows 11`).
galeksandrp added a commit to galeksandrp/search-plugins that referenced this issue Jan 14, 2023
`Jackett` from version [`v0.10.589` Jan 2, 2019](Jackett/Jackett@25ddcb1) does not recommend using `all` indexer.

Problem seems to be that many clients including `nova2.py` will wait for `HTTP` request to `Jackett` to finish before starting to process `HTTP` response.
This effectivly slows `all` indexer to the speed of most slow tracker.

While it's possible to process received parts of `HTTP` response before `HTTP` request is finished, `Jackett` instead recommends separate requests to trackers.

Finally rework `search` function to:
1. Retrieve list of configured indexers.
2. Make request to trackers in parallel threads. `Multithreading` is chosen instead of `multiprocessing` because we already in separate process thanks to `nova2.py` `multiprocessing` `main()`.
Doing `multiprocessing` in `multiprocessing` is questionable matter and actually not working for me (`python 3.11.0` `Windows 11`).
galeksandrp added a commit to galeksandrp/search-plugins that referenced this issue Jan 14, 2023
`Jackett` from version [`v0.10.589` Jan 2, 2019](Jackett/Jackett@25ddcb1) does not recommend using `all` indexer.

Problem seems to be that many clients including `nova2.py` will wait for `HTTP` request to `Jackett` to finish before starting to process `HTTP` response.
This effectivly slows `all` indexer to the speed of most slow tracker.

While it's possible to process received parts of `HTTP` response before `HTTP` request is finished, `Jackett` instead recommends separate requests to trackers.

Finally rework `search` function to:
1. Retrieve list of configured indexers.
2. Make request to trackers in parallel threads. `Multithreading` is chosen instead of `multiprocessing` because we already in separate process thanks to `nova2.py` `multiprocessing` `main()`.
Doing `multiprocessing` in `multiprocessing` is questionable matter and actually not working for me (`python 3.11.0` `Windows 11`).
galeksandrp added a commit to galeksandrp/search-plugins that referenced this issue Jan 14, 2023
`Jackett` from version [`v0.10.589` Jan 2, 2019](Jackett/Jackett@25ddcb1) does not recommend using `all` indexer.

Problem seems to be that many clients including `nova2.py` will wait for `HTTP` request to `Jackett` to finish before starting to process `HTTP` response.
This effectivly slows `all` indexer to the speed of most slow tracker.

While it's possible to process received parts of `HTTP` response before `HTTP` request is finished, `Jackett` instead recommends separate requests to trackers.

Finally rework `search` function to:
1. Retrieve list of configured indexers.
2. Make request to trackers in parallel threads. `Multithreading` is chosen instead of `multiprocessing` because we already in separate process thanks to `nova2.py` `multiprocessing` `main()`.
Doing `multiprocessing` in `multiprocessing` is questionable matter and actually not working for me (`python 3.11.0` `Windows 11`).
galeksandrp added a commit to galeksandrp/search-plugins that referenced this issue Jan 14, 2023
`Jackett` from version [`v0.10.589` Jan 2, 2019](Jackett/Jackett@25ddcb1) does not recommend using `all` indexer.

Problem seems to be that many clients including `nova2.py` will wait for `HTTP` request to `Jackett` to finish before starting to process `HTTP` response.
This effectivly slows `all` indexer to the speed of most slow tracker.

While it's possible to process received parts of `HTTP` response before `HTTP` request is finished, `Jackett` instead recommends separate requests to trackers.

Finally rework `search` function to:
1. Retrieve list of configured indexers.
2. Make request to trackers in parallel threads. `Multithreading` is chosen instead of `multiprocessing` because we already in separate process thanks to `nova2.py` `multiprocessing` `main()`.
Doing `multiprocessing` in `multiprocessing` is questionable matter and actually not working for me (`python 3.11.0` `Windows 11`).
galeksandrp added a commit to galeksandrp/search-plugins that referenced this issue Jan 14, 2023
`Jackett` from version [`v0.10.589` Jan 2, 2019](Jackett/Jackett@25ddcb1) does not recommend using `all` indexer.

Problem seems to be that many clients including `nova2.py` will wait for `HTTP` request to `Jackett` to finish before starting to process `HTTP` response.
This effectivly slows `all` indexer to the speed of most slow tracker.

While it's possible to process received parts of `HTTP` response before `HTTP` request is finished, `Jackett` instead recommends separate requests to trackers.

Finally rework `search` function to:
1. Retrieve list of configured indexers.
2. Make request to trackers in parallel threads. `Multithreading` is chosen instead of `multiprocessing` because we already in separate process thanks to `nova2.py` `multiprocessing` `main()`.
Doing `multiprocessing` in `multiprocessing` is questionable matter and actually not working for me (`python 3.11.0` `Windows 11`).
galeksandrp added a commit to galeksandrp/search-plugins that referenced this issue Jan 14, 2023
`Jackett` from version [`v0.10.589` Jan 2, 2019](Jackett/Jackett@25ddcb1) recommends separate requests to indexers instead of single request to `all` indexer.

Reason to that is because `Jackett` uses `Torznab` specification, which returns results serialized in single `XML`.
This means `all` indexer must wait for all indexers to finish before providing ANY results. This limits `all` indexer to the speed of slowest indexer.

Finally rework `search` function to:
1. Retrieve list of configured indexers.
2. Make requests to indexers in parallel threads.
`Multithreading` is chosen instead of `multiprocessing` because we already in separate process thanks to `nova2.py` `multiprocessing` `main()`. Doing `multiprocessing` in `multiprocessing` is questionable matter and actually not working for me (`python 3.11.0` `Windows 11`).
galeksandrp added a commit to galeksandrp/search-plugins that referenced this issue May 8, 2023
`Jackett` from version [`v0.10.589` Jan 2, 2019](Jackett/Jackett@25ddcb1) recommends separate requests to indexers instead of single request to `all` indexer.

Reason to that is because `Jackett` uses `Torznab` specification, which returns results serialized in single `XML`.
This means `all` indexer must wait for all indexers to finish before providing ANY results. This limits `all` indexer to the speed of slowest indexer.

Finally rework `search` function to:
1. Retrieve list of configured indexers.
2. Make requests to indexers in parallel threads.
`Multithreading` is chosen instead of `multiprocessing` because we already in separate process thanks to `nova2.py` `multiprocessing` `main()`. Doing `multiprocessing` in `multiprocessing` is questionable matter and actually not working for me (`python 3.11.0` `Windows 11`).
ngosang added a commit to ngosang/search-plugins that referenced this issue May 21, 2023
* Make search faster with multi-thread implementation
* Show error messages per indexer
* Number of threads is configurable in jackett.json config file

Based on qbittorrent#222 Thank you @galeksandrp and @WojtekKowaluk
ngosang added a commit to ngosang/search-plugins that referenced this issue May 21, 2023
* Make search faster with multi-thread implementation
* Show error messages per indexer
* Number of threads is configurable in jackett.json config file

Based on qbittorrent#222 Thank you @galeksandrp and @WojtekKowaluk
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants