Fix bugs and release version 0.2.0

.bumpversion.cfg

@@ -1,7 +1,7 @@
 [bumpversion]
 commit = True
 tag = False
-current_version = 0.1.0
+current_version = 0.2.0
 
 [bumpversion:file:pyproject.toml]
 search = version = "{current_version}"

CHANGELOG.md

@@ -10,6 +10,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 ## [0.2.0] - 2025-01-21
 
 - Allow the definition of a reusable config file from `~/.ai-marketplace-monitor/config.toml`
+- Allow options `exclude_sellers` and `exclude_by_description`
 - Fix a bug that prevents the sending of phone notification
 
 ## [0.1.0] - 2025-01-20

README.md

@@ -25,31 +25,32 @@ This program
 
 1. Starts a browser (can be in headless mode)
 2. Search one or more products
-3. Notify you (and others) of new products with phone notification
+3. Notify one or more users of new products with phone notification
 
 ## Features
 
-- Search one or more products with specified keywords
-- Limit search by price, and location
-- Exclude irrelevant results
-- Exclude previous searched items and only notify new items
-- Send notification via PushBullet
-- Search repeatedly with specified intervals in between
-- Add/remove items dynamically by changing the confirmation file.
+- Search for one or more products using specified keywords.
+- Limit search by minimum and maximum price, and location.
+- Exclude irrelevant results.
+- Exclude explicitly listed spammers.
+- Exclude by description.
+- Exclude previously searched items and only notify about new items.
+- Send notifications via PushBullet.
+- Search repeatedly with specified intervals in between.
+- Add/remove items dynamically by changing the configuration file.
 
-TODO:
+**TODO**:
 
-- Exclude explicitly listed spammers
-- Use embedding-based algorithm to identify likely matches
-- Use AI to identify spammers
-- Support other notification methods
-- Support other marketplaces
+- Use embedding-based algorithms to identify likely matches.
+- Use AI to identify spammers.
+- Support other notification methods.
+- Support other marketplaces.
 
-**NOTE**: This is a recipe for programmers, and you are expected to know some Python and command-line operations to make it work. There is no GUI.
+**NOTE**: This is a tool for programmers, and you are expected to know some Python and command-line operations to make it work. There is no GUI.
 
 ## Quickstart
 
-### Set up a python environment
+### Install `ai-marketplace-monitor`
 
 Install the program by
 
@@ -71,7 +72,7 @@ playwright install
 
 ### Write a configuration file
 
-A minimal example is provided as [`minimal_config.toml`](minimal_config.toml). Basically you will need to let the program know which city you are searching, what item you are searching for, and how do you want to get notified.
+A minimal example is provided as [`minimal_config.toml`](minimal_config.toml). Basically you will need to let the program know which city you are searching in, what item you are searching for, and how you want to get notified.
 
 ```toml
 [marketplace.facebook]
@@ -86,7 +87,7 @@ keywords = 'search word one'
 pushbullet_token = 'xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx'
 ```
 
-A more complete example is provided at [`example_config.toml`](example_config.toml), which allows more complex search and notification patterns. Briefly:
+A more complete example is provided at [`example_config.toml`](example_config.toml), which allows for more complex search and notification patterns. Briefly:
 
 - `marketplace.facebook` allows
 
@@ -96,7 +97,9 @@ A more complete example is provided at [`example_config.toml`](example_config.to
   - `max_search_interval`: (optional) maximum interval in minutes between searches
   - `search_city`: (optional if defined for item) search city, which can be obtained from the URL of your search query
   - `acceptable_locations`: (optional) only allow searched items from these locations
-  - `exclude_sellers`: (optional, not implemented yet) exclude certain sellers
+  - `exclude_sellers`: (optional) exclude certain sellers by their names (not username)
+  - `min_price`: (optional) minimum price.
+  - `max_price`: (optional) maximum price.
   - `notify`: (optional) users who should be notified for all items
 
 - `user.username` where `username` is the name listed in `notify`
@@ -108,6 +111,9 @@ A more complete example is provided at [`example_config.toml`](example_config.to
   - `marketplace`: (optional), can only be `facebook` if specified.
   - `exclude_keywords`: (optional), exclude item if the title contain any of the specified words
   - `exclude_sellers`: (optional, not implemented yet) exclude certain sellers
+  - `min_price`: (optional) minimum price.
+  - `max_price`: (optional) maximum price.
+  - `exclude_by_description`: (optional) exclude items with descriptions containing any of the specified words.
   - `notify`: (optional) users who should be notified for this item
 
 ### Run the program
@@ -118,16 +124,20 @@ Start monitoring with the command
 ai-marketplace-monitor
 ```
 
-You will need to specify the path to the configuration file if it is not named `config.toml`.
+or
+
+```
+ai-marketplace-monitor --config /path/to/config.toml
+```
 
 **NOTE**
 
 1. You need to keep the terminal running to allow the program to run indefinitely.
-2. You will see a browser firing up. **You may need to manually enter any prompt (e.g. CAPTCHA) that facebook asks for authentication** in addition to the username and password that the program enters for you. You may want to click "OK" for save password etc.
+2. You will see a browser firing up. **You may need to manually enter any prompt (e.g. CAPTCHA) that facebook asks for authentication** in addition to the username and password that the program enters for you. You may want to click "OK" to save the password, etc.
 
 ## Advanced features
 
-- A file `~/.ai-marketplace-monitor/config.yml`, if exists, will be read and merged to the specified configuration file. This allows you to save sensitive information like facebook username, password, and pushbullet token in a separate file.
+- A file `~/.ai-marketplace-monitor/config.yml`, if it exists, will be read and merged with the specified configuration file. This allows you to save sensitive information like Facebook username, password, and PushBullet token in a separate file.
 - Multiple configuration files can be specified to `--config`, which allows you to spread items into different files.
 
 ## Credits

example_config.toml

@@ -20,8 +20,10 @@ max_search_interval = 60
 acceptable_locations = ['city', 'surrounding city']
 # OPTIONAL, string or list of strings, can be specified for each search item
 notify = 'user1'
-# NOT-IMPLEMENTED, string or list of strings, can be specified for each search item
+# OPTIONAL, string or list of strings, can be specified for each search item
 exclude_sellers = []
+#
+exclude_by_description = []
 
 # REQUIRED for notification
 [notify.user1]
@@ -34,11 +36,11 @@ pushbullet_token = 'xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx'
 
 
 # OPTIONAL a product
-[query.item2]
+[item.name1]
 keywords = 'search word one'
 
 # Complete item
-[item.name1]
+[item.name2]
 # REUIRED string or list of strings
 keywords = ['search word one', 'search word two']
 
@@ -52,5 +54,7 @@ notify = 'user2'
 search_city = 'another city'
 # OPTIONAL
 acceptable_locations = ['another city', 'surrounding city']
-# NOT-IMPLEMENTED
+# OPTONAL
 exclude_sellers = []
+# OPTIONAL
+exclude_by_description = []

pyproject.toml

@@ -1,6 +1,6 @@
 [tool.poetry]
 name = "ai-marketplace-monitor"
-version = "0.1.0"
+version = "0.2.0"
 description = "An AI-based tool for monitoring facebook marketplace"
 authors = ["Bo Peng <[email protected]>"]
 readme = "README.md"

src/ai_marketplace_monitor/__init__.py

@@ -2,4 +2,4 @@
 
 __author__ = """Bo Peng"""
 __email__ = "[email protected]"
-__version__ = "0.1.0"
+__version__ = "0.2.0"

src/ai_marketplace_monitor/config.py

@@ -65,6 +65,29 @@ def validate_search_items(self) -> None:
                 raise ValueError(f"Item [blue]{item_name}[/blue] keywords must be a list.")
             if len(item_config["keywords"]) == 0:
                 raise ValueError(f"Item [blue]{item_name}[/blue] keywords list is empty.")
+
+            # exclude_sellers should be a list of strings
+            if "exclude_sellers" in item_config:
+                if isinstance(item_config["exclude_sellers"], str):
+                    item_config["exclude_sellers"] = [item_config["exclude_sellers"]]
+                if not isinstance(item_config["exclude_sellers"], list) or not all(
+                    isinstance(x, str) for x in item_config["exclude_sellers"]
+                ):
+                    raise ValueError(
+                        f"Item [blue]{item_name}[/blue] exclude_sellers must be a list."
+                    )
+            #
+            # exclude_by_description should be a list of strings
+            if "exclude_by_description" in item_config:
+                if isinstance(item_config["exclude_by_description"], str):
+                    item_config["exclude_by_description"] = [item_config["exclude_by_description"]]
+                if not isinstance(item_config["exclude_by_description"], list) or not all(
+                    isinstance(x, str) for x in item_config["exclude_by_description"]
+                ):
+                    raise ValueError(
+                        f"Item [blue]{item_name}[/blue] exclude_by_description must be a list."
+                    )
+
             # if there are other keys in item_config, raise an error
             for key in item_config:
                 if key not in [
@@ -73,6 +96,9 @@ def validate_search_items(self) -> None:
                     "notify",
                     "exclude_keywords",
                     "exclude_sellers",
+                    "min_price",
+                    "max_price",
+                    "exclude_by_description",
                 ]:
                     raise ValueError(
                         f"Item [blue]{item_name}[/blue] contains an invalid key {key}."

src/ai_marketplace_monitor/facebook.py

@@ -1,7 +1,7 @@
 import re
 import time
 from logging import Logger
-from typing import ClassVar, List
+from typing import ClassVar, Dict, List
 from urllib.parse import quote
 
 from bs4 import BeautifulSoup
@@ -112,7 +112,14 @@ def search(self, item_config) -> List[SearchedItem]:
                 [x for x in self.get_item_list(html) if self.filter_item(x, item_config)]
             )
             time.sleep(5)
-
+        # go to each item and get the description
+        for item in found_items:
+            self.page.goto(f'https://www.facebook.com{item["post_url"]}', timeout=0)
+            html = self.page.content()
+            item |= self.get_item_details(html)
+            time.sleep(5)
+        #
+        found_items = [x for x in found_items if self.filter_item_by_details(x, item_config)]
         # check if any of the items have been returned before
         return found_items
 
@@ -187,6 +194,7 @@ def get_listing_from_css():
                         "price": price,
                         "post_url": post_url,
                         "location": location,
+                        "seller": "",
                         "description": "",
                     }
                 )
@@ -196,6 +204,25 @@ def get_listing_from_css():
 
         return parsed
 
+    def get_item_details(self, html) -> Dict[str, str]:
+        soup = BeautifulSoup(html, "html.parser")
+        try:
+            cond = soup.find("span", string="Condition")
+            ul = cond.find_parent("ul")
+            description_div = ul.find_next_sibling()
+            description = description_div.get_text(strip=True)
+        except Exception as e:
+            self.logger.debug(e)
+            description = ""
+        #
+        try:
+            profiles = soup.find_all("a", href=re.compile(r"/marketplace/profile"))
+            seller = profiles[-1].get_text()
+        except Exception as e:
+            self.logger.debug(e)
+            description = ""
+        return {"description": description, "seller": seller}
+
     def filter_item(self, item: SearchedItem, item_config) -> bool:
         # get exclude_keywords from both item_config or config
         exclude_keywords = item_config.get(
@@ -225,3 +252,30 @@ def filter_item(self, item: SearchedItem, item_config) -> bool:
             return False
 
         return True
+
+    def filter_item_by_details(self, item: SearchedItem, item_config) -> bool:
+        # get exclude_keywords from both item_config or config
+        exclude_by_description = item_config.get("exclude_by_description", [])
+
+        if exclude_by_description and any(
+            [x.lower() in item["description"].lower() for x in exclude_by_description or []]
+        ):
+            self.logger.debug(
+                f"Excluding specifically listed item by description: [red]{exclude_by_description}[/red]"
+            )
+            return False
+
+        # get exclude_sellers from both item_config or config
+        exclude_sellers = item_config.get("exclude_sellers", []) + self.config.get(
+            "exclude_sellers", []
+        )
+
+        if exclude_sellers and any(
+            [x.lower() in item["seller"].lower() for x in exclude_sellers or []]
+        ):
+            self.logger.debug(
+                f"Excluding specifically listed item by seller: [red]{item['seller']}[/red]"
+            )
+            return False
+
+        return True

src/ai_marketplace_monitor/items.py

@@ -10,4 +10,5 @@ class SearchedItem(TypedDict):
     price: str
     post_url: str
     location: str
+    seller: str
     description: str

tests/test_ai_marketplace_monitor.py

@@ -15,4 +15,4 @@ def version() -> Generator[str, None, None]:
 
 def test_version(version: str) -> None:
     """Sample pytest test function with the pytest fixture as an argument."""
-    assert version == "0.1.0"
+    assert version == "0.2.0"