Merge pull request #28 from bruin-data/feature/introduce-slack-source

Introduce slack as a source

README.md

@@ -10,7 +10,7 @@
   </a>
 </div>
 
------
+---
 
 Ingestr is a command-line application that allows you to ingest data from any source into any destination using simple command-line flags, no code necessary.
 
@@ -20,8 +20,8 @@ Ingestr is a command-line application that allows you to ingest data from any so
 
 ingestr takes away the complexity of managing any backend or writing any code for ingesting data, simply run the command and watch the data land on its destination.
 
-
 ## Installation
+
 ```
 pip install ingestr
 ```
@@ -39,15 +39,17 @@ ingestr ingest \
 That's it.
 
 This command will:
+
 - get the table `public.some_data` from the Postgres instance.
 - upload this data to your BigQuery warehouse under the schema `ingestr` and table `some_data`.
 
 ## Documentation
+
 You can see the full documentation [here](https://bruin-data.github.io/ingestr/getting-started/quickstart.html).
 
 ## Community
-Join our Slack community [here](https://join.slack.com/t/bruindatacommunity/shared_invite/zt-2dl2i8foy-bVsuMUauHeN9M2laVm3ZVg).
 
+Join our Slack community [here](https://join.slack.com/t/bruindatacommunity/shared_invite/zt-2dl2i8foy-bVsuMUauHeN9M2laVm3ZVg).
 
 ## Supported Sources & Destinations
 
@@ -157,6 +159,11 @@ Join our Slack community [here](https://join.slack.com/t/bruindatacommunity/shar
         <td>Shopify</td>
         <td>✅</td>
         <td>-</td>
+    </tr>
+     <tr>
+        <td>Slack</td>
+        <td>✅</td>
+        <td>-</td>
     </tr>
     <tr>
         <td>Stripe</td>
@@ -168,4 +175,5 @@ Join our Slack community [here](https://join.slack.com/t/bruindatacommunity/shar
 More to come soon!
 
 ## Acknowledgements
-This project would not have been possible without the amazing work done by the [SQLAlchemy](https://www.sqlalchemy.org/) and [dlt](https://dlthub.com/) teams. We relied on their work to connect to various sources and destinations, and built `ingestr` as a simple, opinionated wrapper around their work.
+
+This project would not have been possible without the amazing work done by the [SQLAlchemy](https://www.sqlalchemy.org/) and [dlt](https://dlthub.com/) teams. We relied on their work to connect to various sources and destinations, and built `ingestr` as a simple, opinionated wrapper around their work.

docs/.vitepress/config.mjs

@@ -89,6 +89,7 @@ export default defineConfig({
               { text: "HubSpot", link: "/supported-sources/hubspot.md" },
               { text: "Notion", link: "/supported-sources/notion.md" },
               { text: "Shopify", link: "/supported-sources/shopify.md" },
+              { text: "Slack", link: "/supported-sources/slack.md" },
               { text: "Stripe", link: "/supported-sources/stripe.md" },
             ],
           },

docs/supported-sources/slack.md

@@ -0,0 +1,42 @@
+# Slack
+
+[Slack](https://www.Slack.com/) is a messaging platform for teams and organizations where they can collaborate, share ideas and information.
+
+ingestr supports Slack as a source.
+
+## URI Format
+
+The URI format for Slack is as follows:
+
+```plaintext
+slack://?api_key=<api-key-here>
+```
+
+URI parameters:
+
+- `api_key`: The API key used for authentication with the Slack API.
+
+The URI is used to connect to the Slack API for extracting data.
+
+## Setting up a Slack Integration
+
+Slack requires a few steps to set up an integration, please follow the guide dltHub [has built here](https://dlthub.com/docs/dlt-ecosystem/verified-sources/Slack#setup-guide).
+
+Once you complete the guide, you should have an API key with the necessary permissions as mentioned in the guide. Let's say your API key is axb-test-564. Here's a sample command that will copy the data from Slack into a DuckDB database:
+
+```sh
+ingestr ingest --source-uri 'slack://?api_key=axb-test-564' --source-table 'channels' --dest-uri duckdb:///slack.duckdb --dest-table 'dest.channels'
+```
+
+The result of this command will be a table in the `slack.duckdb` database.
+
+## Available Tables
+
+Slack source allows ingesting the following sources into separate tables:
+
+- `channels`: Retrieves information about all the channels.
+- `users`: Retrieves information about all the users.
+- `messages:chan1,chan2`: Retrieves messages from specified channels, where chan1 and chan2 represent user-defined channels (e.g: general, memes). Multiple channels can be listed.
+- `access_logs`: Retrieves all the access logs.
+
+Use these as `--source-table` parameter in the `ingestr ingest` command.

ingestr/main.py

@@ -1,7 +1,7 @@
 import hashlib
+import tempfile
 from datetime import datetime
 from enum import Enum
-import tempfile
 from typing import Optional
 
 import dlt
@@ -413,6 +413,7 @@ def ingest(
         # remove the pipelines_dir folder if it was created by ingestr
         if is_pipelines_dir_temp:
             import shutil
+
             shutil.rmtree(pipelines_dir)
 
         print(

ingestr/src/factory.py

@@ -23,6 +23,7 @@
     MongoDbSource,
     NotionSource,
     ShopifySource,
+    SlackSource,
     SqlSource,
     StripeAnalyticsSource,
 )
@@ -109,8 +110,11 @@ def get_source(self) -> SourceProtocol:
             return ChessSource()
         elif self.source_scheme == "stripe":
             return StripeAnalyticsSource()
+        elif self.source_scheme == "slack":
+            return SlackSource()
         elif self.source_scheme == "hubspot":
             return HubspotSource()
+
         else:
             raise ValueError(f"Unsupported source scheme: {self.source_scheme}")