Skip to content

Commit

Permalink
made changes to docs for prompt governance (#457)
Browse files Browse the repository at this point in the history
* Documentation changes for prompt governance
  • Loading branch information
gr8nishan authored Aug 1, 2024
1 parent df58670 commit 1d66abc
Show file tree
Hide file tree
Showing 5 changed files with 39 additions and 16 deletions.
5 changes: 5 additions & 0 deletions docs/gh_pages/docs/retrieval_chain.md
Original file line number Diff line number Diff line change
Expand Up @@ -203,3 +203,8 @@ Question: Please share the performance summary for John Smith?
Answer:
I don't know, I'm sorry.
```

## Prompt Governance
When a user sends any prompt to the LLM using PebbloRetrieveQA, Pebblo captures the findings from that prompt. This behavior is enabled by default.

***Coming Soon***: Ability to block or anonymize prompts based on policy.
16 changes: 8 additions & 8 deletions docs/gh_pages/docs/safe_loader.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,12 +21,12 @@ This page consist of 4 primary tabs that provides the following details:
The number signifies the proportion of applications with findings out of the total active applications. Additionally, it will present you with a detailed list of these applications, including the count of findings (Topics + Entities), the name of the owner, and the option to download the PDF report for each application.

2. **Findings**:
The figure denotes the cumulative count of Topics and Entities identified across all applications. It will also furnish you with a comprehensive list of these Topics and Entities, along with supplementary information including the count of source files they originate from, the Datasource, and the name of the Application.
The figure denotes the cumulative count of Topics and Entities identified across all applications. It will also furnish you with a comprehensive list of these Topics and Entities, along with supplementary information including the count of source documents they originate from, the Datasource, and the name of the Application.

3. **Files with Findings**:
The number of files that has one or more Findings over the total number of files used in document load across all the applications. This field indicates the number of files that need to be inspected to remediate any potentially text that needs to be removed and/or cleaned for Gen-AI inference.
3. **Documents with Findings**:
The number of documents that has one or more Findings over the total number of documents used in document load across all the applications. This field indicates the number of documents that need to be inspected to remediate any potentially text that needs to be removed and/or cleaned for Gen-AI inference.

It will also provide you with a list of these files, accompanied by additional details such as the file size, the owner's name, the count of topics & entities within each file, and the name of the Datasource.
It will also provide you with a list of these documents, accompanied by additional details such as the file size, the owner's name, the count of topics & entities within each file, and the name of the Datasource.

4. **Datasource**:
The number of data sources used to load documents into the Gen-AI RAG applications. For e.g. this field will be two if a RAG application loads data from two different directories or two different AWS S3 buckets.
Expand All @@ -50,16 +50,16 @@ Load History provides details about latest 5 loads of this app. It provides the

1. **Report Name** - The path to the report file.
2. **Findings** - The number of findings identified in the report.
3. **Files With Findings** - The number of files containing findings.
3. **Documents With Findings** - The number of documents containing findings.
4. **Generated On** - The timestamp, when the report was generated. Time would be in local time zone.

**Report Summary**: Report Summary has 4 primary tabs:

1. **Findings**: The figure denotes the cumulative count of Topics and Entities identified in the application. It will also furnish you with a comprehensive list of these Topics and Entities, along with supplementary information including the count of source files they originate from, and the Datasource name.
1. **Findings**: The figure denotes the cumulative count of Topics and Entities identified in the application. It will also furnish you with a comprehensive list of these Topics and Entities, along with supplementary information including the count of source documents they originate from, and the Datasource name.

2. **Files with Findings**: The number of files that has one or more Findings over the total number of files used in document load across the application. This field indicates the number of files that need to be inspected to remediate any potentially text that needs to be removed and/or cleaned for Gen-AI inference.
2. **Documents with Findings**: The number of documents that has one or more Findings over the total number of documents used in document load across the application. This field indicates the number of documents that need to be inspected to remediate any potentially text that needs to be removed and/or cleaned for Gen-AI inference.

It will also provide you with a list of these files, accompanied by additional details such as the file size, the owner's name, the count of topics & entities within each file, and the name of the Datasource.
It will also provide you with a list of these documents, accompanied by additional details such as the file size, the owner's name, the count of topics & entities within each file, and the name of the Datasource.

3. **Datasource**: The number of data sources used to load documents into the Gen-AI RAG applications. For e.g. this field will be two if a RAG application loads data from two different directories or two different AWS S3 buckets.

Expand Down
34 changes: 26 additions & 8 deletions docs/gh_pages/docs/safe_retriever.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,11 +2,11 @@

This section provides details about the retrievals for all the retrieval based application.

![Pebblo UI](../static/img/pebblo-saferetriever-ui.jpeg)
![Pebblo UI](../static/img/pebblo-saferetriever-ui.png)

## Overview Page

This page consist of 4 primary tabs that provides the following details:
This page consist of 5 primary tabs that provides the following details:

1. **Applications**:
The number signifies total number of retrieval type applications.
Expand All @@ -16,10 +16,13 @@ This page consist of 4 primary tabs that provides the following details:
2. **Retrievals**:
The figure denotes the cumulative count of retrievals happened for all the apps.

3. **Active Users**:
3. **Prompts with Findings**:
The number of prompts that has one or more findings over the total number of prompts used during retrieval process. This field indicates the number of prompts that need to be inspected for Gen-AI inference.

4. **Active Users**:
The number of active users for all the retrieval type applications.

4. **Violations**:
5. **Violations**:
Coming Soon!

## Application Details Page
Expand All @@ -29,20 +32,35 @@ You will be directed to the application details page by clicking on any applicat
**Instance Details**:
This section provide a quick glance of where the RAG application is physically running like in a Laptop (Mac OSX) or Linux VM and related properties like IP address, local filesystem path and Python version.

This page consist of 4 primary tabs that provides the following details:
This page consist of 5 primary tabs that provides the following details:

1. **Retrievals**: The figure denotes the cumulative count of retrievals happened for this app.
This section provides details about all retrievals with information like

**Prompt** : What was the prompt sent by the end user, Who sent the prompt and when it was sent.

**Findings**: One or more key findings extracted from the prompt.

**Context** : What was the context and which vector db was used.

**Response**: This is the final response generate by LLM for the given prompt.

**Retrieved From** : Source file of this context.

2. **Active Users**: The number of active users for this application will be provided, accompanied by a list of these users. This list will include additional details such as the retrieval count for each user and the last time they accessed the application.
2. **Prompts with Findings**: The number of prompts that has one or more findings over the total number of prompts for the given application.

3. **Active Users**: The number of active users for this application will be provided, accompanied by a list of these users. This list will include additional details such as the retrieval count for each user and the last time they accessed the application.

3. **Documents**: The number of documents accessed for retrievals will be provided, accompanied by a list of these documents. This list will include supplementary details such as the owner's name, retrieval count for each document, and the most recent access time for information from each document.
4. **Documents**: The number of documents accessed for retrievals will be provided, accompanied by a list of these documents. This list will include supplementary details such as the owner's name, retrieval count for each document, and the most recent access time for information from each document.

4. **Vector Databases**: The number of vector databases used in retrievals for this application.
5. **Vector Databases**: The number of vector databases used in retrievals for this application.
Details within this tab are coming soon.

## Prompt With Findings Details Page

This page shows all the prompt with findings for all the applications.

1. **Entity Name**: The name of the entity which is present in the prompts.
2. **Prompts**: Number of prompts in which the entity was detected.
3. **Users**: Users who have used this entity in the prompt.
4. **Apps** : Application in which the entity is present.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.

0 comments on commit 1d66abc

Please sign in to comment.