✨ Assistant in desktop client & mobile apps

[jancborchardt]

It would be nice to have the functionality of the Assistant modal also available on the desktop and on the go in the mobile apps.

• Mobile apps: Could be a navigation entry "Assistant"
• Desktop client: Possibly an icon in the header bar of the desktop client tray menu

[jancborchardt]

This needs discussion on how to technically integrate, e.g. which APIs are there and/or which we need. cc @julien-nc @tobiasKaminsky :)

[tobiasKaminsky]

Quick idea

• results are notifications
• with new screen filter all results by app "assistant", so that only relevant notifications are shown

[tobiasKaminsky]

Mobile

• add as first item in "talk, notes, more"
• for branded clients
  • Android: in drawer, below "notifications"
  • iOS: as first item in "more"

Desktop:

• move "talk" to three dot menu
• add assistant
• have a prompt, see result, same like on Mobile

For all:

• have first type selected
• result is in notification id
• should be extracted and shown directly
• show history
  • see integration issue
  • retrieve task list for specific task
  • show list, last 5, then "load more"
  • possible to delete

[julien-nc]

Endpoints available

• get available task types:
  GET /ocs/v2.php/textprocessing/tasktypes
  This returns the types that have at least one provider.
• schedule a task:
  POST /ocs/v2.php/textprocessing/schedule with params:
  • string input
  • string type, the task type id
  • string appId, use "assistant" there
  • string identifier = '', something to identify a task later, leave it empty if not needed (used by text to know in which document a task was created)
• get a task:
  GET /ocs/v2.php/textprocessing/task/{id}
  This returns a serialized task with the output and the status.
• delete a task:
  DELETE /ocs/v2.php/textprocessing/task/{id}
• list tasks by app:
  GET /ocs/v2.php/textprocessing/tasks/app/{appId}
  You only want the assistant tasks.

Task statuses

• STATUS_FAILED = 4
• STATUS_SUCCESSFUL = 3
• STATUS_RUNNING = 2
• STATUS_SCHEDULED = 1
• STATUS_UNKNOWN = 0

Capability

I will add a capability to tell the clients if the assistant is enabled for the current user. It will probably be assistant.enabled: boolean.

Handling result notifications

Result notifications have one action button pointing to /apps/assistant/t/{taskId}. This is a page displaying the task result. This page can only be accessed by authenticated users.
Instead of allowing client users to browse this page, you could extract the task ID from the button target URL and get the task with the /ocs/v2.php/textprocessing/task/{id} OCS endpoint to show the result directly in the client.

@tobiasKaminsky Anything else you need to know?

[tobiasKaminsky]

Looks great so far 👍

[alperozturk96]

Design suggestion @jancborchardt

[jancborchardt]

@alperozturk96 the design should best follow the web interface (of course with Material Design and Android components), since otherwise we have different designs for the same thing. :)

Or are you saying we should also improve this in the web interface? cc @julien-nc @nimishavijay

[alperozturk96]

@alperozturk96 the design should best follow the web interface (of course with Material Design and Android components), since otherwise we have different designs for the same thing. :)

Or are you saying we should also improve this in the web interface? cc @julien-nc @nimishavijay

We should have same design language but we should't do exactly same thing on mobile. Mobile devices have small screen UX/UI needs to change according to that. BottomSheet, AlertDialog etc all these things are helping user to tackle limitation of screen size.

For example are web pages have bottom sheet? No. In web we are showing buttons next to each other because we have huge wide screen. So that you can display Free prompt, Summarize etc next to each other but on mobile devices its impossible, we must use Dropdown menu.

On the other hand Input section should't visible to the user all the time because years of user experience will resist this. Almost all apps have same pattern (Apple Human Interface Guideline, Android Design) show empty list then only have one add button then filter list etc.

In short we must not fight against native iOS and Android nature.

[mpivchev]

For iOS: "add as first item in "talk, notes, more""
I don't think this is a good place for this, it's way too hidden. Should be either as a separate icon on the top navigation bar, or at least in the dropdown menu. @marinofaggiana what do you think?

[marinofaggiana]

mmm I honestly don't know how many will use it ... for me better in "add as first item in "talk, notes, more""

[camilasan]

• have a prompt, see result, same like on Mobile

I don't understand what you mean @tobiasKaminsky.

@jancborchardt
wdyt?

If only Talk is available then it will be displayed as it is now.

@julien-nc
is it possible to have a direct link to open the assistant in the browser?

[camilasan]

The desktop PR: nextcloud/desktop#6637

[julien-nc]

is it possible to have a direct link to open the assistant in the browser?

@camilasan There is a page to see the results of a task: apps/assistant/task/view/TASK_ID but there is nothing to just open it. It's a nice idea, we'll add that soon.

[mpivchev]

iOS PR: nextcloud/ios#2881

[camilasan]

@camilasan There is a page to see the results of a task: apps/assistant/task/view/TASK_ID but there is nothing to just open it. It's a nice idea, we'll add that soon.

how soon?

[julien-nc]

@camilasan It's there but not released nextcloud/assistant#72
If you want to know if this page is available, you can check the 'assistant.version' capability (exposed by the assistant) is >= 1.0.9.