Skip to content

Dev email

Jump to bottom
Joshua Haas edited this page Jul 22, 2017 · 3 revisions

Here we will be writing a protocol so sibyl can talk over e-mail. This will be a simplified version of protocols/sibyl_email.py that we will develop step by step. This version will be missing some config options and checks present in the full version.

Goal

We want to control Sibyl via e-mail. Users should be able to send a command to Sibyl and get a reply. This will be handy to have for library searches that respond with a wall of text, or for users who don't have a supported chat app on their phone but still want to issue Sibyl commands from it.

File and Overview

We will be editing a file called sibyl_mail.py which will be located in the protocols/ directory. In order to write a protocol, we need to sub-class Protocol, User, and Room from the sibyl.lib.protocol module. To make that easier, you should start from protocols/skeleton.py, which you should copy and rename to protocols/sibyl_mail.py. For more details see the coding protocols page.

First, this protocol requires an external e-mail account with remote SMTP and IMAP enabled. This is easy to do with pretty much any provider, including the major ones like Gmail or Hotmail. Making a new account for sibyl's exclusive use is probably a good idea. Sending e-mail with our protocol is done using SMTP via the smtplib module. Doing so is straightforward; we just have to build a message, connect to the server, send it, and disconnect.

For receiving messages, we'll by using IMAP from the imaplib module. We could just poll the server and check for new messages every minute or so, but then messages sent to Sibyl via e-mail would take forever to execute. Further, many IMAP servers do not allow you to connect more often than once a minute. However, rather than check for new messages every so often, we can open an IMAP connection and ask the server to notify us as soon as new mail arrives. Doing so halts execution, so we'll have to run the IMAP listener in its own thread.

Exceptions

We'll need to raise exceptions at several points in our protocol so sibyl can deal with disconnects and other important events correctly. If you weren't editing skeleton.py you would still need a few lines of code from it. Specifically, the imports and class definitions that make exceptions protocol specific. When you raise one of these exceptions, it will then be initialised with the name of your protocol so Sibyl knows where it came from.

from sibyl.lib.protocol import ProtocolError as SuperProtocolError
from sibyl.lib.protocol import PingTimeout as SuperPingTimeout
from sibyl.lib.protocol import ConnectFailure as SuperConnectFailure
from sibyl.lib.protocol import AuthFailure as SuperAuthFailure
from sibyl.lib.protocol import ServerShutdown as SuperServerShutdown

class ProtocolError(SuperProtocolError):
  def __init__(self):
    self.protocol = __name__.split('_')[-1]

class PingTimeout(SuperPingTimeout,ProtocolError):
  pass

class ConnectFailure(SuperConnectFailure,ProtocolError):
  pass

class AuthFailure(SuperAuthFailure,ProtocolError):
  pass

class ServerShutdown(SuperServerShutdown,ProtocolError):
  pass

Config Options

Protocols can add config options just like plug-ins, using the @botconf decorator. For more details see the Adding Config Options section on the Plug-Ins page. We'll need a few config options:

  • address - the e-mail address for sibyl to use (required)
  • password - the password to login to the account (required)
  • imap - the IMAP server to use (default inferred from address)
  • smtp - the SMTP server to use (default inferred from address)
from sibyl.lib.decorators import botconf

@botconf
def conf(bot):
  return [
    {'name':'address','req':True},
    {'name':'password','req':True},
    {'name':'imap'},
    {'name':'smtp'},
  ]

User Sub-class

We'll need to make our own User sub-class for holding e-mail addresses. Other protocols may need to care about nick names and resources, but for us the only piece of information that matters is the e-mail address itself. We need to override every method in skeleton.py, which corresponds to overriding every @abstractmethod in protocol.py.

from sibyl.lib.protocol import User

# here we make our MailUser class inherit methods from User
class MailUser(User):

  # in our case, the user is just a str, so we can just store it
  def parse(self,user):
    self.user = user

  # the "name" of an e-mail address is always just the address itself
  def get_name(self):
    return self.user

  # e-mail doesn't keep track of resources or separate devices
  def get_base(self):
    return self.user

  # MailUsers are equal if their user variables are equal
  def __eq__(self,other):
    if not isinstance(other,MailUser):
      return False
    return self.user==other.user

  # again, the string version of a MailUser is just its user
  def __str__(self):
    return self.user

Room Sub-class

Very similar to our User class, except e-mail doesn't have rooms. We still need to have a working Room sub-class so Sibyl and plug-ins work correctly, but the room-related methods in our Protocol sub-class won't actually do anything meaningful.

from sibyl.lib.protocol import Room

# make our MailRoom class inherit from the Room class
class MailRoom(Room):

  # we'll just store the name like we did with MailUser
  def parse(self,name):
    self.name = name

  # just return the stored name
  def get_name(self):
    return self.name

  # two MailRooms are only equal if their name variables are equal
  def __eq__(self,other):
    if not isinstance(other,MailRoom):
      return False
    return self.name==other.name

IMAP Thread

Before we get to our Protocol sub-class, let's create the separate thread for listening for new messages we mentioned in the Overview. If you aren't familiar with threading, you should check out the Threading Tutorial. We'll be sub-classing the Thread class from the threading module. First, we need to keep track of a few things:

  • self.proto - the Protocol object that created the thread
  • self.imap - our IMAP object created with the imaplib module
  • self.msgs - a thread-safe Queue for storing/getting messages asynchronously
from threading import Thread
from Queue import Queue

class IMAPThread(Thread):

  def __init__(self,proto):

    # we have to call the original __init__() method from the Thread class first
    super(IMAPThread,self).__init__()

    # this settings means this thread will exit if the main thread does
    self.daemon = True

    # instance variables explained above
    self.proto = proto
    self.imap = None
    self.msgs = Queue()

We'll be using the IMAP IDLE functionality as described in the IMAP specification (rfc2177). The python adaptation below was created using info from this StackOverflow answer. In order to make this easier, we'll want a function for sending IMAP commands to the remote server.

  def cmd(self,s):

    self.imap.send("%s %s\r\n"%(self.imap._new_tag(),s))

Next let's define a connect() method; our protocol will have to call this method to login to the e-mail account and wait for new messages. At the end of this function, we'll issue an IMAP IDLE command to tell the server we want to be notified when new mail arrives.

import imaplib

  def connect(self):

    # create a new IMAP connection and try to login
    self.imap = imaplib.IMAP4_SSL(self.proto.imap_serv)
    self.imap.login(self.proto.opt('mail.address'),self.proto.opt('mail.password'))

    # we need to choose an inbox before anything else; we'll pick the default
    self.imap.select()

    # tell the server to notify us when new mail arrives
    self.cmd('IDLE')

However, we're missing something important here; what happens if we can't connect to the server, or our password is wrong? Our calls to imaplib will raise exceptions that we need to catch and translate into exceptions Sibyl understands. Now, if we can't connect to the server we'll raise a ConnectFailure exception, and if login fails we'll raise AuthFailure.

import imaplib

  def connect(self):

    # try to connect to the IMAP server
    try:
      self.imap = imaplib.IMAP4_SSL(self.proto.imap_serv)
    except:
      raise ConnectFailure

    # try to login with our password
    try:
      self.imap.login(self.proto.opt('mail.address'),
          self.proto.opt('mail.password'))
    except:
      raise AuthFailure

    self.imap.select()
    self.cmd('IDLE')

The main logic for our thread will be in its run() method, which will be called when we do IMAPThread().start() to run it asynchronously. This is just going to be a big while loop that runs forever, waiting for new mail to arrive. When it receives a new message, it will put it in self.msgs, for later retrieval by our Protocol sub-class.

We'll call self.imap.readline() which will block until the server sends us a response that will be stored in line. If line has the string 'EXISTS' in it, then new mail has arrived. Any other messages from the server we'll just discard. If line is blank, it means our connection timed out. This is normal when using IMAP-IDLE, since connections only have a lifespan of 30 minutes. So if this happens, we'll just try to reconnect.

import email

  def run(self):

    while True:

      # wait for the server to send us a new notification
      line = self.imap.readline().strip()

      # if the line is blank, the server closed the connection
      if not line:
        self.connect()

      # if the line ends with "EXISTS" then there is a new message waiting
      elif line.endswith('EXISTS'):

        # to end the IDLE state and actually get the message we issue "DONE"
        self.cmd('DONE')

        # find all new messages (those flagged as 'UNSEEN')
        (status,nums) = self.imap.search('utf8','UNSEEN')

        # we have a list of message IDs, now we need to get their content
        for n in nums[0].split(' '):

          # we fetch the message from the server, in the RFC822 format
          msg = self.imap.fetch(n,'(RFC822)')[1][0][1]

          # then convert it to an email.Message object and store it in our Queue
          self.msgs.put(email.message_from_string(msg))

        # after we get the new message(s) and Queue them, we enter IDLE again
        self.cmd('IDLE')

It's also possible that a blank line is due to an actualy connection issue rather than a simple timeout. In that case, since we're calling connect() in here, we need to handle any exceptions it might raise. If we don't, they'll likely kill the thread without letting our protocol know. As explained in the Threading Tutorial, in python the main thread never sees exceptions raised in other threads. So instead, we'll catch any exceptions and just add them to self.msgs. Then in our Protocol sub-class we'll check to see if we get a Message or Exception when we access the Queue and handle it accordingly.

import email

  def run(self):

    while True:

      line = self.imap.readline().strip()
      if not line:

        # catch ConnectFailure and AuthFailure when attempting to connect
        try:
          self.connect()
        except ProtocolError as e:
          self.msgs.put(e)

      elif line.endswith('EXISTS'):
        self.cmd('DONE')
        (status,nums) = self.imap.search('utf8','UNSEEN')
        for n in nums[0].split(' '):
          msg = self.imap.fetch(n,'(RFC822)')[1][0][1]
          self.msgs.put(email.message_from_string(msg))
        self.cmd('IDLE')

Finally, let's have our Protocol class handle reconnection by calling our thread's connect() method. When we catch an Exception in run() we'll also set self.imap = None. When our Protocol tries to reconnect and calls our thread's connect() method and it succeeds, self.imap will then contain the imaplib.IMAP4_SSL object. This means that if our thread disconnects, it will sit idle in time.sleep() until our protocol reconnects it.

import email,time

  def run(self):

    while True:

      # wait for reconnection if needed
      if not self.imap:
        time.sleep(1)
        continue

      line = self.imap.readline().strip()
      if not line:
        try:
          self.connect()
        except ProtocolError as e:
          self.msgs.put(e)

          # update self.imap to wait for reconnection
          self.imap = None

      elif line.endswith('EXISTS'):
        self.cmd('DONE')
        (status,nums) = self.imap.search('utf8','UNSEEN')
        for n in nums[0].split(' '):
          msg = self.imap.fetch(n,'(RFC822)')[1][0][1]
          self.msgs.put(email.message_from_string(msg))
        self.cmd('IDLE')

Protocol Sub-class

Now we need to sub-class Protocol which is what Sibyl is actually going to instantiate and call methods from. During bot startup, Sibyl will call the setup() method shortly followed by the connect() method. Then the bot sits in a loop calling process() every so often to handle new messages.

First we'll do the setup() method and define some variables. We also need to create an IMAPThread and start it with start(). Because of the idling logic we put at the beginning of the thread's run() method, it won't try to do anything before we call connect() on it.

  • self.smtp_serv - server address to use for SMTP
  • self.imap_serv - server address to use for IMAP
  • self.smtp - our SMTP object from smtplib
  • self.thread - our IMAPThread instance
from sibyl.lib.protocol import Protocol

class MailProtocol(Protocol):

  def setup(self):

    # create default values if needed for SMTP and IMAP servers
    server = self.opt('mail.address').split('@')[-1]
    self.smtp_serv = (self.opt('mail.smtp') or ('smtp.'+server))
    self.imap_serv = (self.opt('mail.imap') or ('imap.'+server))

    self.smtp = None

    # create a new IMAPThread and start it
    self.thread = IMAPThread(self)
    self.thread.start()

Next we'll do connect(), where we'll attempt to connect to SMTP and IMAP and raise exceptions as needed. We should also have some helpful log messages so if something goes wrong the user (or a developer) has a chance to figure it out. To make things easier in the future, we'll give connecting to SMTP its own function called _connect_smtp(). The leading underscore is just for convenience; in this case it just helps us remember it's our own function and not required by sibyl.lib.protocol.Protocol.

  def connect(self):

    # our thread's connect() method already catches and re-raises exceptions
    self.log.debug('Attempting IMAP connection')
    self.thread.connect()
    self.log.info('IMAP successful')

    self.log.debug('Attempting SMTP connection')
    self._connect_smtp()
    self.log.info('SMTP successful')

  def _connect_smtp(self):

    # we have to catch and re-raise exceptions here
    try:
      self.smtp = smtplib.SMTP(self.smtp_serv,port=587)

      # we'll use TLS for security and encryption
      self.smtp.starttls()
      self.smtp.ehlo()

    except:
      raise ConnectFailure

    # if the protocol raises AuthFailure, Sibyl will never try to reconnect
    try:
      self.smtp.login(self.opt('mail.address'),self.opt('mail.password'))
    except:
      raise AuthFailure

Next a few easy ones. For is_connected() we'll just use the state of our IMAP thread's imap object. The shutdown() method is called by Sibyl when the bot is quitting, but in our case we don't need it for anything.

  def is_connected(self):

    return self.thread.imap is not None

  def shutdown(self):

    pass

Now we're at the most important function, process(), which Sibyl calls about once a second to process new messages and execute commands. Here we need to check if there are any new messages in our thread's Queue, translate them into sibyl.lib.protocol.Message objects, and send them on to Sibyl. We also need to check if items we get from the Queue are exceptions because of how we implemented that in our IMAPThread class.

from sibyl.lib.protocol import Message

  def process(self):

    # go through every message in our thread's Queue
    while not self.thread.msgs.empty():

      mail = self.thread.msgs.get()

      # if the message is actually an exception, raise it
      if isinstance(mail,Exception):
        raise mail

      # get the sender
      frm = email.utils.parseaddr(mail['From'])[1]
      user = MailUser(self,frm)

      # get the body, throwing out everything except the first line
      body = mail.get_payload().split('\n')[0].strip()

      # create the message and send it to Sibyl
      msg = Message(user,body)
      self.log.debug('Got mail from "%s"' % frm)
      self.bot._cb_message(msg)

Equally important is the send() methods, which lets Sibyl reply. The only real caveat here is that we need to check if our SMTP connection is still open, since like IMAP it has a timeout.

from email.mime.text import MIMEText

  def send(self,mess):

    # test our SMTP connection using noop()
    try:
      status = self.smtp.noop()[0]
    except:
      status = -1

    # a code of 250 means everything is good; otherwise reconnect
    if status!=250:
      self._connect_smtp()

    # build the message using the email module and send it over SMTP
    msg = MIMEText(mess.get_text())
    msg['Subject'] = 'Sibyl reply'
    msg['From'] = self.opt('mail.address')
    msg['To'] = str(mess.get_to())
    self.smtp.sendmail(msg['From'],msg['To'],msg.as_string())

Now we've got a series of functions that deal specifically with rooms. Since e-mail doesn't have rooms, most of these will just be pass. For those that need to return something, it will be some sane empty value, for example an empty list. Notable is join_room(), which always notifies sibyl of failure using _cb_join_room_failure().

  def broadcast(self,mess):
    pass

  def join_room(self,room):
    bot._cb_join_room_failure(room,'Not supported')

  def part_room(self,room):
    pass

  def _get_rooms(self,flag):
    return []

  def get_occupants(self,room):
    return []

  def get_nick(self,room):
    return ''

  def get_real(self,room,nick):
    return nick

Next we need a way for Sibyl or its plug-ins to get our username.

  def get_user(self):
    return MailUser(self,self.opt('mail.address'))

Finally, Sibyl and its plug-ins need a way to create MailUser and MailRoom. These methods almost always look the same in every protocol; they're necessary because the Protocol super-class doesn't have access to our e-mail sub-classes.

  def new_user(self,user,typ=None,real=None):
    return MailUser(self,user,typ,real)

  def new_room(self,name,nick=None,pword=None):
    return MailRoom(self,name,nick,pword)

Putting It All Together

One final note: if you haven't done so already, you should keep all of your imports at the top of your file for organization and clarity. Here is a compiled list of the imports to make it easier:

import imaplib,smtplib,email,time
from email.mime.text import MIMEText
from threading import Thread
from Queue import Queue

from sibyl.lib.protocol import User,Room,Protocol,Message
from sibyl.lib.decorators import botconf

from sibyl.lib.protocol import ProtocolError as SuperProtocolError
from sibyl.lib.protocol import PingTimeout as SuperPingTimeout
from sibyl.lib.protocol import ConnectFailure as SuperConnectFailure
from sibyl.lib.protocol import AuthFailure as SuperAuthFailure
from sibyl.lib.protocol import ServerShutdown as SuperServerShutdown

If you want to test your comprehension, try going through example/sibyl_mail.py and writing a comment for every line. If you really know what everything is doing, you or your comments should be able to explain it to another python coder.

And now we're done! Just copy example/sibyl_mail.py or the file you've been working on to the protocols/ directory and add mail to your list of enabled protocols. (You could even have email and mail enabled at the same time if you wanted.) Then set the config options, remembering they need to start with mail. (e.g. mail.password) and start Sibyl. Of course, you also need a valid e-mail address for sibyl to use.

Home
Commands
--- Bookmark
--- General
--- Library
--- Note
--- Room
--- XBMC
Config
--- Black/White List
Protocols
--- Email
--- Matrix
--- XMPP
--- CLI
--- Socket
Troubleshooting
--- FAQ
Development
--- Config
--- Decorators
--- Examples
--- Plug-Ins
--- Protocols
--- SibylBot
--- Unit Tests

Clone this wiki locally