Truteq transport should fail gracefully on character sets it cannot display.

[smn]

Searching for Tubercolosis results in the following results:

"1. Tuberculosis\\n2. List of aquarium diseases\\n3. Petrus Augustus de Genestet\\n4. Eriks \\u0100damsons\\n5. Deodato Arellano\\n6. Peyo Yavorov"

This results in the following error because:

exceptions.UnicodeEncodeError: 'latin-1' codec can't encode character u'\u0100' in position 85: ordinal not in range(256)

Full traceback:

2014-05-19 06:42:00+0000 [VumiRedis,client] Unhandled Error
        Traceback (most recent call last):
          File "/var/praekelt/vumi-go/ve/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 382, in callback
            self._startRunCallbacks(result)
          File "/var/praekelt/vumi-go/ve/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 490, in _startRunCallbacks
            self._runCallbacks()
          File "/var/praekelt/vumi-go/ve/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 577, in _runCallbacks
            current.result = callback(current.result, *args, **kw)
          File "/var/praekelt/vumi-go/ve/src/vumi/vumi/transports/base.py", line 178, in processor
            d = maybeDeferred(handler, message)
        --- <exception caught here> ---
          File "/var/praekelt/vumi-go/ve/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 139, in maybeDeferred
            result = f(*args, **kw)
          File "/var/praekelt/vumi-go/ve/src/vumi/vumi/transports/truteq/truteq.py", line 185, in handle_outbound_message
            text = '\n'.join(text.splitlines()).encode(self.encoding)
        exceptions.UnicodeEncodeError: 'latin-1' codec can't encode character u'\u0100' in position 85: ordinal not in range(256)

We need to fail more gracefully here because timing out is horrible UX. My suggestion is to use unidecode when we cannot encode in Latin-1.

[smn]

Even foo fails:

2014-05-29 06:15:10+0000 [WorkerAMQClient,client] Processed outbound message for truteq_8864_transport: {"transport_name": "truteq_8864_transport", "in_reply_to": "8fd25db18d6148a78f3411e432ecb46c", "group": null, "from_addr": "*120*8864#", "timestamp": "2014-05-29 06:15:09.838937", "to_addr": "+27764493806", "content": "The terms foobar (/\u02c8fu\u02d0b\u0251r/), fubar, ...\n(Full content sent by SMS.)", "routing_metadata": {"go_hops": [[["CONVERSATION:wikipedia:a33195366a3a4e56957e00965e570114", "default"], ["ROUTER:app_multiplexer:067cdceb8bde4ea9af3cc3b9a244c4aa:OUTBOUND", "wikipedia"]], [["ROUTER:app_multiplexer:067cdceb8bde4ea9af3cc3b9a244c4aa:INBOUND", "default"], ["BILLING:OUTBOUND", "default"]], [["BILLING:INBOUND", "default"], ["TRANSPORT_TAG:vumi_base_starcode:*120*8864#", "default"]]], "endpoint_name": "default"}, "message_version": "20110921", "transport_type": "ussd", "helper_metadata": {"go": {"conversation_type": "wikipedia", "user_account": "5fcd799a47eb46499c0af6d7ee173486", "is_paid": true, "router_key": "067cdceb8bde4ea9af3cc3b9a244c4aa", "router_type": "app_multiplexer", "conversation_key": "a33195366a3a4e56957e00965e570114"}, "tag": {"tag": ["vumi_base_starcode", "*120*8864#"]}, "truteq": {"genfields": {}}, "optout": {"optout": false}}, "transport_metadata": {}, "session_event": "close", "message_id": "623ce695561740729a088271ca4adb08", "message_type": "user_message"}
2014-05-29 06:15:10+0000 [VumiRedis,client] Unhandled Error
        Traceback (most recent call last):
          File "/var/praekelt/vumi-go/ve/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 382, in callback
            self._startRunCallbacks(result)
          File "/var/praekelt/vumi-go/ve/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 490, in _startRunCallbacks
            self._runCallbacks()
          File "/var/praekelt/vumi-go/ve/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 577, in _runCallbacks
            current.result = callback(current.result, *args, **kw)
          File "/var/praekelt/vumi-go/ve/src/vumi/vumi/transports/base.py", line 178, in processor
            d = maybeDeferred(handler, message)
        --- <exception caught here> ---
          File "/var/praekelt/vumi-go/ve/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 139, in maybeDeferred
            result = f(*args, **kw)
          File "/var/praekelt/vumi-go/ve/src/vumi/vumi/transports/truteq/truteq.py", line 185, in handle_outbound_message
            text = '\n'.join(text.splitlines()).encode(self.encoding)
        exceptions.UnicodeEncodeError: 'latin-1' codec can't encode character u'\u02c8' in position 19: ordinal not in range(256)

[hodgestar]

Failed again today.

Not sending out a response over USSD is terrible UI and the transport desperately needs an option to handle this better. Perhaps we need to nack the message and also have an option for sending a degraded response?