Garbled output on Telnet Client

[Robatronic]

I'm trying to write a wrapper around Windows console apps that will allow me to connect to their stdio from network/remote clients. It is a VERY pared down concept of what Terminado does.

Each new connection gets a "reader" task that writes directly to the subprocess stdin and there is a "writer" class that monitors a global queue of pty output and forwards the bytes on to all connected clients. The connected clients are a list of writers in a global variable as is the PTY_PROC object.

There is also the pty_reader, that takes text from the PTY and puts it on the global byte queue.

I am testing using Telnet. I get some good data to the client, but some of it is garbled up. Mainly:

1. when I type in characters, they are doubled up ("ddiirr")
2. The prompt doesn't always line up properly with the edge of the screen
3. Some times escape codes are being printed.
4. It's possible to navigate all over the screen which the use of winpty is meant to restrict

So far, I've tried matching columns (80) and rows (24) and have been looking in to different terminal settings (i.e VT100, etc), but to no avail.

I'm hoping it's something simple and "obvious".

The core essentials (I believe) are here:

async def client_reader(reader, addr = ''):
# Monitor incoming data and put the bytes in to the subproc stdin
    cmd = ''

    while True:
        try:
            s = await reader.read(1024)
        except asyncio.CancelledError as e:
            break

        PTY_PROC.write(s.decode())

        print(f"client_reader ({addr}): Cancelled and tidying up...")

        return(f"client_reader ({addr})")

async def write_clients():
# Watches the global CLIENTS_QUEUE and writes bytes out to
# all the clients in "WRITERS"

    data = None

    while True:
        try:
            data = await CLIENTS_QUEUE.get()

            if not data:
                break

            for writer in WRITERS:
                if data != None:
                    writer.write(data.encode())
                    await writer.drain()

        except asyncio.CancelledError:
            break

    for writer in WRITERS:
    # flush existing data to clients
        if data != None:
            writer.write(data.encode())
        writer.write(b"\r\n\r\n### Server shutting down or restarting...")
        await writer.drain()

        writer.close()
        await writer.wait_closed()

    print(f"write_clients exiting.")

    return("write_clients")

async def pty_output():
# Read chars/strings from pty and place on byte queue.
    global CACHED_OUTPUT
    timeout = 0.1

    s = ''
    while True:
        try:
            r, _, _ = select.select([PTY_PROC.fd], [], [], timeout)

            if not r:
                await asyncio.sleep(0.1)
                continue

            s = PTY_PROC.read(1024)
            await CLIENTS_QUEUE.put(s.encode())

        except asyncio.CancelledError:
            print(f"pty_output: Cancelled and tidying up...")
            break

    if len(s) > 0: # flush any remaining data to the queue
        await CLIENTS_QUEUE.put(s.encode())

    return("pty_output")

[andfoy]

Hi @Robatronic, thanks for reaching out! I have two questions regarding your use-case of pywinpty:

1. Have you tried to use the library in a synchronous fashion? Without using async primitives?
2. Do you know which backend is being used? Right now ConPTY is using VT100 by default: https://github.com/andfoy/winpty-rs/blob/3d5a3a8604be35ad0ce8e37b23dd5e7ab914697e/src/pty/conpty/pty_impl.rs#L126, in which case is better to use the WinPTY backend

[Robatronic]

1. I'm working on building one presently, I'll let you know how that goes. :)
2. I have tried both ConPTY and WinPTY with the same results.

[Robatronic]

Here's what I wrote:

import logging
logging.basicConfig(level=logging.DEBUG)
from winpty import PtyProcess as PtyProcessUnicode, Backend
import time
import codecs
import socketserver

pty = None

OUTPUT_CACHE = b''

class ptyTCPHandler(socketserver.BaseRequestHandler):
    """
    The request handler class for our server.

    It is instantiated once per connection to the server, and must
    override the handle() method to implement communication to the
    client.
    """

    def setup(self):
        super().setup()
        # Add to client list
        print(f"Adding {self.client_address} to client list")
        clients.append(self)
        return(0)

    def handle(self):
        global OUTPUT_CACHE
        # self.request is the TCP socket connected to the client
        # self.request.sendall("Hello there!!".encode())
        self.request.sendall(OUTPUT_CACHE)
        while True:
            self.data = self.request.recv(1024)
            logging.info(f"TCP handler recvd: {self.data}")
            pty.write(self.data.decode())

            try:
                ch = pty.read(65536)

                OUTPUT_CACHE += ch.encode()
                for client in clients:
                    print(f"Writing {ch.encode()} to {client.client_address}")
                    client.request.sendall(ch.encode())
            except Exception as e:
                print(f"#### EXCEPTION: ptyTCPHandler: {e}")
                break
    def finish(self):
        print(f"Removing {self.client_address} from client list..")
        clients.remove(self)



if __name__ == "__main__":

    subproc_name = "cmd.exe"
    # subproc = "python.exe"

    pty = PtyProcessUnicode.spawn(subproc_name, backend=Backend.WinPTY)
    # pty = PtyProcessUnicode.spawn(subproc_name, backend=Backend.ConPTY)

    pty.decoder = codecs.getincrementaldecoder('utf-8')(errors='replace')
    pty.name = subproc_name
    print(f"pty size: (rows, columns): {pty.getwinsize()}")

    # Get first few lines from process startup
    time.sleep(0.5)
    OUTPUT_CACHE += pty.read(65536).encode()

    # Start socket server

    HOST, PORT = "localhost", 11000

    # Create the server, binding to localhost on port 9999
    with socketserver.TCPServer((HOST, PORT), ptyTCPHandler) as server:
        # Activate the server; this will keep running until you
        # interrupt the program with Ctrl-C
        print(f"TCP Server running on {HOST}:{PORT}")
        server.serve_forever()

exit(0)

[Robatronic]

This is the output on connecting:

0;C:\Windows\System32\cmd.exeMicrosoft Windows [Version 10.0.18363.1556][9C
(c) 2019 Microsoft Corporation. All rights reserved.
[52C
D:\sandbox\epics\pyProcServ\area_51>[16C 

[Robatronic]

...then, after hitting enter and typing in dir....

D:\sandbox\epics\pyProcServ\area_51>ddirir                                                                                                                       
Volume in drive D is DATA[35C                                                   
Volume Serial Number is DCBD-282A[27C                                          
[61C                                                                             
 Directory of D:\sandbox\epics\pyProcServ\area_51[12C
[61C                                                                            
2022-08-17  04:51 PM    <DIR>          .[21C                                    
2022-08-17  04:51 PM    <DIR>          ..[20C                                   
2021-12-09  06:06 PM               943 asyncio_echo_client.py                   
2021-12-09  06:05 PM               785 asyncio_echo_server.py                   
2021-12-10  12:41 AM             4,202 asyncio_subprocess.py                    
2021-12-18  05:26 AM             1,389 asyncio_testing.py                       
2022-08-10  03:58 PM             1,001 book_samples.py                         
2022-01-13  05:50 PM               533 common_demo_stuff.py                     
2022-08-12  10:58 AM             3,380 constants.py[10C                         
2022-01-11  01:32 AM             2,260 create_task.py                           
2021-12-09  10:24 PM                18 dir.bat[15C                              
2021-12-24  04:35 PM               854 echoProc.py[11C                          
2021-12-09  05:33 PM               673 echo_server.py                           
0;C:\Windows\System32\cmd.exe - dir861 junk.txt[14C                                                                
                                                          2022-08-10  03:53 PM               158 listin
g_13_11.py[25C                                                                  
2022-08-10  04:16 PM               269 listing_13_13.py[25C                     
2022-08-10  04:28 PM             1,994 listing_13_14.py[25C                     
2022-08-09  09:04 PM             1,275 localShell.py[28C                        
2022-01-11  02:37 AM               892 misc.py[34C                              
2021-12-09  03:52 PM             2,515 pipe_client.py[27C                       
2021-12-09  03:51 PM             2,431 pipe_server.py[27C                                   
2022-01-16  04:45 PM             1,200 prime_numbers.py[25C                     
2022-08-09  09:25 PM             5,591 ptyprocess.py[28C                        
2022-08-17  04:51 PM            16,769 pty_testing.py[27C                       
2021-12-20  04:29 PM             6,675 pynCA.py[33C                             
2021-12-18  10:14 AM                 0 pynCA_helpers.py[25C                    
2022-03-24  03:52 PM                23 run_pty.bat[30C                          
2022-08-17  03:34 PM            38,758 server.log[31C                           
2022-08-17  02:57 PM            25,438 simple_server.py[25C                     
2021-12-09  03:49 PM             3,686 socket_stream_redirect.py[16C            
2021-12-21  01:59 PM             3,963 stream_testing.py[24C                    
2022-08-10  04:41 PM             8,423 sys_calls.py[29C                         
2022-01-13  06:00 PM    <DIR>          templates[32C                            
2021-12-09  03:52 PM               635 test-socket_stream_redirect.py[11C           
2022-01-27  08:09 PM            11,979 test_subprocess.py[23C                   
2022-01-13  08:52 PM             2,498 web_term.py[30C                          
2022-08-10  05:12 PM             4,538 winpsuedoterm.py[25C                     
2022-08-15  10:07 AM    <DIR>          __pycache__[30C                                        
                        34 File(s)        166,609 bytes[35C                                              
                        4 Dir(s)  1,703,480,389,632 bytes free[27C                       
0;C:\Windows\System32\cmd.exe                                                                                
                                                     D:\sandbox\epics\pyProcServ\area_51>[44C   

[Robatronic]

Telnet settings:

Escape Character is 'CTRL+]'
Will auth(NTLM Authentication)
Local echo off
New line mode - Causes return key to send CR & LF
Current mode: Console
Will term type
Preferred term type is ANSI

[Robatronic]

Looking at it, I'm starting to wonder if some of the escape codes aren't being sent as an atomic unit.....

[Robatronic]

The terminado project is very slick.

I'm trying to do similar with regular sockets rather than websockets.....

[andfoy]

Does terminado outputs the program correctly?

[Robatronic]

It does

I first found Terminado in Jupyterlab and dug in to find it uses pyWinPty.

I have another experiment I want to do to try and track down where the issue is.