Non-file-based streams in Node.js do not receive progress events

[Acconut]

Describe the bug

When passing a stream to tus-js-client that is not an fs.ReadStream, tus-js-client only emits progress events for the end and beginning of PATCH requests, but not during PATCH requests.

To Reproduce

/* eslint-disable no-console */

'use strict'

const fs = require('fs')
const tus = require('../..')
const { PassThrough } = require('stream')

const path = `${__dirname}/../../dist/tus.js`
const file = fs.createReadStream(path).pipe(new PassThrough())

const options = {
  endpoint: 'https://tusd.tusdemo.net/files/',
  chunkSize: 1024 ** 3,
  uploadLengthDeferred: true,
  metadata: {
    filename: 'README.md',
    filetype: 'text/plain',
  },
  onError(error) {
    console.error('An error occurred:')
    console.error(error)
    process.exitCode = 1
  },
  onProgress(bytesUploaded, bytesTotal) {
    const percentage = ((bytesUploaded / bytesTotal) * 100).toFixed(2)
    console.log(bytesUploaded, bytesTotal, `${percentage}%`)
  },
  onSuccess() {
    console.log('Upload finished:', upload.url)
  },
}

const upload = new tus.Upload(file, options)
upload.start()

Run it and the output is like this (no intermediate upload progress):

$ node demos/nodejs/index.js
0 201172 0.00%
201172 201172 100.00%

Expected behavior

It should be more like this (as with file-based streams):

$ node demos/nodejs/index.js
0 201172 0.00%
65536 201172 32.58%
201172 201172 100.00%

Details

This is a direct consequence of #385. Before this PR, tus-js-client split a stream into multiple sub-streams. After this PR, a stream is now split into buffers. However, our HTTP stack in Node.js only supports progress events when a stream is used a the request body, and not buffers:

tus-js-client/lib/node/httpStack.js

Lines 90 to 94 in 623ec81

	if (body instanceof Readable) {
	body.pipe(new ProgressEmitter(this._progressHandler)).pipe(req)
	} else {
	req.end(body)
	}

We should find a different way to get progress events from requests in Node.js.

[Acconut]

axios has an onUploadProgress event and seems to achieve this by a similar method of piping the upload body through a transform stream. Non-streams are simply converted into streams to conform here: https://github.com/axios/axios/blob/21a5ad34c4a5956d81d338059ac0dd34a19ed094/lib/adapters/http.js#L332-L347

[mifi]

maybe it could also be achieved by simply:

let progress = 0
stream.on('data', chunk => {
  progress += chunk.size
  onProgress(progress/fileSize)
});

[Acconut]

I don't think that is a fitting solution because it would measure the progress from consuming the stream and not the actual network transmission. Since we are buffering internally, these two progress would be vastly different.

[Acconut]

It tried something like this:

      if (body == null) {
        req.end()
      } else {
        if (!isStream.readable(body)) {
          body = Readable.from(body, { objectMode: false })
        }

        pipeline(body, new ProgressEmitter(this._progressHandler), req, (err) => {
          reject(err)
        })
      }

But it does not work. Progress events are emitted based on the read operations that fills the internal buffers and not based on the writes to the HTTP connection. Maybe we should use a more high-level HTTP client, like Axios (see #570).

How pressing is this issue for Uppy, @mifi?

[mifi]

I don't think it's very pressing. Just a minor UX inconvenience I would say, because users won't see any progress until one chunk has finished. But @arturi brought it up so maybe he can chime in

[arturi]

Agreed, not very, prio-2, but on slow connection this means you can sometimes wait a couple of minutes at 0% progress, so you'll think it's not doing anything and abandon the upload. I was convinced it's downloading the file before upload, not streaming it right away (which seems to not be the case, as tested by Mikael).

[Acconut]

I played a bit around with this issue but didn't find any solution. I tested the latest version of Axios and progress-stream, but both approaches exhibit the same issue as described in #602 (comment): The progress information is skewed because the entire upload data passed to the HTTP module in one go through the stream, without any consideration of how much data is written to the network socket in the end. The result is that we immediately get a 100% progress reporting because the entire upload data has been passed to the HTTP module. This does not reflect the actual upload progress though.

Reproducing this is possible without tus-js-client by sending a large buffer via a POST request:

'use strict'

const https = require('https');
const crypto = require('crypto');
const progress = require('progress-stream');

const dataSizeInBytes = 100 * 1024 * 1024;

// Generate a buffer with random data
const randomDataBuffer = crypto.randomBytes(dataSizeInBytes);

// Set the headers for the POST request
const headers = {
  'Content-Type': 'application/offset+octet-stream',
  'Tus-Resumable': '1.0.0',
  'Upload-Length': dataSizeInBytes.toString(),
};

const options = {
  method: 'POST',
  headers,
  hostname: 'tusd.tusdemo.net',
  path: '/files/',
  port: 443,
};

const req = https.request(options, (res) => {
  let data = '';
  
  res.on('data', (chunk) => {
    data += chunk;
  });

  res.on('end', () => {
    console.log('File uploaded successfully!');
    console.log('Response:', data);
  });
});

req.on('error', (error) => {
  console.error('Error uploading file:', error.message);
});

// Send the random data buffer in the request body
const progresStream = progress({
    length: dataSizeInBytes
}, (p) => console.log(p))

progresStream.pipe(req, { end: true })
progresStream.write(randomDataBuffer)

The only progress report I get is 100% immediately after the upload starts:

{
  percentage: 100,
  transferred: 104857600,
  length: 104857600,
  remaining: 0,
  eta: 0,
  runtime: 0,
  delta: 104857600,
  speed: 104857600
}

A search for alternative solutions to this problem did not show any helpful way for capturing upload progress in Node.js. Unless somebody else can help with this, we will likely not have progress information for Node.js unfortunately.