Add 'range' field to tasks to allow the operation to work only on a portion of the file

[781flyingdutchman]

Range would be an (int?, int?)? where null means start/end byte (inside the record) or the entire range if the whole record is null. the values are 0 based, so the max is one less than content length.

We would JSON encode this as an optional array, eg "[null, 5000]" to indicate up to 5000.

Consider:

• check the range against content length provided by server and fail if incompatible.

[781flyingdutchman]

This would also enable progress % if the server does not provide content length, but it is known by the developer (see #156), where the range extend is effectively the known file size.

[delfme]

(null, null) for entire file makes sense.

[delfme]

Also, how the write file is done?
Coz another way to improve speed might be to buffer the chunks in memory and write file chunk by chunk.

Sort of

// Read chunk
 buffer = (fileSize! - bytes.length  >= chunkSize) ?
                  await reader!.readChunk(chunkSize)
                  : await reader!.readChunk(fileSize! - bytes.length);
// Update bytes
bytes = bytes + buffer;

And then

await file.writeAsBytes(buffer, mode: FileMode.append);

This way the file write is optimized by reducing the numbers of write. If chunks size is 1024*1024, there should be evident optimization for small files (150-500KB), which are much used on mobile apps.

[781flyingdutchman]

It depends a bit on the platform, but typically the file is written using a buffer of 8k in length. I have never experimented with it, but on Android you could try setting bufferSize in the Taskworker.kt file (in the companion object) to a different value and see if that makes a difference. I doubt it, but please let me know!

[delfme]

Saw the code. It is almost identical to what I shared above. Increasing bufferSize will improve speed co will reduce the number of write operations. Like if buffer is 500KB and file is 1MB, it needs 2 writes to create file, the rest will be saved to memory, which is fast. If buffer is 10KB, it needs 100writes 👍 Do iOS do the same?

[delfme]

As for the high level logic, Im not sure if that idea is the best one, but lets say we have a file with fileSize 1.5MB , and we download it with ParallelDownload with tasks: 3 and chunkSize: 1024 * 1024

Result would be 3 DownloadTasks downloading in parallel 500KB each.
Not sure about how many downloadTasks can be safely run concurrently.

[781flyingdutchman]

That is what ParallelDownloadTask does: it 'spawns' download tasks (one for each chunk, the size of which depends on the size of the file, eg for a 1Gb file in 4 chunks each chunk would be 250Mb, with 10 chunks it would be 100Mb per chunk), and it manages status and progress updates for each of those child tasks to make the ParallelDownloadTask 'feel' like a regular download. When all the child tasks have completed, we stitch the files together into one big one.
The downloader & Android OS manage how many actually happen in parallel - there should be no issue with having a large number, though at some point the OS will not start more child tasks until others have completed, and that is your maximum download throughput.

[iosephmagno]

This is the class, lmk if you see per improvement

library chunked_downloader;

/// Inspired by plugin https://pub.dev/packages/chunked_downloader
/// It was reworked for the most part

import 'dart:async';
import 'dart:io';

import 'package:async/async.dart';
import 'package:flutter/cupertino.dart';
import 'package:http/http.dart' as http;
import 'package:http/http.dart';

/// Progress Callback
/// [progress] is the current progress in bytes
/// [total] is the total size of the file in bytes
typedef ProgressCallback = void Function(int progress, int total, double speed);

/// On Done Callback
/// [file] is the downloaded file
typedef OnDoneCallback = void Function(File file);

/// On Error Callback
/// [error] is the error that occured
typedef OnErrorCallback = void Function(dynamic error);

/// Custom Downloader with ChunkSize
///
/// [chunkSize] is the size of each chunk in bytes
///
/// [onProgress] is the callback function that will be called when the download is in progress
///
/// [onDone] is the callback function that will be called when the download is done
///
/// [onError] is the callback function that will be called when the download is failed
///
/// [onCancel] is the callback function that will be called when the download is canceled
///
/// [onPause] is the callback function that will be called when the download is paused
///
/// [onResume] is the callback function that will be called when the download is resumed
///
class ChunkedDownloader {
  final String url;
  final String saveFilePath;
  final int chunkSize;
  final ProgressCallback? onProgress;
  final OnDoneCallback? onDone;
  final OnErrorCallback? onError;
  final Function? onCancel;
  final Function? onPause;
  final Function? onResume;
  StreamSubscription<StreamedResponse>? stream;
  ChunkedStreamReader<int>? reader;
  Map<String, String>? headers;
  Map<String, String>? queryParameters;
  double speed = 0;
  bool paused = false;
  bool done = false;
  int? fileSize;
  static const bool kDebugMode = false;
  static bool downloadCompleted = false;
  static bool downloadError = false;

  ChunkedDownloader({
    required this.url,
    required this.saveFilePath,
    this.fileSize,
    this.headers,
    this.queryParameters,
    this.chunkSize = 1024 * 1024, // default is 1 MB
    this.onProgress,
    this.onDone,
    this.onError,
    this.onCancel,
    this.onPause,
    this.onResume,
  });

  static const String _debugPrefix = 'ChunkedDownloader:';

  /// Start the download
  /// @result {Future<ChunkedDownloader>} the current instance of the downloader
  Future<void> start() async {
    // Download file
    try {
      int offset = 0;
      var httpClient = http.Client();
      // Set queryParameters
      var uri;
      if (queryParameters != null) {
        uri = Uri.parse(url).replace(queryParameters: queryParameters);
      } else {
        uri = Uri.parse(url);
      }
     // Uri uri = Uri.parse(url);
     // var finalUri = uri.replace(queryParameters: queryParameters);
      var request = http.Request('GET', uri);
      // Set headers
      if (headers != null) {
        request.headers.addAll(headers!);
      }
      //debugPrint('$_debugPrefix uri ${uri}');
      var response = httpClient.send(request);

      // Initialize file
      File file = File(saveFilePath);
      // Number of iterations of while/do loop
      // while the file is downloaded in chunks
      int cycles = 0;
      // Buffer is the bytes readed from readerChunk
      List<int> buffer = [];
      // bytes if the total numbers od bytes at each cycle
      List<int> bytes= [];

      stream = response.asStream().listen(null);
      stream?.onData((http.StreamedResponse r) async {
        // Get file size. Many servers don't provide [content-length], so
        // standard use of this class is to pass fileSize as parameter
        // In case of Presence we can do it coz fileSize is known.
        fileSize = fileSize ?? int.tryParse(r.headers['content-length'] ?? '-1') ?? -1;
        reader = ChunkedStreamReader(r.stream);
        try {

          do {
            // Wait indefinitely if paused
            await waitWhile(() => (paused));

            // Set start time for speed calculation
            int startTime = DateTime.now().millisecondsSinceEpoch;

            // Read chunk
            buffer = (fileSize! - bytes.length  >= chunkSize) ?
                  await reader!.readChunk(chunkSize)
                  : await reader!.readChunk(fileSize! - bytes.length);
            bytes = bytes + buffer;

            // Calculate speed
            int endTime = DateTime.now().millisecondsSinceEpoch;
            int timeDiff = endTime - startTime;
            if (timeDiff > 0) {
              speed = (buffer.length / timeDiff) * 1000;
            }
            // Add buffer to chunks list
            offset += buffer.length;
            if (kDebugMode) {
              debugPrint('$_debugPrefix Downloading ${offset ~/ 1024 ~/ 1024}MB '
                  'Speed: ${speed ~/ 1024 ~/ 1024}MB/s');
            }
            if (onProgress != null) {
              onProgress!(offset, fileSize!, speed);
            }
            // Write buffer to disk
            // the first cycle we override the initial file
            // coz flutter create new file with 1024 bytes
            // default size. On subsequent cycle we append
            // new bytes to previous ones.
            if (cycles == 0) {
              await file.writeAsBytes(bytes);
            } else {
              await file.writeAsBytes(buffer, mode: FileMode.append);
            }
            cycles++;
          } while (bytes.length < fileSize!);
          //debugPrint('$_debugPrefix downloaded file size ${file.lengthSync()}');
          // Send done callback
          done = true;
          if (onDone != null) {
            onDone!(file);
            downloadCompleted = true;
          }
          if (kDebugMode) {
            debugPrint('$_debugPrefix Downloaded file.');
          }

        } catch (error) {
          downloadError = true;
          bytes.clear();
          buffer.clear();
          if (kDebugMode) {
            debugPrint('$_debugPrefix Error downloading: $error');
          }
          if (onError != null) {
            onError!(error);
          }
        } finally {
          reader?.cancel();
          stream?.cancel();
          bytes.clear();
          buffer.clear();
        }
      });
    } catch (error) {
      if (kDebugMode) {
        debugPrint('$_debugPrefix Error downloading: $error');
      }
      if (onError != null) {
        onError!(error);
      }
    }
    // Do not return until download is !completed and !failed
    // Note: this is required because we cannot
    // await stream?.onData((http.StreamedResponse r)
    // and await won't work with this func
    await waitWhile(() => (!downloadCompleted && !downloadError));
  }

  /// Stop the download
  void stop() {
    stream?.cancel();
    reader?.cancel();
    if (onCancel != null) {
      onCancel!();
    }
  }

  /// Pause the download
  void pause() {
    paused = true;
    if (onPause != null) {
      onPause!();
    }
  }

  /// Resume the download
  void resume() {
    paused = false;
    if (onResume != null) {
      onResume!();
    }
  }
}

// This is used to wait until test condition is not false
Future waitWhile(bool condition(), [Duration pollInterval = Duration.zero]) {
  var completer = new Completer();
  check() {
    if (!condition()) {
      completer.complete();
    } else {
      new Timer(pollInterval, check);
    }
  }
  check();
  return completer.future;
}

Sample code to use it.

final url = 'https://freetestdata.com/wp-content/uploads/2022/02/Free_Test_Data_1MB_JPG.jpg';
var chunkedDownloader = ChunkedDownloader(
          url: url, 
          saveFilePath: filePath,
          fileSize: fileSize,
          chunkSize: 1024 * 1024,                 
          onError: (error) {},
          onProgress: (received, total, speed) {
            onDownloadProgress.call(received, total);
            debugPrint('downloading file: $received of $total');
          },
          onDone: (file) async {
            debugPrint('file $url downloaded');
          });
         await chunkedDownloader.start();

[781flyingdutchman]

Per above, this functionality is now live in the package, so I suggest trying to use that. I think I'll be able to address your concern about files of known size but without content-length header with this 'range' idea, as you woudl simply set the range to (null, your_known_content_length) and the downloader will take that as truth and use it to break up the task in chunks.

[delfme]

That would be great, planning to dismiss flutter_downloader in favour of background_downloder, which I like the most.

Would be great if background_uploader companion could be created with same approach. Currently there is a flutter_uploader which is almost discontinued and other minor plugins which are buggy.

[delfme]

Looks like flutter has an issue and going native can fix it flutter/flutter#136061 (comment)

[delfme]

What about passing just fileSize: my_content_length and then converting it to (null, my_content_lenght) ?

[781flyingdutchman]

We're mixing a few topics here, but

1. background_downloader has full upload capability already - you don't need a separate package, just create an UploadTask or a MultiUploadTask. Read the docs though, because there are a lot of options to set (e.g. binary versus multipart, file and form fields, etc).
2. fileSize as a parameter for the constructor is interesting. It would still effectively set the range though, as that gives more flexibility

[781flyingdutchman]

The more I think about it, I feel we should not add a range option, and let the developer set the range or content-length header if needed. To force a known content length you can then either set the range to end there, or set the content-length header. The logic in the downloader can be updated to take the range or content-length values if the server does not provide one. This keeps the API simple (no need for additional fields in DownloadTask) and still gives you the (unusual) functionality that you need (it's very uncommon to know in advance the file size AND for the server to not provide it).

Thoughts?

[delfme]

Yes that sounds way better.

Feature is common in case of app backups. When app uploads local files to the cloud, it knowns their fileSize. When app downloads files back from the cloud, it knows their fileSize, coz it was saved into local db.

[781flyingdutchman]

@delfme do you happen to have a URL that I can use to test this? It needs to be

• https
• not supply a Content-Length header
• have fixed, reasonable size (eg around 5MB)
• accept ranges
• be public and innocent.

I want to test if I set a Range, or a Content-Length header in the DownloadTask I should get progress updates, whereas if I don't I will not. My current test url for 'not having Content-Length' is www.google.com, but its length is different on every call, and it's short. Every other test file I've looked at so far does already supply Content-Length, so cannot be used for this test.

Hope you can help me out!

[delfme]

Can you try these one?
https://freetestdata.com/
https://testfiledownload.com/
https://www.thinkbroadband.com/download

[781flyingdutchman]

Thanks, but unfortunately they all still send a content-length header.

[781flyingdutchman]

@delfme I've implemented this functionality now on the dev branch. Would you mind trying it out? Currently only works for Android and Desktop, working on iOS but I'd like to make sure the concept works. You can force content length by either setting a Range header (to "bytes=0-999" for a content-length of 1000) or, if the server rejects ranges, a header of "Known-Content-Length" : "1000". In my experiment, setting "Content-Length" on the request header creates issues, so I decided we shoudl go with a special (non-reserved) header that is honestly only used by the downloader. I don't remove the header, so it becomes part of the request.

I've simulated this works in my setting, but would like to her your experience.

[781flyingdutchman]

Implemented in V7.11.0

[delfme]

Great! will have a look and revert. But it will take me a week coz Im out travelling

[delfme]

Hello @781flyingdutchman something came up and I still couldnt test it. Even because I need to test it with our app restore backup function, which is not a quick check. But will try it asap an revert.

How went your test? I mean, can this be used to download say 10 or 100 small files at time without crashing system? That would be our use case. Thx

[781flyingdutchman]

Hi, it's now already published so if you just upgrade to the latest version the functionality is there. And yes this should work for 100 files, but you may want to use the new TaskQueue to ensure you're not choking the main thread when enqueueing a larger number of files. It's very simple, just a buffer.

[delfme]

Great! 🙏 Is there a ready example available for this new feature?

[781flyingdutchman]

From the readme:
To provide progress updates (as a percentage of total file size) the downloader needs to know the size of the file when starting the download. Most servers provide this in the "Content-Length" header of their response. If the server does not provide the file size, yet you know the file size (e.g. because you have stored the file on the server yourself), then you can let the downloader know by providing a {'Range': 'bytes=0-999'} or a {'Known-Content-Length': '1000'} header to the task's header field. Both examples are for a content length of 1000 bytes. The downloader will assume this content length when calculating progress.

For TaskQueues, the docs are here, and for a minimal use case (just buffering and pacing the enqueue calls) use:

final tq = MemoryTaskQueue();
FileDownloader().add(tq); // 'connects' the TaskQueue to the FileDownloader
FileDownloader().updates.listen((update) { // listen to updates as per usual
  print('Received update for ${update.task.taskId}: $update')
});
for (var n = 0; n < 100; n++) {
  task = DownloadTask(url: workingUrl, metData: 'task #$n'); // define task
  tq.add(task); // add to queue. The queue makes the FileDownloader().enqueue call
}