Skip to content

Releases: DanEngelbrecht/longtail

v0.4.3

23 Nov 12:26
@github-actions github-actions
v0.4.3
143e7e6
Compare
Choose a tag to compare
v0.4.3 Latest
Latest
Assets 6
Loading

v0.4.3-pre0

23 Nov 11:18
@github-actions github-actions
v0.4.3-pre0
9923c03
Compare
Choose a tag to compare
v0.4.3-pre0 Pre-release
Pre-release
Assets 6
Loading

v0.4.2

10 Mar 10:11
@github-actions github-actions
v0.4.2
14a0ece
Compare
Choose a tag to compare
v0.4.2
  • CHANGED API Longtail_JobAPI_JobFunc renamed is_cancelled to detected_error, now contains first error returned from a job task in the same job group (if any) or ECANCELLED if job group was cancelled
    If detected_error is non-zero, try to exit (and cleanup) your task directly and return 0.
  • CHANGED_API JobAPI WaitForAllJobs now returns first error encountered in a job group for a task as well as any error in the job api itself, removing the need to book keep the error for tasks separately
  • CHANGED API Longtail_StorageAPI.OpenAppend added to Longtail_StorageAPI to open files without truncating existing data
  • CHANGED API Longtail_CreateConcurrentChunkWriteAPI changed to take source_version_index and version_diff
  • CHANGED API Longtail_ConcurrentChunkWriteAPI refactored to use asset index and open/close files instead of keeping all open during entire lifetime
    • Longtail_ConcurrentChunkWriteAPI.CreateDir now takes asset index instead of version local path
    • Longtail_ConcurrentChunkWriteAPI.Open now takes asset index instead of version local path and dropping chunk_write_count parameter
    • Longtail_ConcurrentChunkWriteAPI.Write now takes asset index instead of version local path and dropping chunk_write_count parameter
  • CHANGED API Longtail_SetMonitor callback functions refactored to accomodate changes in Longtail_ConcurrentChunkWriteAPI
  • NEW API Longtail_SetReAllocAndFree
  • NEW API Longtail_ReAlloc
  • NEW API Longtail_MemTracer_ReAlloc
  • NEW API Longtail_CompareAndSwap compare and swap with platform implementations
  • NEW API Longtail_RunJobsBatched runs jobs in batched mode to handle a job count larger than Longtail_JobAPI::GetMaxBatchCount()
  • NEW API Longtail_GetFilesRecursively2 that executes using parallel jobs improving execution speed for large file trees significantly
  • REMOVED API Longtail_SetAllocAndFree is replaced by Longtail_SetReAllocAndFree
  • REMOVED API Longtail_MemTracer_Alloc is replaced by Longtail_MemTracer_ReAlloc
  • REMOVED API Remove platform api for Read/Write mutex
    • Longtail_GetRWLockSize
    • Longtail_CreateRWLock
    • Longtail_DeleteRWLock
    • Longtail_LockRWLockRead
    • Longtail_LockRWLockWrite
    • Longtail_UnlockRWLockRead
    • Longtail_UnlockRWLockWrite
  • ADDED memtracer now tracks allocations in stb_ds
  • ADDED memtracer now tracks allocations in zstd
  • FIXED Fixed memory leaks in command tool
  • FIXED Longtail_ChangeVersion2() can now handle workloads with a block count larger than 65535
  • FIXED Bikeshed JobAPI implementation does efficient wait when task queue is full
  • FIXED Bikeshed JobAPI::CreateJobs implementation now properly drains both task channels when task queue is full
  • FIXED Make sure we retain order of assets with equal length when sorting them
  • FIXED Fixed excessive "Disk Used" increase during Longtail_ChangeVersion2 execution causing Out Of Disk space errors.
    The changes also improves performance for more common cases with smaller archive sizes (60 Gb raw data/many files) but causes a small regression compared to 0.4.1 for archives with many very large files. It is still performing much more reasonable than 0.4.0 for these cases.
    Version Files Raw Size Compressed Size Unpack Time Peak Memory
    0.4.0 1019 735 GB 214 GB 2h44m26s 7.9 GB
    0.4.1 1019 735 GB 214 GB 0h12m14s 1.9 GB
    0.4.2 1019 735 GB 214 GB 0h13m25s 2.2 GB
    0.4.0 239 340 60 GB 17 GB 0h01m24s 4.2 GB
    0.4.1 239 340 60 GB 17 GB 0h02m48s 0.9 GB
    0.4.2 239 340 60 GB 17 GB 0h01m12s 0.9 GB
  • CHANGED Refactored all internal usage of JobAPI ReadyJobs with new error handling
  • UPDATED Update of ZStd: 1.5.5 https://github.com/facebook/zstd/releases/tag/v1.5.5
  • UPDATED Update of Blake3: 1.5.0 https://github.com/BLAKE3-team/BLAKE3/releases/tag/1.5.0
  • UPDATED Update of Brotli: 1.1.0 https://github.com/google/brotli/releases/tag/v1.1.0
  • EXPERIMENTAL NEW API Longtail_SetMonitor to enable more detailed feedback on Longtail_ChangeVersion2 and Longtail_WriteContent. Includes simple progress UI using MiniFB and console output for upsync/downsync/pack/unpack via --detailed-progress option. MiniFB version only works on Windows so far, other platforms fall back to console text output.
Assets 6
Loading

v0.4.2-pre5

04 Mar 22:12
@github-actions github-actions
v0.4.2-pre5
b77d049
Compare
Choose a tag to compare
v0.4.2-pre5 Pre-release
Pre-release
  • CHANGED API Longtail_JobAPI_JobFunc renamed is_cancelled to detected_error, now contains first error returned from a job task in the same job group (if any) or ECANCELLED if job group was cancelled
    If detected_error is non-zero, try to exit (and cleanup) your task directly and return 0.
  • CHANGED_API JobAPI WaitForAllJobs now returns first error encountered in a job group for a task as well as any error in the job api itself, removing the need to book keep the error for tasks separately
  • CHANGED API Longtail_StorageAPI.OpenAppend added to Longtail_StorageAPI to open files without truncating existing data
  • CHANGED API Longtail_CreateConcurrentChunkWriteAPI changed to take source_version_index and version_diff
  • CHANGED API Longtail_ConcurrentChunkWriteAPI refactored to use asset index and open/close files instead of keeping all open during entire lifetime
    • Longtail_ConcurrentChunkWriteAPI.CreateDir now takes asset index instead of version local path
    • Longtail_ConcurrentChunkWriteAPI.Open now takes asset index instead of version local path and dropping chunk_write_count parameter
    • Longtail_ConcurrentChunkWriteAPI.Write now takes asset index instead of version local path and dropping chunk_write_count parameter
  • CHANGED API Longtail_SetMonitor callback functions refactored to accomodate changes in Longtail_ConcurrentChunkWriteAPI
  • NEW API Longtail_SetReAllocAndFree
  • NEW API Longtail_ReAlloc
  • NEW API Longtail_MemTracer_ReAlloc
  • NEW API Longtail_CompareAndSwap compare and swap with platform implementations
  • NEW API Longtail_RunJobsBatched runs jobs in batched mode to handle a job count larger than Longtail_JobAPI::GetMaxBatchCount()
  • NEW API Longtail_GetFilesRecursively2 that executes using parallel jobs improving execution speed for large file trees significantly
  • REMOVED API Longtail_SetAllocAndFree is replaced by Longtail_SetReAllocAndFree
  • REMOVED API Longtail_MemTracer_Alloc is replaced by Longtail_MemTracer_ReAlloc
  • REMOVED API Remove platform api for Read/Write mutex
    • Longtail_GetRWLockSize
    • Longtail_CreateRWLock
    • Longtail_DeleteRWLock
    • Longtail_LockRWLockRead
    • Longtail_LockRWLockWrite
    • Longtail_UnlockRWLockRead
    • Longtail_UnlockRWLockWrite
  • ADDED memtracer now tracks allocations in stb_ds
  • ADDED memtracer now tracks allocations in zstd
  • FIXED Fixed memory leaks in command tool
  • FIXED Longtail_ChangeVersion2() can now handle workloads with a block count larger than 65535
  • FIXED Bikeshed JobAPI implementation does efficient wait when task queue is full
  • FIXED Bikeshed JobAPI::CreateJobs implementation now properly drains both task channels when task queue is full
  • FIXED Make sure we retain order of assets with equal length when sorting them
  • FIXED Fixed excessive "Disk Used" increase during Longtail_ChangeVersion2 execution causing Out Of Disk space errors.
    The changes also improves performance for more common cases with smaller archive sizes (60 Gb raw data/many files) but causes a small regression compared to 0.4.1 for archives with many very large files. It is still performing much more reasonable than 0.4.0 for these cases.
    Version Files Raw Size Compressed Size Unpack Time Peak Memory
    0.4.0 1019 735 GB 214 GB 2h44m26s 7.9 GB
    0.4.1 1019 735 GB 214 GB 0h12m14s 1.9 GB
    0.4.2 1019 735 GB 214 GB 0h13m25s 2.2 GB
    0.4.0 239 340 60 GB 17 GB 0h01m24s 4.2 GB
    0.4.1 239 340 60 GB 17 GB 0h02m48s 0.9 GB
    0.4.2 239 340 60 GB 17 GB 0h01m12s 0.9 GB
  • CHANGED Refactored all internal usage of JobAPI ReadyJobs with new error handling
  • UPDATED Update of ZStd: 1.5.5 https://github.com/facebook/zstd/releases/tag/v1.5.5
  • UPDATED Update of Blake3: 1.5.0 https://github.com/BLAKE3-team/BLAKE3/releases/tag/1.5.0
  • UPDATED Update of Brotli: 1.1.0 https://github.com/google/brotli/releases/tag/v1.1.0
  • EXPERIMENTAL NEW API Longtail_SetMonitor to enable more detailed feedback on Longtail_ChangeVersion2 and Longtail_WriteContent. Includes simple progress UI using MiniFB and console output for upsync/downsync/pack/unpack via --detailed-progress option. MiniFB version only works on Windows so far, other platforms fall back to console text output.
Assets 6
Loading

v0.4.2-pre3

03 Mar 11:09
@github-actions github-actions
v0.4.2-pre3
37cca28
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Learn about vigilant mode.
Compare
Choose a tag to compare
v0.4.2-pre3 Pre-release
Pre-release
  • CHANGED API Longtail_JobAPI_JobFunc renamed is_cancelled to detected_error, now contains first error returned from a job task in the same job group (if any) or ECANCELLED if job group was cancelled
    If detected_error is non-zero, try to exit (and cleanup) your task directly and return 0.
  • CHANGED_API JobAPI WaitForAllJobs now returns first error encountered in a job group for a task as well as any error in the job api itself, removing the need to book keep the error for tasks separately
  • CHANGED API Longtail_StorageAPI.OpenAppend added to Longtail_StorageAPI to open files without truncating existing data
  • CHANGED API Longtail_CreateConcurrentChunkWriteAPI changed to take source_version_index and version_diff
  • CHANGED API Longtail_ConcurrentChunkWriteAPI refactored to use asset index and open/close files instead of keeping all open during entire lifetime
    • Longtail_ConcurrentChunkWriteAPI.CreateDir now takes asset index instead of version local path
    • Longtail_ConcurrentChunkWriteAPI.Open now takes asset index instead of version local path and dropping chunk_write_count parameter
    • Longtail_ConcurrentChunkWriteAPI.Write now takes asset index instead of version local path and dropping chunk_write_count parameter
  • CHANGED API Longtail_SetMonitor callback functions refactored to accomodate changes in Longtail_ConcurrentChunkWriteAPI
  • NEW API Longtail_SetReAllocAndFree
  • NEW API Longtail_ReAlloc
  • NEW API Longtail_MemTracer_ReAlloc
  • NEW API Longtail_CompareAndSwap compare and swap with platform implementations
  • NEW API Longtail_RunJobsBatched runs jobs in batched mode to handle a job count larger than Longtail_JobAPI::GetMaxBatchCount()
  • NEW API Longtail_GetFilesRecursively2 that executes using parallel jobs improving execution speed for large file trees significantly
  • REMOVED API Longtail_SetAllocAndFree is replaced by Longtail_SetReAllocAndFree
  • REMOVED API Longtail_MemTracer_Alloc is replaced by Longtail_MemTracer_ReAlloc
  • REMOVED API Remove platform api for Read/Write mutex
    • Longtail_GetRWLockSize
    • Longtail_CreateRWLock
    • Longtail_DeleteRWLock
    • Longtail_LockRWLockRead
    • Longtail_LockRWLockWrite
    • Longtail_UnlockRWLockRead
    • Longtail_UnlockRWLockWrite
  • ADDED memtracer now tracks allocations in stb_ds
  • ADDED memtracer now tracks allocations in zstd
  • FIXED Fixed memory leaks in command tool
  • FIXED Longtail_ChangeVersion2() can now handle workloads with a block count larger than 65535
  • FIXED Bikeshed JobAPI implementation does efficient wait when task queue is full
  • FIXED Bikeshed JobAPI::CreateJobs implementation now properly drains both task channels when task queue is full
  • FIXED Make sure we retain order of assets with equal length when sorting them
  • FIXED Fixed excessive "Disk Used" increase during Longtail_ChangeVersion2 execution causing Out Of Disk space errors.
    The changes also improves performance for more common cases with smaller archive sizes (60 Gb raw data/many files) but causes a small regression compared to 0.4.1 for archives with many very large files. It is still performing much more reasonable than 0.4.0 for these cases.
    Version Files Raw Size Compressed Size Unpack Time Peak Memory
    0.4.0 1019 735 GB 214 GB 2h44m26s 7.9 GB
    0.4.1 1019 735 GB 214 GB 0h12m14s 1.9 GB
    0.4.2 1019 735 GB 214 GB 0h13m25s 2.2 GB
    0.4.0 239 340 60 GB 17 GB 0h01m24s 4.2 GB
    0.4.1 239 340 60 GB 17 GB 0h02m48s 0.9 GB
    0.4.2 239 340 60 GB 17 GB 0h01m12s 0.9 GB
  • CHANGED Refactored all internal usage of JobAPI ReadyJobs with new error handling
  • UPDATED Update of ZStd: 1.5.5 https://github.com/facebook/zstd/releases/tag/v1.5.5
  • UPDATED Update of Blake3: 1.5.0 https://github.com/BLAKE3-team/BLAKE3/releases/tag/1.5.0
  • UPDATED Update of Brotli: 1.1.0 https://github.com/google/brotli/releases/tag/v1.1.0
  • EXPERIMENTAL NEW API Longtail_SetMonitor to enable more detailed feedback on Longtail_ChangeVersion2 and Longtail_WriteContent. Includes simple progress UI using MiniFB and console output for upsync/downsync/pack/unpack via --detailed-progress option. MiniFB version only works on Windows so far, other platforms fall back to console text output.
Assets 6
Loading

v0.4.2-pre2

04 Feb 15:12
@github-actions github-actions
v0.4.2-pre2
f56252c
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Learn about vigilant mode.
Compare
Choose a tag to compare
v0.4.2-pre2 Pre-release
Pre-release
  • CHANGED API Longtail_JobAPI_JobFunc renamed is_cancelled to detected_error, now contains first error returned from a job task in the same job group (if any) or ECANCELLED if job group was cancelled
    If detected_error is non-zero, try to exit (and cleanup) your task directly and return 0.
  • CHANGED_API JobAPI WaitForAllJobs now returns first error encountered in a job group for a task as well as any error in the job api itself, removing the need to book keep the error for tasks separately
  • ADDED memtracer now tracks allocations in stb_ds
  • ADDED memtracer now tracks allocations in zstd
  • NEW API Longtail_SetReAllocAndFree
  • NEW API Longtail_ReAlloc
  • NEW API Longtail_MemTracer_ReAlloc
  • NEW API Longtail_CompareAndSwap compare and swap with platform implementations
  • NEW API Longtail_RunJobsBatched runs jobs in batched mode to handle a job count larger than Longtail_JobAPI::GetMaxBatchCount()
  • NEW API Longtail_GetFilesRecursively2 that executes using parallel jobs improving execution speed for large file trees significantly
  • REMOVED API Longtail_SetAllocAndFree is replaced by Longtail_SetReAllocAndFree
  • REMOVED API Longtail_MemTracer_Alloc is replaced by Longtail_MemTracer_ReAlloc
  • CHANGED API Longtail_ConcurrentChunkWriteAPI::Write has new parameter out_chunks_remaining which is set to the remaining number of chunks to write to asset after call completes
  • FIXED Fixed memory leaks in command tool
  • FIXED Longtail_ChangeVersion2() can now handle workloads with a block count larger than 65535
  • FIXED Bikeshed JobAPI implementation does efficient wait when task queue is full
  • FIXED Bikeshed JobAPI::CreateJobs implementation now properly drains both task channels when task queue is full
  • FIXED Make sure we retain order of assets with equal length when sorting them
  • CHANGED Refactored all internal usage of JobAPI ReadyJobs with new error handling
  • UPDATED Update of ZStd: 1.5.5 https://github.com/facebook/zstd/releases/tag/v1.5.5
  • UPDATED Update of Blake3: 1.5.0 https://github.com/BLAKE3-team/BLAKE3/releases/tag/1.5.0
  • UPDATED Update of Brotli: 1.1.0 https://github.com/google/brotli/releases/tag/v1.1.0
  • EXPERIMENTAL NEW API Longtail_SetMonitor to enable more detailed feedback on Longtail_ChangeVersion2 and Longtail_WriteContent. Includes simple progress UI using MiniFB and console output for upsync/downsync/pack/unpack via --detailed-progress option. MiniFB version only works on Windows so far, other platforms fall back to console text output.
Assets 6
Loading

v0.4.2-pre1

04 Feb 15:11
@github-actions github-actions
v0.4.2-pre1
d1c9c70
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Learn about vigilant mode.
Compare
Choose a tag to compare
v0.4.2-pre1 Pre-release
Pre-release
  • CHANGED API Longtail_JobAPI_JobFunc renamed is_cancelled to detected_error, now contains first error returned from a job task in the same job group (if any) or ECANCELLED if job group was cancelled
    If detected_error is non-zero, try to exit (and cleanup) your task directly and return 0.
  • CHANGED_API JobAPI WaitForAllJobs now returns first error encountered in a job group for a task as well as any error in the job api itself, removing the need to book keep the error for tasks separately
  • ADDED memtracer now tracks allocations in stb_ds
  • ADDED memtracer now tracks allocations in zstd
  • NEW API Longtail_SetReAllocAndFree
  • NEW API Longtail_ReAlloc
  • NEW API Longtail_MemTracer_ReAlloc
  • NEW API Longtail_CompareAndSwap compare and swap with platform implementations
  • NEW API Longtail_RunJobsBatched runs jobs in batched mode to handle a job count larger than Longtail_JobAPI::GetMaxBatchCount()
  • NEW API Longtail_GetFilesRecursively2 that executes using parallel jobs improving execution speed for large file trees significantly
  • REMOVED API Longtail_SetAllocAndFree is replaced by Longtail_SetReAllocAndFree
  • REMOVED API Longtail_MemTracer_Alloc is replaced by Longtail_MemTracer_ReAlloc
  • CHANGED API Longtail_ConcurrentChunkWriteAPI::Write has new parameter out_chunks_remaining which is set to the remaining number of chunks to write to asset after call completes
  • FIXED Fixed memory leaks in command tool
  • FIXED Longtail_ChangeVersion2() can now handle workloads with a block count larger than 65535
  • FIXED Bikeshed JobAPI implementation does efficient wait when task queue is full
  • FIXED Bikeshed JobAPI::CreateJobs implementation now properly drains both task channels when task queue is full
  • FIXED Make sure we retain order of assets with equal length when sorting them
  • CHANGED Refactored all internal usage of JobAPI ReadyJobs with new error handling
  • UPDATED Update of ZStd: 1.5.5 https://github.com/facebook/zstd/releases/tag/v1.5.5
  • UPDATED Update of Blake3: 1.5.0 https://github.com/BLAKE3-team/BLAKE3/releases/tag/1.5.0
  • UPDATED Update of Brotli: 1.1.0 https://github.com/google/brotli/releases/tag/v1.1.0
  • EXPERIMENTAL NEW API Longtail_SetMonitor to enable more detailed feedback on Longtail_ChangeVersion2. Includes simple progress UI using MiniFB and console output for downsync/unpack via --detailed-progress option. MiniFB only works on Windows so far, other platforms fall back to console text output.
Assets 6
Loading

v0.4.2-pre0

25 Jan 21:02
@github-actions github-actions
v0.4.2-pre0
034d85b
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Learn about vigilant mode.
Compare
Choose a tag to compare
v0.4.2-pre0 Pre-release
Pre-release
  • CHANGED API Longtail_JobAPI_JobFunc renamed is_cancelled to detected_error, now contains first error returned from a job task in the same job group (if any) or ECANCELLED if job group was cancelled
    If detected_error is non-zero, try to exit (and cleanup) your task directly and return 0.
  • CHANGED_API JobAPI ReadyJobs now returns first error encountered in a job group for a task as well as any error in the job api itself, removing the need to book keep the error for tasks separately
  • ADDED memtracer now tracks allocations in stb_ds
  • ADDED memtracer now tracks allocations in zstd
  • NEW API Longtail_SetReAllocAndFree
  • NEW API Longtail_ReAlloc
  • NEW API Longtail_MemTracer_ReAlloc
  • NEW API Longtail_CompareAndSwap compare and swap with platform implementations
  • NEW API Longtail_RunJobsBatched runs jobs in batched mode to handle a job count larger than Longtail_JobAPI::GetMaxBatchCount()
  • REMOVED API Longtail_SetAllocAndFree is replaced by Longtail_SetReAllocAndFree
  • REMOVED API Longtail_MemTracer_Alloc is replaced by Longtail_MemTracer_ReAlloc
  • CHANGED API Longtail_ConcurrentChunkWriteAPI::Write has new parameter out_chunks_remaining which is set to the remaining number of chunks to write to asset after call completes
  • FIXED Fixed memory leaks in command tool
  • FIXED Longtail_ChangeVersion2() can now handle workloads with a block count larger than 65535
  • FIXED Bikeshed JobAPI implementation does efficient wait when task queue is full
  • FIXED Bikeshed JobAPI::CreateJobs implementation now properly drains both task channels when task queue is full
  • FIXED Make sure we retain order of assets with equal length when sorting them
  • CHANGED Refactored all internal usage of JobAPI ReadyJobs with new error handling
  • EXPERIMENTAL NEW API Longtail_SetMonitor to enable more detailed feedback on Longtail_ChangeVersion2. Includes simple progress UI using MiniFB and console output for downsync/unpack via --detailed-progress option. MiniFB only works on Windows so far, other platforms fall back to console text output.
Assets 6
Loading

v0.4.1

03 Jan 16:58
@github-actions github-actions
v0.4.1
0bcc0a0
Compare
Choose a tag to compare
v0.4.1

  • NEW API Longtail_ChangeVersion2 added

    Implements a new strategy for decompressing/writing version assets which is significantly faster for files that spans multiple blocks while retaining the same speed for assets smaller than a block. It removes redundant decompression of blocks (so the LRU block store is no longer needed) at the expense of doing random access when writing files.

    It uses significantly less memory and less CPU while avoiding redundant reads from source data so it is an overall win*.
    If your main target devices are SSD drives I highly recommend switching to Longtail_ChangeVersion2.

    Version Files Raw Size Compressed Size Unpack Time Peak Memory
    0.4.0 1024 736 GB 178 GB 2h47m39s 9.5 GB
    0.4.1 1024 736 GB 178 GB 0h12m02s 3.5 GB
    0.4.0 239 340 60 GB 17 GB 0h03m34s 1.3 GB
    0.4.1 239 340 60 GB 17 GB 0h01m37s 0.4 GB

    *This strategy is well suited for target storage mediums with fast random access time (such as SDD drives) but will suffer when seek times are higher (such as mechanical drives).
    Longtail_ChangeVersion is still available if you need to cater for mediums with slower seek times.

  • NEW API Longtail_ConcurrentChunkWriteAPI added

  • NEW API Longtail_CreateConcurrentChunkWriteAPI added

  • NEW API Added Longtail_MemTracer_GetAllocationCount to check the number of current memory allocations

  • NEW API Added platform api for Read/Write mutex

    • Longtail_GetRWLockSize
    • Longtail_CreateRWLock
    • Longtail_DeleteRWLock
    • Longtail_LockRWLockRead
    • Longtail_LockRWLockWrite
    • Longtail_UnlockRWLockRead
    • Longtail_UnlockRWLockWrite

  • ADDED SAFE_DISPOSE_STORED_BLOCK macro for easy dispose of stored blocks

  • ADDED Check at test run exit that no memory allocations are left

  • FIXED Updated premake5.lua to set the correct defines for debug vs release builds

  • FIXED Use non-binary units for memtracer counts

  • FIXED Added workaround for calling hmgeti_ts on an empty hash map as that is not thread safe

  • FIXED Use fixed v3 of actions/upload-artifact/actions/download-artifact

  • FIXED Propagate error from FSStorageAPI_Write to caller

  • FIXED Make sure EnsureParentPathExists can create a folder structure and not just parent path

  • FIXED Fixed potential uint32 arithmetic overflow issues

  • CHANGED Command tool uses Longtail_ChangeVersion2 instead of Longtail_ChangeVersion

  • CHANGED Command tool no longer uses LRU block store layer (obsolete with Longtail_ChangeVersion2)

Assets 6
Loading

v0.4.1-pre8

03 Jan 15:48
@github-actions github-actions
v0.4.1-pre8
6c54445
Compare
Choose a tag to compare
v0.4.1-pre8 Pre-release
Pre-release

  • NEW API Longtail_ChangeVersion2 added

    Implements a new strategy for decompressing/writing version assets which is significantly faster for files that spans multiple blocks while retaining the same speed for assets smaller than a block. It removes redundant decompression of blocks (so the LRU block store is no longer needed) at the expense of doing random access when writing files.

    It uses significantly less memory and less CPU while avoiding redundant reads from source data so it is an overall win*.
    If your main target devices are SSD drives I highly recommend switching to Longtail_ChangeVersion2.

    Version Files Raw Size Compressed Size Unpack Time Peak Memory
    0.4.0 1024 736 GB 178 GB 2h47m39s 9.5 GB
    0.4.1 1024 736 GB 178 GB 0h12m02s 3.5 GB
    0.4.0 239 340 60 GB 17 GB 0h03m34s 1.3 GB
    0.4.1 239 340 60 GB 17 GB 0h01m37s 0.4 GB

    *This strategy is well suited for storage mediums with fast random access time (such as SDD drives) but will suffer when seek times are higher (such as mechanical drives).
    Longtail_ChangeVersion is still available if you need to cater for mediums with slower seek times.

  • NEW API Longtail_ConcurrentChunkWriteAPI added

  • NEW API Longtail_CreateConcurrentChunkWriteAPI added

  • NEW API Added Longtail_MemTracer_GetAllocationCount to check the number of current memory allocations

  • NEW API Added platform api for Read/Write mutex

    • Longtail_GetRWLockSize
    • Longtail_CreateRWLock
    • Longtail_DeleteRWLock
    • Longtail_LockRWLockRead
    • Longtail_LockRWLockWrite
    • Longtail_UnlockRWLockRead
    • Longtail_UnlockRWLockWrite

  • ADDED SAFE_DISPOSE_STORED_BLOCK macro for easy dispose of stored blocks

  • ADDED Check at test run exit that no memory allocations are left

  • FIXED Updated premake5.lua to set the correct defines for debug vs release builds

  • FIXED Use non-binary units for mem tracer counts

  • FIXED Added workaround for calling hmgeti_ts on an empty hash map as that is not thread safe

  • FIXED Use fixed v3 of actions/upload-artifact/actions/download-artifact

  • FIXED Propagate error from FSStorageAPI_Write to caller

  • FIXED Make sure EnsureParentPathExists can create a folder structure and not just parent path

  • FIXED Fixed potential uint32 arithmetic overflow issues

  • CHANGED Command tool uses Longtail_ChangeVersion2 instead of Longtail_ChangeVersion2

  • CHANGED Command tool no longer uses LRU block store layer (obsolete with Longtail_ChangeVersion2)

Assets 6
Loading