💾 Copying 16TB with CopyQueue
I know I’m behind the times with spindle drives in external enclosures, but after deleting terabytes worth of duplicates and unneeded data, I need to move huge chunks of data.
When you have tens of thousands of files, ranging in size from 200k to 10GB, how do you reliably move all that?
Well, duh, you use Finder. Or use one of the many products with the word Finder in the name. Or, you use one of the dozen tools with the name Commander.
That works when moving a few files. They’re fine transferring files from one SSD to another, but this is an older Mac Pro with spindle drives inside. There’s also spindle drives in enclosures. SSDs are awesome, but there is no such thing as a $99 8TB SSD. That’s the price of hard drives, so that’s what I have.
The problem is, since I have large blocks of free space, I want to consolidate files and reorganize. I’m moving about 15,000 files of all sizes and types. Regular file management tools start to lose their minds with that many. They spend hours trying to calculate what they’re moving, then struggle adapting from small .jpg files to multi-gig .mp4 files. The transfer speeds drops to almost nothing making the transfer take days.
I’d previously done some of this work using rsync. Even that spent far too much time calculating what files to work with rather than moving them.
Enter CopyQueue a purpose built file copying utility that can handle thousands of files of differing types and sizes without choking. For those familiar with the Windows world, it’s like TeraCopy.
It supports batch copying, starting and stopping queues, moving/copying, and verifying files. It can be set to run on a schedule, it can throttle so as not to overwhelm a network connection, and if a file fails, it gets marked, but doesn’t derail the rest of the operation.
A big advantage for me, when moving a file, if the source file exists in the target location, and the name and size are the same, the file is deleted from source. This is exactly what I was looking for from rsync.
CopyQueue starts copying within seconds, not hours or 30 minutes later after it’s gone through and parsed every file in the list.
You can keep adding files to the list and CopyQueue will add them to the queue.
It may seem like a small thing, but it’s a massively helpful tool. Maybe not that many people need it, but CopyQueue is exactly what I was looking for to move 16TB of files around.
I set them in batches and let CopyQueue work. I could still handle other file actions without waiting for those massive copies to complete. CopyQueue shows the number of files to copy, how many are left and transfer speed. I was moving files in one window and deleting them in another, like a Boss!
If you need to move huge chunks of data to external drives, this is the tool to get. The only downside is there hasn’t been updates in more than a year. I’m not sure if that means the product is totally fine and doesn’t need adjustment or the dev has abandoned it.
I’m on an older model Mac, it works without, so I don’t care.
