Chunking files into sets of a certain size or number of files with PowerShell

Using PowerShell to scratch another itch. This time though it wasn’t a work or SQL Server related itch for once. I’m a keen cyclist and like to keep track of all my rides, so I’ve been experimenting with various online ride trackers. And I’ve settled on Strava¬†as I can have cool little online badges like this:

And race myself on my own segments.

The only problem was transferring a couple of years worth of data across from other sites or off of my hard drive. Strava has a handy multiple file uploader, but it only allows you to upload up to 25MB or 25 files at a time. To streamline this I wanted to run through all my old files and put them into folders so that each folder contained less than 25MB of data or less than 25 files. Enter PowerShell:

#folder containing files to be "chunked"
$file_to_process = Get-ChildItem F:\strava-tmp

#Hold the current total size of files
$current_total_size = 0

#Set the maximum total size of files for one folder (in this case 24MB)
$max_size = 24*1024*1024

#Set maximum number of files in a folder
$max_count = 25

#Folder numbering count
$current_folder_index = 1

#Count of files in folder
$current_file_count = 0

#Base folder for output folders
$output_base = "f:\strava2"

new-item "$output_base\folder$current_folder_index" -ItemType directory
$fullout = "$output_base\folder$current_folder_index"
foreach ($file in $file_to_process){
    $tmp_size = $file.Length + $current_total_size
    if (($tmp_size -lt $max_size) -and ($current_file_count -lt $max_count)){
        copy-item $file.FullName -destination $fullout
        $current_total_size = $current_total_size + $file.Length
        new-item "$output_base\folder$current_folder_index" -ItemType directory
        $fullout = "$output_base\folder$current_folder_index"
        $current_total_size= $file.length
        copy-item $file.FullName -destination $fullout
        $current_file_count = 1

I’d have then liked to have used <code>Invoke-WebRequest</code> to do the actual uploading for me. But it appears that Strava’s v3 API is invite only, which isn’t that useful really.

This script also comes in handy for splitting files up for emailing if you have a maximum attachment limit on your account.


Leave a Reply

Your email address will not be published. Required fields are marked *