How to move files in AWS S3 to another folder or bucket

This post contains affiliate links. If you use these links, I may earn a commission (at no cost to you). I only recommend products I use myself that solve a specific problem. In this post, you are recommended DigitalOcean Spaces, for S3-compatible storage with affordable and predictable pricing. Get started with a $200 credit when you use my referral.

If you're using S3 for object storage for your app or website, you might come across a use case where you need to move files from one folder to another, or even one bucket to another.

In this blog post, we will look at how to move files/folders using the AWS SDK. I'm using @aws-sdk for Node.js. Unfortunately, there's no move command in the AWS SDK, so we will need to code our own moveFolder function.

There might not be a move command in the AWS SDK, but there is in AWS CLI. If this is a one-off move and you're happy using the command line, check out mv in the AWS CLI.

As discussed in the how to copy an S3 folder blog post, folders don't actually exist in S3 so moving a file from one folder to another isn't possible.

In S3, folders are really just prefixes in filenames:

// ❌ folder
📁 projects/
├── 📁 project1/
 ├── 📄 first-file.png

// ✅ prefix
📄 /projects/project1/first-file.png

To move a file, what you would need to do is rename the prefix in the file name - but there's no rename command either!

So we'll code our own moveFolder function for AWS S3 that will move files by:

• copying the S3 files in the folder to the new folder (or bucket) location
• deleting the S3 files in the original location

And since you can only copy files one at a time we will make our function recursive so that all the files in the folder get moved.

As luck would have it, I've already written how to copy folders and delete folders in s3 in two previous blog posts. So we can use that code to make our moveFolder function. Yay!

NodeJS @aws-sdk set up

If you already have @aws-sdk installed and configured, you can skip to the next section. Let's start by using the latest version (v3.x) of AWS SDK.

npm install @aws-sdk/client-s3

Now we will configure it with an S3-compatible service.

I use DigitalOcean Spaces as my S3-compatible object storage provider. I'm pretty pleased with the performance and price predictability, and I found it easier to get started than going directly with AWS.

My setup looks like this:

const { S3Client } = require('@aws-sdk/client-s3');

const s3 = new S3Client({
  endpoint: "https://ams3.digitaloceanspaces.com",
  forcePathStyle: false,
  region: "ams3",
  credentials: {
    accessKeyId: process.env.S3_KEY,
    secretAccessKey: process.env.S3_SECRET
  }
});

If you use AWS directly, the setup is similar, but it will look more like this:

const { S3Client } = require('@aws-sdk/client-s3');

const s3 = new S3Client({
  region:'eu-west-1',
  credentials: {
    accessKeyId: process.env.S3_KEY,
    secretAccessKey: process.env.S3_SECRET
  }
});

Don't forget to switch the region to wherever your buckets are located, and pass in your own credentials.

I use DigitalOcean Spaces for S3-compatible object storage and I'd recommend it to a friend - you can get a $200 credit to try out DigitalOcean Spaces here.

Ok, now we have S3 set up in NodeJS, let's start coding a moveFolder function.

moveFolder function

function moveFolder() {
 // we will add the code here
};

Because folders don't exist in S3, what our moveFolder function will need to do is:

• Get a list of all the items at a specific prefix (a.k.a. the "folder")
• Copy each of those items to a new prefix (a.k.a. the new "folder")
• Delete the items at the original prefix (a.k.a the "folder")

To do this, our moveFolder function is going to need to take a few arguments:

• fromBucket - the bucket where we are copying the folder from
• fromLocation - the path to the folder/prefix
• toBucket - in case we want to copy the folder to a different bucket
• toLocation - the path for the new folder/prefix

Since we will be providing a few arguments, and one of them toBucket will be optional, I'm going to pass an object as a parameter to the moveFolder function so we don't have to worry about the order.

function moveFolder({ fromBucket, fromLocation, toBucket, toLocation }) {
  // we will add the code here
};

Now let's default the toBucket to fromBucket so if we are moving a folder in the same bucket, we only need to give it the fromBucket argument.

function moveFolder({ fromBucket, fromLocation, toBucket = fromBucket, toLocation }) {
  // we will add the code here
};

Now if you've read my other S3 blog posts, and are thinking that this all looks a bit similar to the copyFolder function - you're right! And since we are starting by copying files to the new folder, we can re-use that function here.

Use the copyFolder function

The first major task for moving the files to another folder is to copy the files to the new location.

I've already coded a recursive function (the copyFolder S3 function) that does this task. And I wrote a blog post about it.

I'm just going to call the copyFolder function inside moveFolder, to copy the files, so you'll need to grab the code from how to copy an S3 folder with aws-sdk in Node.js. You can also read the blog post to get a detailed breakdown of how the function works.

Since that's an asynchronous function, we will make the moveFolder function async - so that we can await the copyFolder function.

async function moveFolder({ fromBucket, fromLocation, toBucket = fromBucket, toLocation }) {
  // copy the files to the new location
  await copyFolder({ fromBucket, fromLocation, toBucket, toLocation });
};

The copyFolder function takes the same arguments as moveFolder, so we can pass those arguments along to copyFolder. Nice!

Use the deleteFolder function

Now the files have been copied we need to delete the files from the original location (as we want them moved - not copied!). Since we previously coded a deleteFolder S3 function, we can call that function to do this job!

Again, I'm just going to call the deleteFolder function inside moveFolder, to delete the files, so you'll need to grab the code from how to delete an S3 folder recursively with aws-sdk in Node.js.

async function moveFolder({ fromBucket, fromLocation, toBucket = fromBucket, toLocation }) {
  // copy the files to the new location
  await copyFolder({ fromBucket, fromLocation, toBucket, toLocation });
  // delete the files from the original location
  await deleteFolder({ bucket: fromBucket, location: fromLocation });
  
  return `Files moved from ${fromBucket}/${fromLocation} to ${toBucket}/${toLocation}.`;
};

The deleteFolder function takes the bucket and location arguments, so we will pass the fromBucket as bucket and fromLocation as location since they are the original files we want to delete.

Final Code

const { S3Client } = require('@aws-sdk/client-s3');

// s3 client
const s3 = new S3Client({
  region: "your-region",
  credentials: {
    accessKeyId: process.env.S3_KEY,
    secretAccessKey: process.env.S3_SECRET
  }
});

// copy folder S3
async function copyFolder({ fromBucket, fromLocation, toBucket = fromBucket, toLocation }) {
  // ...
};

// delete folder S3
async function deleteFolder({ bucket, location }) {
  // ...
};

// move folder S3 🆕
async function moveFolder({ fromBucket, fromLocation, toBucket = fromBucket, toLocation }) {
  // copy the files to the new location
  await copyFolder({ fromBucket, fromLocation, toBucket, toLocation });
  // delete the files from the original location
  await deleteFolder({ bucket: fromBucket, location: fromLocation });
  
  return `Files moved from ${fromBucket}/${fromLocation} to ${toBucket}/${toLocation}.`;
};

Super!

Don't forget to read the other posts in this series for copyFolder and deleteFolder as you will need the code for these two functions!