Build a Scalable Node.js File Upload Pipeline with S3, Busboy, Sharp, and Presigned URLs

Have you ever found yourself staring at a pile of uploaded files, wondering if your server is about to choke? I have. A production incident taught me the hard way. Naive file upload code – the kind that buffers everything in memory or writes to disk – works for a demo but fails at scale. That night, a single 200MB PDF brought our Node.js server to its knees. The event loop stalled, users got timeouts, and I spent hours debugging. That’s when I decided to build a secure, scalable upload pipeline. Not just any pipeline. One that streams files directly to AWS S3, transforms images on the fly with Sharp, and never blocks the event loop. Let me walk you through how I did it.

Why start with streaming instead of buffering? Think about a typical upload: the client sends a multipart form with a file. Without streaming, your server reads the entire payload into memory or writes it to a temporary file. That works for a few small uploads, but what happens when 100 users simultaneously upload high‑resolution photos? Memory usage spikes, garbage collection struggles, and your app becomes unresponsive. The solution is to pipe the incoming data directly to S3 using Node.js streams. This way, your server acts as a tiny relay – it never stores the full file anywhere. Handle a single chunk, send it to S3, rinse and repeat.

But before diving into code, I needed to decide on the upload strategy. Do I accept files on my server and then push them to S3? Or do I generate presigned URLs so clients can upload directly to S3? Both have tradeoffs. Server‑side upload gives me full control over validation and processing, but it ties up my server resources. Presigned URLs offload the heavy lifting to AWS, but then I lose the ability to inspect the file before it hits the bucket. For my use case – a photo‑sharing app that needs image thumbnails and virus scanning – I chose a hybrid approach: accept small uploads on the server after validation, and for large files, generate presigned URLs for direct client upload, then post‑process asynchronously.

Enough theory. Let me show you the core of the pipeline. I use Busboy, a streaming multipart parser, to handle form data without touching the file system. Here’s a simplified version of how I wire it up:

import { Request, Response } from 'express';
import Busboy from 'busboy';
import { S3Client, PutObjectCommand } from '@aws-sdk/client-s3';

const s3 = new S3Client({ region: 'us-east-1' });

app.post('/upload', (req: Request, res: Response) => {
  let fileCount = 0;
  const busboy = Busboy({ headers: req.headers, limits: { fileSize: 50 * 1024 * 1024 } });

  busboy.on('file', (fieldname, file, info) => {
    const { filename, mimeType } = info;
    // Validate mime type – reject if not allowed
    if (!['image/jpeg', 'image/png', 'image/webp'].includes(mimeType)) {
      file.resume(); // discard the stream
      return res.status(400).send('Invalid file type');
    }

    const key = `uploads/${Date.now()}-${filename}`;
    const passThrough = new PassThrough();
    // Pipe file data to S3
    const upload = s3.send(new PutObjectCommand({
      Bucket: process.env.S3_BUCKET,
      Key: key,
      Body: passThrough,
      ContentType: mimeType,
    }));

    file.pipe(passThrough);
    fileCount++;

    file.on('limit', () => {
      console.error('File size exceeded');
      file.destroy();
    });
  });

  busboy.on('finish', () => {
    res.json({ message: `Uploaded ${fileCount} files` });
  });

  req.pipe(busboy);
});

Notice how I never write to disk. The file readable stream is piped through a PassThrough and then directly into the S3 PutObjectCommand body. Busboy handles backpressure automatically. If a client sends a 2GB video, my server only holds a few kilobytes in memory at any moment. That’s the beauty of streaming.

But what if you need to generate image thumbnails? Here’s where Sharp enters. Sharp is a fast, streaming image processing library. I can pipe the incoming file through a Sharp transform before it reaches S3. For example, I create a thumbnail variant:

import sharp from 'sharp';

busboy.on('file', (fieldname, file, info) => {
  const thumbnailKey = `thumbnails/${filename}`;
  const thumbnailStream = sharp().resize(200, 200).webp({ quality: 80 });
  const uploadThumbnail = s3.send(new PutObjectCommand({
    Bucket: process.env.S3_BUCKET,
    Key: thumbnailKey,
    Body: file.pipe(thumbnailStream),
    ContentType: 'image/webp',
  }));
  // ... also upload original file via another stream
});

Wait – can you pipe the same stream to two destinations? No, a Node.js stream can only be consumed once. You’d need to tee the stream or use a PassThrough to fork it. A cleaner approach is to write both variants in parallel using separate PassThrough instances. But watch out: Sharp operations are CPU‑intensive and synchronous in older APIs. Sharp’s current version uses asynchronous pipelines, but heavy image processing can still block the event loop if not handled with worker threads. For production, I offload Sharp processing to a separate worker pool using worker_threads. That keeps the main thread responsive.

Now, security. You can’t trust any uploaded file. MIME type validation from the client is easily spoofed. I check the file’s magic bytes using a library like file-type after receiving the first few chunks. For virus scanning, I integrate ClamAV. With clamscan, I can scan the stream as it passes through. However, scanning a live stream is tricky: you need to buffer enough of the file to pass to ClamAV, but not the whole file. A practical compromise is to scan the file after it’s stored in S3 using an AWS Lambda triggered by S3 events. That way, you don’t slow down the upload path. For extra caution, you can pre‑scan small files in‑memory using clamscan with a socket connection.

Another lesson I learned: always set strict size limits and timeouts. Use Busboy’s limits.fileSize and also an overall request timeout. Malicious clients can send headers that promise a tiny file but then stream forever. I also generate a unique fileId with uuid and store metadata in a database after upload – but for the article’s sake, let’s keep it simple.

But what about large files? If a user wants to upload a 1GB video, server‑side streaming works, but it ties up your server for a long time. That’s where presigned URLs shine. Using the AWS SDK’s @aws-sdk/s3-request-presigner, I generate a temporary URL that allows the client to upload directly to my bucket. Here’s how:

import { getSignedUrl } from '@aws-sdk/s3-request-presigner';
import { PutObjectCommand } from '@aws-sdk/client-s3';

const command = new PutObjectCommand({
  Bucket: bucketName,
  Key: `user-uploads/${filename}`,
  ContentType: mimeType,
});
const url = await getSignedUrl(s3Client, command, { expiresIn: 3600 });

Return this URL to the client, and they can PUT the file directly to S3. After upload, an S3 event triggers a Lambda function that processes the image, scans for viruses, and stores metadata. This architecture scales horizontally – your server only handles the presigned URL generation, not the data transfer.

I remember the first time I implemented this hybrid approach. The relief was immediate. Our server CPU dropped from 80% to 5% during peak upload hours. The only thing left to worry about was proper error handling and retries. For instance, when Sharp fails to process a corrupted image, I need to delete the already‑uploaded S3 object and return a meaningful error. That’s why I use a transactional model: store temporary objects, process them, and then move them to a permanent location upon success.

Now, let’s talk about testing. I use Vitest with Supertest to mock the S3 client and test the streaming logic. I create a readable stream from a buffer to simulate file uploads. For image processing tests, I generate simple test images with Sharp. And I always test boundary conditions: zero‑byte files, extremely large files, invalid content types, and concurrent uploads.

One personal habit: I always log every upload with a unique request ID. When something goes wrong, I can trace the entire flow from the incoming request to the S3 object. It saved me hours during debugging.

So, what have we learned? Streaming file uploads is not just about performance – it’s about reliability and security. Use Busboy to parse multipart data without buffering. Pipe directly to S3 using the AWS SDK’s streaming support. Integrate Sharp for on‑the‑fly image transformation, but be mindful of CPU usage. Validate every byte, scan for viruses asynchronously, and never trust client data. And for large files, let the client talk directly to S3 with presigned URLs.

If you’re still buffering files in memory, stop now. Your users and your server will thank you. I’ve seen too many applications crash under the simplest upload load. Don’t let that be your story.

Did this help you rethink your file upload strategy? Did I miss any edge case you’ve encountered? I’d love to hear your experiences. Drop a comment below if you’ve built something similar or if you have questions about the code. And if you found this useful, share it with a teammate who’s still using multer.diskStorage. Don’t forget to like the article – it tells me you want more deep, practical guides like this.

---

As a best-selling author, I invite you to explore my books on Amazon. Don’t forget to follow me on Medium and show your support. Thank you! Your support means the world!

---

101 Books

101 Books is an AI-driven publishing company co-founded by author Aarav Joshi. By leveraging advanced AI technology, we keep our publishing costs incredibly low—some books are priced as low as $4—making quality knowledge accessible to everyone.

Check out our book Golang Clean Code available on Amazon.

Stay tuned for updates and exciting news. When shopping for books, search for Aarav Joshi to find more of our titles. Use the provided link to enjoy special discounts!

---

📘 Checkout my latest ebook for free on my channel!
Be sure to like, share, comment, and subscribe to the channel!

---

Our Creations

Be sure to check out our creations:

Investor Central | Investor Central Spanish | Investor Central German | Smart Living | Epochs & Echoes | Puzzling Mysteries | Hindutva | Elite Dev | JS Schools

---

We are on Medium

Tech Koala Insights | Epochs & Echoes World | Investor Central Medium | Puzzling Mysteries Medium | Science & Epochs Medium | Modern Hindutva