Episode 3 — NodeJS MongoDB Backend Architecture / 3.7 — Handling Files with Express

3.7.c — Storage Engines

In one sentence: Multer uses pluggable storage engines to decide where uploaded files go — disk storage writes files to the filesystem with full control over naming and directory structure, while memory storage holds files as Buffers in RAM for immediate processing or direct cloud upload.

Table of Contents

1. What Is a Storage Engine?

A storage engine tells Multer where and how to store the incoming file. Multer ships with two built-in engines:

┌──────────────────────────────────────────────────────────────┐
│                    STORAGE ENGINES                            │
│                                                              │
│  ┌────────────────────┐    ┌────────────────────┐            │
│  │   DISK STORAGE     │    │  MEMORY STORAGE    │            │
│  │                    │    │                    │            │
│  │  File → Filesystem │    │  File → RAM Buffer │            │
│  │  req.file.path     │    │  req.file.buffer   │            │
│  │  req.file.filename │    │                    │            │
│  │                    │    │  No file on disk   │            │
│  └────────────────────┘    └────────────────────┘            │
│                                                              │
│  Third-party engines:                                        │
│  • multer-storage-cloudinary (upload to Cloudinary)          │
│  • multer-s3 (upload to AWS S3)                              │
│  • multer-gridfs-storage (upload to MongoDB GridFS)          │
└──────────────────────────────────────────────────────────────┘

The three ways to configure storage

// Way 1: dest shortcut (uses diskStorage internally with random names)
const upload = multer({ dest: 'uploads/' });

// Way 2: Explicit diskStorage with custom functions
const upload = multer({
  storage: multer.diskStorage({ destination, filename })
});

// Way 3: Memory storage (file stays in RAM)
const upload = multer({
  storage: multer.memoryStorage()
});

2. Disk Storage — multer.diskStorage()

Disk storage saves files directly to the server's filesystem. You control two things:

  1. destination — which folder to save in
  2. filename — what to name the file
const multer = require('multer');

const storage = multer.diskStorage({
  // destination can be a string or a function
  destination: function (req, file, cb) {
    cb(null, 'uploads/');  // first arg is error (null = no error)
  },

  // filename function — you decide the saved filename
  filename: function (req, file, cb) {
    cb(null, file.originalname);  // WARNING: not unique, can overwrite
  }
});

const upload = multer({ storage: storage });

How the callback pattern works

Both destination and filename use the Node.js error-first callback pattern:

// cb(error, value)
cb(null, 'uploads/');           // success — save to 'uploads/'
cb(new Error('Bad folder'));    // error — Multer stops processing

What req.file looks like with disk storage

{
  fieldname: 'avatar',
  originalname: 'my-photo.jpg',
  encoding: '7bit',
  mimetype: 'image/jpeg',
  destination: 'uploads/',           // ← disk storage property
  filename: '1699123456789-my-photo.jpg', // ← disk storage property
  path: 'uploads/1699123456789-my-photo.jpg', // ← disk storage property
  size: 234567
}

3. Custom Destination Function

The destination function receives req, file, and cb. This means you can make folder decisions based on the request or the file.

Static destination

destination: (req, file, cb) => {
  cb(null, 'uploads/');
}

Dynamic destination by MIME type

const fs = require('fs');

destination: (req, file, cb) => {
  let folder = 'uploads/misc';

  if (file.mimetype.startsWith('image/'))       folder = 'uploads/images';
  else if (file.mimetype.startsWith('video/'))  folder = 'uploads/videos';
  else if (file.mimetype === 'application/pdf') folder = 'uploads/docs';

  // Ensure directory exists
  fs.mkdirSync(folder, { recursive: true });
  cb(null, folder);
}

Dynamic destination by date

destination: (req, file, cb) => {
  const date = new Date();
  const folder = `uploads/${date.getFullYear()}/${String(date.getMonth() + 1).padStart(2, '0')}`;
  fs.mkdirSync(folder, { recursive: true });
  cb(null, folder);
  // Result: uploads/2025/04/
}

Dynamic destination by authenticated user

destination: (req, file, cb) => {
  const userId = req.user?.id || 'anonymous';
  const folder = `uploads/users/${userId}`;
  fs.mkdirSync(folder, { recursive: true });
  cb(null, folder);
  // Result: uploads/users/64a1b2c3d4e5/
}

Important: Always call fs.mkdirSync(folder, { recursive: true }) before saving. If the directory does not exist, Multer throws an error.

4. Custom Filename Function

The default dest shortcut generates random hex strings with no extension. Custom filename functions solve this:

Strategy 1: Timestamp + original name

filename: (req, file, cb) => {
  cb(null, Date.now() + '-' + file.originalname);
  // 1699123456789-profile-photo.jpg
}

Pros: Human-readable, sortable by time.
Cons: Special characters in original name could cause issues.

Strategy 2: Crypto random hex + extension

const crypto = require('crypto');
const path = require('path');

filename: (req, file, cb) => {
  const ext = path.extname(file.originalname);
  const name = crypto.randomBytes(16).toString('hex');
  cb(null, name + ext);
  // a3f8b2c1d4e5f6a7b8c9d0e1f2a3b4c5.jpg
}

Pros: Guaranteed unique (collision probability is astronomically low), no special characters.
Cons: Not human-readable.

Strategy 3: UUID + extension

const { v4: uuidv4 } = require('uuid');
const path = require('path');

filename: (req, file, cb) => {
  const ext = path.extname(file.originalname);
  cb(null, uuidv4() + ext);
  // 550e8400-e29b-41d4-a716-446655440000.jpg
}

Strategy 4: Fieldname + timestamp + extension

filename: (req, file, cb) => {
  const ext = path.extname(file.originalname);
  cb(null, `${file.fieldname}-${Date.now()}${ext}`);
  // avatar-1699123456789.jpg
}

Filename strategies compared

StrategyExample OutputUnique?Readable?Best For
Timestamp + original1699123456789-photo.jpgMostlyYesDev/small apps
Crypto hexa3f8...c5.jpgYesNoProduction
UUID550e8400-...0000.jpgYesNoProduction (standard format)
Fieldname + timestampavatar-1699123456789.jpgMostlyYesSingle-purpose uploads

5. Memory Storage — multer.memoryStorage()

Memory storage does not write the file to disk. Instead, it stores the entire file as a Buffer object in req.file.buffer.

const storage = multer.memoryStorage();
const upload = multer({
  storage: storage,
  limits: { fileSize: 10 * 1024 * 1024 } // IMPORTANT: set limits!
});

app.post('/api/upload', upload.single('avatar'), (req, res) => {
  // File is in memory — NOT on disk
  console.log(req.file.buffer);        // <Buffer ff d8 ff e0 ...>
  console.log(req.file.buffer.length); // file size in bytes

  // req.file.path     → undefined (no file on disk)
  // req.file.filename  → undefined
  // req.file.destination → undefined

  res.json({
    originalname: req.file.originalname,
    mimetype: req.file.mimetype,
    size: req.file.size
  });
});

What req.file looks like with memory storage

{
  fieldname: 'avatar',
  originalname: 'my-photo.jpg',
  encoding: '7bit',
  mimetype: 'image/jpeg',
  buffer: <Buffer ff d8 ff e0 00 10 4a 46 ...>,  // ← memory storage
  size: 234567
  // NO destination, filename, or path properties
}

Common uses for memory storage

  1. Image processing — resize/crop with Sharp before saving
  2. Direct cloud upload — send buffer to Cloudinary/S3 without touching disk
  3. File content analysis — scan for viruses, read CSV data, parse JSON
// Example: Process image with Sharp, then save
const sharp = require('sharp');

app.post('/api/avatar', upload.single('avatar'), async (req, res) => {
  // Resize to 200x200, convert to webp, save to disk
  await sharp(req.file.buffer)
    .resize(200, 200)
    .webp({ quality: 80 })
    .toFile(`uploads/avatars/${Date.now()}.webp`);

  res.json({ message: 'Avatar processed and saved' });
});

6. Disk vs Memory Storage — When to Use Which

FactorDisk StorageMemory Storage
File locationSaved to filesystemHeld in RAM as Buffer
Access viareq.file.pathreq.file.buffer
Large files (>50 MB)Safe — streams to diskDangerous — eats RAM
Small files (<5 MB)FineFine
Image processingRead from disk, process, re-saveProcess buffer directly (faster)
Cloud uploadRead from disk → upload → delete temp fileUpload buffer directly (simpler)
Memory usageLow (file on disk)High (file in RAM)
Disk I/OYes (write + possible read)None
Cleanup neededYes (delete temp files)No (garbage collected)
ScalabilityBetter for large/many filesBetter for small files + cloud pipeline
┌─────────────────────────────────────────────────────────┐
│            DECISION FLOWCHART                            │
│                                                          │
│  Is the file large (>10 MB)?                             │
│    YES → Use Disk Storage                                │
│    NO ↓                                                  │
│                                                          │
│  Will you upload directly to cloud (S3/Cloudinary)?      │
│    YES → Use Memory Storage (buffer → cloud)             │
│    NO ↓                                                  │
│                                                          │
│  Will you process the file (resize, convert)?            │
│    YES → Use Memory Storage (buffer → Sharp → disk/cloud)│
│    NO ↓                                                  │
│                                                          │
│  Will you serve the file from this server?               │
│    YES → Use Disk Storage                                │
│    NO → Either works; Memory is simpler                  │
└─────────────────────────────────────────────────────────┘

7. Complete Disk Storage Example

const express = require('express');
const multer = require('multer');
const path = require('path');
const crypto = require('crypto');
const fs = require('fs');

const app = express();

// --- Disk Storage Configuration ---
const diskStorage = multer.diskStorage({
  destination: (req, file, cb) => {
    const uploadDir = 'uploads/images';
    fs.mkdirSync(uploadDir, { recursive: true });
    cb(null, uploadDir);
  },
  filename: (req, file, cb) => {
    const ext = path.extname(file.originalname);
    const name = crypto.randomBytes(16).toString('hex');
    cb(null, `${name}${ext}`);
  }
});

const diskUpload = multer({
  storage: diskStorage,
  limits: { fileSize: 10 * 1024 * 1024 },
  fileFilter: (req, file, cb) => {
    const allowed = ['image/jpeg', 'image/png', 'image/gif', 'image/webp'];
    if (allowed.includes(file.mimetype)) {
      cb(null, true);
    } else {
      cb(new Error('Only JPEG, PNG, GIF, and WebP images are allowed'), false);
    }
  }
});

// --- Route ---
app.post('/api/upload-disk', diskUpload.single('photo'), (req, res) => {
  res.json({
    message: 'File saved to disk',
    file: {
      originalName: req.file.originalname,
      savedAs: req.file.filename,
      path: req.file.path,
      size: `${(req.file.size / 1024).toFixed(1)} KB`,
      mimetype: req.file.mimetype
    }
  });
});

// Serve uploaded files
app.use('/uploads', express.static('uploads'));

app.listen(3000, () => console.log('Disk storage server running on port 3000'));

8. Complete Memory Storage Example

const express = require('express');
const multer = require('multer');
const fs = require('fs').promises;
const path = require('path');
const crypto = require('crypto');

const app = express();

// --- Memory Storage Configuration ---
const memoryUpload = multer({
  storage: multer.memoryStorage(),
  limits: { fileSize: 5 * 1024 * 1024 },  // 5 MB — keep small for memory
  fileFilter: (req, file, cb) => {
    if (file.mimetype.startsWith('image/')) {
      cb(null, true);
    } else {
      cb(new Error('Only images allowed'), false);
    }
  }
});

// --- Route: Process in memory, then save ---
app.post('/api/upload-memory', memoryUpload.single('photo'), async (req, res) => {
  try {
    // File is in req.file.buffer — do whatever you need
    const ext = path.extname(req.file.originalname);
    const name = crypto.randomBytes(16).toString('hex');
    const filename = `${name}${ext}`;
    const savePath = path.join('uploads', 'processed', filename);

    // Ensure directory exists
    await fs.mkdir(path.dirname(savePath), { recursive: true });

    // Option A: Save buffer directly to disk
    await fs.writeFile(savePath, req.file.buffer);

    // Option B: Process with Sharp first (uncomment if Sharp is installed)
    // const sharp = require('sharp');
    // await sharp(req.file.buffer)
    //   .resize(800, 600, { fit: 'inside' })
    //   .jpeg({ quality: 85 })
    //   .toFile(savePath);

    res.json({
      message: 'File processed and saved',
      file: {
        originalName: req.file.originalname,
        savedAs: filename,
        size: `${(req.file.size / 1024).toFixed(1)} KB`
      }
    });
  } catch (error) {
    res.status(500).json({ error: 'Failed to process file' });
  }
});

app.listen(3000, () => console.log('Memory storage server running on port 3000'));

9. Building a Custom Storage Engine

Multer allows you to create your own storage engine by implementing _handleFile and _removeFile. This is how third-party packages like multer-s3 work.

// custom-storage.js
class CustomStorage {
  constructor(opts) {
    this.getDestination = opts.destination || ((req, file, cb) => cb(null, 'uploads/'));
  }

  _handleFile(req, file, cb) {
    // file.stream is a readable stream of the uploaded file
    // You can pipe it anywhere: disk, cloud, database, etc.

    const chunks = [];
    file.stream.on('data', (chunk) => chunks.push(chunk));
    file.stream.on('end', () => {
      const buffer = Buffer.concat(chunks);
      console.log(`Received ${buffer.length} bytes for ${file.originalname}`);

      // Call cb with file info to attach to req.file
      cb(null, {
        buffer: buffer,
        size: buffer.length,
        customProperty: 'hello from custom engine'
      });
    });
    file.stream.on('error', cb);
  }

  _removeFile(req, file, cb) {
    // Called when an error occurs after file was partially stored
    // Clean up here
    cb(null);
  }
}

module.exports = function (opts) {
  return new CustomStorage(opts);
};

// Usage:
// const customStorage = require('./custom-storage');
// const upload = multer({ storage: customStorage({ destination: 'uploads/' }) });

In practice: You rarely need a custom engine. Use multer-storage-cloudinary, multer-s3, or memory storage + your own upload logic.

10. Key Takeaways

  1. Disk storage saves files to the filesystem — use multer.diskStorage() with destination and filename functions for full control.
  2. Memory storage holds files as Buffers in RAM — use multer.memoryStorage() for processing or cloud upload pipelines.
  3. Always create directories with fs.mkdirSync(dir, { recursive: true }) in the destination function.
  4. Prevent filename collisions with crypto hex, UUID, or timestamps — never use file.originalname directly.
  5. Use disk storage for large files and when serving files from the same server.
  6. Use memory storage for small files that will be processed (Sharp) or uploaded to cloud storage (Cloudinary, S3).
  7. Set file size limits — especially critical with memory storage where large files consume RAM.
  8. Custom storage engines are possible but rarely needed — third-party packages exist for common cloud providers.

Explain-It Challenge

Can you explain to a friend: "When should I use disk storage vs memory storage in Multer, and what are the trade-offs?" If you can draw the decision flowchart from memory, you have mastered this topic.

Next → 3.7.d — File Validation and Security