Something Odd!

I share a photo library between two Immich users, my photos visible in my partner’s account, using an external library. Instead of uploading, Immich points at a folder on disk and imports whatever it finds there. It’s a lovely setup right up until the library gets big. Mine is big: over 750,000 photos, with more than half a million in a single external library.

Running out of memory

My server was struggling, chewing into swap all the time and running out of memory.

microservices worker error: Error [ERR_WORKER_OUT_OF_MEMORY]:
Worker terminated due to reaching memory limit: JS heap out of memory

I had Immich running in a 6GB Docker container but reduced that to 4GB for other reasons, and it appeared to work ok for a while, but it took a while for me to notice it was using the entirety of the 4GB of RAM. I increased the RAM allocated to the container, but it was still a heavy user.

The library watcher

External libraries have two ways to notice new files. There’s a periodic scan, which I’d already disabled because it’s brutal at this size. And there’s the watcher, which uses filesystem notifications to spot new files the moment they land. That’s what makes photos I add show up automatically.

The watcher is the convenient one. It’s also the expensive one. Watching a folder tree with 750,000 files means holding an enormous number of filesystem watches plus the bookkeeping that goes with them, and on my library that was around 3GB of resident memory, sitting there permanently doing nothing most of the time.

I turned it off in the admin settings. Memory use went from ~4GB pinned at the limit to about 1GB. Nightly jobs suddenly had room to breathe and the crash loop stopped.

Except now new photos don’t get imported, because the watcher was the thing importing them. So: how do you add a photo to a huge external library without a watcher and without a full scan?

The obvious fix that doesn’t work

Immich has an API endpoint to scan a library:

POST /api/libraries/{id}/scan

It scans the whole library. On my photo archive that’s a long, heavy crawl, and I’d already learned the hard way what a scan does to this box. Not something you run every time you drop in a picture.

Enqueue the import job yourself

So I went looking at what the watcher actually does when it sees a new file, expecting something complicated. It’s almost nothing. It queues one background job:

jobRepository.queue({
    name: 'LibrarySyncFiles',
    data: { libraryId, paths: [path] },
})

LibrarySyncFiles takes an explicit list of paths and imports exactly those files, without crawling the other 500,000. The per-file import I wanted was already sitting there. The watcher was only ever the trigger, and I was paying 3GB for the trigger.

I can pull that trigger myself. Immich uses BullMQ (backed by Redis/Valkey) for its job queue, so I can push the same job onto the same queue. The photos I add already go through a little script that moves them into the library folder by date, so the import hooks straight into that. When the script moves a new file in, it queues a LibrarySyncFiles job for that exact file:

docker exec -w /usr/src/app/server immich_server node -e '
  const { Queue } = require("bullmq");
  const q = new Queue("library", {
    prefix: "immich_bull",
    connection: { host: "redis", port: 6379 },
  });
  q.add("LibrarySyncFiles", {
    libraryId: "your-library-id",
    paths: ["/path/inside/the/container/to/new-photo.jpg"],
  }).then(() => q.close());
'

Immich picks it up, runs its normal import pipeline on that one file (metadata, thumbnails, the lot) and the photo appears. No watcher sitting on gigabytes of RAM, no full-library scan, and new photos still land within a cycle of my import script.

The caveats, because this is a hack

It leans on Immich’s internals, so: only send files that are genuinely new. The job handler inserts unconditionally, so hand it a path that’s already imported and you get a duplicate-key error in the logs. My script only queues files it actually moved.

More seriously, this is an internal job queue, not a public API. The queue name, the job name and the payload shape are implementation details, and an upgrade could rename or reshape any of them. The failure mode is silent: photos would just quietly stop importing and I wouldn’t find out until I went looking for one. My script shouts if the enqueue fails, which covers some of that but not all of it. This was working on Immich 3.0.2.

And turning off the watcher also turns off deletion detection, since the watcher is what removes assets when their files vanish. My workflow only ever adds files, so it doesn’t bite me. It might bite you.

There is a discussion here asking for an API endpoint to scan single files, but it hasn’t received much attention in 2 years, so I guess not many people have this issue, or realise it is an issue.

Was it worth it?

For a library this size, yes. About 3GB of RAM back, no more crash loop, and photos still import on their own. If your external library is small this is all irrelevant, so leave the watcher on and get on with your life. But if you’re in the hundreds of thousands of files and your server keeps falling over, check the watcher first.