Fixed it. See the full answer in the Sophos forums.
Sophos on Linux has two different ways of knowing when files are being accessed: Talpa and fanotify. Fanotify is the recommended approach and solves some performance issues. I thought we were already using it because the preferFanotify flag was set but it wasn’t having any effect because the disableFanotify flag was also set. Configuring the antivirus correctly has made the builds run at their normal speed.