Skip to content
8 min read

The Hidden Culprit: How macOS Quarantine Attributes Broke Our Lambda Deployment

A trainee’s journey through a strange AWS deployment bug

Originally published on Substack

As a trainee on our development team, I ran into one of those bugs that makes you scratch your head for hours. What should have been a simple Lambda layer update turned into a learning experience that taught us something unexpected about macOS and AWS.

Here’s how we solved it, with help from our senior developers and an AI assistant.

The Problem

We needed to update our Lambda layers for two environments: staging and production. Both had the same Node.js code, but only production was failing.

The error was:

Error: Cannot find module ‘jsonwebtoken’
Require stack:
- /var/task/lib/helper/utils.js
- /var/task/lib/validate/schema/validation.js
- /var/task/lib/validate/index.js
- /var/task/index.js
- /var/runtime/index.mjs

The weird part? The package was definitely there in the `node_modules` folder.

The Different Approaches

Here’s what we did differently:

Staging (worked fine):
• Downloaded existing layer ZIP
• Added one new package using `yarn add crypto-js`
• Uploaded it — worked perfectly ✅

Production (failed):
• Downloaded existing layer ZIP
• Deleted the entire `node_modules` folder (for a clean install)
• Ran `yarn install` to reinstall everything
• Uploaded it — got the “Cannot find module” error ❌

Both used `yarn` initially.

The Senior Developer’s Insight

After banging our heads against this problem for a while, we decided to ask the senior developers on our team. One of them immediately said, “This might be related to which package manager you’re using. Try installing with npm instead of yarn.”

So we did exactly that. We deleted the `node_modules` folder again, ran `npm install` instead of `yarn install`, and… it worked! The production layer deployed successfully.

But this left us more confused than satisfied. Why would the package manager make a difference? The `package.json` files were identical. The resulting `node_modules` should be the same, right?

This is when our curiosity kicked in, and we decided to investigate deeper.

Starting Our Investigation

We’re the kind of developers who can’t sleep until we understand why something works 😅 (or doesn’t work). So we started comparing everything we could think of:

• File structures: Identical ✅
• Package versions: Identical ✅
• File contents: Identical ✅

Everything looked the same on the surface. That’s when we decided to get some help from an AI assistant to dig deeper into this mystery.

The AI Detective Joins the Case

This is where things got really interesting. We started working with an AI assistant (GitHub Copilot, to be precise) to help investigate this mystery. We suggested the AI to look beyond just file contents and examine file metadata.

The first breakthrough came when the AI checked the extended file attributes on macOS using the `xattr` command. I had never heard of this before, but willing to try anything.

xattr -l /path/to/staging/node_modules/jsonwebtoken/index.js
# No output - clean file
xattr -l /path/to/production/node_modules/jsonwebtoken/index.js
# com.apple.quarantine: 0081;6880d219;Browser;
# com.apple.provenance:

Bingo! The production directory had these mysterious “quarantine” attributes that the staging directory didn’t have.

The Investigation Deepens

Working with the AI, we set up some controlled experiments to understand this better. We created fresh installations using both npm and yarn:

# Fresh npm install
mkdir test-npm && cd test-npm && npm install [email protected]
xattr -l node_modules/jsonwebtoken/index.js
# No output - clean
# Fresh yarn install
mkdir test-yarn && cd test-yarn && yarn add [email protected]
xattr -l node_modules/jsonwebtoken/index.js
# com.apple.provenance: (yarn adds this attribute)

This was fascinating! Yarn was adding extended attributes that npm wasn’t adding.

The Eureka Moment

With the AI’s help, we finally understood what was happening. The issue wasn’t really about npm vs yarn — it was about macOS security features called “quarantine attributes.”

Here’s what was actually going on:

The Real Culprit: macOS Quarantine System

macOS has a security feature that automatically “quarantines” files downloaded from the internet. When you download a ZIP file and extract it, macOS marks those files with special attributes to prevent potentially malicious code from running.

The key insight was understanding how our different workflows triggered this:

Staging (the working one):

  1. Downloaded existing layer ZIP

  2. Extracted it (files got quarantined, but we didn’t touch node_modules)

  3. Added one package with `yarn add crypto-js` (this didn’t trigger fresh downloads of existing packages)

  4. The existing modules kept their clean attributes ✅

Production (the broken one):

  1. Downloaded existing layer ZIP

  2. Extracted it (files got quarantined)

  3. Deleted node_modules (removed the quarantined files)

  4. Ran `yarn install` (fresh downloads got quarantined again!)

  5. ZIP compression preserved these quarantine attributes

  6. AWS Lambda couldn’t execute quarantined files ❌

Why npm “Fixed” It

npm doesn’t add extended attributes the way yarn does. When we switched to npm, we got clean files without the problematic attributes that were preventing Lambda from loading the modules.

The Lambda Environment Issue

Important discovery : Locally, both yarn and npm installations work fine for importing `jsonwebtoken`. The issue only manifests in AWS Lambda’s containerized Linux environment where:

  1. macOS quarantine attributes get preserved in ZIP files

  2. Lambda’s runtime can’t execute files with these macOS-specific attributes

  3. This causes the “Cannot find module” error despite files being physically present

The Solution

We found the actual fix was removing these quarantine attributes or changing the package manager :

# Remove quarantine attributes from all files
find ./production-layer -exec xattr -d com.apple.quarantine {} \; 2>/dev/null
find ./production-layer -exec xattr -d com.apple.provenance {} \; 2>/dev/null

After running these commands, we verified the attributes were gone:

xattr -l ./production-layer/node_modules/jsonwebtoken/index.js
# No output - clean!

We repackaged the layer and deployed it. Success! The production Lambda function was working again.

What We Learned

This whole experience taught us several valuable lessons:

1. Sometimes the Problem Isn’t What You Think

We initially thought this was a package manager issue, but it was actually a macOS security feature interfering with AWS Lambda’s execution environment.

2. AI Can Be an Amazing Debugging Partner

The AI assistant significantly reduced the time we spent on this investigation. It guided us through the `xattr` commands I never used before, and helped us set up controlled experiments to understand the issue.

3. Workflow Matters More Than Tools

The issue wasn’t really about npm vs yarn — it was about how we prepared the deployment packages. Incremental updates preserve file attributes, while fresh installations can trigger security measures.

4. macOS Security Features Can Affect Cloud Deployments

Who would have thought that downloading a file on macOS could affect AWS Lambda deployment? These kinds of cross-platform issues are tricky to debug because they’re not obvious.

5. Local Testing vs Production Environment

Both yarn and npm work fine locally, but the extended attributes only cause issues in Lambda’s containerized environment. This makes the problem even harder to diagnose.

How to Prevent This Issue

If you’re developing on macOS and deploying to AWS Lambda, here are some tips to avoid this problem:

For Quick Fixes

# Clean extended attributes before packaging your layer
find ./your-lambda-layer -exec xattr -c {} \; 2>/dev/null || true

For Build Pipelines
Add this to your deployment scripts:

# Remove macOS quarantine attributes
xattr -rc ./build/lambda-layers/ 2>/dev/null || echo “No attributes to remove”

For Fresh Installations
If you need to delete and reinstall node_modules:

rm -rf node_modules
yarn install # or npm install
# Clean any quarantine attributes
find node_modules -exec xattr -rc {} \; 2>/dev/null || true

The Docker Alternative
The most reliable solution is to build your Lambda layers in a Linux environment using Docker:

FROM node:18-alpine
WORKDIR /app
COPY package*.json ./
RUN npm ci - production

This completely avoids macOS-specific issues.

Final Thoughts

This bug was one of those rare cases where the solution was completely unrelated to what we initially suspected. It reinforced our belief that debugging is as much about asking the right questions as it is about having the right tools.

Having an AI assistant during this investigation was game-changing. It helped us explore areas I wouldn’t have thought of on my own and guided me through unfamiliar territory. The combination of human curiosity and AI’s broad knowledge base made for a powerful debugging team.

If you’re facing similar mysterious deployment issues, don’t give up! Sometimes the answer is hiding in places you’d never think to look. And don’t hesitate to get help — whether from senior colleagues, AI assistants, or the broader developer community.

Environment Details

• Operating System: macOS 15.5 (24F74)
• Node.js Version: v22.12.0
• npm Version: 11.2.0
• Yarn Version: 1.22.22
• AWS Lambda Runtime: Node.js

The macOS Quarantine System

macOS automatically marks files downloaded from the internet with quarantine attributes:

com.apple.quarantine: [flags];[timestamp];[source_app];[UUID]
Example: 0081;6880d219;Brave;

Package Manager Differences

| Package Manager | Extended Attributes | Quarantine Risk  |
| - - - - - - - - | - - - - - - - - - - | - - - - - - - - -|
| npm 11.2.0      | None                | Low              |
| Yarn 1.22.22    | Adds provenance     | Medium           |

AWS Lambda Requirements

AWS Lambda expects POSIX permissions and clean files:

• Files: 644 (rw-r — r — )
• Directories: 755 (rwxr-xr-x)
• Layer structure: `layer.zip/nodejs/node_modules/`

When files have macOS quarantine attributes, they can’t be executed properly in Lambda’s containerized Linux environment.

Want to avoid similar issues?

Always clean extended attributes before packaging Lambda layers on macOS, or better yet, use Docker to build in a Linux environment from the start.

Sometimes the best debugging happens when human curiosity meets AI’s vast knowledge base.

References

npm vs yarn extended attributes: This behaviour difference stems from how Yarn handles package metadata and integrity checks. Yarn maintains additional provenance information and uses more sophisticated caching mechanisms that can trigger macOS extended attributes. See: [here]

AWS Lambda deployment best practices : AWS Lambda Deployment Package Documentation - Guidelines for packaging Lambda functions and layers.

Lambda runtime execution environment: [AWS Lambda Execution Environment] — Details about Lambda’s containerized Linux environment and file execution requirements.

Building Lambda layers with Docker: [AWS Lambda Layers Documentation] and [Building Lambda functions with Docker] - Official guidance on using Docker for Lambda development.

← All posts