Background
I've been using Nginx as a reverse proxy for my daily development work. After upgrading Nginx, it stopped serving large files correctly: a vendor.js file built from webpack wasn't being served properly. Chrome console showed:
GET http://nginx.example.com/dist/js/vendor.js net::ERR_CONTENT_LENGTH_MISMATCH
Initial observations
- Chrome DevTools Network tab shows
vendor.jsrequest with status code 200 - Response headers appear correct, but no content preview in Chrome DevTools
- Response Content-Length header matches actual file size (4.6MB)
# ls -l : get the size in bytes
-rw-r--r-- 1 michaelzheng staff 4844863 Jul 2 17:00 vendor.js
# ls -lh : get the size in human readable format
-rw-r--r-- 1 michaelzheng staff 4.6M Jul 2 17:00 vendor.js
- The next diagnostic step is to compare the failing proxied URL with the upstream URL shown in the Nginx error log. Record which route each browser or
curlrequest uses. A successful request alone does not establish whether it bypassed Nginx.
Understanding Nginx proxy buffering
Before debugging, it's important to understand how Nginx proxy buffering works:
What is proxy buffering?
When Nginx acts as a reverse proxy, it can buffer responses from the upstream server before sending them to the client. This allows Nginx to:
- Free up upstream connections faster - The upstream can move on to other requests
- Handle slow clients - Nginx buffers the response and sends it at the client's pace
- Optimize throughput - Reduces number of system calls and improves overall performance
When buffering is used
With proxy_buffering on (the default), Nginx reads upstream responses into memory buffers while sending data to the client. If the response cannot be handled within those buffers, it can write part of the response to a temporary file, subject to the proxy temp-file settings. Buffering and disk spill are different stages.
See the Nginx proxy buffering documentation.
Buffer hierarchy
Nginx uses a two-tier buffering system:
- Memory buffers - Controlled by
proxy_buffers(fast, limited) - Disk buffers - Controlled by
proxy_temp_path(slower, larger capacity)
When memory buffers are full, Nginx writes overflow data to temporary files in proxy_temp_path.
Debugging steps
1. Locate Nginx configuration
nginx -t
# nginx: the configuration file /usr/local/etc/nginx/nginx.conf syntax is ok
# nginx: configuration file /usr/local/etc/nginx/nginx.conf test is successful
This command shows the config file location and validates syntax.
2. Find error log location
Check nginx.conf for the error_log directive. E.g.
error_log /usr/local/var/log/nginx/error.log warn;
3. Check error logs
tail -f /usr/local/var/log/nginx/error.log
The critical error revealed:
2018/07/02 22:29:27 [crit] 14586#0: *3641 open() "/usr/local/var/run/nginx/proxy_temp/1/08/0000000081" failed
(13: Permission denied) while reading upstream, client: 127.0.0.1, server: nginx.example.com,
request: "GET /dist/js/vendor.js HTTP/1.1", upstream: "http://127.0.0.1:8080/dist/js/vendor.js",
host: "nginx.example.com", referrer: "http://nginx.example.com/testabc"
Key insight: The file was too large for memory buffers, so Nginx tried to write to proxy_temp but lacked permissions.
4. Verify Nginx process owner
ps -eo user,group,args | grep nginx
# Inspect both USER and GROUP for the nginx worker process.
# The master can run as root while workers use a different identity.
The worker process runs as nobody, which needs write access to proxy_temp.
5. Check directory permissions
ls -la /usr/local/var/run/nginx/proxy_temp
# drwx------ 2 root wheel 64 Jul 2 17:00 proxy_temp
The directory is owned by root with 700 permissions, preventing nobody from writing.
Solutions
Solution 1: fix directory permissions (recommended)
Use the worker user and group observed on your machine. The original ownership command below uses nobody:admin. Verify the worker's actual user and group on your installation before adapting it. The worker needs write and traversal access to the temp directory. Changing a root-owned directory to 755 alone does not grant a non-owner write access:
# Change owner to nginx worker user
sudo chown -R nobody:admin /usr/local/var/run/nginx/proxy_temp
# Or set broader permissions (less secure)
sudo chmod 755 /usr/local/var/run/nginx/proxy_temp
Pros:
- Maintains optimal performance with buffering enabled
- Allows Nginx to handle large files efficiently
- No configuration changes needed
Cons:
- May be reset after Nginx upgrades
- Requires root access
Solution 2: disable proxy buffering
Add to your nginx.conf or server block:
location / {
proxy_pass http://upstream_server;
proxy_buffering off;
}
Pros:
- Eliminates permission issues entirely
- Simpler configuration
- Lower memory usage
Cons:
- Slower performance for large responses
- Upstream connections held open longer
- Not recommended for production with slow clients
Solution 3: change buffer directory location
Specify a custom path where Nginx has permissions:
http {
proxy_temp_path /var/tmp/nginx/proxy_temp;
}
Create and set permissions:
sudo mkdir -p /var/tmp/nginx/proxy_temp
sudo chown -R nobody:admin /var/tmp/nginx/proxy_temp
sudo chmod 755 /var/tmp/nginx/proxy_temp
Pros:
- Maintains buffering performance
- More control over buffer location
- Can specify larger disk space
Cons:
- Requires configuration change
- Need to ensure directory persists across reboots
Conclusion
In this incident, the Nginx error log identified a temp-directory permission failure. The browser error alone only indicated that the response did not match its declared length. Other causes require different fixes. Understanding Nginx's buffering mechanism helps you choose the right solution:
- For most cases: Fix permissions (Solution 1)
- For streaming/real-time: Disable buffering (Solution 2)
- For complex setups: Custom temp path (Solution 3)
After changing ownership or configuration, validate the configuration, reload Nginx if needed, and repeat the same proxied request. Confirm that the complete response arrives and the permission error no longer appears in the log. Compare the upstream response if the mismatch remains; then test with a slow client before changing buffering behavior in production.