How to: Pass NginX the correct client IP address when using Varnish reverse proxy

How to: Pass NginX the correct client IP address when using Varnish reverse proxy

Varnish is a great piece of software that you can use to serve static http content without having to hit your web servers. Many of the issues that come with using a reverse proxy like Varnish is that there is another layer now in-between your webserver and your client and that tends to complicate things at least a little. One of those complications is obtaining the client ip address of the original visitor. by default all the activity for your webserver will appear to be coming from Varnish not the actual visitor. This could be an issue for example in the next scenarios:

• GEOIP does not resolve the country
• Apache/NginX logs write 127.0.0.1 as request client IP (or if Varnish is in another server the IP address from that server)
• PHP / .htaccess / NginX logic cannot longer apply filters by IP (Think blocking bots, WP Better Security, etc.)

So it seems important to be able to identify the original client IP in case we have logic at any point that uses that information.

But that has an easy fix. All you need to do is to tell Varnish  to send the client IP to the backend. This can be easily  accomplished by setting in your VLC  (/etc/varnish/default.vcl) the following:

1. sub vcl_recv {
3. remove req.http.X-Forwarded-For;
4. set req.http.X-Forwarded-For = client.ip;
5. …

With this change your  application will be able to get the  client IP and resolve correctly the country using GeoIP. When  your application needs the client IP you only have to access to  the header X-Forwareded-For. But caution! If you ALWAYS want that header set in every request,  you need to add this:

1. sub vcl_pipe {
2. # Note that only the first request to the backend will have
3. # X-Forwarded-For set. If you use X-Forwarded-For and want to
4. # have it set for all requests, make sure to have:
5. # set bereq.http.connection = “close”;
6. # here. It is not set by default as it might break some broken web
7. # applications, like IIS with NTLM authentication.
9. set bereq.http.connection = “close”;
10. return (pipe);
11. }

Now,  you might want to see the original IP in the apache  logs too. To do that you have to use a custom log that prints the   X-Forwareded-For header. Just add inside your  Virtualhost declaration:

1. LogFormat “%{X-Forwarded-For}i %l %u %t \”%r\” %>s %b \”%{Referer}i\” \”%{User-Agent}i\”” varnish
2. CustomLog /var/log/httpd/access_combined_SOMEDOMAIN.log varnish

You should also have in mind that this configuration is appended,  so it might happen that you vcl has already a line like this, so  you are doing it twice. See the commented code in the default.vcl  file.

Now on NginX you need to make modifications as well:

1) Make nginx aware that you are behind a proxy

Nginx is smart enough to have a dedicated module for this work: Real Ip This module allows to change the client’s IP address to value from request header (e. g. X-Real-IP or X-Forwarded-For), it is useful if nginx works behind some proxy of L7 load balancer, and the request comes from a local IP, but proxy add request header with client’s IP. The configuration is really simple, you just have to add these 2 lines in your /etc/nginx/nginx.conf file (section: http) or directly on your Virtual host file (section : server)

set_real_ip_from   127.0.0.1; #Put the Ip of your varnish/proxy here
real_ip_header      X-Forwarded-For; #Put the Header that your varnish/proxy set

and restart nginx. You have also to modify your vcl file (usually /etc/varnish/default.vcl), in the vcl_recv part add this rule:

set req.http.X-Forwarded-For = client.ip;

And restart Varnish, this will set the header X-Forwarded-For correctly. Drawback: This module is usually not enabled by default, you can enable it rebuilding nginx with the configure option: --with-http_realip_module If you use Nginx from a binary package verify the description of the package, or simply run from the teminal nginx -V that will give a verbose output like this one:

nginx -V
nginx version: nginx/1.2.6
TLS SNI support enabled
configure arguments: --prefix=/usr/share/nginx --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --http-client-body-temp-path=/var/lib/nginx/body --http-fastcgi-temp-path=/var/lib/nginx/fastcgi --http-log-path=/var/log/nginx/access.log --http-proxy-temp-path=/var/lib/nginx/proxy --lock-path=/var/lock/nginx.lock --pid-path=/var/run/nginx.pid --with-pcre-jit --with-file-aio --with-http_gzip_static_module --with-http_ssl_module --with-ipv6 --without-http_browser_module --without-http_geo_module --without-http_limit_req_module --without-http_limit_zone_module --without-http_memcached_module --without-http_referer_module --without-http_scgi_module --without-http_split_clients_module --with-http_stub_status_module --without-http_ssi_module --without-http_userid_module --without-http_uwsgi_module --add-module=/usr/src/nginx/source/nginx-1.2.6/debian/modules/nginx-echo

in this example the module is NOT built in the Nginx webserver, so this solution would not work, let’s move to solution 2:

2) Change the format of your Nginx log files

This solution uses the header X-Forwarded-For too, so you have to set it on varnish as done in the former solution to set it. The idea behind this solution is that Nginx has all the information about the remote IP, just in a different header, so it’s just a matter of making nginx use that variable in its access logs instead of the default variable defining the referring IP. Edit your nginx.conf file and in the http section add this line: log_format varnish_log '$http_x_forwarded_for - $remote_user [$time_local] ' '"$request" $status $body_bytes_sent "$http_referer" ' '"$http_user_agent"' ; You also have to change the access_log directive of you virtual host, to use the varnish_log format:

access_log /var/log/nginx/access.log varnish_log;

Note: log_format controls how nginx logs the requests, in this example we have added a new log format named “varnish_log” where the first field is the header $http_x_forwarded_for and not the standard $remote_addr, in this way you’ll correctly logs the remote IP of your visitors. Drawback if you have many virtual hosts, you have to change for everyone of them the access_log directive to use the new log_format.

My Prefered Solution With default config nginx all IP addresses shows as localhost. Here is configuration to show real client IP. Snippet from varnish config:

    sub vcl_pipe {
        set bereq.http.connection = "close";
        if (req.http.X-Forwarded-For) {
                set bereq.http.X-Forwarded-For = req.http.X-Forwarded-For;
        } else {
                set bereq.http.X-Forwarded-For = regsub(client.ip, ":.*", "");
          }
    }
    sub vcl_pass {
        set bereq.http.connection = "close";
        if (req.http.X-Forwarded-For) {
                set bereq.http.X-Forwarded-For = req.http.X-Forwarded-For;
        } else {
          set bereq.http.X-Forwarded-For = regsub(client.ip, ":.*", "");
          }
    }

Snippet from nginx config:

set_real_ip_from 127.0.0.1;
real_ip_header X-Forwarded-For;