0

I'm having a an issue after I added varnish to magento 2.

website is not working now it give me this error

Error 503 Backend fetch failed Backend fetch failed

Guru Meditation: XID: 261

Varnish cache server

this is my confs files

file : /etc/default/varnish

# Configuration file for varnish
#
# /etc/init.d/varnish expects the variables $DAEMON_OPTS, $NFILES and $MEMLOCK
# to be set from this shell script fragment.
#
# Note: If systemd is installed, this file is obsolete and ignored.  Please see 
# /usr/share/doc/varnish/examples/varnish.systemd-drop-in.conf

Should we start varnishd at boot?  Set to "no" to disable.


START=yes


Maximum number of open files (for ulimit -n)


NFILES=131072


Maximum locked memory size (for ulimit -l)


Used for locking the shared memory log in memory.  If you increase log size,


you need to increase this number as well


MEMLOCK=82000


Default varnish instance name is the local nodename.  Can be overridden with


the -n switch, to have more instances on a single server.


You may need to uncomment this variable for alternatives 1 and 3 below.


INSTANCE=$(uname -n)


This file contains 4 alternatives, please use only one.


Alternative 1, Minimal configuration, no VCL




Listen on port 6081, administration on localhost:6082, and forward to


content server on localhost:8080.  Use a 1GB fixed-size cache file.




This example uses the INSTANCE variable above, which you need to uncomment.




DAEMON_OPTS="-a :6081 \


-T localhost:6082 \


-b localhost:8080 \


-u varnish -g varnish \


-S /etc/varnish/secret \


-s file,/var/lib/varnish/$INSTANCE/varnish_storage.bin,1G"


Alternative 2, Configuration with VCL




Listen on port 6081, administration on localhost:6082, and forward to


one content server selected by the vcl file, based on the request.




DAEMON_OPTS="-a :6081 

             -T localhost:6082 

             -f /etc/varnish/default.vcl 

             -S /etc/varnish/secret 

             -s malloc,256m"
             -p http_resp_hdr_len=70000 

             -p http_resp_size=100000 \


Alternative 3, Advanced configuration




This example uses the INSTANCE variable above, which you need to uncomment.




See varnishd(1) for more information.




# Main configuration file. You probably want to change it :)


VARNISH_VCL_CONF=/etc/varnish/default.vcl




# Default address and port to bind to


# Blank address means all IPv4 and IPv6 interfaces, otherwise specify


# a host name, an IPv4 dotted quad, or an IPv6 address in brackets.


VARNISH_LISTEN_ADDRESS=




# Telnet admin interface listen address and port


VARNISH_ADMIN_LISTEN_ADDRESS=127.0.0.1


VARNISH_ADMIN_LISTEN_PORT=6082




# Cache file location


VARNISH_STORAGE_FILE=/var/lib/varnish/$INSTANCE/varnish_storage.bin




# Cache file size: in bytes, optionally using k / M / G / T suffix,


# or in percentage of available disk space using the % suffix.


VARNISH_STORAGE_SIZE=1G




# File containing administration secret


VARNISH_SECRET_FILE=/etc/varnish/secret




# Backend storage specification


VARNISH_STORAGE="file,${VARNISH_STORAGE_FILE},${VARNISH_STORAGE_SIZE}"




# Default TTL used when the backend does not specify one


VARNISH_TTL=120




# DAEMON_OPTS is used by the init script.  If you add or remove options, make


# sure you update this section, too.


DAEMON_OPTS="-a ${VARNISH_LISTEN_ADDRESS}:${VARNISH_LISTEN_PORT} \


-f ${VARNISH_VCL_CONF} \


-T ${VARNISH_ADMIN_LISTEN_ADDRESS}:${VARNISH_ADMIN_LISTEN_PORT} \


-t ${VARNISH_TTL} \


-S ${VARNISH_SECRET_FILE} \


-s ${VARNISH_STORAGE}"




Alternative 4, Do It Yourself




DAEMON_OPTS=""

file : /etc/nginx/sites-enabled/mydomian.conf

upstream mydomian_backend {
 server   unix:/var/run/php/php8.1-fpm-mydomian.sock;
}

server {
    server_name mydomian.com www.mydomian.com;
    listen 127.0.0.1:8080;
    set $MAGE_ROOT /var/www/mydomian-upgrade;
    set $MAGE_MODE production;
    include /etc/nginx/magento/mydomian.conf;


location / {
try_files $uri $uri/ /index.php$is_args$args;
}




}
server {
   listen 80;
   server_name mydomian www.mydomian;


return 301 https://mydomian.com$request_uri;
}
server {
   listen 443 ssl http2;
   server_name mydomian www.mydomian;


ssl on;

# ssl_certificate /etc/nginx/pems/public_cert.pem;
# ssl_certificate_key /etc/nginx/pems/privat_key.pem;
ssl_certificate /etc/letsencrypt/live/mydomian/fullchain.pem; # managed by Certbot
ssl_certificate_key /etc/letsencrypt/live/mydomian/privkey.pem; # managed by Certbot
include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot





access_log /var/log/nginx/mydomian-access.log;
   error_log /var/log/nginx/mydomian-error.log;


location / {
        proxy_pass http://127.0.0.1:6081;
        proxy_set_header Host $http_host;
        proxy_set_header X-Forwarded-Host $http_host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header Ssl-Offloaded "1";
        proxy_set_header X-Forwarded-Proto https;
        proxy_set_header X-Forwarded-Port 443;
        proxy_http_version 1.1;
    proxy_set_header Accept-Encoding "";
    proxy_set_header GEOIP_COUNTRY_CODE $geoip_country_code;
    proxy_set_header Connection "";


proxy_buffer_size          128k;
proxy_buffers              4 256k;
proxy_busy_buffers_size    256k;


proxy_read_timeout 300s;
proxy_connect_timeout 75s;




}
}

file : /etc/varnish/default.vcl

# VCL version 5.0 is not supported so it should be 4.0 even though actually used Varnish version is 6
vcl 4.0;

import std;


The minimal Varnish version is 6.0


For SSL offloading, pass the following header in your proxy server or load balancer: 'X-Forwarded-Proto: https'


backend default {
    .host = "127.0.0.1";
    .port = "8080";
    .first_byte_timeout = 600s;
    .probe = {
        .url = "/pub/health_check.php";
        .timeout = 2s;
        .interval = 5s;
        .window = 10;
        .threshold = 5;
   }
}
acl purge {
    "127.0.0.1";
}


sub vcl_recv {
    if (req.restarts > 0) {
        set req.hash_always_miss = true;
    }


if (req.method == "PURGE") {
    if (client.ip !~ purge) {
        return (synth(405, "Method not allowed"));
    }
    # To use the X-Pool header for purging varnish during automated deployments, make sure the X-Pool header
    # has been added to the response in your backend server config. This is used, for example, by the
    # capistrano-magento2 gem for purging old content from varnish during it's deploy routine.
    if (!req.http.X-Magento-Tags-Pattern && !req.http.X-Pool) {
        return (synth(400, "X-Magento-Tags-Pattern or X-Pool header required"));
    }
    if (req.http.X-Magento-Tags-Pattern) {
      ban("obj.http.X-Magento-Tags ~ " + req.http.X-Magento-Tags-Pattern);
    }
    if (req.http.X-Pool) {
      ban("obj.http.X-Pool ~ " + req.http.X-Pool);
    }
    return (synth(200, "Purged"));
}

if (req.method != "GET" &&
    req.method != "HEAD" &&
    req.method != "PUT" &&
    req.method != "POST" &&
    req.method != "TRACE" &&
    req.method != "OPTIONS" &&
    req.method != "DELETE") {
      /* Non-RFC2616 or CONNECT which is weird. */
      return (pipe);
}

# We only deal with GET and HEAD by default
if (req.method != "GET" && req.method != "HEAD") {
    return (pass);
}

# Bypass customer, shopping cart, checkout
if (req.url ~ "/customer" || req.url ~ "/checkout") {
    return (pass);
}

# Bypass health check requests
if (req.url ~ "/pub/health_check.php") {
    return (pass);
}

# Set initial grace period usage status
set req.http.grace = "none";

# normalize url in case of leading HTTP scheme and domain
set req.url = regsub(req.url, "^http[s]?://", "");

# collect all cookies
std.collect(req.http.Cookie);

# Compression filter. See https://www.varnish-cache.org/trac/wiki/FAQ/Compression
if (req.http.Accept-Encoding) {
    if (req.url ~ "\.(jpg|jpeg|png|gif|gz|tgz|bz2|tbz|mp3|ogg|swf|flv)$") {
        # No point in compressing these
        unset req.http.Accept-Encoding;
    } elsif (req.http.Accept-Encoding ~ "gzip") {
        set req.http.Accept-Encoding = "gzip";
    } elsif (req.http.Accept-Encoding ~ "deflate" && req.http.user-agent !~ "MSIE") {
        set req.http.Accept-Encoding = "deflate";
    } else {
        # unknown algorithm
        unset req.http.Accept-Encoding;
    }
}

# Remove all marketing get parameters to minimize the cache objects
if (req.url ~ "(\?|&)(gclid|cx|ie|cof|siteurl|zanpid|origin|fbclid|mc_[a-z]+|utm_[a-z]+|_bta_[a-z]+)=") {
    set req.url = regsuball(req.url, "(gclid|cx|ie|cof|siteurl|zanpid|origin|fbclid|mc_[a-z]+|utm_[a-z]+|_bta_[a-z]+)=[-_A-z0-9+()%.]+&?", "");
    set req.url = regsub(req.url, "[?|&]+$", "");
}

# Static files caching
if (req.url ~ "^/(pub/)?(media|static)/") {
    # Static files should not be cached by default
    return (pass);

    # But if you use a few locales and don't use CDN you can enable caching static files by commenting previous line (#return (pass);) and uncommenting next 3 lines
    #unset req.http.Https;
    #unset req.http.X-Forwarded-Proto;
    #unset req.http.Cookie;
}

# Bypass authenticated GraphQL requests without a X-Magento-Cache-Id
if (req.url ~ "/graphql" && !req.http.X-Magento-Cache-Id && req.http.Authorization ~ "^Bearer") {
    return (pass);
}

return (hash);



}


sub vcl_hash {
    if ((req.url !~ "/graphql" || !req.http.X-Magento-Cache-Id) && req.http.cookie ~ "X-Magento-Vary=") {
        hash_data(regsub(req.http.cookie, "^.?X-Magento-Vary=([^;]+);.*$", "\1"));
    }


# To make sure http users don't see ssl warning
if (req.http.X-Forwarded-Proto) {
    hash_data(req.http.X-Forwarded-Proto);
}


if (req.url ~ "/graphql") {
    call process_graphql_headers;
}



}


sub process_graphql_headers {
    if (req.http.X-Magento-Cache-Id) {
        hash_data(req.http.X-Magento-Cache-Id);


    # When the frontend stops sending the auth token, make sure users stop getting results cached for logged-in users
    if (req.http.Authorization ~ "^Bearer") {
        hash_data("Authorized");
    }
}

if (req.http.Store) {
    hash_data(req.http.Store);
}

if (req.http.Content-Currency) {
    hash_data(req.http.Content-Currency);
}



}


sub vcl_backend_response {


set beresp.grace = 3d;

if (beresp.http.content-type ~ "text") {
    set beresp.do_esi = true;
}

if (bereq.url ~ "\.js$" || beresp.http.content-type ~ "text") {
    set beresp.do_gzip = true;
}

if (beresp.http.X-Magento-Debug) {
    set beresp.http.X-Magento-Cache-Control = beresp.http.Cache-Control;
}

# cache only successfully responses and 404s that are not marked as private
if (beresp.status != 200 &&
        beresp.status != 404 &&
        beresp.http.Cache-Control ~ "private") {
    set beresp.uncacheable = true;
    set beresp.ttl = 86400s;
    return (deliver);
}

# validate if we need to cache it and prevent from setting cookie
if (beresp.ttl > 0s && (bereq.method == "GET" || bereq.method == "HEAD")) {
    unset beresp.http.set-cookie;
}




If page is not cacheable then bypass varnish for 2 minutes as Hit-For-Pass


if (beresp.ttl <= 0s ||
       beresp.http.Surrogate-control ~ "no-store" ||
       (!beresp.http.Surrogate-Control &&
       beresp.http.Cache-Control ~ "no-cache|no-store") ||
       beresp.http.Vary == "*") {
        # Mark as Hit-For-Pass for the next 2 minutes
        set beresp.ttl = 120s;
        set beresp.uncacheable = true;
   }


If the cache key in the Magento response doesn't match the one that was sent in the request, don't cache under the request's key


if (bereq.url ~ "/graphql" && bereq.http.X-Magento-Cache-Id && bereq.http.X-Magento-Cache-Id != beresp.http.X-Magento-Cache-Id) {
      set beresp.ttl = 0s;
      set beresp.uncacheable = true;
   }


return (deliver);



}


sub vcl_deliver {
    if (resp.http.x-varnish ~ " ") {
        set resp.http.X-Magento-Cache-Debug = "HIT";
        set resp.http.Grace = req.http.grace;
    } else {
        set resp.http.X-Magento-Cache-Debug = "MISS";
    }


# Not letting browser to cache non-static files.
if (resp.http.Cache-Control !~ &quot;private&quot; &amp;&amp; req.url !~ &quot;^/(pub/)?(media|static)/&quot;) {
    set resp.http.Pragma = &quot;no-cache&quot;;
    set resp.http.Expires = &quot;-1&quot;;
    set resp.http.Cache-Control = &quot;no-store, no-cache, must-revalidate, max-age=0&quot;;
}

if (!resp.http.X-Magento-Debug) {
    unset resp.http.Age;
}
unset resp.http.X-Magento-Debug;
unset resp.http.X-Magento-Tags;
unset resp.http.X-Powered-By;
unset resp.http.Server;
unset resp.http.X-Varnish;
unset resp.http.Via;
unset resp.http.Link;



}


sub vcl_hit {
    if (obj.ttl >= 0s) {
        # Hit within TTL period
        return (deliver);
    }
    if (std.healthy(req.backend_hint)) {
        if (obj.ttl + 300s > 0s) {
            # Hit after TTL expiration, but within grace period
            set req.http.grace = "normal (healthy server)";
            return (deliver);
        } else {
            # Hit after TTL and grace expiration
            return (restart);
        }
    } else {
        # server is not healthy, retrieve from cache
        set req.http.grace = "unlimited (unhealthy server)";
        return (deliver);
    }
}

file : /etc/systemd/system/varnish.service

[Unit]
Description=Varnish HTTP accelerator
Documentation=https://www.varnish-cache.org/docs/6.1/ man:varnishd

[Service]
Type=simple
LimitNOFILE=131072
LimitMEMLOCK=82000
ExecStart=/usr/sbin/varnishd -j unix,user=vcache -F -a :6081 -T localhost:6082 -f /etc/varnish/default.vcl -S /etc/varnish/secret -s malloc,256m
ExecReload=/usr/share/varnish/varnishreload
ProtectSystem=full
ProtectHome=true
PrivateTmp=true
PrivateDevices=true


[Install]
WantedBy=multi-user.target

I am looking how to fix this for 2 days now and most people just copy past the same thing from each other they all talking about the default.vcl file and try to remove "/pub" from this line "url = "/pub/health_check.php";" I tried that but still not working.

can someone please look into my config files and tell me what is wrong with it.

minaax
  • 1

1 Answers1

0

Please have a look at Magento 2.4 Varnish error 503 and follow the steps described there.

  • Run varnishlog -g raw -i Backend_health and check the output
  • If the varnishlog command talks about the backend being sick, it means that the health probe isn't returning the right value
  • In that case access the health probe /health_check.php yourself and check the output it returns
  • While you're running the health probe yourself, check the var/log/debug.log logs for relevant output

If this information doesn't solve your problem, please provide additional information under the form of logs. Either Magento logs, or varnishlog output.

Thijs Feryn
  • 1,061
  • 4
  • 6
  • I get this in varnishlog ( - Begin req 37074 rxreq
    • Timestamp Start: 1679534555.472317 0.000000 0.000000
    • Timestamp Req: 1679534555.472317 0.000000 0.000000
    • ReqStart 127.0.0.1 39338 a0
    • ReqMethod GET
    • ReqURL /en/girls/37-54-71-72-73-74-78-79-83-5-39-59-slim-1-6-12.html
    • ReqProtocol HTTP/1.0
    • ReqHeader Host: aeropostale-eg.com
    • ReqHeader X-Real-IP: 66.249.66.73
    • ReqHeader X-Forwarded-For: 66.249.66.73
    • ReqHeader Connection: close )
     – minaax Mar 23 '23 at 01:29
  • and get this from php pool logs ( - - 22/Mar/2023:02:23:18 +0200 "GET /pub/health_check.php" 404
      • 22/Mar/2023:02:23:18 +0200 "GET /errors/404.php" 404

    )

     – minaax Mar 23 '23 at 01:29
  • @minaax It would be easier to read the logs if you attach the full log transaction to your question and format it using code blocks. However, I also see that your health check returns an HTTP 404 implying that /pub/health_check.php doesn't exist. This could explain why Varnish fails its health check. Please analyze and if necessary attach the full logs to your question – Thijs Feryn Mar 23 '23 at 08:56