Description
Encountered an issue wherein certain product listing pages with 60 products per page loads properly when using M2's built-in FPC but returns Error 500 when using Varnish.
Preconditions
- Magento 2.0.x up to 2.1.x
- Store has products with many product options
- All cache types are enabled
Steps to reproduce
- Configure the store to make use of Varnish cache instead of built-in FPC
- Set the number of products to display in the catalog to a high value (in our case, 60)
- Load a product listing page
Expected result
The product listing page should display properly
Actual result
Page returns Error 500
After a lot of hours spent in investigating the issue and even trying to tune Varnish by configuring certain parameters, I noticed this recently submitted issue here in the M2 repository and tried to look into the X-Magento-Tags header.
#7640 X-Magento-Tags header containing whitespaces causes exception
I didn't find any pages that has spaces in the said header, nor blocks with block ID containing spaces, but I noticed something: perhaps due to the sheer amount of products that is being displayed in our product listing pages, some of them have X-Magento-Tags header lengths that reach up to more than 16k characters. I commented-out the "unset" statements in the default.vcl Varnish configuration file to check the contents of the said header and one of them are as follows:
store,cms_block,cms_block_header_mini_banner,catalog_category_42,catalog_category,catalog_category_product_42,catalog_product_13897,catalog_product,catalog_product_8512,catalog_product_8498,catalog_product_8499,catalog_product_8500,catalog_product_8501,catalog_product_8502,catalog_product_8503,catalog_product_8504,catalog_product_8513,catalog_product_8506,catalog_product_8507,catalog_product_8508,catalog_product_8509,catalog_product_8510,catalog_product_8511,catalog_product_13898,catalog_product_8514,catalog_product_8515,catalog_product_8516,catalog_product_8517,catalog_product_8518,catalog_product_8519,catalog_product_8520,catalog_product_8521,catalog_product_8522,catalog_product_8523,catalog_product_8524,catalog_product_8525,catalog_product_8526,catalog_product_8527,catalog_product_8528,catalog_product_13932,catalog_product_8590,catalog_product_8591,catalog_product_8592,catalog_product_8593,catalog_product_8594,catalog_product_8595,catalog_product_8603,catalog_product_8596,catalog_product_8597,catalog_product_8598,catalog_product_8599,catalog_product_8600,catalog_product_8601,catalog_product_8602,catalog_product_8632,catalog_product_8633,catalog_product_8634,catalog_product_8635,catalog_product_8636,catalog_product_8637,catalog_product_16189,catalog_product_16190,catalog_product_16191,catalog_product_16192,catalog_product_16193,catalog_product_16194,catalog_product_16195,catalog_product_16196,catalog_product_13959,catalog_product_8721,catalog_product_8722,catalog_product_8723,catalog_product_8724,catalog_product_8725,catalog_product_8726,catalog_product_8727,catalog_product_8728,catalog_product_8729,catalog_product_8730,catalog_product_15754,catalog_product_15755,catalog_product_15756,catalog_product_15757,catalog_product_15758,catalog_product_15759,catalog_product_15760,catalog_product_15761,catalog_product_15762,catalog_product_18991,catalog_product_18992,catalog_product_18993,catalog_product_18994,catalog_product_18995,catalog_product_13971,catalog_product_8793,catalog_product_8788,catalog_product_8789,catalog_product_8790,catalog_product_8791,catalog_product_8792,catalog_product_8794,catalog_product_8795,catalog_product_8796,catalog_product_8797,catalog_product_8798,catalog_product_8799,catalog_product_8812,catalog_product_8813,catalog_product_8814,catalog_product_8815,catalog_product_8816,catalog_product_8817,catalog_product_16026,catalog_product_16027,catalog_product_16028,catalog_product_16029,catalog_product_16030,catalog_product_16031,catalog_product_16032,catalog_product_16033,catalog_product_16034,catalog_product_16035,catalog_product_16036,catalog_product_16038,catalog_product_16039,catalog_product_16040,catalog_product_16041,catalog_product_16042,catalog_product_16043,catalog_product_17363,catalog_product_17364,catalog_product_17365,catalog_product_17366,catalog_product_17367,catalog_product_17368,catalog_product_13977,catalog_product_8837,catalog_product_8838,catalog_product_8839,catalog_product_8840,catalog_product_8841,catalog_product_8842,catalog_product_8845,catalog_product_8846,catalog_product_8847,catalog_product_8848,catalog_product_8849,catalog_product_8850,catalog_product_8851,catalog_product_8852,catalog_product_14004,catalog_product_8999,catalog_product_9000,catalog_product_9001,catalog_product_9002,catalog_product_9003,catalog_product_16175,catalog_product_9004,catalog_product_9005,catalog_product_9006,catalog_product_9007,catalog_product_9008,catalog_product_9009,catalog_product_9012,catalog_product_9013,catalog_product_9014,catalog_product_9015,catalog_product_9016,catalog_product_16176,catalog_product_16183,catalog_product_16184,catalog_product_16185,catalog_product_16186,catalog_product_16187,catalog_product_16188,catalog_product_16177,catalog_product_16178,catalog_product_16179,catalog_product_16180,catalog_product_16181,catalog_product_16182,catalog_product_17357,catalog_product_17358,catalog_product_17359,catalog_product_17360,catalog_product_17361,catalog_product_17362,catalog_product_14014,catalog_product_17411,catalog_product_9055,catalog_product_9056,catalog_product_9057,catalog_product_9058,catalog_product_9059,catalog_product_17412,catalog_product_9060,catalog_product_9061,catalog_product_9062,catalog_product_9063,catalog_product_9064,catalog_product_18526,catalog_product_18527,catalog_product_18528,catalog_product_18529,catalog_product_18530,catalog_product_18531,catalog_product_17413,catalog_product_9071,catalog_product_9072,catalog_product_9073,catalog_product_9074,catalog_product_9075,catalog_product_15866,catalog_product_15867,catalog_product_15868,catalog_product_15870,catalog_product_18532,catalog_product_18533,catalog_product_18534,catalog_product_18535,catalog_product_18536,catalog_product_18537,catalog_product_17415,catalog_product_17406,catalog_product_17407,catalog_product_18821,catalog_product_17410,catalog_product_14228,catalog_product_10288,catalog_product_10289,catalog_product_10290,catalog_product_10291,catalog_product_10292,catalog_product_10295,catalog_product_10296,catalog_product_10297,catalog_product_10298,catalog_product_10299,catalog_product_10300,catalog_product_10302,catalog_product_10303,catalog_product_10304,catalog_product_10305,catalog_product_10306,catalog_product_10307,catalog_product_10310,catalog_product_10311,catalog_product_10313,catalog_product_10314,catalog_product_15961,catalog_product_15962,catalog_product_15963,catalog_product_15964,catalog_product_15965,catalog_product_15966,catalog_product_15967,catalog_product_15968,catalog_product_15969,catalog_product_15971,catalog_product_15972,catalog_product_15974,catalog_product_14231,catalog_product_10318,catalog_product_10319,catalog_product_10320,catalog_product_10321,catalog_product_10322,catalog_product_10323,catalog_product_10324,catalog_product_14240,catalog_product_10406,catalog_product_10407,catalog_product_10408,catalog_product_10409,catalog_product_10410,catalog_product_10411,catalog_product_15860,catalog_product_15861,catalog_product_15862,catalog_product_15863,catalog_product_17530,catalog_product_17531,catalog_product_17532,catalog_product_17533,catalog_product_17534,catalog_product_14272,catalog_product_10660,catalog_product_10661,catalog_product_10662,catalog_product_10663,catalog_product_10664,catalog_product_10665,catalog_product_10666,catalog_product_10667,catalog_product_10668,catalog_product_10669,catalog_product_10670,catalog_product_10671,catalog_product_10672,catalog_product_10673,catalog_product_15883,catalog_product_15884,catalog_product_15885,catalog_product_15886,catalog_product_15887,catalog_product_15888,catalog_product_15889,catalog_product_14277,catalog_product_10701,catalog_product_10702,catalog_product_10703,catalog_product_10704,catalog_product_10705,catalog_product_10711,catalog_product_10712,catalog_product_10713,catalog_product_10714,catalog_product_10715,catalog_product_10706,catalog_product_10707,catalog_product_10708,catalog_product_10709,catalog_product_10710,catalog_product_15692,catalog_product_15693,catalog_product_15694,catalog_product_15695,catalog_product_15696,catalog_product_15697,catalog_product_15698,catalog_product_15699,catalog_product_15700,catalog_product_15701,catalog_product_17834,catalog_product_17835,catalog_product_17836,catalog_product_17837,catalog_product_17838,catalog_product_14343,catalog_product_11360,catalog_product_11361,catalog_product_11362,catalog_product_11363,catalog_product_11364,catalog_product_11365,catalog_product_11354,catalog_product_11355,catalog_product_11356,catalog_product_11357,catalog_product_11358,catalog_product_16012,catalog_product_16013,catalog_product_16014,catalog_product_16015,catalog_product_16016,catalog_product_16017,catalog_product_16019,catalog_product_16020,catalog_product_16021,catalog_product_16022,catalog_product_16023,catalog_product_17304,catalog_product_17305,catalog_product_17306,catalog_product_17307,catalog_product_17308,catalog_product_17309,catalog_product_17310,catalog_product_18996,catalog_product_18997,catalog_product_18998,catalog_product_18999,catalog_product_19000,catalog_product_19001,catalog_product_19002,catalog_product_19003,catalog_product_14357,catalog_product_11543,catalog_product_11544,catalog_product_11545,catalog_product_11546,catalog_product_11547,catalog_product_11548,catalog_product_15891,catalog_product_15893,catalog_product_15894,catalog_product_15895,catalog_product_15897,catalog_product_15898,catalog_product_15899,catalog_product_15901,catalog_product_18234,catalog_product_18235,catalog_product_18236,catalog_product_18237,catalog_product_18238,catalog_product_14359,catalog_product_11572,catalog_product_11573,catalog_product_11574,catalog_product_11575,catalog_product_11576,catalog_product_11577,catalog_product_11595,catalog_product_11596,catalog_product_11597,catalog_product_11598,catalog_product_11599,catalog_product_11600,catalog_product_11589,catalog_product_11590,catalog_product_11591,catalog_product_11592,catalog_product_11593,catalog_product_11594,catalog_product_16171,catalog_product_17313,catalog_product_17314,catalog_product_17315,catalog_product_17316,catalog_product_17317,catalog_product_17318,catalog_product_17344,catalog_product_17345,catalog_product_17346,catalog_product_17347,catalog_product_17348,catalog_product_17349,catalog_product_17828,catalog_product_17829,catalog_product_17830,catalog_product_17831,catalog_product_17832,catalog_product_14366,catalog_product_11653,catalog_product_11654,catalog_product_11655,catalog_product_11656,catalog_product_11657,catalog_product_11658,catalog_product_11659,catalog_product_11660,catalog_product_11661,catalog_product_11662,catalog_product_11663,catalog_product_11664,catalog_product_11680,catalog_product_11681,catalog_product_11682,catalog_product_11683,catalog_product_11684,catalog_product_11685,catalog_product_16071,catalog_product_16072,catalog_product_16073,catalog_product_16074,catalog_product_16075,catalog_product_18208,catalog_product_18209,catalog_product_18210,catalog_product_18211,catalog_product_18212,catalog_product_18213,catalog_product_18214,catalog_product_18215,catalog_product_18216,catalog_product_18217,catalog_product_14423,catalog_product_12004,catalog_product_12005,catalog_product_12006,catalog_product_12007,catalog_product_12008,catalog_product_12009,catalog_product_12010,catalog_product_12011,catalog_product_12012,catalog_product_12013,catalog_product_12014,catalog_product_12016,catalog_product_12017,catalog_product_12018,catalog_product_12019,catalog_product_12020,catalog_product_12021,catalog_product_12022,catalog_product_12023,catalog_product_16077,catalog_product_16078,catalog_product_16079,catalog_product_16080,catalog_product_16081,catalog_product_16082,catalog_product_16083,catalog_product_16084,catalog_product_16085,catalog_product_16086,catalog_product_16087,catalog_product_16088,catalog_product_18353,catalog_product_14436,catalog_product_12214,catalog_product_12215,catalog_product_12216,catalog_product_12217,catalog_product_12218,catalog_product_12219,catalog_product_14441,catalog_product_12293,catalog_product_12294,catalog_product_12295,catalog_product_12296,catalog_product_12297,catalog_product_12298,catalog_product_12305,catalog_product_12306,catalog_product_12307,catalog_product_12308,catalog_product_12309,catalog_product_12310,catalog_product_12311,catalog_product_12312,catalog_product_12313,catalog_product_12314,catalog_product_12315,catalog_product_12316,catalog_product_16094,catalog_product_16095,catalog_product_16096,catalog_product_16097,catalog_product_16098,catalog_product_16099,catalog_product_16101,catalog_product_16102,catalog_product_16103,catalog_product_16104,catalog_product_16105,catalog_product_16106,catalog_product_14456,catalog_product_12450,catalog_product_12451,catalog_product_12452,catalog_product_12453,catalog_product_12454,catalog_product_12455,catalog_product_15930,catalog_product_15924,catalog_product_15925,catalog_product_15926,catalog_product_15927,catalog_product_15928,catalog_product_15929,catalog_product_15931,catalog_product_15918,catalog_product_15919,catalog_product_15920,catalog_product_15921,catalog_product_15922,catalog_product_15923,catalog_product_15932,catalog_product_14513,catalog_product_12692,catalog_product_12693,catalog_product_12694,catalog_product_12695,catalog_product_12696,catalog_product_12697,catalog_product_15878,catalog_product_15879,catalog_product_15880,catalog_product_15881,catalog_product_15882,catalog_product_14515,catalog_product_12707,catalog_product_12708,catalog_product_12709,catalog_product_12710,catalog_product_12711,catalog_product_12712,catalog_product_12713,catalog_product_15910,catalog_product_15911,catalog_product_15912,catalog_product_15913,catalog_product_15914,catalog_product_15915,catalog_product_15916,catalog_product_15903,catalog_product_15904,catalog_product_15905,catalog_product_15906,catalog_product_15907,catalog_product_15908,catalog_product_15909,catalog_product_18411,catalog_product_18412,catalog_product_18413,catalog_product_18414,catalog_product_18415,catalog_product_18416,catalog_product_18405,catalog_product_18406,catalog_product_18407,catalog_product_18408,catalog_product_18409,catalog_product_18410,catalog_product_14516,catalog_product_12714,catalog_product_12715,catalog_product_12716,catalog_product_12717,catalog_product_12718,catalog_product_12719,catalog_product_15934,catalog_product_15935,catalog_product_15936,catalog_product_15937,catalog_product_15938,catalog_product_15939,catalog_product_15940,catalog_product_15941,catalog_product_15942,catalog_product_15943,catalog_product_15944,catalog_product_15945,catalog_product_15946,catalog_product_15947,catalog_product_15948,catalog_product_15949,catalog_product_15950,catalog_product_15951,catalog_product_15952,catalog_product_15953,catalog_product_15954,catalog_product_15955,catalog_product_17443,catalog_product_17444,catalog_product_17445,catalog_product_17446,catalog_product_17447,catalog_product_17448,catalog_product_17449,catalog_product_15460,catalog_product_15436,catalog_product_15437,catalog_product_15438,catalog_product_15439,catalog_product_15440,catalog_product_15441,catalog_product_15443,catalog_product_15444,catalog_product_15445,catalog_product_15446,catalog_product_15448,catalog_product_15449,catalog_product_15450,catalog_product_15452,catalog_product_15455,catalog_product_15456,catalog_product_15457,catalog_product_15458,catalog_product_15461,catalog_product_15462,catalog_product_15463,catalog_product_15464,catalog_product_17394,catalog_product_17395,catalog_product_17396,catalog_product_17397,catalog_product_17398,catalog_product_17399,catalog_product_16859,catalog_product_16838,catalog_product_16839,catalog_product_16840,catalog_product_16841,catalog_product_16842,catalog_product_16843,catalog_product_16844,catalog_product_16845,catalog_product_16846,catalog_product_16847,catalog_product_16848,catalog_product_16849,catalog_product_16850,catalog_product_16851,catalog_product_16852,catalog_product_16853,catalog_product_16854,catalog_product_16855,catalog_product_16856,catalog_product_16857,catalog_product_16858,catalog_product_18591,catalog_product_18592,catalog_product_18593,catalog_product_18594,catalog_product_18595,catalog_product_18596,catalog_product_18597,catalog_product_16964,catalog_product_16958,catalog_product_16959,catalog_product_16960,catalog_product_16961,catalog_product_16962,catalog_product_16963,catalog_product_16971,catalog_product_17043,catalog_product_17037,catalog_product_17038,catalog_product_17039,catalog_product_17040,catalog_product_17041,catalog_product_17042,catalog_product_17044,catalog_product_17051,catalog_product_17045,catalog_product_17046,catalog_product_17047,catalog_product_17048,catalog_product_17049,catalog_product_17050,catalog_product_18691,catalog_product_18488,catalog_product_18489,catalog_product_18490,catalog_product_18491,catalog_product_18692,catalog_product_18693,catalog_product_17053,catalog_product_17054,catalog_product_17055,catalog_product_17056,catalog_product_17057,catalog_product_17058,catalog_product_17060,catalog_product_18694,catalog_product_18493,catalog_product_18494,catalog_product_18495,catalog_product_18496,catalog_product_18497,catalog_product_18695,catalog_product_18696,catalog_product_16965,catalog_product_16967,catalog_product_16968,catalog_product_16969,catalog_product_18119,catalog_product_18120,catalog_product_18121,catalog_product_18122,catalog_product_18123,catalog_product_18124,catalog_product_19004,catalog_product_19005,catalog_product_19006,catalog_product_19007,catalog_product_19008,catalog_product_19009,cms_block_customer-register,cms_block_,cms_block_newsletter_popup
I paginated through the category and found a page that loads properly even with X-Magento-Tags header containing almost 5000 characters.
So what I did was to try modifying the setHeader() function of vendor/magento/framework/HTTP/PhpEnvironment/Response.php to create a sha1 hash of the header's value and then the pages started loading properly.
So I guess a possible fix for this is to either hash the value of the X-Magento-Tags or perhaps compress it if we want a reversible string, e.g.:
$value = base64_encode(gzencode($value));
I'm just not sure of the implications of this solution, like when cleaning or flushing the cache or whatever. I'll just leave it to you guys. Thanks!