Fast & Furious Fashion for 10k customers a minute

Rebellious Fashion Case study

Contract: Active (Since 2018)
Niche Fashion Company UK based

Background

Rebellious Fashion is a fashion company with their headquarters in Manchester UK, with over 20 years experience selling amazing designs at affordable prices, their website handles millions of transactions and have a very high satisfaction rate.

Issues

Rebellious Fashion reached out to us, as they were concerned about their conversion rate (CR). They were running Magento 1, which was already running on a powerful five-tier server infrastructure – enabling them to handle all their traffic. The traffic was decent; however, their CR was very unstable, due to concurrent users of 200/300 hitting their website every minute and of an evening this figure grew to around 600/800. This amount of users should be celebrated, but in this situation the CR was being heavily impacted, lowering their expected revenue by quite a considerable amount.

James Wayland who is the Head of Marketing at Rebellious Fashion had predicted over 2k concurrent users during Black Friday, which would have put their site on standstill – not something a company needs at peak sales periods.

They approached us to help them meet their demands. Now, I must say that this is one of those projects than any system architect/developer dreams about, as this is where they can really show off their talent and help the client.

Discovery

Even though Rebellious Fashion’s contracted hosting company offered great support, their engineers deployed several dedicated servers, but their CR was still fluctuating. After we examined their architecture, we noticed tiny – yet critical support mechanisms were missing. Generally, as we dedicate our time on building/developing specific Magento architectural systems, we have a deep understanding of the logic of Magento’s performance needs and what is needed to make Magento work for your business and not to be a hindrance that would impact your potential business revenue.

The problem breakdown

01. Rebellious Fashion team were using a tool called Visual Merchandiser, which is a great tool and a timesaver, allowing you to manage categories and products visually; you only need to drag and drop product images in order to change a product position within the frontend. For any fashion company it’s a must have tool, which integrates into Magento Enterprise; however, they were utilising Magento Community version and what with the team using Visual Merchandiser at all hours, which included peak periods (when the site was getting thousands of hits); this became a real server performance killer.

02. As previously said, Visual Merchandiser is an amazing product, but without the right architecture – it can destroy your revenue through strangling the stores database. For this to work – we had to develop an asynchronous indexer algorithm to extend the Magento’s core indexer.

03. You will find caching services are definitely underestimated by most hosting companies, and we often see these incorrect setups being poorly managed, this isn’t the hosting companies fault, as they don’t have the deep understanding of Magento Architecture as a specialist firm like ourselves would have. By setting up Varnish Cache properly – Magento will run like a tornado; every ecommerce companies dream. In this situation UKfast host engineers couldn’t workout how to use this great software to stop the caching bottleneck and felt fit to just switch the software off - that was a big mistake.

As we did a deep dive with a powerful monitoring tool (New Relic), we were able to see the segments that were causing the database to shutdown, producing 501 response server errors. This meant the storefront was returning errors in the middle of peak periods – halting sales.

We placed our Lead Developer Neven (bit of a genius) on the task; he switched Varnish back on – for us to fix the issue - this needed to be done as part of the deep dive. This opened the door to various other historic issues appearing. I am amazed at how many times I have seen hosting companies do quick sloppy workarounds, yeah they’re quick – but when they affect the ecommerce company’s revenue – that just annoys me.

04. The backend had been utilising several APIs and with the usage of Visual Merchandiser tool, this was causing massive bottlenecks in the busy periods.

The Colour swatch (a custom made module purchased at Amasty) was lagging at the category pages during busy hours; this was in addition to the load issue also attacking the category pages.

The core indexer was interfering with all orders and stock management; the indexer needed to be run frequently by the staff, allow for managerial transparency on orders (successful or not), who had a heavy reliance on it for stock management. The present indexer was showing ‘index required’ and didn’t change the status at all during the day; Rebellious Fashion management were working blind.

Issue upon issue were arising, but these enabled us to systematically create a strategic fix. Obviously, Rebellious Fashion were getting worried, as holiday season was knocking on at the door and these issues were appearing in the middle of their busiest quarter – we had to act fast.

Solution

Before performing key code updates, we had to modify their current server environment. The current store was already running on five dedicated servers, which some would think would be sufficient – but it really wasn’t. We can’t work on theories, we base our strategies on experience. James Wayland wanted us to hold off from moving the site to AWS (Amazon Cloud Service). So, we requested another three servers from UKfast host:

01. One additional dedicated server for Varnish cache and Haproxy

02. One additional dedicated server for database replication

03. One additional dedicated server for Magento Backend - allowing backend tasks to be completely separated from the storefront server.

Our strategy was simple: We needed to turn Varnish back on, whilst it sits in front of all other servers, allowing delivery of content directly to the users browsers from cache. We would split heavy backend requests from the frontend requests via the database layer - replication setup. James Wayland head of marketing scheduled an appointment with UKfast engineers. The call didn’t last long, we explained in detail to Ben senior engineer at UKfast what needed to be updated. They acted fast, ETA for new server setup was 5 days. That was more than enough time for us to sort the issues with Varnish and the front Store.

Boost Magento speed with Varnish cache

We couldn’t afford to do any testing on current production. We deployed a quick staging environment. All updates related to Magento Turpentine Module and Varnish Cache had been settled on the staging environment.

This is what we updated to gain maximum out from Varnish cache:

- We redefined TTLs for each ESI block in turpentine configuration file in order to utilize higher cache rates in Varnish. For example: Block for Cart content was passing requests along to backend for the default setup. After our modification, the cart content was cached per user. This update has significantly reduced Varnish MISS rate and it has extended Varnish HIT rate to over 65%.

- We removed all media files from being cached in Varnish. This move allocated more space to Varnish. HIT rate extended for another 18%.

- We fixed the two most common issues that arise with default settings being used on Varnish and Magento: Cart content being lost in the middle of the checkout process and cart content is getting cached. This is a real pain for many storeowners who would like to utilize Varnish caching. Varnish is a synonyms for fast ecommerce websites.

Stress Test Results

The new environment architecture was ready; we reconfigured the load balancer (Haproxy), and switched Varnish back on, which was a great success on the new setup. It was now time to do a stress test on Varnish. Our stress test was based on Gatling Tool, we created 10 different user journeys; the results were outstanding. The new setup was capable of handling 2500 concurrent users. This result gives us a base level to follow, as by now means does it fully cover customer behaviour patterns; the simulation generally works out around 30% to 50% in front of regular behavioural patterns, which is brilliant as this means that the result of our effort – enables around 5000 concurrent users not having any issues. You can imagine Rebellious was relieved, especially with the incoming holidays.

Speeding up the backend

Great, we have managed to speed up their frontend, we still needed to make sure that the backend activity wouldn’t result in delays to the frontend; it all needs to work in harmony – giving the teams confidence in doing their tasks.

We reconfigured the Load Balancer, so it would recognise sales traffic from backend traffic. This means that all heavy export and import cron jobs had been set up to trigger on separate nodes: API requests, Visual Merchandiser requests and Indexer requests.

Heavy third party module requests that are not related to sales traffic are now querying replicated database servers. On the other side, database queries related to sales are now closed on the master database. This was another positive impact on top of what we had done so far. We now had a happy management team, who were able to run their tasks on the backend without any concerns to the frontend sales site.

Results

Or efforts and knowledge kicked in CR from being average 1.5 to 1.8% to be 2.0 to 2.7%. This is an amazing result, which gained us a long-term contract with Rebellious Fashion. We are now team leaders at Rebellious Fashion technical support department, where any procurement of technology is required to go through us before being adopted.

Technologies being utilized

Modules developed