Türchen 1: To earn more money, we need faster shops – Lizards and Pumpkins (?&?)

Dec
1
2017

Every year, we had an adventcalendar on webguys.de. Checkout the posts from the last years 2016, 2014 and 2013. Due to the fact that webguys.de is a blog by three German developers and this started in the German community, many of the posts are in German. But more and more of the new posts are in English, so have a look! A big shout out goes to Tobias,who took care of the calendar in the last years. We nearly forgot it this year. But as always, the Magento community stands together in an emergency, when the distress call happens. The list of posts is not complete yet, so if you have a cool idea please send an email to Fabian.

It is a pleasure for us and especially me that you found the way to this year’s advent calender. Please come back every day, and check out the other posts as well!

But now right into the topic.

Caching is not the solution!

I think we can agree that magento is too slow. Don’t get me wrong, I love magento (especially version 1)! And back in the days (like 2 years ago ;-))  it was kind of fast enough. It was ok to wait a second or two for your content. But today, our customers expect to see the page before they even click the link. The fastest magento I heard about without full page cache (FPC) is 200-300ms, and this was already customised and most of the EAV stuff was replaced with a NoSQL database.

Most of us, use a full page cache to speed up your site. And if this works for you, you can stop reading here. You have a solution which works! What more can one expect. But many of us are unhappy with their FPCs, for a couple of reasons:

  • Caching is complex. One of the easier, yet compelling examples is that if you disable a product, you not only need to refresh the listing page with the product, but every page after this page, because every product moves one position forward.
  • Caching works great in many read-only environments. This means if you have a big team of content writers, price or inventory updates or something alike, your cache is invalidated all the time. You can decide that you don’t invalidate the cache and serve old content. But there is no need to make this tradeoff.
  • If you decide to serve old content or your invalidation is not perfect you have inconsistencies.
  • and if your store is not powerful enough a flush kills your complete site.

Lizards and Pumpkins generates a view-optimized format of your data, right after the data changed. This way the data are always up to date and blazingly fast at the user.

Step by step or only what you need

Lizards and Pumpkins sits right before your store. So every request runs through ?&? and it decides whether it processes the request or not. If the request is not processed by Lizards & Pumpkins, Magento gets its chance. If it can’t process it either, we get the our hated 404 error.

While importing all product data, Lizards & Pumpkins is building an URL storage to easily decide who answers the request. ?&? can currently take care of three different page types. “Can” means it doesn’t have to.

  • Categories (or as we call it: product lists)
  • Product detail pages
  • Product search results

Depending on your store, you decide what you need.

You have this one category which is full of configurable products and loads to death? Then import all products of this category into Lizards & Pumpkins and serve this one category from Lizards and don’t touch the rest.

Your product pages and categories are blazing fast, but your search is horribly slow and the results are even worse? Replace the search with Lizards & Pumpkins and stay with your catalog!

You update your product prices all the time but show them only on product detail pages? Your FPC is working fine for product listings and the query cache solves performance issues with the search? Serve the product detail pages by Lizards & Pumpkins and make sure that prices and stock data are always up to date.

Whatever your problem with your catalog is, Lizards & Pumpkins can solve it, without breaking functionality with the legacy system. And when you started using it, you might find the other pages too slow as well, so move them step by step.

Use tech you already know

Lizards & Pumpkins is built upon PHP. The hard dependencies are: PHP7, php7.0-intl and php7.0-xml. End of list.

Theoretically you can run Lizards & Pumpkins with this setup, even in production, but we highly recommend to at least use a non-php search engine. ?&? needs three systems to help it doing its job: a search engine, a key value storage and a queue. A few implementations already exist, like Solr, Elasticssearch, Redis and ZeroMQ. But staying with the technology you already know is a cheap entry. You already use Amazon’s DynamoDB, Google Cloud’s Pub/Sub or CouchDB? Stay with it! Implementations for the Queue and the Key value storage are done in hours to days. The search engine takes a bit longer. But if you already has the knowledge to maintain the technology the investment is worth it.

Open Source

BSD-3clause. This means, no one can take the code from you. You are free to use it. Do your implementations. There is no obligation to give back in form of open sourcing your implementations, e.g. for a new key-value storage, although we highly encourage you. Beside the fact, that it is very good for your karma, you have more people testing your implementation and maintaining it, so win-win for all parties.

First step away from your legacy system

I love Magento, I really do. But there are edge cases were Magento is simply not the right system. I once had a customer who sold trainings which consist of trainings which consist of trainings. So levels and levels of recursion. Magento was not able to map this. And if you use other functionality, like price rules, wishlist, reviews and such, Magento might be the right choice. If you don’t, you have a huge dependency you can’t get rid off and built all your stuff into it.

You already did this? And you are at a point were your current system, be it Magento, Oxid or something else can’t handle your current and upcoming business needs? Then maybe it is an idea to do the first step away from your current system by replacing it, step by step.

Comments

  • 2. December 2017

    L&P is just a pre-baked cache. It consumes lots of resources for baking snippets. Those snippets might get delivered to the client … if not you’re sitting on a lot of data which gets build for nothing. You also need to maintain a second stack which needs additional knowledge besides M1/M2.

    • 2. December 2017

      Hi John!

      Posts like this make it hard to hold onto myself (1. you post anonymus, 2. it seems, that you haven’t read/understood the first paragraph, 3. and it feels rude, but it might be that this is a cultural misunderstanding on my side)

      I’ll try to give a decent answer anyways, and clear up the misunderstandings you stumbled upon, although I’m, quite sure you won’t see it, because you used a throw-away email address.

      L&P is just a pre-baked cache.

      I disagree. Because my understanding of a cache is, that computed data are stored, so when you have the same computation again, you don’t have to rerun it. Even if you pre-warm a cache, the cache is totally passive. It saves data, some other system spat out. Be it a block, a configuration or a full page.
      The Lizard & Pumpkin approach is active. Lizards & Pumpkins renders the page on its own. It is specialised in rendering pages, therefore it is much faster than Magento (and every other ecommerce system I know). The mindset and the way of implementing a page changes fundamentally if you use Lizards & Pumpkins.

      It consumes lots of resources for baking snippets. Those snippets might get delivered to the client … if not you’re sitting on a lot of data which gets build for nothing.

      1. It consumes much less resources than Magento for generating snippets. Our last project used less than 16MB per worker, where Magento expects 256MB+.

      2. All the processing is made before hand, so the visitor doesn’t wait for their content. We prefer to waste resources to generate snippets we don’t need, before the first customer has to wait half a second (or 2 seconds) for the content. A real pre warmed FPC needs more resource (because Magento does need more) and generates content too, which no one ever might look at.

      You also need to maintain a second stack which needs additional knowledge besides M1/M2.

      I agree. The question is not, do you maintain a second stack, but which second stack do you maintain to speed up your store? It can be varnish (which often has complex flushing and ESI setups), it can be Lizards & Pumpkins, it can be a Magento with replaced EAV and database stuff, so you are running on ElasticSearch, CouchDB or Solr. In any case you have to learn more than Magento.

      Lizards & Pumpkins is a solution for shops with a lot of data, traffic and/or product data/price/stock updates. If the shop’s extend doesn’t hit the point where caching makes more problems than it solves and where magento isn’t slowed down too much – you don’t need it. True.

Kommentieren

Your email address will not be published. Required fields are marked *