Türchen 8: Split off requests to micro services

Magento and additional Microservices: What a dream team!

We started as probably many of you also did: with Magento and some additional extensions to add missing features and bring value to the customer. Over time it became more and more and luckily the business flourishing. By this we encountered areas in which Magento is well known: performance and easy scalability. Much was done but in the end the easiest solution disguise was chosen: Full Page Caching.
Suddenly everything seemed fine again. At least until someone had the brilliant idea of showing “more accurate/real time data” to customers. We started with a simple thing: real delivery time/availability information on product detail pages. Everything was implemented, tested and approved but after deployment nothing changed. Everybody forgot our full page cache.

AJAX hole punching

As we do not use a fpc solution that supports ESI blocks we tried the standard way of making AJAX requests for the stuff we wanted updating and it worked. Yay!! Soon after adding the availability information to the detail pages we added those to product listings, and hell broke loose. With 30 products per page we suddenly created 30 Ajax requests, all to a Magento controller. After a bit of research we found that we tried to access the user session concurrently and there is a slight problem with PHP-session locking for this which caused 15-30% of our AJAX requests to result in “503 Service not available”.

Use no user session

Somehow we were not able to implement this in the preDispatch() method as it should normally work:

public function preDispatch()
{
  $this->getLayout()->setArea($this->_currentArea);
  $this->setFlag('', self::FLAG_NO_START_SESSION, 1); // Should skip session startup
  $this->setFlag('', self::FLAG_NO_PRE_DISPATCH, 1);
  $this->setFlag('', self::FLAG_NO_POST_DISPATCH, 1);
  parent::preDispatch();
  return $this;
}

As we could not figure out how to bypass this we needed to come up with another idea and we agreed upon writing our own product-availability-service.

Availability as a services

We agreed on building this one in node.js. Partially because we wanted hands on experiences in an area that was new to most of us and partially because we thought it would be a good fit for the needed features. For fast implementation and also to prevent us from reinventing everything we use express.js

Express route to a simple CRUD API

Using express.js for this is dead simple. After installing express-generator and mysql for npm for database connection you can simply generate the “project” by

$ express-generate advent

This will result in a base project structure that can be used for our small example. Additionally we need to install the mysql stuff for npm:

$ npm install mysql --save

and set it up afterwards in models/db.js

var mysql = require('mysql');
//local mysql db connection depending on your sql-setup
var connection = mysql.createConnection({
  host : 'localhost',
  port: '3307',
  user : 'adv',
  password : 'testen123',
  database : 'advent'
});
connection.connect(function(err) {
  if (err) throw err;
});

Now we can prepare the real controller for request handling. I put it in routes/stockstatus.js

var express = require('express');
var router = express.Router();
var db = require('../models/db.js');

/* GET stock for product. */
router.get('/:itemid', function(req, res, next) {
  var itemid = req.params.itemid;
  var results = db.query('SELECT qty from stock where id = ?', [itemid], function (error, results, fields) {
    if(error){
      res.send(JSON.stringify({"status": 500, "error": error, "response": null})); 
    } else {
      res.send(JSON.stringify({"status": 200, "error": null, "response": results}));
    }
  });
});
module.exports = router;

Last thing we need is to tell our app how to access this endpoint. This will go to app.js where we need to add

var stockstatusRouter = require('./routes/stockstatus');
app.use('/stockstatus/', stockstatusRouter);

For this example I used a very simplistic table with just a tiny bit of sample data:

CREATE TABLE stock (`id` INT(11) NOT NULL, `qty` INT(11) NOT NULL DEFAULT 0);
ALTER TABLE stock ADD PRIMARY KEY (id);
ALTER TABLE stock MODIFY id INT(11) NOT NULL AUTO_INCREMENT;
INSERT INTO stock (qty) VALUES (4),(5),(0),(10);

This is all you need to run the application with

$ npm start

Normally you can access it now under http://localhost:3000/stockstatus/2 to get a JSON response.

Conclusion
As you can see it is really super simple to split off some functionalities or add new by adding a very small node.js app. Personally i would advice against enlarging those APIs without boundaries, instead I would suggest having specialized services for every need you have as in: “Do one thing, but do it right”. Those are easier to adapt, replace and scale if need be.