Magento Selenium Testing

[TL;DR] Go to the conclusion at the end, get the links for the GitHub repo, contribute.

This past summer (2015) I was tasked with building out automated testing for a Magento 1 module.  I had not really built out a lot of automated testing before and so I was in for a massive shocker. It sucked.  It royally sucked.

I had my list of user stories and I started out as everyone starts out: I downloaded the Selenium IDE.  It worked really well.  I built out about 10-20 test cases and was impressed at my progress.

Then I had to make a simple change to all of the test cases.  That turned all of my progress into a useless pile of crap.  I had wanted do it with as little developer skills as necessary and failed miserably.

So I went to Google to try and find out what was available that suited my needs.  My needs were simple:

  1. I needed repeatable functionality, such as being able to easily execute the checkout process
  2. I needed that repeatable functionality to be modifiable – the module was tightly integrated with checkout and had many options to tests against.  So I needed to be able to modify the checkout consistently.
  3. I needed to do all that in the admin area too
  4. It needed to be written in PHP.  Magento IS written in PHP and it’s silly to require yet-another-programming-language

There were several different options available but they often seemed basic.  Good, but basic.  Or good, but general.  In other words, there was still a lot of work that I needed to do to make it work.

Plus, I didn’t have an existing toolset that I was familiar with enough to force it to make it work.

So I decided to take some things that I was decent at (PHPUnit and PHP in general) and make it work.

Lo and behold there was a PHPUnit Selenium2 library.  So I started writing tests using that library.  I really kind of liked it.  I was under the gun and so didn’t have to time research and architect it well.  What I wrote broke about every SOLID guidance there was but it largely worked.

Except it didn’t do it very well.  Things like category navigation worked, except when I had to switch themes.  Then I had to create conditional statements to see which Xpath query would work.  Checkout would work, except when it didn’t.  Ajax was not happy with the library.  And don’t even get me started on the admin area.  It was quite difficult.  Add on top of that the module had a lot of different configuration settings that needed to be tested.

Which brings me to my last point.  The Selenium2 library for PHPUnit is sssslllllooooowwwwwww!.

At the end of the day I made it work, but there were a lot of lessons learned and a lot of bad testing approaches that I would not have done had I had more experience.

Enter Magium.

Magium is a testing framework that I’ve started work on that originally combined the things I needed to have working together.  Selenium, PHPUnit, and Magento.  However, as I’ve been working on it I’ve found that it could actually be used for other projects as well.  But my work is with Magento and so I’m building out the framework with that in mind.  Hence the name, despite the fact that the core functionality could actually be used in any web-based application.

I looked at some options for a PHP-based Selenium driver and there were basically 3 that came up.  PHPUnit_Selenium2, Instaclick, and Facebook.

  1. PHPUnit_Selenium2 – Good.  Nice integration with PHPUnit.  Slow as molasses in Alaska.
  2. Instaclick – Definitely faster, but I had problems with consistent test results
  3. Facebook – No problems.  Nice, fast, and consistent

So I opted to use the Facebook WebDriver driver.

Now I have the WebDriver driver decided on.  Next up is to decide how to best use it.

Now, “best” is a hard term to quantify.  What is the best way of doing browser testing?  There are a plethora of browser testing tools out there.  Several are available for Magento.  Many of them are good.  But I found that what was available didn’t solve the problems the way I felt they needed to be solved.

Because of that there are a number of choices I made to build a testing foundation that solved the 4 problems I noted earlier.

Repeatable Functionality

This is probably one of the easiest problems to solve.  Create classes that utilize SOLID principles.

Single Responsibility – A class is (generally) responsible for one thing.  For example, filling out the billing form or navigating to a category.

Open/Closed – Checkout may have some customization that need to be tested.  Because each step in the checkout process follows Single Responsibility the checkout can be modified by providing different objects to do different things.  But the checkout class, itself, is only a container for that functionality and does not require any changes in order to handle customization.

Liskov Substitution – Using checkout as an example again.  The checkout may have different payment methods.  You are able to swap them out because they are based off of sub-class that defines how they should work.

Interface Segregation – Payment in checkout is a good example.  It is both a payment method and a checkout step and both are defined separately.

Dependency Inversion – I kind of break this one a little, though not too badly.  The way I break it is by depending on base-level functionality, not abstractions.  That is because what I needed was repeatable functionality that works with the core Magento implementation, but can also be customized.  The proper way, I know, is to provide the abstractions, or interfaces, as the dependency and then configurably provide concrete implementations.  I commit a minor sin in its usage for the purpose of having a working test system out of the box that requires only minor configuration.

Modifiable, Repeatable Functionality

My implementation of this requirement is probably what will give most people the most agida.  I used the SOLID principles not for architecting an application, but architecting tests.  I mis-use them intentionally because it is the principles that are useful, not the implementation.  The test is the application, in the case of building a testing library.

My most egregious assault on software development principles is in how I fill the need for making the repeatable functionality modifiable.  I do that through the use of a Dependency Injection Container as a critical part of the test case.

I have very good reasons for this.

Test code must be configurable

A big, big part of my original problem was that the module I was working on needed to work on CE 1.7, CE 1.8, CE 1.9, EE 1.12, EE 1.13, and EE 1.14.  You might be tempted to think that not much has changed, or, rather, that the change has been iterable.  But that’s not true.  Checkout HTML code is quite different between versions.

As an example consider this code I had to write to place the order.

  if ($this->elementExists('//button[@title="Place Order"]', self::BY_XPATH)) {
    $this->waitUntilElementDisplayed('//button[@title="Place Order"]', self::BY_XPATH);
    $this->byXPath('//button[@title="Place Order"]')->click();
  } else if ($this->elementExists('//*[@onclick=";"]', self::BY_XPATH)) {
    $this->waitUntilElementDisplayed('//*[@onclick=";"]', self::BY_XPATH);
  } else if ($this->elementExists('review_button')) {

That is ghastly code that I never want to write again.  Doesn’t it make you want to throw up?  And what happens if another change is made?  Yet Another If Statement.

Instead, I want to be able to make changes where the behavior is the same, but the selectors might be different.  Using Magium there are two ways of doing this (which I won’t get into here).  Either by setting type preferences for the DIC or by having some classes configurable (extending an AbstractConfigurableElement class).

Test code must be repeatable

One of the things I’ve not liked about many of the libraries I’ve looked at is that they look fine in their examples but that they don’t take you beyond the basics.  I did find some other code that was similar to what I needed but most of it looked pretty old and was less complete than what I’ve written so far.

But I’ve gone a little too far.  What do I mean by “repeatable?”

I mean that I can make one method call to do a checkout.  I don’t have to write it out over and over again.

For example, if I need to do a guest checkout in Magium I can simply run

$guestCheckout = $this->getAction('Checkout\GuestCheckout');

But it’s not enough to just run the same thing over and over.  What if one of your requirements is that you can sell to the US and Canada?

Handling that kind of scenario is really easy in Magium.

$guestCheckout = $this->getAction('Checkout\GuestCheckout');

Test code must be readable

Which brings me to my next point.  A good number of people, i.e. virtually all of them, who run user acceptance testing are not particularly good programmers.  I wanted a library that was both powerful and accessible.

One of the core intentions of Magium is to make tests look clean so they are easy to read.

To see what I mean, consider the Selenium log to navigate to a category and assert that a product exists on that page.

[get: http://magento19.loc/])
[find element: By.xpath: //nav[@id="nav"]/ol/descendant::li[contains(concat(" ",normalize-space(@class)," ")," level0 ")]/a[.="Accessories"]/../a])
[mousemove: 0 false])
[find element: By.xpath: //nav[@id="nav"]/ol/descendant::li[contains(concat(" ",normalize-space(@class)," ")," level0 ")]/a[.="Accessories"]/../descendant::li[contains(concat(" ",normalize-space(@class)," "),"level1 ")]/a[.="Jewelry"]/../a])
[mousemove: 1 false])
[find element: By.xpath: //body[contains(., "Blue Horizons Bracelets")]])

Now compare that to the Magium code.


Which makes more sense to you?  And you’re probably a programmer.

It needs to work in the admin area

Seeing that ^^^ header tag alone probably makes you want to go home and cry.  Have you ever tried to script placing an order in the admin?  You’ll be Nietzsche and Freddy Mercury rolled into one.  You will be sorrowfully singing the Bohemian Rhapsody all the way home from work in your car around when the clock ticks over to Sunday and praying that the priest will accidentally throw some holy water in your eyes.  (being Protestant I have no such hope)

Let’s say you have a feature that turns the site into maintenance mode via a configuration switch.  You gotta test that, right?  You want to automate it, right?

Well because I have such fondness for people who want to automate their Magento testing as much as possible Magium has an action called an Enabler that allows you to enable and disable “enablement” switches in the system configuration.  Let’s say that your hypothetical maintenance mode is under System / Maintenance mode and has switch called “Enable”.

$enabler = $this->getAction('Admin\Configuration\Enabler');

Magium will log in, navigate to the System Configuration page, click on the System Tab, make sure that the Maintenance section is displayed, clicking the section header if it isn’t visible, switches the “Enable” switch to “Yes”, and clicks “Save” (which is surprisingly hard in Magento).


So what is Magium?  It is a fancy batching tool for Selenium tests.

How should you start?  Read the tutorials.  (Very much a work in progress at the time of writing)

How can you contribute?  Check out Magium and MagiumMagento and contribute.  I’m one of the nice open source software maintainers and I promise I won’t be an ass about contributions.

Important notes

  • This is not an official Magento project – it is done entirely on my own time to solve my own problems
  • I have no idea how or even if this will fit into Magento 2. I have barely touched M2 and, more importantly, the Magento Test Framework.  M2 MTF may render this whole thing moot.  I’ll let you know when I actually have time to get to know Magento 2.
  • This is nowhere near a v1 release.

How to examine the reports files for Magento

I was recently asked by a client for a snippet of Linux CLI that I use to get a summary of Magento reports files under var/reports.  I don’t guarantee that this is the best CLI command, but this is what I use

head -n 1 * | grep -v == | grep -v -e '^$' | awk -F \" '{ print $2 }' | sort | uniq -c | sort -n
 1 SQLSTATE[HY000] [1049] Unknown database 'magento19'
 1 SQLSTATE[HY000] [2002] Connection refused
 2 Cannot send headers; headers already sent in , line 0
 3 SQLSTATE[HY000] [2002] No such file or directory

If you want to format it so you can copy and paste into Excel (for charting)

head -n 1 * | grep -v == | grep -v -e '^$' | awk -F \" '{ print $2 }' | sort  | uniq -c | sort -n | sed 's/^ *//;s/ *$//' | awk '{ a=$1;sub("[^"FS"]+["FS"]+",""); printf "\"%s\",%d\n", $0, a}'

If you only want to check the last 1000 reports

head -n 1 `ls -t | head -n 1000`  | grep -v == | grep -v -e '^$' | awk -F \" '{ print $2 }' | sort | uniq -c | sort -n

Feel free to post corrections or other useful CLI commands in the comments if you like.

Configuring MySQL SSL in Magento

I’ve been asked a few times now if there is a way to use encrypted MySQL connections in Magento.  Most of the time it is when merchants are selling medical products and HIPAA requirements come into play.  I am not an expert in HIPAA, nor do I want to be, but with the cost of vulnerabilities on the rise it made sense to at least look into it and get a good answer on how to do it.

The answer, to my surprise, is that there is no way of doing it out of the box.  The reason is actually very, very simple.  The PDO options that allow you to initiate SSL on MySQL are passed in as numerical values.  For example, the value for PDO::MYSQL_ATTR_SSL_KEY is 1007.  Shouldn’t be a problem, right?  Just pass in a numerical value for the config.

That works in the Zend Framework adapter, but it does not work for Magento.  All database configurations are stored in the local.xml file and the XML specification does not allow numbers for XML node names.  So no matter how you try to slice it it looks like getting the SSL settings into the Magento adapter will not work without a code change.  The Internet seems to confirm this.

But that doesn’t mean that it can’t be done.  So  I wrote a quick Magento adapter that allows you to pass in the constant values.  The implementation is about 10 lines of code and it’s super easy to test.

  1. Download the code from github.
  2. Configure MySQL – Note that when you create the server certificate you MUST use the IP address or hostname that you will be using in your Magento configuration
  3. Configure Magento

For step 3 first copy the Github code into your Magento installation.  Then modify your configuration such as this

         <initStatements><![CDATA[SET NAMES utf8]]></initStatements>

The main difference is the node secure_driver_options.  Internally, what the driver does is append that to PDO:: and gets the value of the constant.  Then it adds it to the driver_options configuration array (notably absent since this adapter overwrites it) using the numerical values of the constants instead of node names.

Run the test script secure.mysql.test.php in your browser (or CLI).

There you go.


  1. This is not an officially sanctioned Magento code example
  2. I have not tested it for performance, security, gremlins.
  3. I have built this as a proof of concept only
  4. Use at your own risk.
  5. If you find issues in the code, or a better implementation, feel free to contribute or let me know

How to (properly) harness the Magento EE Full Page Cache (part 2 – application states)

In part 1 we took a look at some of the basics of the Magento EE FPC.  We talked about the general structure and flow of the FPC.

In this article we are going to take a look at the application states in the FPC.  I don’t know if “states” is the official term, but it’s what I use to describe what they are.

There are, essentially, 4 states that the Full Page Cache can be in.

Completely cached

All containers can be rendered by the processor.  This is the best state to be in.  No DB connection is required.  The only infrastructure required is a connection to the cache.  Each container will pull content from the cache and inject its content in the page.

Completely cached, with processing

Most containers will pull their content from the cache, but some might render content inline.  This is done by overriding the applyWithoutApp() method.  We’ll take a look at this in a future article

Partially bootstrapped without blocks

The state is when the applyWithoutApp() method returns false.  The FPC will first call that method.  If ANY of the containers returns false the FPC will render as many containers as it can but keep track of the ones that did not.  Then it will bootstrap the request and forward it to the FPC’s request controller called Enterprise_PageCache_RequestController.  The processAction() method call each un-processed container’s applyWithinApp() method.

Partially bootstrapped without layout

The difference between the two partially bootstrapped states depends on if the individual container overrides the applyInApp() method.  The abstract container will try to render a block when its applyInApp() method is called.  So if you do not want to render the blocks (they will likely have recursion and hierarchy to them) you will need to override the applyInApp() method for that container so it does not automatically try to render its block.

All of the different states can also be categorized by how they are processed.  The following diagram splits them up.


So which should you use?  Well, that’s the wrong question to ask.  The right question is “which should I build for?”  And for that question the answer is obvious.  Build to make your application use the request processor as much as possible.  Why?  Take a look at the code. Here is Mage_Core_Model_App:run().

public function run($params)
  $options = isset($params['options']) ? $params['options'] : array();
  Mage::register('application_params', $params);
  if ($this->_cache->processRequest()) {
  } else {
    $this->loadAreaPart(Mage_Core_Model_App_Area::AREA_GLOBAL, Mage_Core_Model_App_Area::PART_EVENTS);
    if ($this->_config->isLocalConfigLoaded()) {
      $scopeCode = isset($params['scope_code']) ? $params['scope_code'] : '';
      $scopeType = isset($params['scope_type']) ? $params['scope_type'] : 'store';
      $this->_initCurrentStore($scopeCode, $scopeType);
  return $this;

The full page cache is executed in the line that says processRequest().  If that method returns true, meaning that all of the containers were able to process their content, then the response object is called and the request is essentially done.  No routing, no controllers, and, importantly, no layout.  All of those are integral to executing a Magento request, but they do come at a cost.  If you can bypass that cost as often as possible your application will be much, much faster.

How much faster?  Consider the response times of this quick load test on my very, very under-powered virtual machine.


How to (properly) harness Magento EE’s Full Page Cache (part 1)


  1. The Full Page Cache is one of the most important performance features in Magento EE and very few people know how to use it
  2. Containers control your content
  3. Processors manage containers
  4. Knowing containers and processors gets you 90% of the way to solving #1

With my work with ECG one of the more common things I see are performance problems with web sites.  Most often it is due to problems that developers have caused with how they interact with database.  However, that represents the problems I see.  How about the things that people could do better. A big issue that a lot of Enterprise Magento merchants have is that their full page cache is underutilized.  You may be saying to yourself “Why don’t I just use Varnish?  It’s loads faster anyway.”  You can… as part of an overall caching strategy.  Varnish is really good for static content.  But what happens when someone adds an item to the shopping cart.  You are now running fully bootstrapped and laid out Magento requests.  That, or you’re now using Ajax and cookies to do things to your layout to make it appear like the shopping cart is being rendered. But it doesn’t just have to be the shopping cart.  Anything that needs some level of unique display is a potential cache invalidation feature.  What is unique about the Magento Enterprise Full Page Cache is that you get the best of both worlds.  You get cached pages AND you get non-static content.  Varnish is still going to be faster, but now the difference is 0.5ms vs 20ms instead of 0.5ms vs 2000ms.  In many cases, a properly interfaced EE FPC negates the need for Varnish. Following is a diagram that roughly corresponds to how the FPC works and where it hooks in to the process. FPC_Order There are two places where the FPC hooks into. The first is the request processors.  This is at the front of the request.  It is done immediately after the app/etc/*.xml files have been loaded.  This is because that is where the cache configuration exists.  Can use a cache without cache configuration.  The FPC is a processor.  The processor (Enterprise_PageCache_Model_Processor) is responsible for loading the cache metadata and initial cache content.  It then delegates the processing of the cached content to a sub-processor.  A sub-processor is in charge of retrieving cached content and merging content into the correct place.  There are three types of sub-processors

  1. Default (typically used by the CMS)
  2. Category
  3. Product

For the most part you do not need to worry about the sub-processors. The real magic occurs in the containers.  The containers are responsible for caching and rendering block output.  The caching component injects itself into the block output and the rendering injects itself into the sub-processor. Containers are the glue between the cached and uncached content. 2015-05-15_1646 A container “injects” itself at certain places in the code and it does this by hooking in to the core_block_abstract_to_html_after event.  This happens when the layout is being rendered.  If there is a container defined for the block being executed then the FPC will do two things.

  • Extract the block’s HTML and save it to the cache
  • Surround the HTML with a placeholder tag.

The placeholder tag is used on subsequent requests to find cached output for certain containers when rendering the page from the cache. We’ll take a look at the structure of a full page cached page in a future article. But next up will be the states that the full page cache can be in.

Be wary of functions in SQL in Magento, such as MAX(). There may be a better way.

I just wrapped up a Healthcheck for a Magento ECG customer and I ran into an interesting issue.  IIRC, the problem wasn’t a client one but rather code that a community extension used.  I won’t mention which one.

This extension was generating a query that was looking for the maximum minimum price for all products in a category.  The resulting query was something like this.

SELECT MAX(min_price) FROM catalog_product_index_price AS cpip
INNER JOIN catalog_category_product_index AS ccpi
    ON ccpi.product_id = cpip.entity_id
WHERE ccpi.category_id = 2;

So, it was doing a lot of things right.  It was using the index tables, not the attribute tables.  It was also using indexes, so that was right too.  But take out the MAX() call and see what happens.  I’m running this on the Magento sample data set and it returns 1275 results.  And as we all know the sample data is not like most production systems.  The customer I was working with saw about 40k results on some queries and under load this was taking 20 seconds in some cases, though never less than 200ms.  The reason is because it’s essentially doing a table scan on the results to find the record with the highest value.

So how do we make this faster?

It’s super easy, and fun to do.

SELECT min_price FROM catalog_product_index_price AS cpip
INNER JOIN catalog_category_product_index AS ccpi
    ON ccpi.product_id = cpip.entity_id
WHERE ccpi.category_id = 2
ORDER BY min_price DESC

Sorting by min_price and limiting by 1 gives you the same result with a much more efficient query.  On the client’s test system it went from a minimum of 200ms (possibly 10s of seconds) to >1ms.

If you are well versed in SQL this makes a lot of sense, and might have been what you did normally.  But for this community extension it isn’t what it did and I suspect that given that a lot of Magento developers are API-oriented this is what they tend to do.

He put this in a blog and was amazed at what happened.

Sorry.  I saw about 50 of those linkbait posts and it bugged me so I did one of my own.  I didn’t click on them but those just really, really bug me.  Call this link bait, call it a protest piece or call it performance art.  I just really, really hate those posts.

I don’t know why I’m writing this post.  Maybe it’s because the fire alarm went of for a few seconds at 3:00am.  Geez, fire alarm, if you’re going to startle me awake like that have the common decency to have something to check for.  It is easier to sleep when things happen for reasons you can explain.

Maybe that’s a life lesson.  Who knows?



The server requested authentication method unknown to the client [mysql_old_password]

I updated my server this morning, moving from Zend Server PHP 5.3 to 5.6 but when I did I got a really weird error.

The server requested authentication method unknown to the client [mysql_old_password]

I spent way too much time debugging it but it turns out that this is an incompatibility between MySQLnd and MySQL 5.5.  For the defaults, at least.  MySQLnd does not know about the mysql_old_password authentication method and so this error was being thrown.

The information I found said to update your user’s password and things should be fine.  I did, and they weren’t.  When I was updating the password it wasn’t using the new password hash for some reason.  So what I did was execute the following commands.

set session old_passwords=0;
update mysql.user set Password=PASSWORD('password') where User = 'user';
flush privileges;

For whatever reason I needed to disable old passwords on the MySQL session that I was connected on.

Load Testing the Magento Checkout with JMeter

Load Testing the Magento checkout is, in theory, difficult.  Well, not so much difficult as time consuming.  It is my personal opinion that most Magento load testing is done either poorly or incompletely.  Load testing read-only pages usually means hitting the same page over and over again.  This is counter productive because it allows the system to cache re-used data and skew load testing results to the positive.  I have yet to see a load test that yields performance results that matches production.

Testing the checkout is often even worse.  It is also the point in the load test that is most likely to change system behavior due to changing inventory, API calls, add-to-cart operations, etc.  It’s also hard to build a JMeter test for.  Most of the time orgs don’t have a JMeter expert on staff and so rely on the developers to build the load test.  Developers spend most of their time writing code, not load tests, and so the tests tend to not include some of the harder tests, such as checkout.  I don’t blame anyone for that, a devs core skill is not load testing.

To help with that I’ve written a basic JMeter checkout scenario for JMeter.  It uses the JMeter Plugins Extras Set.  The test uses the Guest Checkout and the product added to the cart is from the Magento sample data, so you will need to change that.  In addition you will need to change the HTTP Default host name, as shown below.


You can download the JMeter script here.

Magento, HHVM and the Enterprise Full Page Cache

*** Personal note: It has been suggested that this may reflect an official Magento position.  It does not.  I like doing research and I like publishing what I find as long as I find it interesting and it isn’t tied to a customer.  So if you’re wondering if this reflects a change in Magento or some kind of future roadmap, it does not.  ***

I was recently asked by a client about Magento and HHVM support.  While the answer is “don’t touch it in production and there’s no guarantee of support” it did spur in me the insatiable desire to create charts.  I’m curious about it and I’m sure you are too.

One of the things I’ve been curious about is how much juice could we squeeze out of the HHVM/Magento combo?  In other words, how fast could Magento Enterprise 1.14 actually be?

My test setup is my crappy local VM.  8GB of RAM with 4x Intel(R) Core(TM) i5-4440 CPU @ 3.10GHz.  I had intended to test with Redis (boy do I have a fascinating blog post coming up there!) but there seems to be an odd thing that occurs to HHVM when dealing with reading from sockets that have buffered data 16k or greater.  Since HHVM uses FastCGI that meant that for a proper comparison I needed to use Nginx with PHP-FPM.  So I used the Cm_Cache_Backend_File instead.  While my goal was to see if HHVM was faster than standard PHP for Magento I also wanted to see how many more requests per second I could squeeze out of it.

The answer to the latter question is; a whole shitload.

The test was simple.  Hit the front page of a FPC-enabled site with JMeter and see what happened (0 – 25 concurrent connections and down again).  Here’s what happened.

PHP-FPM peaks at just under 200 requests per second


HHVM Peaks at, wait for it, over 300 requests per second.

HHVM Requests Per Second

That is 300 freaking MAGENTO requests per second on a 4 CPU box!

Now, lest you think I’m suggesting you use HHVM, I’m not.  It was not easy getting it to work and I wouldn’t trust it with a full workload.  PHP 7 is also supposed to have similar improvements.  And this was a test of one page, to see what was possible with one piece of functionality.  But it’s an interesting indication of the improvements that are being made in the PHP ecosystem as a whole thanks to the introduction of HHVM.  Perhaps PHP 7 will render HHVM moot.  If that’s what happens, it’s fine by me.  I’m just glad that we are finally starting to see some forward momentum on addressing some of the performance issues that have been inherent in PHP for quite some time.  A little bit of creative destruction, now and then, is a good thing.

Web Analytics