Using PHP WITH Jetty, Camel, ActiveMQ, WebSockets and other things that might score high on SEO

Leave a comment

Over the past several days (more than I would like to admit) I’ve been working on getting a system based off of Jetty and ActiveMQ to work out of the box for PHP developers.  While “out of the box” isn’t quite true I have been able to get everything to work in a single downloadable container.  It uses ActiveMQ as the messaging system, which also has a WebSocket listener.  ActiveMQ also has a queue called “worker” which is used to send job queue requests to the queue.  That queue has an HTTP camel component which listens on the queue and forwards the requests to an HTTP pool.  Running the whole thing is a pre-configured Jetty server.  Getting that set up takes a smidgen of work, but it’s not that hard.  grep -R is your friend.

The example application is a chat application (of course), but it has some added functionality.  You can create a pure WebSocket chat server quite easily, no PHP needed.  But I wanted to include PHP functionality so what I did was make the (ugly) frontend push the chat message to the queue and the worker would do three things.

  1. Parse it for Markdown-based content
  2. Regex the output for URLs and send those URLs back to the queue
  3. When dequeued the URLs are parsed for a content snippet, title and an image URL.  If those are found they are sent back to the frontend which then attaches the snip to the original message container.

It’s kind of a neat little setup.  And, what’s cool about it, is if you change the document root for the website you can run a Magento instance out of it.

You can download the whole thing here.

Configuration

It is configured to run on port 8161.

There are a few things you are going to need to do to make it work.

First, make sure you have PHP-FPM running on port 9000.

Second, edit the etc/vhosts/localhost.xml file to change the document root for the web application in the download’s html directory.

Third, edit the html/WEB-INF/web.xml to change the document root in that file as well.

To run PHP in Jetty you need to have the WEB-INF/web.xml file in there.  There is a configuration setting where the only path to change it is in that file.  I spent, literally, days trying to get this to work without building my own into Jetty.  I could have done that, but I wanted this to run as much out-of-the-box as possible.  The setting is async-supported, in case you are wondering.  There was a change in one of the Jetty versions where FastCGI went from announcing it supported async by default to saying it didn’t.  And the only way to change that setting is in the WEB-INF setting.

The Application

The application, as noted before, is a simple chat application.  You can have multiple people sharing the same chat window.  Everyone can chat with everyone else.  There is no concept of a room, this is a demo after all.

To make it work, go to either http://localhost:8161/ or http://localhost/ if you set up iptables to route to port 8161.  From there the chat window will open and you can start typing.  Note that I have not built anything resembling security into this application.

Some notes on the packages installed.  The first is the eschrade/async_pack composer package.  I have used only one component, the Stomp client.  That client extends the Fusesource Stomp package.  I needed to extend it because I was getting connection errors when creating the WebSocket connection and it’s easier to extend a package than to get a fix done… especially a fix that could break other adapters.

One note on how I do queuing.  It is wrong.  Horribly wrong.  Don’t use it as an example.  In the eschrade/async_pack I created a queue client/server model that can be used.  But for a simple demo what I did was sufficient.  Maybe when I get some free time I’ll make it better.

Moving On

I hope you try it out and like it.  I’ve been a little bit surprised by how much I’ve liked it despite the fact that I’m using it largely as a web server.  Perhaps it’s because this approach opens up a world of opportunities with PHP developers that, beforehand, required a lot of infrastructure chicanery.  This package has most of what I’ve wanted in an all-in-one PHP environment for almost a decade.


Configuring Jetty to run Magento

Leave a comment

Yesterday I wrote a blog post on my surprising finding that Jetty was able to out-perform, or at minimum keep up (depending on how you looked at the numbers), Nginx when it comes to static files.  Today I wanted to get it up and running and able to serve Magento requests.

It turns out that it is stupid easy.  That said, it was stupid hard figuring out how stupid easy it was.

To get Magento running with Jetty

  1. Set up PHP-FPM as you would with Nginx, making sure that you use TCP and not Unix sockets
  2. Download Jetty.  Do not use the RPM, if you are using CentOS.  Yum does not have all the components you need.  Use the full download from the Jetty page on the Eclipse site.  Install into /usr/share/java/jetty
  3. Configure Jetty.

So let’s look at number 3.

There are two things you need to know about this.

First of all is that FastCGI is not enabled by default.  You need to add it to its start.d directory.

# export JETTY_HOME=/usr/share/java/jetty
# cd /usr/share/java/jetty/etc 
# java -jar $JETTY_HOME/start.jar --add-to-start=fcgi
INFO: fcgi initialised in ${jetty.base}/start.ini
INFO: Base directory was modified

FastCGI is now installed.  Now you need to configure a default context.  Create a file called $JETTY_HOME/etc/webapps/fcgi.xml (or whatever.xml) and add the following.

<?xml version="1.0" encoding="UTF-8"?>



 
    
        /var/www/magento19/magento/
    
 
    /
    
    
        index.php
    
 
    
        org.eclipse.jetty.fcgi.server.proxy.TryFilesFilter
        /*
        
            
                
            
        
        
            files
            $path /index.php
        
    
 
    
        
            
                default
                
                    
                        org.eclipse.jetty.servlet.DefaultServlet
                    
                
                
                    dirAllowed
                    false
                
            
        
        /
    
 
    
        org.eclipse.jetty.fcgi.server.proxy.FastCGIProxyServlet
        *.php
        
            proxyTo
            http://localhost:9000
        
        
            prefix
            /
        
        
            scriptRoot
            
        
        
            scriptPattern
            (.+?\\.php)
        
    
 


THAT  is a lot of stuff.  Well, actually it’s not, but it’s very verbose.  There are a few things you need to know

  1. //New[@id=”root”] – this is document root for this web application
  2. //Set[@name=”contextPath”] – This is the base URI where your application lives.  http://example.com/ would be /, whereas http://example.com/blog/ would be /blog/
  3. Find org.eclipse.jetty.fcgi.server.proxy.TryFilesFilter.  That is the equivalent of Nginx’s try_files setting.  Here, if it doesn’t find the file, route it to index.php.
  4. The last addServlet item is the FastCGI component.  I trust you can see the two important settings.

And there you have it.  A Jetty instance configured to run Magento.

Oh, and how does this compare to Nginx in terms of Magento throughput?

2016-06-03_1250


3 Best Practices for Selenium Testing when constructing your page

Leave a comment

Having spent now about two months building out Magium there are a couple of things that are worth sharing as I’ve been working through several different versions of Magento. These practices (best practices?) may or may not be “officialized” by the Overlords of the Internet but what I have found is that I have the most trouble when these practices are not executed.

The basic premise behind each of these is that the quickest path to the most specific element is best. Long Xpaths may be prone to break and do not lend themselves well to re-use. Specificity and predictability make building Selenium tests not quite a dream, but definitely not a nightmare.

Read more at the Magium blog.


80-ish% of Magento Developers Do Not Automate Browser Testing

3 Comments
small-white-transparent-logo

A few weeks back I asked a question on Twitter about browser testing.  There weren’t a ton of responses but the responses I got were about what I expected.  Around 80% of Magento developers (and I would probably say web-developers in general) do NOT do any significant browser testing automation.

The reason that doesn’t surprise me is because browser testing is kind of hard, time consuming, and often requires a fairly significant investment to do it.  I know this because I spent a modest portion of my summer last year trying to do it for an important project.  If you are a really good developer, or a professional QA tester, then this may sound like absolute nonsense to you.  But I think that when you look at the market at large this is a truth that you will see.

Note that I don’t believe that this is a symptom of developer competency or anything like that.  My perception is that this is largely due to the cost/investment required to automate browser testing.

For that reason I started working on a project creatively called Magium.  I won’t speak much about it here (check out the website for more information).  My goal was to remove many of the barriers to doing effective browser testing on Magento.  I believe that is something I am achieving, though have not achieved yet.  As it stands Magium supports a lot of things

  • Add-to-cart functionality (automated (“best guess”) and specific)
  • Guest Checkout
  • Registered Customer Checkout
  • New Customer Checkout
  • Frontend Navigation
  • Customer Login
  • Customer Creation
  • Customer Navigation
  • Customer Order Extraction
  • Search (including search suggestions)
  • Layered Navigation
  • Product Extraction (Category List, Category Grid, Search List, Search grid)
  • Admin Navigation
  • System Configuration Changes (this was not easy to do)
  • Admin Order Handling (this was even harder to do)

plus a bunch more

If all of this interests you I would invite you to take a look at this video.  It’s a little long, but it is a tutorial that will take you through the basics of building Magium tests through doing a full checkout all the way through managing the order in the admin.

Using Selenium.

In about 20 lines of code.

Yeah.


Magento Selenium Testing

4 Comments

[TL;DR] Go to the conclusion at the end, get the links for the GitHub repo, contribute.  Follow MagiumLib on Twitter (nothing posted as of writing)

[update] The Magium site is live.  Lots of stuff there.  More coming.

This past summer (2015) I was tasked with building out automated testing for a Magento 1 module.  I had not really built out a lot of automated testing before and so I was in for a massive shocker. It sucked.  It royally sucked.

I had my list of user stories and I started out as everyone starts out: I downloaded the Selenium IDE.  It worked really well.  I built out about 10-20 test cases and was impressed at my progress.

Then I had to make a simple change to all of the test cases.  That turned all of my progress into a useless pile of crap.  I had wanted do it with as little developer skills as necessary and failed miserably.

So I went to Google to try and find out what was available that suited my needs.  My needs were simple:

  1. I needed repeatable functionality, such as being able to easily execute the checkout process
  2. I needed that repeatable functionality to be modifiable – the module was tightly integrated with checkout and had many options to tests against.  So I needed to be able to modify the checkout consistently.
  3. I needed to do all that in the admin area too
  4. It needed to be written in PHP.  Magento IS written in PHP and it’s silly to require yet-another-programming-language

There were several different options available but they often seemed basic.  Good, but basic.  Or good, but general.  In other words, there was still a lot of work that I needed to do to make it work.

Plus, I didn’t have an existing toolset that I was familiar with enough to force it to make it work.

So I decided to take some things that I was decent at (PHPUnit and PHP in general) and make it work.

Lo and behold there was a PHPUnit Selenium2 library.  So I started writing tests using that library.  I really kind of liked it.  I was under the gun and so didn’t have to time research and architect it well.  What I wrote broke about every SOLID guidance there was but it largely worked.

Except it didn’t do it very well.  Things like category navigation worked, except when I had to switch themes.  Then I had to create conditional statements to see which Xpath query would work.  Checkout would work, except when it didn’t.  Ajax was not happy with the library.  And don’t even get me started on the admin area.  It was quite difficult.  Add on top of that the module had a lot of different configuration settings that needed to be tested.

Which brings me to my last point.  The Selenium2 library for PHPUnit is sssslllllooooowwwwwww!.

At the end of the day I made it work, but there were a lot of lessons learned and a lot of bad testing approaches that I would not have done had I had more experience.

Enter Magium.

Magium is a testing framework that I’ve started work on that originally combined the things I needed to have working together.  Selenium, PHPUnit, and Magento.  However, as I’ve been working on it I’ve found that it could actually be used for other projects as well.  But my work is with Magento and so I’m building out the framework with that in mind.  Hence the name, despite the fact that the core functionality could actually be used in any web-based application.

I looked at some options for a PHP-based Selenium driver and there were basically 3 that came up.  PHPUnit_Selenium2, Instaclick, and Facebook.

  1. PHPUnit_Selenium2 – Good.  Nice integration with PHPUnit.  Slow as molasses in Alaska.
  2. Instaclick – Definitely faster, but I had problems with consistent test results
  3. Facebook – No problems.  Nice, fast, and consistent

So I opted to use the Facebook WebDriver driver.

Now I have the WebDriver driver decided on.  Next up is to decide how to best use it.

Now, “best” is a hard term to quantify.  What is the best way of doing browser testing?  There are a plethora of browser testing tools out there.  Several are available for Magento.  Many of them are good.  But I found that what was available didn’t solve the problems the way I felt they needed to be solved.

Because of that there are a number of choices I made to build a testing foundation that solved the 4 problems I noted earlier.

Repeatable Functionality

This is probably one of the easiest problems to solve.  Create classes that utilize SOLID principles.

Single Responsibility – A class is (generally) responsible for one thing.  For example, filling out the billing form or navigating to a category.

Open/Closed – Checkout may have some customization that need to be tested.  Because each step in the checkout process follows Single Responsibility the checkout can be modified by providing different objects to do different things.  But the checkout class, itself, is only a container for that functionality and does not require any changes in order to handle customization.

Liskov Substitution – Using checkout as an example again.  The checkout may have different payment methods.  You are able to swap them out because they are based off of sub-class that defines how they should work.

Interface Segregation – Payment in checkout is a good example.  It is both a payment method and a checkout step and both are defined separately.

Dependency Inversion – I kind of break this one a little, though not too badly.  The way I break it is by depending on base-level functionality, not abstractions.  That is because what I needed was repeatable functionality that works with the core Magento implementation, but can also be customized.  The proper way, I know, is to provide the abstractions, or interfaces, as the dependency and then configurably provide concrete implementations.  I commit a minor sin in its usage for the purpose of having a working test system out of the box that requires only minor configuration.

Modifiable, Repeatable Functionality

My implementation of this requirement is probably what will give most people the most agida.  I used the SOLID principles not for architecting an application, but architecting tests.  I mis-use them intentionally because it is the principles that are useful, not the implementation.  The test is the application, in the case of building a testing library.

My most egregious assault on software development principles is in how I fill the need for making the repeatable functionality modifiable.  I do that through the use of a Dependency Injection Container as a critical part of the test case.

I have very good reasons for this.

Test code must be configurable

A big, big part of my original problem was that the module I was working on needed to work on CE 1.7, CE 1.8, CE 1.9, EE 1.12, EE 1.13, and EE 1.14.  You might be tempted to think that not much has changed, or, rather, that the change has been iterable.  But that’s not true.  Checkout HTML code is quite different between versions.

As an example consider this code I had to write to place the order.

1
2
3
4
5
6
7
8
9
10
11
  $this->waitUntilElementDisplayed('review-buttons-container');
  if ($this->elementExists('//button[@title="Place Order"]', self::BY_XPATH)) {
    $this->waitUntilElementDisplayed('//button[@title="Place Order"]', self::BY_XPATH);
    $this->byXPath('//button[@title="Place Order"]')->click();
  } else if ($this->elementExists('//*[@onclick="review.save();"]', self::BY_XPATH)) {
    $this->waitUntilElementDisplayed('//*[@onclick="review.save();"]', self::BY_XPATH);
    $this->byXPath('//*[@onclick="review.save();"]')->click();
  } else if ($this->elementExists('review_button')) {
     $this->waitUntilElementDisplayed('review_button');
     $this->byId('review_button')->click();
  }

That is ghastly code that I never want to write again.  Doesn’t it make you want to throw up?  And what happens if another change is made?  Yet Another If Statement.

Instead, I want to be able to make changes where the behavior is the same, but the selectors might be different.  Using Magium there are two ways of doing this (which I won’t get into here).  Either by setting type preferences for the DIC or by having some classes configurable (extending an AbstractConfigurableElement class).

Test code must be repeatable

One of the things I’ve not liked about many of the libraries I’ve looked at is that they look fine in their examples but that they don’t take you beyond the basics.  I did find some other code that was similar to what I needed but most of it looked pretty old and was less complete than what I’ve written so far.

But I’ve gone a little too far.  What do I mean by “repeatable?”

I mean that I can make one method call to do a checkout.  I don’t have to write it out over and over again.

For example, if I need to do a guest checkout in Magium I can simply run

1
2
$guestCheckout = $this->getAction('Checkout\GuestCheckout');
$guestCheckout->execute();

But it’s not enough to just run the same thing over and over.  What if one of your requirements is that you can sell to the US and Canada?

Handling that kind of scenario is really easy in Magium.

1
2
3
$this->getIdentity()->setBillingCountryId('CA');
$guestCheckout = $this->getAction('Checkout\GuestCheckout');
$guestCheckout->execute();

Test code must be readable

Which brings me to my next point.  A good number of people, i.e. virtually all of them, who run user acceptance testing are not particularly good programmers.  I wanted a library that was both powerful and accessible.

One of the core intentions of Magium is to make tests look clean so they are easy to read.

To see what I mean, consider the Selenium log to navigate to a category and assert that a product exists on that page.

[get: http://magento19.loc/])
[find element: By.xpath: //nav[@id="nav"]/ol/descendant::li[contains(concat(" ",normalize-space(@class)," ")," level0 ")]/a[.="Accessories"]/../a])
[mousemove: 0 false])
[find element: By.xpath: //nav[@id="nav"]/ol/descendant::li[contains(concat(" ",normalize-space(@class)," ")," level0 ")]/a[.="Accessories"]/../descendant::li[contains(concat(" ",normalize-space(@class)," "),"level1 ")]/a[.="Jewelry"]/../a])
[mousemove: 1 false])
[find element: By.xpath: //body[contains(., "Blue Horizons Bracelets")]])

Now compare that to the Magium code.

1
2
$this->commandOpen('http://magento19.loc/]');
$this->getNavigator()->navigateTo('Accessories/Jewelry');

Which makes more sense to you?  And you’re probably a programmer.

It needs to work in the admin area

Seeing that ^^^ header tag alone probably makes you want to go home and cry.  Have you ever tried to script placing an order in the admin?  You’ll be Nietzsche and Freddy Mercury rolled into one.  You will be sorrowfully singing the Bohemian Rhapsody all the way home from work in your car around when the clock ticks over to Sunday and praying that the priest will accidentally throw some holy water in your eyes.  (being Protestant I have no such hope)

Let’s say you have a feature that turns the site into maintenance mode via a configuration switch.  You gotta test that, right?  You want to automate it, right?

Well because I have such fondness for people who want to automate their Magento testing as much as possible Magium has an action called an Enabler that allows you to enable and disable “enablement” switches in the system configuration.  Let’s say that your hypothetical maintenance mode is under System / Maintenance mode and has switch called “Enable”.

1
2
$enabler = $this->getAction('Admin\Configuration\Enabler');
$enabler->enable('System/Maintenance');

Magium will log in, navigate to the System Configuration page, click on the System Tab, make sure that the Maintenance section is displayed, clicking the section header if it isn’t visible, switches the “Enable” switch to “Yes”, and clicks “Save” (which is surprisingly hard in Magento).

Conclusion

So what is Magium?  It is a fancy batching tool for Selenium tests.

How should you start?  Read the tutorials.  (Very much a work in progress at the time of writing)

How can you contribute?  Check out Magium and MagiumMagento and contribute.  I’m one of the nice open source software maintainers and I promise I won’t be an ass about contributions.

Follow MagiumLib on Twitter (nothing posted as of writing).

Important notes

  • This is not an official Magento project – it is done entirely on my own time to solve my own problems
  • I have no idea how or even if this will fit into Magento 2. I have barely touched M2 and, more importantly, the Magento Test Framework.  M2 MTF may render this whole thing moot.  I’ll let you know when I actually have time to get to know Magento 2.
  • This is nowhere near a v1 release.


How to examine the reports files for Magento (and not burn your eyes)

1 Comment
magento

I was recently asked by a client for a snippet of Linux CLI that I use to get a summary of Magento reports files under var/reports.  I don’t guarantee that this is the best CLI command, but this is what I use

head -n 1 * | grep -v == | grep -v -e '^$' | awk -F \" '{ print $2 }' | sort | uniq -c | sort -n
 1 SQLSTATE[HY000] [1049] Unknown database 'magento19'
 1 SQLSTATE[HY000] [2002] Connection refused
 2 Cannot send headers; headers already sent in , line 0
 3 SQLSTATE[HY000] [2002] No such file or directory

If you want to format it so you can copy and paste into Excel (for charting)

head -n 1 * | grep -v == | grep -v -e '^$' | awk -F \" '{ print $2 }' | sort  | uniq -c | sort -n | sed 's/^ *//;s/ *$//' | awk '{ a=$1;sub("[^"FS"]+["FS"]+",""); printf "\"%s\",%d\n", $0, a}'

If you only want to check the last 1000 reports

head -n 1 `ls -t | head -n 1000`  | grep -v == | grep -v -e '^$' | awk -F \" '{ print $2 }' | sort | uniq -c | sort -n

Feel free to post corrections or other useful CLI commands in the comments if you like.


Configuring MySQL SSL in Magento (to get your HIPAA auditor off your back)

Leave a comment
mysql-magento-openssl

I’ve been asked a few times now if there is a way to use encrypted MySQL connections in Magento.  Most of the time it is when merchants are selling medical products and HIPAA requirements come into play.  I am not an expert in HIPAA, nor do I want to be, but with the cost of vulnerabilities on the rise it made sense to at least look into it and get a good answer on how to do it.

The answer, to my surprise, is that there is no way of doing it out of the box.  The reason is actually very, very simple.  The PDO options that allow you to initiate SSL on MySQL are passed in as numerical values.  For example, the value for PDO::MYSQL_ATTR_SSL_KEY is 1007.  Shouldn’t be a problem, right?  Just pass in a numerical value for the config.

That works in the Zend Framework adapter, but it does not work for Magento.  All database configurations are stored in the local.xml file and the XML specification does not allow numbers for XML node names.  So no matter how you try to slice it it looks like getting the SSL settings into the Magento adapter will not work without a code change.  The Internet seems to confirm this.

But that doesn’t mean that it can’t be done.  So  I wrote a quick Magento adapter that allows you to pass in the constant values.  The implementation is about 10 lines of code and it’s super easy to test.

  1. Download the code from github.
  2. Configure MySQL – Note that when you create the server certificate you MUST use the IP address or hostname that you will be using in your Magento configuration
  3. Configure Magento

For step 3 first copy the Github code into your Magento installation.  Then modify your configuration such as this

<config>
 <global>
   <resources>
     <default_setup>
       <connection>
         <host><![CDATA[127.0.0.1]]></host>
         <username><![CDATA[root]]></username>
         <password><![CDATA[]]></password>
         <dbname><![CDATA[magentoee_114]]></dbname>
         <initStatements><![CDATA[SET NAMES utf8]]></initStatements>
         <model><![CDATA[mysql4]]></model>
         <type><![CDATA[secure_pdo_mysql]]></type>
         <pdoType><![CDATA[]]></pdoType>
         <active>1</active>
         <secure_driver_options>
           <MYSQL_ATTR_SSL_KEY>/etc/mysql-ssl/client-key.pem</MYSQL_ATTR_SSL_KEY>
           <MYSQL_ATTR_SSL_CERT>/etc/mysql-ssl/client-cert.pem</MYSQL_ATTR_SSL_CERT>
           <MYSQL_ATTR_SSL_CA>/etc/mysql-ssl/ca.pem</MYSQL_ATTR_SSL_CA>
         </secure_driver_options>
       </connection>
     </default_setup>
   </resources>
 </global>
</config>

The main difference is the node secure_driver_options.  Internally, what the driver does is append that to PDO:: and gets the value of the constant.  Then it adds it to the driver_options configuration array (notably absent since this adapter overwrites it) using the numerical values of the constants instead of node names.

Run the test script secure.mysql.test.php in your browser (or CLI).

There you go.

Note

  1. This is not an officially sanctioned Magento code example
  2. I have not tested it for performance, security, gremlins.
  3. I have built this as a proof of concept only
  4. Use at your own risk.
  5. If you find issues in the code, or a better implementation, feel free to contribute or let me know

How to (properly) harness the Magento EE Full Page Cache (part 2 – application states)

Leave a comment

In part 1 we took a look at some of the basics of the Magento EE FPC.  We talked about the general structure and flow of the FPC.

In this article we are going to take a look at the application states in the FPC.  I don’t know if “states” is the official term, but it’s what I use to describe what they are.

There are, essentially, 4 states that the Full Page Cache can be in.

Completely cached

All containers can be rendered by the processor.  This is the best state to be in.  No DB connection is required.  The only infrastructure required is a connection to the cache.  Each container will pull content from the cache and inject its content in the page.

Completely cached, with processing

Most containers will pull their content from the cache, but some might render content inline.  This is done by overriding the applyWithoutApp() method.  We’ll take a look at this in a future article

Partially bootstrapped without blocks

The state is when the applyWithoutApp() method returns false.  The FPC will first call that method.  If ANY of the containers returns false the FPC will render as many containers as it can but keep track of the ones that did not.  Then it will bootstrap the request and forward it to the FPC’s request controller called Enterprise_PageCache_RequestController.  The processAction() method call each un-processed container’s applyWithinApp() method.

Partially bootstrapped without layout

The difference between the two partially bootstrapped states depends on if the individual container overrides the applyInApp() method.  The abstract container will try to render a block when its applyInApp() method is called.  So if you do not want to render the blocks (they will likely have recursion and hierarchy to them) you will need to override the applyInApp() method for that container so it does not automatically try to render its block.

All of the different states can also be categorized by how they are processed.  The following diagram splits them up.

Magento-EE-FPC-Types

So which should you use?  Well, that’s the wrong question to ask.  The right question is “which should I build for?”  And for that question the answer is obvious.  Build to make your application use the request processor as much as possible.  Why?  Take a look at the code. Here is Mage_Core_Model_App:run().

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
public function run($params)
{
  $options = isset($params['options']) ? $params['options'] : array();
  $this->baseInit($options);
  Mage::register('application_params', $params);
 
  if ($this->_cache->processRequest()) {
    $this->getResponse()->sendResponse();
  } else {
    $this->_initModules();
    $this->loadAreaPart(Mage_Core_Model_App_Area::AREA_GLOBAL, Mage_Core_Model_App_Area::PART_EVENTS);
 
    if ($this->_config->isLocalConfigLoaded()) {
      $scopeCode = isset($params['scope_code']) ? $params['scope_code'] : '';
      $scopeType = isset($params['scope_type']) ? $params['scope_type'] : 'store';
      $this->_initCurrentStore($scopeCode, $scopeType);
      $this->_initRequest();
      Mage_Core_Model_Resource_Setup::applyAllDataUpdates();
    }
 
    $this->getFrontController()->dispatch();
  }
  return $this;
}

The full page cache is executed in the line that says processRequest().  If that method returns true, meaning that all of the containers were able to process their content, then the response object is called and the request is essentially done.  No routing, no controllers, and, importantly, no layout.  All of those are integral to executing a Magento request, but they do come at a cost.  If you can bypass that cost as often as possible your application will be much, much faster.

How much faster?  Consider the response times of this quick load test on my very, very under-powered virtual machine.

cached-vs-uncached


How to (properly) harness Magento EE’s Full Page Cache (part 1)

Leave a comment
2015-05-15_1646

TL;DR

  1. The Full Page Cache is one of the most important performance features in Magento EE and very few people know how to use it
  2. Containers control your content
  3. Processors manage containers
  4. Knowing containers and processors gets you 90% of the way to solving #1

With my work with ECG one of the more common things I see are performance problems with web sites.  Most often it is due to problems that developers have caused with how they interact with database.  However, that represents the problems I see.  How about the things that people could do better. A big issue that a lot of Enterprise Magento merchants have is that their full page cache is underutilized.  You may be saying to yourself “Why don’t I just use Varnish?  It’s loads faster anyway.”  You can… as part of an overall caching strategy.  Varnish is really good for static content.  But what happens when someone adds an item to the shopping cart.  You are now running fully bootstrapped and laid out Magento requests.  That, or you’re now using Ajax and cookies to do things to your layout to make it appear like the shopping cart is being rendered. But it doesn’t just have to be the shopping cart.  Anything that needs some level of unique display is a potential cache invalidation feature.  What is unique about the Magento Enterprise Full Page Cache is that you get the best of both worlds.  You get cached pages AND you get non-static content.  Varnish is still going to be faster, but now the difference is 0.5ms vs 20ms instead of 0.5ms vs 2000ms.  In many cases, a properly interfaced EE FPC negates the need for Varnish. Following is a diagram that roughly corresponds to how the FPC works and where it hooks in to the process. FPC_Order

There are two places where the FPC hooks into. The first is the request processors.  This is at the front of the request.  It is done immediately after the app/etc/*.xml files have been loaded.  This is because that is where the cache configuration exists.  Can use a cache without cache configuration.  The FPC is a processor.  The processor (Enterprise_PageCache_Model_Processor) is responsible for loading the cache metadata and initial cache content.  It then delegates the processing of the cached content to a sub-processor.  A sub-processor is in charge of retrieving cached content and merging content into the correct place.  There are three types of sub-processors

  1. Default (typically used by the CMS)
  2. Category
  3. Product

For the most part you do not need to worry about the sub-processors. The real magic occurs in the containers.  The containers are responsible for caching and rendering block output.  The caching component injects itself into the block output and the rendering injects itself into the sub-processor. Containers are the glue between the cached and uncached content. 2015-05-15_1646

A container “injects” itself at certain places in the code and it does this by hooking in to the core_block_abstract_to_html_after event.  This happens when the layout is being rendered.  If there is a container defined for the block being executed then the FPC will do two things.

  • Extract the block’s HTML and save it to the cache
  • Surround the HTML with a placeholder tag.

The placeholder tag is used on subsequent requests to find cached output for certain containers when rendering the page from the cache. We’ll take a look at the structure of a full page cached page in a future article. But next up will be the states that the full page cache can be in.


Be wary of functions in SQL in Magento, such as MAX(). There may be a better way.

Leave a comment
2015-07-23_0943

I just wrapped up a Healthcheck for a Magento ECG customer and I ran into an interesting issue.  IIRC, the problem wasn’t a client one but rather code that a community extension used.  I won’t mention which one.

This extension was generating a query that was looking for the maximum minimum price for all products in a category.  The resulting query was something like this.

1
2
3
4
SELECT MAX(min_price) FROM catalog_product_index_price AS cpip
INNER JOIN catalog_category_product_index AS ccpi
    ON ccpi.product_id = cpip.entity_id
WHERE ccpi.category_id = 2;

So, it was doing a lot of things right.  It was using the index tables, not the attribute tables.  It was also using indexes, so that was right too.  But take out the MAX() call and see what happens.  I’m running this on the Magento sample data set and it returns 1275 results.  And as we all know the sample data is not like most production systems.  The customer I was working with saw about 40k results on some queries and under load this was taking 20 seconds in some cases, though never less than 200ms.  The reason is because it’s essentially doing a table scan on the results to find the record with the highest value.

So how do we make this faster?

It’s super easy, and fun to do.

1
2
3
4
5
6
SELECT min_price FROM catalog_product_index_price AS cpip
INNER JOIN catalog_category_product_index AS ccpi
    ON ccpi.product_id = cpip.entity_id
WHERE ccpi.category_id = 2
ORDER BY min_price DESC
LIMIT 1;

Sorting by min_price and limiting by 1 gives you the same result with a much more efficient query.  On the client’s test system it went from a minimum of 200ms (possibly 10s of seconds) to >1ms.

If you are well versed in SQL this makes a lot of sense, and might have been what you did normally.  But for this community extension it isn’t what it did and I suspect that given that a lot of Magento developers are API-oriented this is what they tend to do.