Improving WordPress Caching: Batcache and Redirects

BatmanThere’s a great plugin for WordPress called Batcache by Andy Skelton, which uses memcached as the cache backend for object cache and full page caches. It’s a great piece of code, and using it is a must if you want a scalable distributed cache for your WordPress install (avoiding the local file system caching WP-Super-Cache offers).

I’ve been running batcache for a while, and noticed that although I set up a query-heavy homepage to be cached for a long while, the homepage query was still running every couple of minutes. So after some digging and logging I came to realize that batcache was missing an important part — caching redirects.

Why is it important to cache redirects? Here’s an example. If I go to http://example.com/, my homepage will be cached. But when I go to http://www.example.com/, the homepage query still runs. This is because the default www to no-www redirect happens at ‘template-redirect’ action time (with canonical_redirect()), which occurs after the main posts query is run. And, batcache wouldn’t cache the redirect itself nor would it associate the www with the no-www version (it differentiates pages by a couple of parameters, including HTTP_HOST, which is different in this case). So any request to http://www.example.com would actually first run the homepage query, then redirect to http://example.com and serve the cached version only then.

After a short correspondence with Andy, he committed a changeset to advanced-cache.php (and readme.txt ;) ) that now enables to cache redirects (off by default, be sure to turn it on!). Great to see such responsiveness from a plugin author, and I hope it helps others as well.

Apache mod_ssl Makes Your Clients Crawl Compared To Nginx SSL Module

The SSL/TLS process is a heavy one, it involves algorithm negotiation between client and server, key exchanges, cyphering, decyphering and authentication. But what’s surprising is, that the server you’re connecting to can directly influence the performance of your client and its CPU consumption.

I had a php command line process spawning child processes and connecting through SSL to a web server, in 2 scenarios. The first scenario was to an out of the box Apache httpd server with mod_ssl, and the second scenario was to an out of the box Nginx with the SSL module. Both were using the exact same box, and were “out of the box” meaning I used the default configuration for both.

In the first scenario I was able to spawn no more than 6 (!) php processes before the box running them began to show load, and the CPU queue started to fill up. Each php child was taking between 15%-30% cpu at any given moment.

In the second scenario, I was able to spawn 40 (!!) php child processes without the box being loaded. Each php child was taking around 1.5% cpu.

I’m no SSL expert, and there might be a way to configure Apache to inflict less load on the connecting client. There is also SSLSessionCache which might relieve load from both the server and the client. But the “out of the box” configuration shows that Nginx is a real winner again.

If you can, avoid SSL altogether. If not, terminate it at a front-end before proceeding to Apache.

Google Chrome 2, CSS Content-Type and Amazon S3

Google ChromeIt seems that ever since Google Chrome 2 was released, some of the CSS files I was serving from S3 were not being treated as valid by it, and the page layouts would break because of it. Firefox and IE were fine with it, and Chrome 1 was ok with it too. It was just Chrome 2.

A little inspection showed that the CSS files I stored on S3 were not being served with a Content-Type header, while from a standard apache web server they were. This combined with the new strictness of Chrome 2 (actually resulting from a new strictness in WebKit) made Chrome not treat these files as actual CSS, and break the page.

So the obvious solution was to make the CSS files be delivered from S3 with the correct “Content-Type: text/css” header. Fortunately enough, this is very easy to do with S3 API. Just pass the “Content-Type:text/css” header when you’re uploading the file to S3, and it will be present in the response headers whenever someone requests the file.

Here’s to the browser wars, that never end and got more complicated with the new player in town, Google Chrome.

Cross Domain AJAX Calls and Iframe Communication How To

I set out to write a short tutorial about cross domain ajax calls and communication between iframes using url hashes (fragments), but I found this great tutorial by Michael Mahemoff covering the subject inside and out.

It is important to note that the code on his demo uses window.location.hash, although it seems this practice should currently be avoided because of a known open bugs in the way Firefox URI-decodes document.location.hash when getting its value (could be considered an expected behavior and not a bug, but still different than any other browser). Facebook’s code uses substring on pathname, and it seems to work cross browser. Facebook also have on their wiki a nice visual explanation of how it works:

Until the HTML 5 postMessage is widely adopted, we will have to keep using this somewhat of a hack, but it works, that’s what’s important.

Connecting Several PHP Processes To The Same MySQL Transaction

The following concept is still under examination, but my initial tests proved successful, so I thought it’s time to share.

Here is a problem: There is a batch job reading remote data through XML-RPC, and updating a local MySQL database according to the XML-RPC responses. The db is InnoDB, and the entire batch job should be transacted. That is, in any case of failure, there should be rollback, and on success there should be commit.

So, the simple way is of course a single process script that uses a linear workflow:

  1. START TRANSACTION locally.
  2. Make XML-RPC request, fetch data.
  3. Insert data into db as needed.
  4. Repeat 2-3 until Error or Finish.
  5. If Error ROLLBACK, if Finish COMMIT.

This works, but you may notice a bottleneck, being the XML-RPC request. It’s using http, and it’s connecting to a remote server. Sometimes the XML-RPC server also takes time to perform the work that generates the response. Add the network latency, and you get a single process that most of the time sits idle and waits for response.

So if we have a process that just sits and waits most of the time, let’s spread its work over several processes, and assume that while most of the processes will be waiting, at least one can be free to deal with the local database. This way we will get maximum utilization of our resources.

So the multi-process workflow:

  1. START TRANSACTION locally.
  2. Fork children as necessary.
  3. From child, make XML-RPC request, fetch data.
  4. From child, acquire database access through semaphore.
  5. From child, insert data into db as needed.
  6. From child, release database access through semaphore.
  7. From child, repeat 3-6 until Error or Finish.
  8. From parent, monitor children until Error or Finish.
  9. From parent, if Error ROLLBACK, if FINISH COMMIT.

Now, the workflow seems all and well in theory, but can it work in practice? Can we connect to the same transaction from several different PHP processes?

I was surprised to find out that the answer is positive. As long as all processes share the same connection resource, they all use the same connection. And in MySQL, the same connection means the same transaction, given that a transaction was started and not yet committed or rolled back (either explicitly or implictly).

The secret is to create the connection resource with the parent, and when forking children, they have a reference to the same connection. The caveat is that they must access the resource atomically, otherwise unexpected behavior occurs (usually the connection hangs, I am guessing that it is when one child tries to read() from the socket and the other to write() to it). So in order to streamline the access to the db connection, we use a semaphore. Each child can access the connection only when it’s available, and it’s blocking if not available.

In the end of the workflow, our parent process acts much like a Transaction Manager in an XA Transaction, and according to what the children report, decides whether to commit or rollback.

Here is a proof of concept code (not tested in this version, but similar code tested and succeeded):

The DBHandler Class

class DBHandler
{
	private $link;
	private $result;
	private $sem;

	const SEMKEY = '123456';

	public function __construct($host, $dbname, $user, $pass, $new_link = false, $client_flags = 0)
	{
		$this->link = mysql_connect($host, $user, $pass, $new_link, $client_flags);
		if (!$this->link)
			throw new Exception ('Could not connect to db. MySQL error was: '. mysql_error());
		$isDb = mysql_select_db($dbname,$this->link);
		if (!$isDb)
			throw new Exception ('Could not select db. MySQL error was: '. mysql_error());
	}

	private function enterSemaphore()
	{
		$this->sem = sem_get(self::SEMKEY,1);
		sem_acquire($this->sem);
	}

	private function exitSemaphore()
	{
		sem_release($this->sem);
	}


	public function query($sql)
	{
		$this->enterSemaphore();

		$this->result = mysql_unbuffered_query($sql, $this->link);
		if (!$this->result)
			throw new Exception ('Could not query: {' . $sql . '}. MySQL error was: '. mysql_error());
		if ($this->result === true)
		{
			// INSERT, UPDATE, etc..., no result set
			$ret = true;
		}
		else
		{
			// SELECT etc..., we have a result set
			$retArray = array();
			while ($row = mysql_fetch_assoc($this->result))
				$retArray[] = $row;
			mysql_free_result($this->result);
			$ret = $retArray;
		}

		$this->exitSemaphore();

		return $ret;
	}

	public function beginTransaction()
	{
		$this->query('SET AUTOCOMMIT = 0');
		$this->query('SET NAMES utf8');
		$this->query('START TRANSACTION');
	}

	public function rollback()
	{
		$this->query('ROLLBACK');
	}

	public function commit()
	{
		$this->query('COMMIT');
	}
}

The Forking Process

$pid = 'initial';
$maxProcs = $argv[1];
if (!$maxProcs)
{
	 $maxProcs = 3;
}
$runningProcs = array(); // will be $runningProcs[pid] = status;
define('PRIORITY_SUCCESS','-20');
define('PRIORITY_FAILURE','-19');

try
{
	$dbh = new DBHandler(DBHOST,DBNAME,DBUSER,DBPASS);

	$dbh->beginTransaction();

		// fork all needed children
		$currentProcs = 0;
		while ( ($pid) && ($currentProcs < $maxProcs))
		{
			$pid = pcntl_fork();
			$currentProcs++;
			$runningProcs[$pid] = 0;
		}

		if ($pid==-1)
		{
			throw new Exception ("fork failed");
		}
		elseif ($pid)
		{
			// parent
			echo "+++ in parent +++n";
			echo "+++ children are: " . implode(",",array_keys($runningProcs)) . "n";

			// wait for children
			// NOTE -- here we do it with priority signaling
			// @TBD -- posix signaling or IPC signaling.
			while (in_array(0,$runningProcs))
			{
				if (in_array(PRIORITY_FAILURE,$runningProcs))
				{
					echo "+++ some child failed, finish waiting for children +++n";
					break;
				}
				foreach ($runningProcs as $child_pid => $status)
				{
					$runningProcs[$child_pid] = pcntl_getpriority($child_pid);
					echo "+++ children status: $child_pid, $status +++n";
				}
				echo "n";
				sleep(1);
			}

			echo "+++ checking if should commit or rollback +++n";
			if (in_array(PRIORITY_FAILURE,$runningProcs) || in_array(0,$runningProcs))
			{
				echo "+++ some child had problem! rollback! +++n";
				$dbh->rollback();
			}
			else
			{
				echo "+++ all my sons successful! committing! +++n";
				$dbh->commit();
			}

			// signal all children to exit
			foreach ($runningProcs as $child_pid => $status)
			{
				echo "+++ killing child $child_pid +++n";
				posix_kill($child_pid,SIGTERM);
			}
		}
		else
		{
			// child
			$mypid = getmypid();
			echo "--- in child $mypid ---n";
			//sleep(1);
			echo "--- child $mypid current priority is " . pcntl_getpriority() . " ---n";

			// NOTE -- following queries do not work, for example only
			$dbh->query("select ...");

			echo "--- child $mypid finished, setting priority to success and halting ---n";
			pcntl_setpriority(PRIORITY_SUCCESS);
			while (true)
			{
				echo "--- child $mypid waiting to be killed ---n";
				sleep(1);
			}
		}

}
catch (Exception $e)
{
	// output error
	print "Error!: " . $e->getMessage() . "n";

	// if parent -- rollback, signal children to exit
	// if child  -- make priority failure to signal
	if ($pid)
	{
		// rollback
		$dbh->rollBack();
		foreach ($runningProcs as $child_pid => $status)
			posix_kill($child_pid,SIGTERM);
	}
	else
	{
		pcntl_setpriority(PRIORITY_FAILURE);
		$mypid = getmypid();
		while (true)
		{
			echo "--- child $mypid waiting to be killed ---n";
			sleep(1);
		}
	}

}

Well, all of this sounds well, and also worked well on a development environment. But it should be taken out of the lab and tested on a production environment. Once I give it a shot, I will update with benchmarks.

Zend Framework Database Admin

If you’re looking for a simple tool that uses Zend Framework’s robust database classes (such as Zend_Db and Zend_Db_Table), you can check out zdbform. It’s a short yet effective library that let’s you perform simple administration tasks on your database, with minimal coding.

It’s not a full blown phpMyAdmin, but it’s a simple way to view, edit and add your tables rows in a web interface. Also, don’t expect it to scale, because I am sure this library was written to serve some quick table administration needs, and that it is not ready to handle large datasets. But, it is very convenient if you have a small database to administer.

I implemented it with the Zend MVC components, and following is a brief overview.

In the front controller, or front plugin, or any class that your controller subclasses:

$this->db = Zend_Db::factory('Pdo_Mysql', array(
				'host'     => DB_HOST,
				'username' => DB_USER,
				'password' => DB_PASS,
				'dbname'   => DB_NAME
		));
Zend_Db_Table_Abstract::setDefaultAdapter($this->db);

Then set your controller and view scripts as necessary. Let’s say you have two tables to admin, “clients” and “history”. First make sure they are declared as subclasses of Zend_Db_Table:

require_once "Zend/Db/Table/Abstract.php";

class Clients extends Zend_Db_Table_Abstract
{
    protected $_name = 'clients';
}

class History extends Zend_Db_Table_Abstract
{
    protected $_name = 'history';
}

Your controller would look like:

require_once "Zend/Controller/Action.php";

class AdminController extends Zend_Controller_Action
{

	public function init()
	{
		require_once 'zdbform/zdbform.class.php';
		require_once 'zdbform/zdbform_widgets.class.php';
		require_once 'zdbform/zdbform_validations.php';

		parent::init();

		$this->view->headLink()->appendStylesheet('/zdbform/zdbform.css');
		$this->_helper->viewRenderer('index');
	}

	public function indexAction()
	{
	}

	public function clientsAction()
	{
		$this->view->dbform = new Zdbform('Clients');
		$this->view->dbform->setWidget('description', 'textarea');
		$this->view->dbform->processForms();
	}

	public function historyAction()
	{
		$this->view->dbform = new Zdbform('History');
		$this->view->dbform->processForms();
	}

}

And the single view script you need is admin/index.phtml:

<?php
echo $this->headLink();

include_once "Zend/Filter/Word/CamelCaseToDash.php";
include_once "Zend/Filter/Word/CamelCaseToUnderscore.php";
$cctd = new Zend_Filter_Word_CamelCaseToDash();
$cctu = new Zend_Filter_Word_CamelCaseToUnderscore();
$classes = get_declared_classes();

foreach ($classes as $class)
{
	if (is_subclass_of($class,'Zend_Db_Table_Abstract'))
	{
	?>
		<a href="/admin/<?= strtolower($cctd->filter($class)) ?>"><?= strtolower($cctu->filter($class)) ?></a>&nbsp;&nbsp;
	<?php
	}
}

if ($this->dbform)
{
	?>
	<h1>Table: <?= $this->dbform->tableName ?></h1>
	<?php
	$this->dbform->showForms();
	$this->dbform->showTable();
}
?>

There were also a couple things that needed changing in the zdbform class itself:

  • Replace all PHP_SELF with REQUEST_URL. On the mvc case, PHP_SELF is empty or index.php, and we don’t want all the forms posted there, we want them to go back to /admin/clients or /admin/history
  • After this line
    $this->pk = $tableInfo['primary'];

    I had to add this:

    if (is_array($this->pk))
    	$this->pk = $this->pk[1];
  • zdbform->orderBy is treated as a single column, of you want multiple column sorting you have to hack a bit with getAllRows().

That’s it, point your browser to /admin and you’re good to go. In a very short time and with a little bit of code, you can get something similar to a stripped down version of phpMyAdmin, using the power of Zend Framework.

Flex on Ubuntu: The Complete How To Guide

ubuntu_flex

I usually live with the axiom that whatever you can find in the realm of Windows and proprietary software, you can easily find in the realm of Linux (any flavor) and open source. While this is indeed usually the case, when it comes to a Flex IDE for Ubuntu, there’s a real gap. Adobe has their Flash IDE for willing and paying Windows users, and I am happy to say that I was one of these happy customers a while ago. But since then, Ubuntu has taken over my life, and when I set out to make a small Flex app a couple of days ago, I came across some hurdles. Not impossible to overcome, but not trivial as well.

The Options

There is no one complete solution for developing Flex on Linux. Many folks are looking for one, but there is still none to be found. There are many open source tools that cover vast areas of the ActionScript and SWF world, most of the listed on the wonderful osflash.org. Some of them are just right for you if you have a very specific task (like converting between formats, just compiling a bit of code, etc.), but none provide a complete IDE that lets you both drop in WYSIWYG elements and manually code some stuff, while easily maintaining complete control of what libraries are used.

The Choice

The choice then is to combine as few tools as possible. I have succeeded to get along with 2 tools: Flex Builder Linux Alpha, and Open Dialect.

Flex Builder Linux Alpha is an Adobe free product, which is a Flex build environment as a plugin for Eclipse. Don’t worry about the Alpha part, it seems like a very stable product, and besides eating up some memory, I had no problems with it. It is actually an exact replica of the Flex Builder for Windows, without the features of Design View, some wizards and the profiler.

Open Dialect is the most comprehensive attempt i’ve seen at creating a graphic WYSYWIG IDE for developing Flash. It has some basic characteristics of such an IDE, like a timeline with frames and key frames, and a graphic interface for creating shapes and editing their properties.

The Method

The development cycle is quite simple once you have these two tools running. Use Open Dialect for whatever graphic or animation you need, then grab the code from the “Document Script” tab, paste it to the Flex Builder in Eclipse, and start tweaking whatever is needed, add the MXML code etc. Open Dialect has great potential if it were to enable manual script editing, but currently it doesn’t.

Getting Things To Work

Requirements and Installation of Flex Builder Linux Alpha are covered on Adobe’s release notes. In short, you got to have Eclipse 3.3.x, Sun JRE 1.5.x and Firefox, and just follow the installation instructions there. Be sure to set Eclipse’s browser to Firefox, as mentioned in the release notes, and there are also a couple of guides to walk you through. Oh, and it’s very important to follow these instructions in order for your existing Eclipse plugins to survive the install. I installed the builder and lost my subclipse and collabnet merge plugin, and had to reinstall them.

Open Dialect uses the .NET framework, so you need Mono to run it. According to the installation instruction, you can either download pre-compiled binaries of Open Dialect, or download the source and compile, but then you need MonoDevelop as well. In my case, using apt-get install mono was enough, and Open Dialect ran like a charm.

Tweaking and Real Life Example

Let’s go through an example of how to make a rounded rectangle that gets its color through a FlashVar.

In Open Dialect:

  1. Fire up Open Dialect.
  2. Choose Rounded Rectangle from the Shapes list on the Items pane.
  3. Draw a rounded rectangle on the canvas.
  4. Set its X and Y to 0.
  5. Go to the Document Script tab, select all the code and copy.screenshot-open-dialect

In Eclipse:

  1. Fire up Eclipse, create a new Flex Project, name it “Test”, and go through the creation wizard (next, next, finish).
  2. You are now editing Test.mxml which is a skeleton Flex app file. Paste everything you copied from Open Dialect into Test.mxml, instead of its current content.
  3. Save.
  4. Hey, we’ve got errors! That’s right, BListBox type is not defined. It’s because the script uses the “ActionScriptComponents” library that comes with Open Dialect, in which BListBox is declared, but we haven’t imported it yet, let’s do it.
  5. Copy the directory /path/to/opendialect/ActionScriptComponents to /path/to/workspace/Test/src/
  6. Run again.
  7. Viola! Rounded rectangle showing up!

Now that we’ve covered the basics, let’s see how to pass FlashVars to our app. In order to do so, we need to understand what the Flex Builder environment does in build time — besides compiling the SWF, it also takes the file called index.template.html found in the html-template directory, and compiles it to a file called Test.html in the bin-debug directory, then fires this file in Firefox. So to pass FlashVars and process them in the script:

  1. Open index.template.html
  2. Scroll down to the javascript part under “hasRequestedVersion”, this is the part that runs the swf on our page (assuming we have javascript enabled and the correct version of the flash player).
  3. Under
    "src", "${swf}",

    add

    "FlashVars", "color=0x000000",
  4. Run once to see everything is working, and that we did not screw up the html template.
  5. Add the following variable declaration where all other variable declarations are:
    private var rectColor:Number;
  6. In init(), add the following line at the top of the function:
    rectColor = Application.application.parameters.color;
  7. In the next line, where the shape properties are defined, change the last 2 colors (which in my case were 0x000000,0xFF0000) to rectColor,rectColor:
    RRect1A.Properties[0] = new ShapeProperties(0,0,120,172,"BRoundedRectangle",1,rectColor,rectColor);
  8. Run again, see the rectangle is now black!

Conclusion

The field of Flex development on Linux is of course bound to change as time goes by, but for now it seems like it is still in its early unstable days. This was a brief demonstration of how to harness the IDE power of Open Dialect, and the development and build power of Adobe’s Flex Builder Linux Alpha, to create a working environment for developing Flex apps on Linux. It is by no means the simplest solution — my guess is that running Flex Builder on a Windows VMware could be easier, albeit costly — but this solution that I presented is of course free and conforms to the open source spirit. And, most importantly, can get you developing flex apps quickly and easily.

Have fun flexing!

A Couple of Sidenotes

  • You don’t have to use the ActionScriptComponents library that comes with Open Dialect if you don’t intend to use tweening or frame events. You can use the regular mx.controls classes for just plain drawing.
  • Flex applications tend to be big because the whole framework is included in the build. You can use RSL loading to cache the framework on the client side, or you can make a plain ActionScript Project instead of a Flex Project, still use mxml and just import the needed flash.* libraries. Both of these subjects are out of the scope of this post.