Crudder in action

It took a while.. a long while actually.. but I finally made a new release of Crudder. This new release 0.50 has a lot of new features and bugfixes, to name a few;

• New Field-type: HTML-editor
• New Field-type: File-upload
• New Field-type: Enum
• Export to CSV
• Adding records from the many-2-many-editor
• Bugfixes; language-support works better, datefield has been fixed, the many-2-many-editor should work better

Check out the DEMO or read more

Op http://wps.i-v-o.nl/ staat sinds vandaag een vernieuwde demo-website van WPS.  Met deze demo-website kan je lezen over wat WPS is, wat het kan, hoe het werkt en je kan spelen met het systeem.

Wat is WPS

WPS staat voor Web Publishing System. Geen spannende afkorting, maar het dekt de lading wel goed. WPS is namelijk geen klassiek Content Management Systeem (CMS) maar veel meer. Een klassiek CMS zorgt er eigenlijk alleen maar voor dat je de inhoud van je website kunt bewerken, dat doe je dan eigenlijk altijd op een aparte website (ook wel backend genoemd). Zo’n CMS is harstikke mooi, maar er moet ook een ‘voorkant’, je website, geprogrammeerd worden, en dat kan best een tijdrovende klus worden. WPS is een publicatiesysteem en zorgt dus niet alleen voor de CMS functie, maar ook voor het publiceren; de voorkant van je website.

Publicatie-systeem

Er zijn gerust wel andere publicatie-systemen, bijvoorbeeld Joomla of Wordpress, maar toch heeft I-V-O de keuze gemaakt om een heel nieuw systeem te ontwikkelen. Waarom? Flexibiliteit.
De meeste andere systemen zijn nogal rigide in functionaliteit of opmaak, en dan nog maar niet te spreken over veiligheid. WPS is extreem flexibel door het gebruik van XML. Alle gegevens die in WPS rondgaan worden omgezet in XML en komen in 1 groot document samen; het WPSDoc. XML is een speciale manier van gegevens opmaak die zich uitermate goed leent tot ‘translaties’. Stel je voor dat je een tekst hebt ingevoerd en je wilt daar een deel van dikgedrukt maken; normaliter wordt dat in html aangegeven met de b-tag. In WPS wordt dit de bold-tag, niet “b”, en bij het publiceren van de pagina kan de programmeur dit transleren/vertalen naar de standaard “b”-tag, maar ook naar iets heel anders, bijvoorbeeld “strong” of “h1″.
De taal die gebruikt wordt voor dit vertalen heet XSL-t, een soort HTML, maar dan met logica; je kan als programmeur o.a if-then-else, loops, en variabelen gebruiken. Hierdoor wordt de XML een stuk slimmer, en je site dus ook!

Bewerken en beheren

Wat WPS nog meer bijzonder maakt is de manier waarop je de teksten in je website kunt beheren; dit doe je namelijk gewoon in je site en niet op een aparte website of backend. Hierdoor zie je veel beter wat en waar je wat doet. Dit is veel gebruiksvriendelijker dan de standaard CMS’en en geeft veel beter weer wat de cohesie is tussen website en content.

Plugins

Een standaard WPS-site kan al heel veel;

• Pagina’s aanmaken, teksten maken en aanpassen
• Meta-tags (voor zoekmachines) beheren
• Linkjes in je teksten controleren of ze nog goed zijn
• Bestanden (pdf’s, Word-documenten, plaatjes) uploaden
• Een site in meerdere talen publiceren (en dat doet WPS heel slim want als je een pagina in het nederlands naar bijvoorbeeld het engels hebt vertaald dan weet WPS welke pagina’s bij elkaar horen, als een engels-sprekende op een nederlandse pagina komt kan hij/zij met 1 klik naar de engelse versie zonder de pagina opnieuw te moeten zoeken)
• Rechten instellen per pagina

Verder zijn er heel veel extra functies gemaakt in de vorm van ‘plugins’, een greep uit de collectie;

• Nieuws-plugin: zorgt voor het beheer en publicatie van nieuws-berichten (ook naar RSS!)
• Poll-plugin: maak zelf een poll en zorg dat je bezoekers kunnen stemmen
• Search-plugin : zorgt ervoor dat bezoekers kunnen zoeken in je website
• Shop-plugin: volledige webwinkel, met betalingsmodule voor IDEAL en Mollie (micropayments)
• Enquete-plugin: zeer uitgebreide plugin voor het maken van grote enquetes, compleet met uitnodigings-email-functie, toegangscodes en export-functies.
• Catalogus/Portofolio-plugin: laat je bezoekers zien wat je verkoopt of wat je hebt gemaakt.
• Download-beveiliging voor Audio: als je bijvoorbeeld muziek wilt laten horen vanaf je website via een zg. FlashPlayer dan zorgt WPS ervoor dat bezoekers niet stiekem de bestanden kunnen downloaden.
• Google-Maps-plugin: zorgt ervoor dat je bezoekers op een kaartje kunnen zien waar je bedrijf is gevestigd
• Reactie-plugin: geeft je bezoekers de mogelijkheid om korte reacties op je website te plaatsen (en ze kunnen ook op elkaar reageren)
• PDF-output; maakt automatisch PDF-bestanden van de tekst op een pagina die bezoekers kunnen downloaden
• Blog-Plugin: samenvoeging van de nieuws-plugin en de reactie-plugin
• Gallery-plugin: maakt automatisch een mooie foto-gallery van foto’s
• En in ontwikkeling; Google WAVE-integratie; een mogelijkheid om bezoekers via Google Wave te laten reageren op nieuws-berichten.

Verder maakt WPS gebruik van slimme caching; je site zal altijd rete-snel blijven omdat alle veel gebruikte informatie in het geheugen van de webserver bewaard blijven, zodat ze snel weer voor handen zijn als het nodig is.

Voor meer informatie over WPS, klikkerdeklik!

Installation is quite easy; there’s a perfect howto available on Howtoforge.

There are some pretty good uses for a Singleton though; a registry and the database-connection;

Download example code here

Database-connection

I use ADODB to access databases, don’t ask me why, I just do. ADODB uses a DSN to connect to a database, which looks something like:

mysql://dbUser:dbPass@localhost/someDb

Normally you’d use $connection = &ADONewConnection($DSN) to connect to the database and run your query ($rs = $connection->execute($sqlQuery) ). The problem here; you keep creating new connections anywhere you run a query, you need to pass the value for $DSN into your objects,  you’ve made your class depend on ADODB and it take 2 lines of code   You can bypass these ‘problems’ by using a Singleton to connect to your database and run a query:

$rs = basicDBConn::getConnection()->execute($sqlQuery);

Now, what is wrong here? Did you see $DSN anywhere?
Nope… This statement assumes you’ve passed $DSN already somewhere before, in fact the first time you call getConnection() from this singleton you must pass $DSN as argument in getConnection(). The first time you call getConnection() is the moment the singleton creates an instance of itself, and uses $DSN to connect to the database (&ADONewConnection($DSN)). After this initialization the connection is kept within the Singleton, and $DSN isn’t needed anymore. Take a look at the files, basicDBConn.php in particular to get an idea of how the Singleton creates an instance of itself and the database-connection.

Registry

Another good use for Singletons is a Registry-object; A registry-object is used for configuration; global settings that you might need in your code (path’s, DB-configuration etc.). Since I’m infected by the XML-virus my configuration-files always are XML :

<config>
    <item index="DSN">mysql://dbUser:dbPass@localhost/singletondemo</item>
    <item index="someKey">someValue</item>
</config>

There is a drawback though; these XML-files are not protected by Apache by default, anyone can open them from a browser (if they are somewhere in the public root of the website). There are 2 solutions for this; prefix the file with .hta (in which case Apache will protect them just as .htaccess files are protected) or add the following lines to your Apache-configuration :

<Files ~ "configuration_filename.xml">
   Order allow,deny
   Deny from all
</Files>

The registry-object works the same as the Database-object; it needs a reference to a XML-file the first time you call it, after that it’s fully initialized and doesn’t need that reference anymore.

The first call to a registry object (and get the value for ’some_key’):

$someValue = basicRegistry::getInstance('/path/to/configuration/file.xml')->get("some_key");

Any call after that:

$someValue = basicRegistry::getInstance()->get("some_key");

Why is a singleton a good thing?
Getting configuration-values or a $DSN-string from within objects should be simple, you do not want to pass the path and filename to your configuration file or $DSN to a database every time you create an instance of an object. It will create dependencies and spaghetti in your code. If you make sure the registry and/or database-object are created before your other objects start doing something you do not need to worry about file-names and/or $DSN’s. They have been safely put away in a singleton.

Another very nice side-effect ; you’ve decoupled database-connections and configuration-settings. In my examples I use XML to setup the values in the registry, if you want to use some other method of keeping key-value-pairs, just modify the inner workings of the Registry-singleton, the objects using the registry won’t notice and work just as well. I usually keep session-variables in the registry, if I can’t use PHP’s $_SESSION, I can fall back onto Cookies, my classes won’t notice.

Download example code here

[download code]

This is how these RPC-calls are done:

Since I like XML a lot, I wanted to use XML as transport mechanism for doing requests to the RPC-interface. Because I use XML I was also able to group RPC-calls into one single request. A simple method of authenticating these RPC-calls is something to consider if you’re not running a public service, but that’s beyond the scope of this article

A typical call would look like this:

<command>
	<method name=”remote_method” object=”some_object”>
		<arg key=”some_argument”>value_of_some_argument</arg>
		<arg key=”another_argument”>value_of_another_argument</arg>
	</method>
	<method name=”another_remote_method” object=”another_object”>
		<arg key=”some_argument”>value_of_some_argument</arg>
		<arg key=”another_argument”>value_of_another_argument</arg>
	</method>
</command>

Now, what does this XML actually mean?

First of all; the methods need to be nested into a single node: <command>, the XML would not be valid otherwise. Each method-node has an name-attribute and object-attribute, the object-attribute defines the remote PHP-object which contains the method you’d like to call, the method is defined in the name-attribute.

Each method-node contains one or more arg-nodes, these nodes are passed to the method as a hashed-array and used as arguments for the method. Attribute “key” will be the key or name in the arguments-array, the node-value will become the value.

The example above will eventually call 2 methods,

1. the first method, “remote_method” is part of object “some_object”, and receives 2 arguments;
   • “some_argument” with value “value_of_some_argument”
   • “another_argument” with value “value_of_another_argument”
2. the second method, “another_remote_method” is part of object “another_object”, and also receives 2 arguments;
   • “some_argument” with value “value_of_some_argument”
   • “another_argument” with value “value_of_another_argument”

Now, let get some coding done, let’s focus on the RPC-server first, download this code-snippet and unpack it in a fresh webroot (it contains all the files described in this article, also the RPC-client).

First take a look at /include/classes:

• tools.php : contains static “helper” methods, I hate procedural spaghetti-code and I always put methods which are not part of a larger object-model into several helper-classes. In this particular example a single helper-class is enough. All methods in this class are static and can be called without first creating an object.
• abstract_object.php : this is important and can be confusing if you’re not used to an object oriented approach. This class contains methods which are inherited into child-classes, but you cannot create an object-instance of this class, because it’s abstract. Why is it abstract? It’s arguable to just let it be a normal class in this particular example, however; using abstract classes in larger projects really keeps your code clean and separates the implementation of your object-design from the structure you’re using.
• some_object.php : inherited from abstract_class.php; example implementation.
• another_object.php : inherited from abstract_class.php; another example implementation.

The actual object available for the RPC-interface are some_object and another_object. Now let’s take a look at rpc.php, which is in the root. The only really interesting part is from line 56

	// gather 'method' nodes and execute them;
	$methods = $dom->getElementsByTagName('method');

	for ($i=0;$i<$methods->length;$i++){
		$methodNode = $methods->item($i);
		$method = $methodNode->getAttribute('name');
		$obj = $methodNode->getAttribute('object');
		$obj = new $obj();
		$arguments = tools::getArgumentsAsArray($methodNode);
		$obj->$method($arguments);
		$messages .= $obj->getMessages();
	}

	echo tools::wrapMessages($messages);

So, what’s really happening here?

The first statement; $methods = $dom->getElementsByTagName(‘method’); will retrieve all method-nodes from the request. PHP-DOM (or any other DOM-implementation for that matter) will always return the nodes in a nodeset, even if it’s only one method-node, but not if there aren’t any nodes found.
This nodeset is used to loop upon, the classic for-next statement on the next line.

In each loop the method-node is examined; the attribute-values for object and method are determined ($method = $methodNode->getAttribute(‘name’) and $obj = $methodNode->getAttribute(‘object’)) and using a static function from the tools-class the arguments are converted into a PHP-array ($arguments = tools::getArgumentsAsArray($methodNode) ).

Now we’re ready gathering all the values needed to really create the object ( $obj = new $obj() ) and execute the requested method ( $obj->$method($arguments) ).

The method should not return anything, instead it’s return-values are stored into a protected variable (defined in abstract_object). To get these values, use $obj->getMessages(). This method doesn’t seem to be available in the class of $obj, it’s defined in abstract_object.

After the loop has finished the $message-string should be wrapped into a node to make sure the XML is valid, this is done with the tools’ class wrapMessages-method.

Ok, the RPC-interface is ready, let’s test it! Enter the following URL into a browser (assuming you’re running PHP on localhost, in a folder called rpc:

http://localhost/rpc.php?req=<command><method name="remote_method" object="some_object"> <arg key="some_argument">value_of_some_argument</arg><arg key="another_argument">value_of_another_argument</arg> </method><method name="another_remote_method" object="another_object"> <arg key="some_argument">value_of_some_argument</arg><arg key="another_argument">value_of_another_argument</arg> </method></command>

You’ve just called a 2 functions from your RPC-interface! The output should look like this:

<servermessages>
    <message object="some_object" method="remote_method" code="200">remote_method was executed
        succesfully with arguments : value_of_some_argument, value_of_another_argument </message>
    <message object="another_object" method="another_remote_method" code="200">another_remote_method
        was executed succesfully with arguments : value_of_some_argument, value_of_another_argument
    </message>
</servermessages>

Now let’s move on to the RPC-client: take a look at rpcClient.php in the root. It’s quite large, and I will not go into all the details (the comments in the file should be enough). Most of the functions in rpcClient take care of constructing the XML needed to execute the RPC-call, absolutely no rocket-science.  However; I do want to highlight the actual “executing” of the RPC-call, take a look at the private function sendToHost:

 private function sendToHost($data) {
	$postdata = http_build_query(
	    array(
	        'req' => $data
	    )
	);
	$opts = array('http' =>
	    array(
	        'method'  => 'POST',
	        'header'  => 'Content-type: application/x-www-form-urlencoded',
	        'content' => $postdata
	    )
	);
	$context  = stream_context_create($opts);
	$this->result = file_get_contents($this->rpcURI, false, $context);
}

This method uses file_get_contents, http_build_query and stream_context_create to POST the constructed XML to the RPC-server and stores the reply in a local variable ($result). I’ve tried using fopen like this:

function sendToHost($host,$path,$data) {
	$fp  = @fsockopen($host,80);
	$buf = '';
	if ($fp) {
		@fputs($fp, "POST $path HTTP/1.0\n");
		@fputs($fp, "Host: $host\n");
		@fputs($fp, "Content-type: application/x-www-form-urlencoded\n");
		@fputs($fp, "Content-length: " . strlen($data) . "\n");
		@fputs($fp, "Connection: close\n\n");
		@fputs($fp, $data);
		while (!feof($fp)) {
		  $buf .= fgets($fp,128);
		}
		fclose($fp);
	}
	return $buf;
}

This method failed. I kept getting weird responses from my RPC-server (command not understood etc.). And honestly;  I like the clean call using file_get_contents, http_build_query and stream_context_create much better.

Ok; we have a nice object-structure, a RPC-server and a RPC-Client. What’s left is a real world implementation of the client. Take a look at example.php:

 include('rpcClient.php');

 $hsClient = new rpcClient();
 $hsClient->createCall('some_object', 'remote_method', array('some_argument'=>'some_value', 'another_argument'=>'another_value'));
 $hsClient->createCall('another_object', 'another_remote_method', array('some_argument'=>'some_value', 'another_argument'=>'another_value'));
 $hsClient->execute();
 $str = $hsClient->getXMLResponse();
 echo "<pre>";
 echo $str;
 echo "</pre>"; 

If you’d leave out the debugging-statement at the end you end up using 4 lines of code to create, execute and retrieve a RPC-call (ok; 5 lines if you consider I’ve made 2 RPC calls). First you create an instance of the RPC-client, secondly you create a call using createCall and last but not least; you execute the call and intepret the response. If you’d enter the URL to example.php into your browser you’ll get:

remote_method was executed succesfully with arguments : some_value, another_value
another_remote_method was executed succesfully with arguments : some_value, another_value

That’s it! You have a RPC-server, RPC-client and a working implementation! You should take care of authenticating your RPC-calls, and design a nice Object-structure to implement ‘behind’  the RPC-server, this all depends on the kind of logic you need. I’ve used this method of RPC-ing to allow 3rd-parties to update certain values in a shop-database, and with a little effort you could make your RPC-server handle AJAX-requests and return JSON to clients. But that something for the next article.

[download code]

I had a problem running executables (like identify.exe and convert.exe from ImageMagick) from php under windows using the exec() funciton. The solution provided by others was kind off stupid… Give full permisions on cmd.exe to IURS… Like you wanna do that…..
I wrote a exec() replacement function using the bypass_shell option proc_open has. It works great for me

Check out this article on bokko.nl to bypass using exec() on Windows

The first hurdle most developers have to take when introduced to AJAX is S.O.P.
S.O.P stands for Same Origin Policy and basically means that you can only do AJAX-requests to the (sub)-domain the client is currently on. First thing I thought when I read about AJAX couple of years ago; ‘wow! cool! I can strip content from other sites and incorporate them in my websites’ .. and immediately I started hammering away on a RSS-reader which was supposed to grab news from a RSS-feed directly. My efforts failed. The RSS-feed was on another domain and S.O.P kicked in spoiling all the fun.

There are ways to get around this, you could create a server-side script which retrieves content from another domain and proxy it to your AJAX-request. However; you will need a server that has either PHP, ASP(.net), CGI, Rails, or at least something more intelligent than a plain HTML-only environment. This is OK for most people, and proxying content from other domains to AJAX-request is used a lot. But what if you’d like to create something, like a widget, which needs to grab content from any domain, and which people can use without having a PHP/ASP/CGI or Ruby-server? You wouldn’t be able to get around S.O.P, right?
How can you bypass S.O.P, while still maintain a certain level of security?

TimeLiner
I ran into this problem when I created TimeLiner. TimeLiner retrieves and displays messages linked to a flashmovie or mp3-file. These messages are stored on one of my servers and TimeLiner could be running anywhere. Retrieving those messages should be done by AJAX-like requests but I didn’t want users to install all kinds of server-side scripts on their server for proxying the messages from my server to overcome SOP. I wanted TimeLiner to be able to run from HTML-only-websites, or as a widget from social networking websites. But with SOP this wouldn’t be possible.
With this in mind I started experimenting. I started off by creating script-tags and insert those tags into the DOM on the fly and see if variables in the referenced js file were actually available. Expecting limitations due to security measures within browsers the results actually suprised me; the variables were globally available! I started on designing a proof of concept and tried to take this idea to the next level. At the end of the day I got a working prototype and called it AJP; the Authenticated JSON Proxy.

RPC, AJAX, Objects
First a bit of history. The past few years I’ve been busy writing WPS; a publishing system much like Joomla (only much better;). It’s written in PHP5, object orientated, follows a observer-pattern(plugins) and runs entirely on XML/XSL-translations. If you’d like a shop-plugin on your WPS-driven-website you only need to include the shop-object and you’re ready to sell your stuff.

Because all data within WPS is actually XML it really doesn’t matter what you want to do with with it; you could translate it to HTML by using XSL-t and write it to the browser or you can pickup this data by using AJAX-calls. KISS at work

WPS uses its own RPC-interface for AJAX-requests. You can create a request by defining which WPS-object you’d like to call, which method within the choosen object must be executed and the arguments that need to be passed to the method. This WPSRPC-interface consists of two parts; the server-side pickup and dispatch, and a javascript-class that simplifies the actual RPC-calls.
A typical WPSRPC-request would look like this:

Javascript:

var wpsRPC = new wps.rpc;
wpsRPC.createCall('shop', this.setSaved.bind(this));
wpsRPC.call('storeItemToBasket', {id:itemID, uid:someUID});

Line 1: create a RPC-object
Line 2: define the PHP-object you’d like to call (in this example; the shop-object) and define the callback-method (in this example; the method setSaved from the same object-instance, hence the .bind(this)-statement )
Line 3 : define the method which needs to executed remotely, define the arguments that need to be passed and execute the request.

On the server-side (PHP) the request is handled by the WPS-controller. WPS uses _autoload to include the requested class, instantiates the requested object (in this case the Shop-object) executes the requested method (storeItemToBasket) and passes the arguments to that method (id & uid). The output of the requested method should be XML or JSON which will be returned to the javascript-callback (setSaved) for further handling by the browser.
This proofed to be a powerful method of remotely executing stuff; it’s clean, quick and flexible and I’ve used this structure for 5 years without any problems whatsoever.

Now; why this elaboration about WPS?
AJP uses exactly the same structure, a Javascript object that contains methods for doing RPC-calls, and a server-side RPC-interface to handle those requests, but there is a slight difference; the calls can be done across domains!

Back to AJP: how does it work?
I’ll use the example of the WPS-shop-object and explain each step; let’s assume a user is visiting your webshop and clicks “add to basket”, what will happen?

By clicking an event is triggered, this event will create an AJP-object:

var ajpObject = new ajp;
ajpObject.createCall('shop', this.setSaved.bind(this));
ajpObject.call('storeItemToBasket', {id:itemID, uid:someUID});

Internally the AJP-object creates a random ID and prepends “jsonProxy”, let’s say this variable is called jsonProxy12345. This variable is used to identify the request. Next step is to create a script-tag and insert it into the head of your HTML-document. The src-attribute of the script-tag points to the URI where a RPC-interface is waiting for requests. This is not a .js file, but a .php-file; which allows you to use http-requests ($_REQUEST in PHP) to pass along variables.
A script-tag from this example would look like this;

<script src="http://www.yourdomain.com/
rpc.php?jsonProxy=12345&object=shop&method=storeItemToBasket&id=1&uid=12" >
</script>

The server-side pretty much works the same as the WPS-example above; the $_REQUEST is mapped to an object, a method and the arguments and is executed. There’s a small difference; the output can only be JSON, and put into a variable called jsonProxy12345.

Ok, let’s assume the RPC-call executed OK and the ID of the inserted row should be returned to javascript. How does AJP know there’s output, and how does AJP actually use the output?
In ‘real’ AJAX you can use an event to notify Javascript the RPC-call returned something, but AJP is not ‘real’ AJAX… This is where the jsonProxy12345 comes in: the moment the script-tag is inserted into the DOM a timer is started which keeps checking for the variable jsonProxy12345. If the variable exists the value is picked up and passed to the callback-function, otherwise the timer keeps running until it encounters the variable (there’s a timeout to catch errors). This effectively mimics the events used in real AJAX, which we want.

Now for some security
If you use this kind of RPC you really do not want to expose your PHP-objects to the world. If you created a populair service your server would run out of resources if anyone can do RPC-calls to it. To overcome this you need to authenticate the requests, but how?
It’s really quite simple; before actually doing a request, ask the server if you may do a request!
I’ve done this before with traditional forms; before posting a form an AJAX-call retrieves a unique ID from the server (a mysql GUID), this GUID is appended to the form in a hidden field and the form is posted. The server checks if the GUID is valid, and if that’s the case the form is handled.
But how does the server know the GUID is valid? The moment the server creates the GUID it’s stored into a database, along with the originating IP-address, the browser-string and the time it’s requested. If the form is actually posted the server checks if the GUID is available, if it’s from the same IP, with the same browserstring and within 5 seconds after it’s been requested. Only if all conditions are met the form is handled and the GUID is removed from the database.

This principle can also be used to authenticate AJP-requests, and AJP can use itself to request a GUID by being clever in it’s design and inheritance-scheme.
I realize it’s quite easy to mimic the requesting of GUID’s, but it’s a barrier, and AJP does something else to tighten up security even more; whenever AJP receives a result the script-tag will be removed immediately. It removes any evidence of requests, making it harder to find out what happened.

AJP is certainly not rocket-science, I’ve seen a lot of scripts using this kind of technique. I do believe the structure of AJP itself works great and allows developers to extend upon. As a bonus; this also works great on very old browsers; i’ve tested it on MSIE 4, and AJP works fine!

I’ve compiled a small (LGPL’ed) download which contains all the files for using AJP, it’s not a full implementation, but enough to get you going.

HDX-1000 Networked Media Tank

I have a HDX-1000; a Networked Media Tank.  A kind of dedicated audio/video-player with harddisk and capable of playing about anything you throw at it (matroska, DVD-iso’s, DIVX, XVid’s, mpg’s) in Full HD (1080P).  Absolutely a toy to recommend if you’re into HD-video. If you’re looking for one I can strongly recommend buying one at DivxPlayer.nl , their service is absolutely superb.

Anyway; the HDX-1000 is also capable of displaying (simple) websites, aka. webservices (in fact; the GUI is HTML-based), and it’s very good at playing Shoutcast-streams. But; there were no webservices for browsing through the Shoutcast-directory.  Ok; there’s a top10 shoutcast station-list, but with over 50000 stations this does not even scratch the surface. So.. i gave up a sundaymorning and started hammering away on a Shoutcast-directory Browser for NMT’s.

Fortunately Shoutcast has a very nice API to request all kinds of information, and it returns XML, which I became quite an expert on the last 10 years. Because I didn’t want to put a heavy load on Shoutcast’s servers by requesting data on each visit, and to keep my directory-browser at blazing speed i’ve written a sync-script which pulls all categories and stations from the Shoutcast-API and dump them into a database.  The script runs every 2 days (if I don’t forget it:)) and refreshes the stationlist in my database.

If you have a NMT-player; check out http://hdx.i-v-o.nl (the stations will only play when started from a NMT, not from your browser)

• Updates without any affected_rows will not trigger an error anymore, SQL-errors are still caught though
• MultiLinked fields _should_ work better on MSIE, but you never know; IE sucks at being any kind of predictable (and all other things a browser should do).
• Language-settings of the interface-texts actually work
• new feat: thouroughCheck; checks whether a table has non-nullable fields and will make those fields mandatory
• new feat: exporting data to CSV (currently the output with default-settings is suitable for openOffice)
• new feat: Advanced filtering / Querybuilder

The idea was to send 4000 cd’s to 15 cities around the world, and ask people to copy the cd and give the original to someone else, or leave it at a public spot in town. And while you’re at it, leave a message with picture on a website. Inspired by J.A.L.D (just another link dump) we came to a design that would fit all images that would be uploaded, whether it would be just 10 pictures, or 1000, it had to fit into one page.

So I started some tryouts:

first runner up: using Javascript to resize all images. The major drawback using this technique; bandwidth. If there would be 1000 pictures, each having an original resolution of 1024×768 (which is quite small) and about 300 kb each, the client would have to download 1000×300kb of pictures, a whopping 292 MB worth of data, which will eventually be displayed as tiny thumbnails. A huge waste of bandwidth! Not only bandwidth would be a problem, what about the javascript itself? I’d probably be using getElementsByTagName, or getElementsByClassName, loop through 1000 images, resizing each images. IMHO. a pretty expensive operation, even for a modern computer/browser.

So.. no javascript for resizing, what other options are there?

Second runner up: using GD or Imagick to resize the images on the  fly. Verrrrry expensive, but with the (visually) best results. Each uploaded image would be dynamically resized to fit the grid of images. I’d use an Ajax call after the page is loaded, pass the size of the viewport to a PHP-script which is able to calculate the max-width & height of each image, resizing a copy of the original and send something (xml/json/plain html) back to the client. Well.. I own a server which has a lot of  processing power, but 10 concurrent sessions of resizing-images-script would even bring this Xeon Quad-core to it’s knees (even if i’d cache the resized images for later use). Not an option!

Third runner up:
Use GD/Imagick to resize the original into several resized copies. After a client uploads his/hers image the server would make a certain amount of resized copies. When a client opens the page, the size of the viewport is determined and sent back to the server using an Ajax-call. The server determines the total area and determines how may pixels one image may have, loops through the available sizes, and returns the size that fits the amount of pixels the image may have. I actually build this solution in Ruby on Rails. I mixed PHP and Ruby before (well.. mixed.. just called RoR-pages from a PHP-based webpage using Ajax and some smart proxy-ing), and liked the idea of spreading performance among different
techniques. If the site were to be a HUGE success the actual calculation would become the bottleneck; since RoR is extremely scalable I could setup another (RoR) server to handle the load.
The Proof Of Concept did hold well, even under a bit of stress, however; RoR itself uses quite a lot or resources and I’m not an expert on RoR (yet). I did not want to leave a potentially dangerous script (remember; people are uploading stuff) on a
production-server while i’m not really able to assess the potential security issues.  So; I rewrote the thing in PHP.

Another problem I came across was the aspect-ratio of the uploaded images. The design would be a grid-like solution, all images would have to fill up a certain amount of pixels, leaving no space between them. But; what if an image has a  spect-ratio of 16×10, and the image next to it, say 10×16. One of them has to be resized to match the other height, or width, leaving no space above. And how would I calculate the optimal size of each image? Not knowing the length of each side? So I dug up some old math-books to check out whether I forgot some rule or calculation.. No results.
I did write some code to roughly guess the average size of one image, but the results were quite strange and unpredictive. I came to the conclusion that the only way to get those images aligned and keep a reasonable quality was to crop them to have a common aspect-ratio. Fortunately Imagick has a build-in method to accomplish this: cropThumbnailImage.

OK, So, how did I solve this ?

Pretty easy! I wrote one function which resized one uploaded image into 15 different sizes, store each of those images into a folder named after the size of the image and wrote a function to calculate the optimal size of those images (based on x and y from the ajax-call) to output to the client. I tried to resize the images again using javascript, to get them to fit exactly into the page, but this was too expensive and images didn’t look as crisp as they should be. PHP-Function for resizing an image:

private function getImages($country = '', $city=''){
	if ($country){
		$sqlExtra = " AND country LIKE '" .  mysql_escape_string($country) . "' ";
	}
	if ($city){
		$sqlExtra .= " AND city LIKE '" . mysql_escape_string($city). "'";
	}
	$sql = "SELECT * FROM reactions WHERE `show` = 1 " . $sqlExtra . " ORDER BY created_at DESC";
	$rs = $this->wps->getADO()->execute($sql);
	$count = $rs->recordCount() + 2;

	$totalArea = $this->totalArea();
	$areaPerImage = ($totalArea/$count);
	// determine which image path to use:
	foreach($this->sizes as $size){
		$sizeH = $size + $this->padCorrectionH;
		$sizeW = $size + $this->padCorrectionW;
		$imgArea = ($sizeH*$sizeW);
		if ($imgArea < $areaPerImage){
			$imgPath = $size;
			break;
		}
	}
	// make up the xml for the images in $imgPath;
	$xml ='';
	while (!$rs->EOF){
		if (!$rs->fields['has_image']){
			$xml .='<gerrit_image id="' . $rs->fields['id'] . '" color="#' . $this->getRndColor(). '">';
		}else{
			$xml .='<gerrit_image file="' . $imgPath . '/' . $rs->fields['id'] . '.jpg" id="' . $rs->fields['id'] . '">';
		}
		$xml .='<name>' . $rs->fields['name'] . '</name>';
		$xml .='<city>' . $rs->fields['city'] . '</city>';
		$xml .='<country>' . $rs->fields['country'] . '</country>';
		$xml .='</gerrit_image>';
		$rs->moveNext();
	}

	$xml = '<gerrit_images size="' . $imgPath . '" count="' . ($count-2). '">' . $xml . '</gerrit_images>';
	$this->WPS_DB_ado2dom = new DOMDocument();
	$this->WPS_DB_ado2dom->loadXML($xml);
}

Javascript code : Ajax call to get the images from the server:

function resize(){ // executed at page-loaded-event, and on page-resize.
	var screenSize = Element.getDimensions(document.body);
	var absBottom = screenSize.height;
	var absRight = screenSize.width;
	var width = (absRight - 305 - 30);
 	$('mainContainer').style.width = width + 'px';
 	$('mainContainer').style.height = (absBottom - 60) + 'px';
 	loadDefaultContent(currCountry, currCity);
}

function loadDefaultContent(country, city){
// Use Prototype-Ajax-updater to update the image-container
	if (!country) country = '';
	if (!city) city = '';
	var screenSize = Element.getDimensions($('mainContainer'));
	try{
		new Ajax.Updater(
			'mainContainer',
			gRelRootPath + 'get_images/&city=' + city + '&c=' + country + '&w='+
				(screenSize.width) + '&h=' +(screenSize.height ),
				{
					asynchronous:true,
					evalScripts:true,
					onLoading : waiter,
					onLoaded : unWaiter,
					onComplete :function(transport){ makeSortable(); }
				}
			);
	}catch(e){
	}
	return false;
}

Checkout the result : www.homeless-songs.com

Other features/wannahave’s:

• More/Better Help-functions, possibly integrated into WPS itself.
• Multi-language-support, not only for editMode, but also for the publishing modules themselves
• Sandbox-support out of the box, no need for a specific subdomain or VirtualHost
• New site-manager; an overview of pages/contentblocks from a grid/treeview.
• Link-checker/mover; Checks internal links to WPS-pages or renames them when a page is renamed
• More wizards for general functions (add page/block etc).
• Template-editting : access and edit XSL-stylesheets that make up the layout/design of your WPS-site
• Empty-cache-button
• Filemanager needs to be rewritten to be MUCH faster and uses less resources
• Custom-XML-tags-support in content-editors, more editing-facilities in content-editors
• Install/Configuration/Security-audit – scripts
• Version-control

I’ve been playing with a new webserver for a week now. It’s a brand new Dell PowerEdge 980, with Xeon Quadcore (2.4ghz), SAS-raid1-sata (7200rpm) and 4 GB of ECC-ram. Should be enough for a relatively large website with high concurrency.. The application I write do not cause a lot of Mysql-queries, I try to cache as much as possible, the real performance-gain for me is getting PHP5 to run faster. So I installed the machine and got to work on some stress-tests.

FreeBSD 6.2-STABLE
I installed FreeBSD 6.2-stable, build the world and a fresh kernel and configured FreeBSD to run several Jails. Among these jails I have a Mysql-5.1 database-server (bound to a loopback-interface) and an Apache2.2 instance, build as MPM-worker and (eventually) fully loaded with modules. Apache and mysql each run in their own jail, and natd ipfw is forwarding web-traffic to Apache, and allows apache to connect to Mysql. Jails were build by using EZ-Jail and all compiling was done from a fresh FreeBSD-ports-collection.

MPM-Worker VS MPM-Prefork
Apache can be build as worker-module of as prefork-module (see apache.org ). After some research I assumed the Worker-MPM should be better for this system, because it has a Quad-Core.

Benchmarking: AB (apacheBench)

For benchmarking I used AB and Siege. Siege didn’t allow the amount of concurrent sessions I initially wanted (&gt;300) because of memory-problems on my laptop, so; the real benchmarking was done with AB. The command I executed each test (twice): ab -n 5000 -c 50 http://192.168.254.103/index.php (except forr the first run, I had to use index.html ‘coz there was no PHP). The PHP-file did nothing more than calling phpinfo();

MPM-WORKER tests:

1. Apache without any PHP, but build with almost all default-options/modules :

Concurrency Level: 50
Time taken for tests: 12.305248 seconds
Complete requests: 5000
Failed requests: 1
(Connect: 0, Length: 1, Exceptions: 0)
Write errors: 0
Requests per second: 406.33 [#/sec] (mean)
Time per request: 615.262 [ms] (mean)
Time per request: 2.461 [ms] (mean, across all concurrent requests)
Transfer rate: 11344.35 [Kbytes/sec] received

2. Apache Build with PHP 5.25 (default options) with mysql module:
* unfortunately PHP was NOT build with the suhosin-patch, which is unstable in jail-environments

Concurrency Level: 50
Time taken for tests: 13.169914 seconds
Complete requests: 5000
Failed requests: 4
(Connect: 0, Length: 4, Exceptions: 0)
Write errors: 0
Total transferred: 153016345 bytes
HTML transferred: 151995475 bytes
Requests per second: 379.65 [#/sec] (mean)
Time per request: 658.496 [ms] (mean)
Time per request: 2.634 [ms] (mean, across all concurrent requests)
Transfer rate: 11346.32 [Kbytes/sec] received

3. Apache PHP with mysql, DOM, PCRE and SimpleXML :

Concurrency Level: 50
Time taken for tests: 14.363650 seconds
Complete requests: 5000
Failed requests: 5
(Connect: 0, Length: 5, Exceptions: 0)
Write errors: 0
Total transferred: 166824023 bytes
HTML transferred: 165801959 bytes
Requests per second: 348.10 [#/sec] (mean)
Time per request: 718.182 [ms] (mean)
Time per request: 2.873 [ms] (mean, across all concurrent requests)
Transfer rate: 11342.10 [Kbytes/sec] received

4. Apache PHP, with mysql, DOM, PCRE, SimpleXML, xml and XSL

Concurrency Level: 50
Time taken for tests: 14.643427 seconds
Complete requests: 5000
Failed requests: 6
(Connect: 0, Length: 6, Exceptions: 0)
Write errors: 0
Total transferred: 170059685 bytes
HTML transferred: 169041004 bytes
Requests per second: 341.45 [#/sec] (mean)
Time per request: 732.171 [ms] (mean)
Time per request: 2.929 [ms] (mean, across all concurrent requests)
Transfer rate: 11341.13 [Kbytes/sec] received

5. Apache PHP, with mysql, DOM, PCRE, SimpleXML, xml, XSL and ImageMagick (PECL)

Didn’t run well, see conclusion

PREFORK-Tests:

1. PHP Only, No Modules

Concurrency Level: 50
Time taken for tests: 12.295282 seconds
Complete requests: 5000
Failed requests: 0
Write errors: 0
Total transferred: 141711640 bytes
HTML transferred: 140707685 bytes
Requests per second: 406.66 [#/sec] (mean)
Time per request: 122.953 [ms] (mean)
Time per request: 2.459 [ms] (mean, across all concurrent requests)
Transfer rate: 11255.54 [Kbytes/sec] received

2. PHP mysql

Concurrency Level: 50
Time taken for tests: 12.927480 seconds
Complete requests: 5000
Failed requests: 0
Write errors: 0
Total transferred: 150875864 bytes
HTML transferred: 149879073 bytes
Requests per second: 386.77 [#/sec] (mean)
Time per request: 129.275 [ms] (mean)
Time per request: 2.585 [ms] (mean, across all concurrent requests)
Transfer rate: 11397.35 [Kbytes/sec] received

3. PHP Mysql, PCRE, SPL , DOM SimpleXML
Concurrency Level: 50
Time taken for tests: 14.93142 seconds
Complete requests: 5000
Failed requests: 0
Write errors: 0
Total transferred: 164663496 bytes
HTML transferred: 163661133 bytes
Requests per second: 354.78 [#/sec] (mean)
Time per request: 140.931 [ms] (mean)
Time per request: 2.819 [ms] (mean, across all concurrent requests)
Transfer rate: 11410.09 [Kbytes/sec] received

4.PHP Mysql, PCRE, SPL , DOM SimpleXML, XML XSL
Concurrency Level: 50
Time taken for tests: 14.584835 seconds
Complete requests: 5000
Failed requests: 0
Write errors: 0
Total transferred: 168910632 bytes
HTML transferred: 167907871 bytes
Requests per second: 342.82 [#/sec] (mean)
Time per request: 145.848 [ms] (mean)
Time per request: 2.917 [ms] (mean, across all concurrent requests)
Transfer rate: 11309.76 [Kbytes/sec] received

5. PHP Mysql, PCRE, SPL , DOM SimpleXML, XML XSL, IMAGICK

Concurrency Level: 50
Time taken for tests: 15.321272 seconds
Complete requests: 5000
Failed requests: 0
Write errors: 0
Total transferred: 176969536 bytes
HTML transferred: 175968367 bytes
Requests per second: 326.34 [#/sec] (mean)
Time per request: 153.213 [ms] (mean)
Time per request: 3.064 [ms] (mean, across all concurrent requests)
Transfer rate: 11279.81 [Kbytes/sec] received

Conclusion

During the testing I rebuild the whole thing to run Apache Preforked. I noticed that adding ImageMagick to a Apache Worker-instance caused segmentation faults, leading to dropped clients. I didn’t find anything conclusive at Google on this. I also noticed the distribution of clients among the httpd-processes. Worker caused 8 http-processes to run at 40% processor (out of 400%), the prefork-httpd-version had 50 httpd-processes spawned, each taking about 5% at init, and got down to 1% soon after init. Using Imagick on worker caused each httpd-process to use every cycle it could (350% on 1 client). Preforked httpd’s with ImageMagick also used a lot more cycles; 8-10% at init!

So: if you want to get every cycle worth of PHP from your box you’re better of keeping PHP as slim as possible, and try to keep away from loading Imagick. I’m considering building a special Jail for imagick-php which will not be used that much. This way the-real-everyone-sees-me-PHP5-instance is ready for a lot more concurrent session.

Still want more performance?
Consider serving static content (images, css, javascript and HTML-files) from lighttpd. Lighttpd is a LOT faster, doesn’t need a lot of memory. and uses less cycles compared to Apache (remember to build lighttpd without support for php, fcgi). I’m still figuring out whether I can use mod_rewrite for redirection to lighttpd, I guess not, because Apache still has to handle those requests. Maybe mod_proxy will work.

Update

Session-support in Worker-threads seems to break too, so I finally build a debug-build of PHP and the extensions, trying to figure out why Imagick made the Worker-thread barf on me. No results; no clue anywhere in the logs. At this time I have a stable Prefork-apache running with both session and imagick-support, including some other extensions. This means giving up on 100 req/s based on the tests. In still want to build a production environment capable of handling heavy load (without imagick), a script/jail for manipulation of images and lighttpd for static content (which is already up, but cannot be tested yet).