One day, just when I was seriously thinking about implementing some kind of caching, I found about mtop – small tool like top to display running MySQL queries in real-time, running it I found that every second there were many queries (same query but made by different requests) that got stuck in the “PREPARING” state and sometimes took as long as 2 secs. to complete. The evil query was:

SELECT c.post_id FROM category_post_relationship c
  WHERE  (c.cat_id IN
  (SELECT c2.id AS c2__id FROM categories c2
  WHERE (c2.user_id =  ?)))

Thinking what it does? Let me make it easier – it fetches “post ids” of all the posts of a user that are categorized.

1. Running the query

184 rows, Query took 0.0812 sec

2. Profiling:

   starting                0.000132
   checking permissions    0.000011
   checking permissions    0.000011
   Opening tables          0.000037
   System lock             0.000015
   Table lock              0.000022
   init                    0.000065
   optimizing              0.000025
   statistics              0.000027
   preparing               0.000027
   executing               0.000011
   Sending data            0.000047
   optimizing              0.000023
   statistics              0.000086
   preparing               0.075275
   end                     0.000015
   query end               0.000010
   freeing items           0.000033
   logging slow query      0.000009
   cleaning up             0.000011

3. EXPLAIN:

id 	    select_type         table       type 	       possible_keys 	    key         key_len    ref 	    row     Extra
1 	    PRIMARY 	        c 	    index 	       NULL 	            PRIMARY 	8 	   NULL     53898   Using where; Using index
2 	    DEPENDENT SUBQUERY 	c2 	    unique_subquery    PRIMARY,user_id 	    PRIMARY 	4 	   func     1 	    Using where

If you’d ask some novice to do what the query does many of them would use two separate queries and believe me (I’ve done the testing as well) that’ll be much faster!

Googling “subquery optimization” got me this:

MySQL evaluates queries “from outside to inside.” That is, it first obtains the value of the outer expression outer_expr, and then runs the subquery and captures the rows that it produces.

From http://dev.mysql.com/doc/refman/5.1/en/in-subquery-optimization.html

The page also tells us some tips on how to optimize subqueries, going by the suggestions we get the following query:

SELECT c.post_id FROM category_post_relationship c
  WHERE  EXISTS
  (SELECT 1 FROM categories c2
  WHERE c2.user_id = 7639 AND c.cat_id =  c2.id)

Still, running it takes similar query times and as such doesn’t help.

Again from the same page:

After the conversion, MySQL can use the pushed-down equality to limit the number of rows that it must examine when evaluating the subquery.

Though I didn’t read the whole page thoroughly (I’m lazy and since I got the job done by some other technique) still I’m sure that this “optimized” query only “optimizes” the subquery while our biggest problem is that the outer table (with about 50000 rows) is getting evaluated.

I felt, this was one query you’d rather not “optimize” and be happy with two separate queries.

But, this is not to say it can’t be optimized, it can be very easily – by NOT using subqueries at all. I used JOINS:

SELECT cc.post_id FROM categories c
  RIGHT JOIN category_post_relationship cc
  ON  c.id = cc.cat_id<br>
  WHERE  c.user_id = ?

Now the query takes 0.0008 sec compared to 0.0812, that’s a hundred fold improvement!

Some tips:

1. If you’re using subqueries and are having some problems with it being slow read the article I mentioned.
2. If possible, use WHERE condition on the outer query.
3. Don’t use subqueries when the outer query table contains lots of rows. (I may be wrong but this is what I’ve found out)

Before listing the class, let me first show how it’s used:

require 'Validator.class.php';

$validator = new Validator();

// Add rules
$validator->addRule('name', array('minlength' => 5, 'maxlength' => 20));
$validator->addRule('age', array('min' => 13, 'max' => 35));
$validator->addRule('url', array('url'));
$validator->addRule('about', array('require'));

// Data to be validated, would normally come from a form
$data = array(
        'name' => 'Arvind Gupta',
        'age' => '80',
        'url' => 'http:www.arvindgupta',
);

// Set data to be validated
$validator->setData($data);

// Check
if ($validator->isValid())
{
    echo '<h1>Data is valid!</h1>';
}
else
{
    echo '<h1>Data is not valid!</h1>';
    echo '<ol>';

    // Get and print errors in a nice manner
    foreach ($validator->getErrors() as $field => $messages)
    {
        if (count($messages) == 1)
        {
            echo "<li><strong>$field</strong>: $messages[0]</li>";
        }
        else
        {
            // If a field has more than one error
            echo "<li><strong>$field</strong>:</li>";
            echo '<ol>';
            foreach ($messages as $message)
            {
                echo "<li>$message</li>";
            }
            echo '</ol>';
        }
    }
    echo '</ol>';
}

Very straightforward!

Validating a form is equally straightforward. The best way would be to have the form elements named like arrays, for example:

<form ...>
	<input type="text" name="form1[text1]" value="" />
	<input type="text" name="form1[text2]" value="" />
</form>

And, use something the following line to provide the form data to the validator at one go:

$validator->setData($_REQUEST['form1']);

That’s it! Here is the class code:

/**
 * Validator
 *
 * Data validation class
 *
 * @author      Arvind Gupta <contact [ AT ] arvindgupta [ DOT ] co [ DOT ] in>
 * @copyright   Arvind Gupta (c) 2011
 * @link        http://www.arvindgupta.co.in
 * @license     You're free to do whatever with this as long as this notice
 *              remains intact.
 */
class Validator
{

    protected $_rules = array();
    protected $_data = array();
    protected $_messages = array();
    protected $_errors = array();

    public function __construct()
    {
        $this->setDefaultMessages();
    }

    /**
     * Add a rule
     *
     * @param string $field	Field name (index of data to be validated)
     * @param array  $rules	Array of rule(s)
     */
    public function addRule($field, array $rules)
    {
        $this->_rules[$field] = $rules;
    }

    /**
     * Set data to be validated
     *
     * @param array $data   Data to be validated
     */
    public function setData(array $data)
    {
        $this->_data = $data;
    }

    /**
     * Set error message for rule
     *
     * @param string $rule	Rule name
     * @param string $message	New message
     */
    public function setMessage($rule, $message)
    {
        $this->_messages[$rule] = $message;
    }

    /**
     * Validates current data with current rules
     *
     * @return boolean
     */
    public function isValid()
    {
        $valid = true;
        foreach ($this->_rules as $field => $rules)
        {
            $value = isset($this->_data[$field]) ? $this->_data[$field] : '';

            foreach ($rules as $rule => $parameter)
            {
                // If rule does not require parameter
                if (is_int($rule))
                {
                    $rule = $parameter;
                    $parameter = null;
                }

                if (!$this->check($value, $rule, $parameter))
                {
                    $valid = false;

                    if (stripos($this->_messages[$rule], '%s') !== false)
                    {
                        $this->_errors[$field][] = sprintf($this->_messages[$rule], $parameter);
                    }
                    else
                    {
                        $this->_errors[$field][] = $this->_messages[$rule];
                    }
                }
            }
        }

        return $valid;
    }

    /**
     * Get error messages if validation fails
     *
     * @return array	Error messages
     */
    public function getErrors()
    {
        return $this->_errors;
    }

    protected function check($value, $rule, $parameter)
    {
        switch ($rule)
        {
            case 'require' :
                return!(trim($value) == '');

            case 'maxlength' :
                return (strlen($value) <= $parameter);

            case 'minlength' :
                return (strlen($value) >= $parameter);

            case 'numeric' :
                return is_numeric($value);

            case 'int' :
                return is_int($value);

            case 'min' :
                return $value > $parameter ? true : false;

            case 'max' :
                return $value < $parameter ? true : false;

            case 'url':
                // Regex taken from symfony
                return preg_match('~^
			      (https?)://                               # protocol
			      (
				([a-z0-9-]+\.)+[a-z]{2,6}               # a domain name
				  |                                     #  or
				\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}      # a IP address
			      )
			      (:[0-9]+)?                                # a port (optional)
			      (/?|/\S+)                                 # a /, nothing or a / with something
			    $~ix', $value);

            case 'email':
                return preg_match('/^([^@\s]+)@((?:[-a-z0-9]+\.)+[a-z]{2,})$/i', $value);

            case 'regex':
                return preg_match($parameter, $value);

            case 'pass':
                return true;

            default :
                return false;
        }
    }

    protected function setDefaultMessages()
    {
        $this->_messages = array(
                'require' => 'Field is required.',
                'maxlength' => 'Too long (%s characters max).',
                'minlength' => 'Too short (%s characters min).',
                'numeric' => 'Value must be numeric.',
                'int' => 'Value must be an integer.',
                'max' => 'Value must be at most %s',
                'min' => 'Value must be at least %s',
                'url' => 'Value must be a valid URL.',
                'email' => 'Value must be a valid email.',
                'regex' => 'Invalid value.',
        );
    }
}

Now that I have finally created a blog here, I plan on publishing posts on everything from programming to publishing photographs I shoot on a regular basis (not all that frequently though). Read my About Me page for more information about me.

Screenshot of the previous website