PHP and cURL in parallel

I was recently working on a php based project for a client. I needed to pull in content from 10 different URLs, then process that data into something useful.

I have a growing love affair with cURL, so it was my natural selection for the project.

In my first approach, I looped over the list of URLs, sent the cURL request, wait for the response, stored the data. repeat until done.

That worked, but it was very slow process. The page was taking between 30 – 45 seconds to load. That doesn’t sound long, but in terms of web applications, that’s an eternity.

I tried several tricks to speed the application up, but the bottle neck was the cURL calls. Each call was done in a synchronous manner. Every call to cURL had to be completed before the next could be made.

After doing to research on how to speed this up, I came across the following php cURL functions:

curl_multi_init();
curl_multi_add_handle();
curl_multi_select();
curl_multi_exec();
curl_multi_getcontent();
curl_multi_info_read();
curl_multi_remove_handle();

Using these methods allow for cURL to send asynchronous requests, solving my pervious problem.

A little more searching, and I found an awesome wrapper library that takes the guesswork out of the using the curl_multi methods. “ParallelCurl” https://github.com/petewarden/ParallelCurl

The sample code provided with the library was very straight forward and super easy to use.

After using asynchronous cURL calls via the ParallelCurl library, I was able to reduce the page load time from 45 seconds to 15 seconds. It’s still slow, but it’s a HUGE improvement, and it makes the application usable, and reduces load on my server. It’s a win-win-win situation!

Optimizing the performance of WordPress

It’s no secret that I LOVE WordPress. I’ve been using the software for 6 years and have no plans to stop any time soon. Last year I attended WordCamp LA with @JasonKeller, it was awesome. I knew that I had to be involved with WordCamp 2010.

When WordCamp 2010 was announced, I signed up as a speaker. My topic, “Improving the performance of WordPress”.

I had a great time preparing and delivering my presentation. A lot of people asked me to post the slides online. I decided to do one better, and post a recorded version of my presentation.

Download the slides in PDF format