PHP and cURL in parallel

I was recently working on a php based project for a client. I needed to pull in content from 10 different URLs, then process that data into something useful.

I have a growing love affair with cURL, so it was my natural selection for the project.

In my first approach, I looped over the list of URLs, sent the cURL request, wait for the response, stored the data. repeat until done.

That worked, but it was very slow process. The page was taking between 30 – 45 seconds to load. That doesn’t sound long, but in terms of web applications, that’s an eternity.

I tried several tricks to speed the application up, but the bottle neck was the cURL calls. Each call was done in a synchronous manner. Every call to cURL had to be completed before the next could be made.

After doing to research on how to speed this up, I came across the following php cURL functions:

curl_multi_init();
curl_multi_add_handle();
curl_multi_select();
curl_multi_exec();
curl_multi_getcontent();
curl_multi_info_read();
curl_multi_remove_handle();

Using these methods allow for cURL to send asynchronous requests, solving my pervious problem.

A little more searching, and I found an awesome wrapper library that takes the guesswork out of the using the curl_multi methods. “ParallelCurl” https://github.com/petewarden/ParallelCurl

The sample code provided with the library was very straight forward and super easy to use.

After using asynchronous cURL calls via the ParallelCurl library, I was able to reduce the page load time from 45 seconds to 15 seconds. It’s still slow, but it’s a HUGE improvement, and it makes the application usable, and reduces load on my server. It’s a win-win-win situation!

Using cURL and PHP to upload files through a form post

Lately I have been working on a project that requires me to use PHP to interact with a REST based service. cURL is the logical choice for making HTTP calls to the REST service.

I love cURL, I’ve blogged about it before, but I recently ran into some major issues.

The REST service I was using required me to send two files along with some meta information. easy enough. I used the following code:

$postFields = array();
 
//files
$postFields['file'] = "@$filePath";
$postFields['thumbnail'] = "@$thumbnailPath";
 
//metaData
$postFields['title'] = "$title";
$postFields['description'] = "$description";
$postFields['tags'] = "$tags";
$postFields['licenseinfo'] = "$licenseinfo";
$postFields['token'] = "$userToken";
 
$curl_handle = curl_init();
 
curl_setopt($curl_handle, CURLOPT_URL, $api_url);
curl_setopt($curl_handle, CURLOPT_RETURNTRANSFER, true);
curl_setopt($curl_handle, CURLOPT_POST, true);
curl_setopt($curl_handle, CURLOPT_POSTFIELDS, $http_post_fields);
 
//execute the API Call
$returned_data = curl_exec($curl_handle);

The code worked great, sending an “at” sign (@) before the file path makes sure that cURL sends the file as part of a “multipart/form-data” post. Exactly what we needed.

The form post from cURL worked great, but the REST service was retuning a 400 error and saying “The specified thumbnail file is not supported.”. I was at a loss. The service documentation stated the “jpg, jpeg, gif, and png” files were supported.

I ended up contacting the developers of the service who told me that the content type for the file had to be set to “image/jpg” (for jpg).

After pouring through the cURL documentation and not finding anything about how to set the content type for a single file in a “multipart/form-data” post, I turned to Goolge. My searches with about as helpful as the cURL docs. I sent a few hours hacking my code and trying some things, I ever read some posts from 2008 saying that is was not possible to do. Then, I got a break through, a single ray of light. On a message board was a single sentence replay. “You should try this… $image;type=image/jpg”.

That was the break through I needed. Below is final updated code:

$postFields = array(); 
 
//files
 $postFields['file'] = "@$filePath";
 
//get the extension of the image file
$tumbnailExtention = preg_replace('/^.*\.([^.]+)$/D', '$1', $thumbnailPath);
$postFields['thumbnail'] = "@$thumbnailPath;type=image/$tumbnailExtention";
 
//metaData
$postFields['title'] = "$title";
$postFields['description'] = "$description";
$postFields['tags'] = "$tags";
$postFields['licenseinfo'] = "$licenseinfo";
$postFields['token'] = "$userToken"; 
 
$curl_handle = curl_init();
 
curl_setopt($curl_handle, CURLOPT_URL, $api_url);
curl_setopt($curl_handle, CURLOPT_RETURNTRANSFER, true);
curl_setopt($curl_handle, CURLOPT_POST, true);
curl_setopt($curl_handle, CURLOPT_POSTFIELDS, $http_post_fields); 
 
//execute the API Call
$returned_data = curl_exec($curl_handle);

In summary, if you need to set the content type of a file being sent an image through cURL, via a POST, use the following format:
$postFields[‘file’] = “@PATHTOFILE;type=CONTENTTYPEHERE”;

Controlling Twitter SMS notifications through code

twitter sms

It’s not secret my current favorite tech toys are my iPhone and Twitter.

I follow several hundred people of interest, and regularly use TwitterFon Pro, TweetDeck and Twitter.com to keep up with them all. For an elite group of people I follow on twitter I choose to have their updates pushed to me via SMS (text message) to my phone.

Sometimes I’m in area where I dont have reliable service to my phone (most of the meeting rooms at my work for some reason), and Twitter will try to send me an SMS message telling me that one of my good friends just made a tweet. If Twitter cant deliver that tweet for some reason, like the phone doesnt have service, Twitter simply stops sending message to your phone. There has been many times where i have thought to myself “damn Twitter is quiet today”, then I log onto Twitter.com and see a “is you device have trouble?” error. If i don’t go check the website, I have no idea that Twitter turned off SMS updates to my phone. Twitter is cool but they aren’t cool enough to send you an email saying “Hey your phone is screwed up, we are going to stop sending you SMS message until you fix it”. Atleast then the poor bastards like me who are addicted to twitter would know what we have to do.

Because I am such a Twitter addict, I have come up with a plan to end my frustrations with this problem. The solution? Use some evil “robots” to do my bidding. I have blogged about them before, I am talking about Cron and Curl. okokok, so they aren’t evil robots they are software running on my computer.

Twitter has an awesome API (Application Programming Interface) that allows you to do all sorts of sweet things using REST based http requests, including turning on and off SMS notifications to your phone.

cURL is an application that allows you to connect to web addresses from a command line.

cron is an application that allows you run application at scheduled intervals.

Can you guess where this is all going? Here is mathematical representation for you
((Twitter API + cURL) * cron(30 minutes)) = Happy Josh Highland

I have cron run the following command every 30 minutes (replace username and password with your own of course)

/usr/bin/curl -u user:password -d device=sms http://twitter.com/account/update_delivery_device.xml

I will never need to log onto twitter to turn on my SMS notifications ever again. In the event that Twitter disables notifications to my phone, every 30 minutes my evil robots enable MS messages to my phone. At most I will have 30 minutes of down time. I can live with that.

This is a perfect example of what one of my Computer Science professors told me… “If you have to do it more then 2 times write a script to do it for you”.

Let cURL do your heavy lifting

curlrobot

In a previous post, I talked about using cron to do your bidding. If you do a lot of web development like I do, you may have some web based tasks that you need to automate. A good example of this could be clearing a file cache or tipping of a script to rebuild your xml site map.

One of the best ways to do that would to be cURL and cron together. cURL is an application used to transfer the contents of a url. Insistently, to transfer the contents of a url, curl has to connect to that url, and the server has to process that url/script… do you see what I am going with this? cURL can be used to do anything you would normally do from the url of your browser.

This is the usage of cURL:

curl [options] [URL…]

lets say you have the following URL that you want to connect to on a regular schedule
http://SomeDomain.com/SomeFile.php?var1=abc&var2=zyz

you would use the following cURL command:
/usr/bin/curl -G -d var1=abc -d var2=zyz http://SomeDomain.com/SomeFile.php

We are using the “-G” argument to simulate a “GET” command, and each “-d” is a variable/value pair

note: “/usr/bin/curl” is the location on my computer that cURL is installed

running cURL from the command line will show you results that get returned from the processing of the script.

To fully automate this processs, add the curl command into cron. now you are one step closer to world domination!