Twitter’s API returning HTML instead of XML / JSON
I wasn’t going to write about this, but it’s been two weeks now, and more than a few people are agonizing over their Twitter API based apps not working. In case the title didn’t give it away, Twitter has been intermittently returning an empty HTML page as a response to API calls. I’m making XML requests, but the issue is also plaguing JSON requests. More about it here…
In an ideal world, a RESTful response should always be in the format requested. From what I’ve read, it appears this issue may be caused by recent measures to help prevent DoS attacks. Either way, I couldn’t wait any longer for Twitter to address the issue. Hopefully this helps someone else out…
My issues began when I first started building an application based on the twitter datasource. My Twitter data iterators kept throwing errors. By debugging the response before letting the datasource convert it into an array, I saw the request was retuning a 200 success header, and the following HTML:
<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/1999/REC-html401-19991224/strict.dtd"> <!-- <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd"> --> <HTML> <HEAD> <META HTTP-EQUIV="Refresh" CONTENT="0.1"> <META HTTP-EQUIV="Pragma" CONTENT="no-cache"> <META HTTP-EQUIV="Expires" CONTENT="-1"> <TITLE></TITLE> </HEAD> <BODY><P></BODY> </HTML>
Based on other user’s experience, it seemed waiting about .3 of a second usually cleared up the HTML response issue, and within 2-3 additional requests, you retrieved the data you were after. So I modified my _twitterReqest method, a simple wrapper for my calls to the datasource, to make the same request up to 5 times with a .3 second pause between requests until the data was returned in an acceptable format.
If 5 times wasn’t enough, I output a message explaining to the user that the API is unresponsive at the moment. So far, this message has yet to appear, and looking at my logs, I’ve never had to retry a request more than 2 times.
As soon as I get the data I want, I cache it for 24 hours, hopefully avoiding further issues for another day.
Here’s a quick look at my method:
function _twitterRequest($method = null, $params = null, $tries = 5, $delay = 300000){
/*
* This wrapper method is necessary b/c of issue documented here:
* http://code.google.com/p/twitter-api/issues/detail?id=968
* http://code.google.com/p/twitter-api/issues/detail?id=1014
*
* iterates through api calls with a .3 second delay to allow twitter responses to clear
* has yet to fail
*/
if($method):
$try = 1;
while($try <= $tries){
$response = $this->Twitter->$method($params);
if(isset($response['Hash']['error']) || isset($response['User']) || isset($response['Users'])){
return $response;
} else {
$this->log($method . ': retry # ' . $try);
usleep($delay);
$try ++;
}
}
return false;
endif;
}
It’s frustrating to have to test against 1 + 1 equaling 2, but I guess it’s just the price of developing with 3rd party platforms… The moral here is always check your data against the data format you’re expecting. Don’t assume anything!
very nice catch. i was actually having the exact same problem with the twitter datasource today. thanks.
@dave rupert
Np, hope it helps…