Using cURL for HTTP transactions

Using cURL for HTTP transactions

By GTADarkDude | 12768 Reads |

0     0

/* Introduction */

Well hello my fellow HBH members and welcome to my first article. Since a lot of topics are already discussed in articles, I started looking around which subjects were not very popular. It caught my eye that lots of members haven't tried HBH's Timed Challenges yet and since I thought they were great fun :-P, I decided to write an article about them. On the other hand, I wouldn't want to give you lots of spoilers so I won't be going into many details regarding the challenges. Of course, the challenges can be solved by using a whole variety of languages, but I, as many others, prefer PHP. When visiting the forums, I noticed cURL is most commonly used to do these challenges. Therefore this article will explain to you how to use cURL to connect to a web page. Knowledge which can be applied to HBH's Timed Challenges.

/* cURL */

This is for the people who have never heard of cURL before. Let me cite something from php.net:

PHP supports libcurl, a library created by Daniel Stenberg, that allows you to connect and communicate to many different types of servers with many different types of protocols. libcurl currently supports the http, https, ftp, gopher, telnet, dict, file, and ldap protocols. libcurl also supports HTTPS certificates, HTTP POST, HTTP PUT, FTP uploading (this can also be done with PHP's ftp extension), HTTP form based upload, proxies, cookies, and user+password authentication. These functions have been added in PHP 4.0.2. Perhaps a bit easy to just copy-paste that, but it does explain what cURL actually does. As you can see, cURL can be used to retrieve information from websites and also to submit information to websites. And that's exactly what you'll be using it for when doing the challenges.

/* I don't have the cURL library? */

Most systems automatically include the cURL library. You can check phpinfo() to see if you've got it. If you don't, I'd be glad to redirect you to the PHP Manual, where'll you'll find all the information you need concerning the installation of cURL on your server. http://nl2.php.net/manual/en/curl.setup.php Most hosting sites support cURL, so if you haven't got your own website or server, you can always subscribe at freehostia or something like that. I shouldn't be hard to find a host which supports cURL.

/* Retrieving data */

To retrieve data from a web page, you'll only need 4 simple functions: curl_init(), curl_setopt(), curl_exec() and curl_close(). To start a cURL session, we'll use curl_init(), which returns a handle to the cURL session. Then, we'll set several options for the session using curl_setopt($handle, OPTION, value). One of these options - and perhaps the most important one - is the URL. Then we'll use curl_exec($handle) to perform the cURL session. Finally, we'll use curl_close($handle) to end the session. The only part that really needs further explaining is the curl_setopt() part, as lots of options can be set. A nice list of all the options can be found here: http://nl2.php.net/manual/en/function.curl-setopt.php In this example, we'll be using the GET method to show HBH's home page. Here are the cURL options I am going to use in this example:

• CURLOPT_URL | Self-explanatory (STRING)
• CURLOPT_COOKIE | Basically to tell HBH you are a logged-in member. Only the PHPSESSID and fusion_user cookie will be needed. (STRING)
• CURLOPT_USERAGENT | Self-explanatory (STRING)
• CURLOPT_RETURNTRANSFER | To return the transfer as a string of the return value of curl_exec() (BOOL) (Yes, that is quoted from PHP.net)
• CURLOPT_VERBOSE | To write the output to the browser (BOOL)

Here's an example on how to use the GET method to show the HBH index page:

$c = curl_init();
curl_setopt($c, CURLOPT_URL, \"http://www.hellboundhackers.org/\");
curl_setopt($c, CURLOPT_COOKIE, \"PHPSESSID=*****; fusion_user=*****.**********\");
// You have to use your own cookies for this. I assume you know how to check your cookies...
// Make sure you are logged in when you check your PHPSESSID cookie.
curl_setopt($c, CURLOPT_USERAGENT, \"Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.11) Gecko/20071127 Firefox/2.0.0.11\");
// Of course it doesn\'t really matter which USERAGENT you enter, as long as it is valid. This is just an example.
curl_setopt($c, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($c, CURLOPT_VERBOSE, 1);
$page = curl_exec($c);
curl_close($c);
echo $page;
?>```
I just show you the entire code at once, so you can get an idea of what a cURL session might look like. As you can see, $page holds the entire HTML source of HBH\'s index page. Because it is a string, you can easily manipulate the data or extract that certain part you\'re looking for.


/* Using the POST method */

Just image you want to show HBH\'s index page and then you want to submit something to the code bank in less than a second. To do this, you can write a cURL script that POSTs the data for you. Basically, this goes the same as retrieving the data.  The only difference is that you\'ll have to change the URL and the HTTP-request type (from GET to POST). In this example, we assume the URL has to change and we have to use POST to submit the answer.
You can copy the whole cURL session from above and start a new session with some changed/new OPTIONs, but you can also use curl_copy_handle($handle), which is quite self-explanatory. In the new session, you can just reset the CURLOPT_URL option and add some options. Here are the two new options we\'re going to add:
- CURLOPT_POST | To use the POST method. (BOOL, default=false)
- CURLOPT_POSTFIELDS | To set the POST data. (STRING)
Here\'s how we\'re going to do this:
```markup<?php
$c = curl_init();
// See above for the code, I\'m not going to copy it.
$page = curl_exec($c);
//ATTENTION: Don\'t close the cURL session just yet!!! We\'ll need it to copy for the second session.

// $page holds HBH\'s index page. You can do with it whatever you want.

$c2 = curl_copy_handle($c); //Copy the first cURL session, including all it\'s options.
curl_close($c); // Now you can close the first one, if you like.
curl_setopt($c2, CURLOPT_URL, \"http://www.hellboundhackers.org/code/codesubmit.php\");
curl_setopt($c2, CURLOPT_POST, 1); // Sets the POST method.
curl_setopt($c2, CURLOPT_POSTFIELDS, \"code_category=24&code_subject=\" . $subject . \"&code_description=\" . $description . \"&code_body=\" . $content . \"&submit_code=Submit+Code\");
// You could for instance load $subject, $description and $content out of a text file which holds your code. 
$final = curl_exec($c2);
curl_close($c2);
echo $final; // Shows the result of what happens when you submit that POST data.```
Of course this whole scenario is quite ridiculous. It wouldn\'t really be a great help if you could show HBH\'s index page and then submit something to the code bank, but nevertheless is shows you how to use curl_copy_handle() and how to use CURLOPT_POST.

NOTE: you can also use the same cookie session for the whole script. I\'ve chosen to start a new session, because I\'m used to do that. I can\'t really see if one way is better than the other, you should choose for yourself. If you want to use the same session, do it like this:
```markup<?php
$c = curl_init();
curl_setopt($c, CURLOPT_URL, \"http://www.example.com/page.php\");
// Other cURL session options go here...
$firstpage = curl_exec($c);
curl_setopt($c, CURLOPT_URL, \"http://www.example.com/otherpage.php\");
$secondpage = curl_exec($c);
curl_close($c);```


/* Some other functions that might be useful */

- curl_getinfo($handle) | This returns an array containing a lot of information about the cURL session. To name some examples: HTTP headers, total transaction time, up-/download speed, content type, etc. You can also use curl_getinfo($handle, OPTION) to return a specific option as a string. curl_getinfo() can be used between curl_exec() and curl_close(). A list of all options can be found in the PHP manual.
- curl_version() | Returns an array containing information about the cURL version that is used.
- curl_error($handle) | This returns a string containing a potential error, which occured during curl_exec(). It can be used to investigate why a certain cURL session doesn\'t give the desired result.
- curl_errno($handle) | Same as above, but gives the result as an integer. A list of all cURL error codes and their corresponding integer values can be found here:
http://curl.haxx.se/libcurl/c/libcurl-errors.html
- curl_setopt_array($handle, $options) | Where $options contains an array with all desired cURL session options. Basically the same as curl_setopt(), but this could be more useful when you want to set multiple options without constantly using curl_setopt().
There are also a lot of functions available for multi cURL sessions, where you can process multiple sessions in parallel. However, these functions are not that hard and I can\'t really think of a lot of scenario\'s where you\'ll be using it. If you do want to know about this, read the PHP manual.


/* Some other options that might be useful */

Even though I\'ve discussed the most important cURL session options, there are some more I\'d like to name here. A complete list can be found in the PHP manual, but here are some options I\'d like to name, because I think you might need these. I\'m not really going into details about using cURL for FTP connections or the SSL encrypted protocol, because I think you won\'t need it a lot. If you do want to learn something about that, you can look it up in the PHP Manual, where everything is nicely documented. 
http://nl2.php.net/manual/en/function.curl-setopt.php
Anyway:
- CURLOPT_HTTPPROXYTUNNEL | TRUE if you want to tunnel through a given HTTP proxy. As in: \"your communications will go as if the proxy is NOT THERE\" (quoted from andrabr at gmail dot com on php.net)(BOOL)
- CURLOPT_HTTPPROXYAUTH | See CURLOPT_HTTPAUTH. (INTEGER)
- CURLOPT_PROXY | The HTTP proxy (if CURLOPT_HTTPPROXYTUNNEL == TRUE) to tunnel through. Can include port number, but CURLOPT_PROXYPORT can also be used. (STRING)
- CURLOPT_PROXYUSERPWD | [username]:[password] to connect to the given proxy. (STRING)
- CURLOPT_PROXYPORT  | See CURLOPT_PORT. (INTEGER)
- CURLOPT_PORT | Connect to an alternative port. (INTEGER, default=80)
- CURLOPT_HTTPAUTH | Choose between CURLAUTH_BASIC, CURLAUTH_DIGEST, CURLAUTH_GSSNEGOTIATE (or combine them using bitwise | [OR])or CURLAUTH_ANY (or _ANYSAFE). Sets the authentication method. (INTEGER)
- CURLOPT_USERPWD | [username]:[password] to connect. Used with NTLM authentication e.a. (STRING)
- CURLOPT_COOKIESESSION | Start a new cookie session. (BOOL)
- CURLOPT_COOKIEJAR | Where all internal cookies are stored when the cURL session closes. (STRING)
- CURLOPT_REFERER | Sets the referer HTTP header for the cURL session. (STRING)
- CURLOPT_FILE | The file that the transfer should be written to. For example when using cURL to store images locally. (STREAM RESOURCE)
I\'ve selected these because I think you just might need these once. Not that that is very likely though, as some people will probably only use cURL to get challenge points, but nevermind that now.


/* HBH\'s Timed Challenges */

Now I\'m not going to tell you how to do every single challenge, but here are some tips for each challenge:

1. I used some basic string functions to find the word. You can also work with REGEX patterns if you like. (Personally, I don\'t.)

2. Loop through the hash and check if each char is a number. This is what you have to do: 845d1e -> 8+4+5+1 = 18.

3. Loop through the file and md5() each word. Then, compare the hashes.

4. Rewrite the pseudocode in a PHP function. You\'re supposed to reverse it, but I did something else. Try rescrambling the scrambled word a couple times.

5. Easy math. No explaining necessary.

6. Perhaps the hardest part is to identify the URLs on Google Search. Perhaps you can look in the HTML source and find some tags that are used around each URL.

7. Definitely my favourite one. I\'m going to let this one remain a challenge, so I won\'t be giving any hints here.


/* Epilogue */

I didn\'t want to spoil everything, so I didn\'t. However, I did want to make clear how to use cURL to execute some basic HTTP transactions. If you didn\'t know a lot about cURL before, I hope you learned something by reading this article. By the way, I know my English is not that outstanding, but I think it is good enough to be clear for any reader of this article. If you\'ve still got some questions regarding the timed challenges or this article, feel free to send me a PM. Hope you guys liked my article. ;)

GTADarkDude