Working with HttpWebRequest and HttpWebResponse in ASP.NET

Introduction

I will explain about the usage of HttpWebRequest and HttpWebResponse in this article. As you all probably know or heard about this class before. HttpWebRequest and HttpWebResponse class is inside the System.NET namespace, and this two classes is designed to communicate by using the Http Protocol

You can use this two classes to make requests to other Web Pages via HTTP and parse the resulting text to extract data. This is what we know as screen scraping.

In ASP world, you normally need to rely on third party components called ASPTear to grab the contents from other site. But now with the help of HttpWebRequest and HttpWebResponse, you can do that easily without have to invoke third party components.

Using HttpWebRequest and HttpWebResponse

In the code below, I provide very basic sample code on how to use HttpWebRequest and HttpWebResponse. In the first example I will list out the code on how to do screen scraping and the second example would be doing HttpPost data to another website

1. Sample Code on Grabbing Contents (Screen Scraping)

C#

protected void Page_Load(object sender,EventArgs e) {
    Uri uri = new Uri("http://www.microsoft.com/default.aspx");
    if(uri.Scheme = Uri.UriSchemeHttp) {
        HttpWebRequest request = HttpWebRequest.Create(uri);
        request.Method = WebRequestMethods.Http.Get;
        HttpWebResponse response = request.GetResponse();
        StreamReader reader = new StreamReader(response.GetResponseStream());
        string  tmp = reader.ReadToEnd();
        response.Close();
        Response.Write(tmp);
    }
}

Protected Sub Page_Load(ByVal sender as Object,ByVal e as System.EventArgs)
    Dim uri as New Uri("http://www.microsoft.com/default.aspx");
    
    If(uri.Scheme == uri.UriSchemeHttp) Then
        Dim request as HttpWebRequest = HttpWebRequest.Create(uri)
        request.Method = WebRequestMethods.Http.Get
        Dim response As HttpWebResponse = request.GetResponse()
        Dim reader As New StreamReader(response.GetResponseStream())
        Dim tmp As String = reader.ReadToEnd()
        response.Close()
        Response.Write(tmp)
    End If
End Sub

If you try to run the code, you can see that all the HTML code from Microsoft site has been grabbed and display on your local web server.

2. Sample Code on how to Post Data to remote Web Page using HttpWebRequest

protected void Page_Load(object sender,EventArgs e) {
    Uri uri = new Uri("http://www.amazon.com/exec/obidos/search-handle-form/102-5194535-6807312");
    string data = "field-keywords=ASP.NET 2.0";
    if (uri.Scheme == Uri.UriSchemeHttp)
    {
        HttpWebRequest request = (HttpWebRequest)HttpWebRequest.Create(uri);
        request.Method = WebRequestMethods.Http.Post;
        request.ContentLength = data.Length;
        request.ContentType = "application/x-www-form-urlencoded";

        StreamWriter writer = new StreamWriter(request.GetRequestStream());
        writer.Write(data);
        writer.Close();

        HttpWebResponse response = (HttpWebResponse)request.GetResponse();
        StreamReader reader = new StreamReader(response.GetResponseStream());
        string tmp = reader.ReadToEnd();
        response.Close();
        Response.Write(tmp);
    }
}

Protected Sub Page_Load(ByVal sender As Object, ByVal e As System.EventArgs) Handles Me.Load
        Dim uri As New Uri("http://www.amazon.com/exec/obidos/search-handle-form/102-5194535-6807312")
        Dim data As String = "field-keywords=ASP.NET 2.0"
        If uri.Scheme = uri.UriSchemeHttp Then
            Dim request As HttpWebRequest = HttpWebRequest.Create(uri)
            request.Method = WebRequestMethods.Http.Post
            request.ContentLength = data.Length
            request.ContentType = "application/x-www-form-urlencoded"

            Dim writer As New StreamWriter(request.GetRequestStream)
            writer.Write(data)
            writer.Close()

            Dim oResponse As HttpWebResponse = request.GetResponse()
            Dim reader As New StreamReader(oResponse.GetResponseStream())
            Dim tmp As String = reader.ReadToEnd()
            oResponse.Close()

            Response.Write(tmp)
        End If

    End Sub

Conclusion

Based on  the sample code above, you can see that it is quite simple to do Http Post and Http Get to remote Website by using two built in class HttpWebRequest and HttpWebResponse. There is another set of classes called FtpWebRequest and FtpWebResponse that allow you to do ftp post and get to remote ftp Server.

Other Related and Popular Articles :

Author Profile : Sanjay Shravan

Comments