Introduction
I will explain about the usage of HttpWebRequest and HttpWebResponse in this article. As you all probably know or heard about this class before. HttpWebRequest and HttpWebResponse class is inside the System.NET namespace, and this two classes is designed to communicate by using the Http Protocol
You can use this two classes to make requests to other Web Pages via HTTP and parse the resulting text to extract data. This is what we know as screen scraping.
In ASP world, you normally need to rely on third party components called ASPTear to grab the contents from other site. But now with the help of HttpWebRequest and HttpWebResponse, you can do that easily without have to invoke third party components.
Using HttpWebRequest and HttpWebResponse
In the code below, I provide very basic sample code on how to use HttpWebRequest and HttpWebResponse. In the first example I will list out the code on how to do screen scraping and the second example would be doing HttpPost data to another website
1. Sample Code on Grabbing Contents (Screen Scraping)
C#
protected void Page_Load(object sender,EventArgs e) {
Uri uri = new Uri("http://www.microsoft.com/default.aspx");if(uri.Scheme = Uri.UriSchemeHttp) {
HttpWebRequest request = HttpWebRequest.Create(uri);
request.Method = WebRequestMethods.Http.Get;
HttpWebResponse response = request.GetResponse();
StreamReader reader = new StreamReader(response.GetResponseStream());string tmp = reader.ReadToEnd();
response.Close();
Response.Write(tmp);
}
}
VB.NET
Protected Sub Page_Load(ByVal sender as Object,ByVal e as System.EventArgs)
Dim uri as New Uri("http://www.microsoft.com/default.aspx");If(uri.Scheme == uri.UriSchemeHttp) Then
Dim request as HttpWebRequest = HttpWebRequest.Create(uri)
request.Method = WebRequestMethods.Http.Get
Dim response As HttpWebResponse = request.GetResponse()
Dim reader As New StreamReader(response.GetResponseStream())
Dim tmp As String = reader.ReadToEnd()
response.Close()
Response.Write(tmp)
End If
End SubIf you try to run the code, you can see that all the HTML code from Microsoft site has been grabbed and display on your local web server.
2. Sample Code on how to Post Data to remote Web Page using HttpWebRequest
C# protected void Page_Load(object sender,EventArgs e) {
Uri uri = new Uri("http://www.amazon.com/exec/obidos/search-handle-form/102-5194535-6807312");string data = "field-keywords=ASP.NET 2.0";if (uri.Scheme == Uri.UriSchemeHttp)
{
HttpWebRequest request = (HttpWebRequest)HttpWebRequest.Create(uri);
request.Method = WebRequestMethods.Http.Post;
request.ContentLength = data.Length;
request.ContentType = "application/x-www-form-urlencoded";
StreamWriter writer = new StreamWriter(request.GetRequestStream());
writer.Write(data);
writer.Close();
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
StreamReader reader = new StreamReader(response.GetResponseStream());string tmp = reader.ReadToEnd();
response.Close();
Response.Write(tmp);
}
}
VB.NET Protected Sub Page_Load(ByVal sender As Object, ByVal e As System.EventArgs) Handles Me.LoadDim uri As New Uri("http://www.amazon.com/exec/obidos/search-handle-form/102-5194535-6807312")
Dim data As String = "field-keywords=ASP.NET 2.0"If uri.Scheme = uri.UriSchemeHttp Then
Dim request As HttpWebRequest = HttpWebRequest.Create(uri)
request.Method = WebRequestMethods.Http.Post
request.ContentLength = data.Length
request.ContentType = "application/x-www-form-urlencoded"
Dim writer As New StreamWriter(request.GetRequestStream)
writer.Write(data)
writer.Close()
Dim oResponse As HttpWebResponse = request.GetResponse()
Dim reader As New StreamReader(oResponse.GetResponseStream())
Dim tmp As String = reader.ReadToEnd()
oResponse.Close()
Response.Write(tmp)
End If
End SubConclusion
Based on the sample code above, you can see that it is quite simple to do Http Post and Http Get to remote Website by using two built in class HttpWebRequest and HttpWebResponse. There is another set of classes called FtpWebRequest and FtpWebResponse that allow you to do ftp post and get to remote ftp Server.