Using Microsoft's XMLHTTP Object to Get Data From Other Web Pages

PART-I
There are several popular commercial COM components that retrieve data from other web sites via HTTP requests, such as AspTear and AspHTTP. There is another lesser know option from Microsoft, however. Microsoft's suite of XML DOM (Document Object Model) components includes the XMLHTTP object. This object was originally designed to provide client-side access to XML documents on remote servers through the HTTP protocol. It exposes a simple API which allows you to send requests (yes, even POSTS) and get the resultant XML, HTML or binary data!

(There is one side note at the time of this writing: Microsoft's MSXML 3.0 has just been released. It contains a server-side specific version of the XMLHTTP component for use in web or application servers. Therefore, unless you download v3.0 you are going to have to use the original version, which shouldn't be a problem for most applications.)

Getting Started
If you've been keeping up with versions of Internet Explorer, you probably have some version of MSXML installed. If you need to install it, you can either install the latest version of MDAC or the MSXML component itself. If you don't have it installed, you may get the error: Invalid ProgID when you execute Server.CreateObject("Microsoft.XMLHTTP").

The most basic function of XMLHTTP is simply returning the entire text of a web page, here is the syntax:

<%
  Response.Buffer = True
  Dim objXMLHTTP, xml

  ' Create an xmlhttp object:
  Set xml = Server.CreateObject("Microsoft.XMLHTTP")
  ' Or, for version 3.0 of XMLHTTP, use:
  ' Set xml = Server.CreateObject("MSXML2.ServerXMLHTTP")

  ' Opens the connection to the remote server.
  xml.Open "GET", "http://www.4Guysfromrolla.com/", False
	
  ' Actually Sends the request and returns the data:
  xml.Send

  'Display the HTML both as HTML and as text
  Response.Write "<h1>The HTML text</h1><xmp>"
  Response.Write xml.responseText
  Response.Write "</xmp><p><hr><p><h1>The HTML Output</h1>"

  Response.Write xml.responseText
 
  
  Set xml = Nothing
%>

[View the live demo!]

This code will write the entire 4GuysFromRolla.com default page to the browser that requests it. The responseText property contains a the entire text of the page returned as a string.

Getting Binary Data
It's extremely simple to return binary data using the XMLHTTP object, the syntax is exactly the same, but you return the respondsBody property, which is an array of unsigned bytes:

<%
  Response.Buffer = True
  Dim objXMLHTTP, xml
  Set xml = Server.CreateObject("Microsoft.XMLHTTP")
	
  xml.Open "GET", _
      "http://www.4guysfromrolla.com/webtech/code/mitchell-pres.zip", _
      False

  xml.Send
	
  ' Add a header to give it a file name:
  Response.AddHeader "Content-Disposition", _
                     "attachment;filename=mitchell-pres.zip"
	
  ' Specify the content type to tell the browser what to do:
  Response.ContentType = "application/zip"
	
  ' Binarywrite the bytes to the browser
  Response.BinaryWrite xml.responseBody

  Set xml = Nothing
%>

PART-II
In PART-I we examined how to use XMLHTTP to get simple HTML and binary data. In this part we'll look at how to POST data and how to request secured pages via XMLHTTP!

Posting Data
It's also very easy to do POST requests to remote web sites. This example searches the Internet Movie Database for "The Usual Suspects".

<%
  Response.Buffer = True
  Dim objXMLHTTP, xml
  Set xml = Server.CreateObject("Microsoft.XMLHTTP")
	
  ' Notice the two changes in the next two lines:
  xml.Open "POST", "http://www.imdb.com/Find", False
  xml.Send "select=All&for=The Usual Suspects"
	
  Response.Write xml.responseText

  Set xml = Nothing
%>

[View the live demo!]

The Open method takes the argument POST instead of GET and the Send method contains the POST data. POST data has exactly the same syntax as query string data, only you don't need to include the ? at the beginning of the string. The only way to know what to post is to examine the form doing the posting, and see what the elements are and what sort of values they submit.

Accessing Secure Sites
XMLHttp can also access data that requires authentication. The Open method also accepts two optional arguments for a user name and password, as seen in the example below. Note that this user name and password only applies to web sites that request credentials (using the WWW-Authenticate header) like IIS does when you have Basic and/or challenge-response authentication enabled. For sites that use a database-style authentication (like Hotmail or Yahoo mail) you should simply post the user name and password like you would post any other data.

<%
  Response.Buffer = True
  Dim objXMLHTTP, xml

  Set xml = Server.CreateObject("Microsoft.XMLHTTP")

  xml.Open "GET", "http://www.4Guysfromrolla.com/", False, _
           "Richard", "Welcome"
	
  ' Actually Sends the request and returns the data:
  xml.Send

  Response.Write xml.responseText

  Set xml = Nothing
%>

There are other methods and properties of the Microsoft XMLHttp component, but the vast majority of uses only requires the ones discussed in the examples above.

Happy Programming!

By Richard Lowe
Richard Lowe works as a Development Services consultant for Spherion in the Technology Group