K2HttpRequest: asynchronous HTTP request using IWebBrowser2

kiZZ kiZZ — Mon, 05 May 2008 15:34:01 +0000

In my previous battle against Internet Explorer add-on (BHO), I discovered some valuable (and scarce) resources to my aid. Pete‘s TinyBHO and CPeteHttpRequest probably saved the most of my day.

Then I realized a problem: for the browser add-on that I was writing, I would have to fetch a Web page from a server that requires user login. Under typical usage scenarios, the users are assumed to have logged in the server in a previous session, so the Web server will authorize the user from cookies. However, using WinHttp on which CPeteHttpRequest is based on, there seems to be no (simple) ways to obtain the user’s cookies and embed it in the HTTP request. I couldn’t find another piece that is as simplistic as CPeteHttpRequest and also able to deal with my problem at hand.

So I wrote one myself.

K2HttpRequest has the usage of:

1
2
3
4
5
6
7
8
9

K2HttpRequest request;

// Synchronous
wstring innerHTML = request.request ( L"http://www.google.com" );

// Asynchronous
void foo ( wstring innerHTML ){}
request.SetOnComplete ( foo );
request.request ( L"http://www.google.com", true );

K2HttpRequest uses IWebBrowser2 internally, which means it opens a new Internet Explorer instance under the hood and loads the Web page with it. Thus, the server will receive login cookies as if the user visited the Web page. It will fetch the Web page at the specified URL and return the HTML in the <body> element (I actually intended return the whole <html> element but I can’t find an easy way to do it, as IHTMLDocument2 doesn’t have get_innerHTML())

This is still a very preliminary version, and will probably improve as my COM/ATL skills improve some day.

Download K2HttpRequest(source code ~5KB, Visual C++ 2008)

(Update: The asynchronous part of this doesn’t work very well. I didn’t know COM enough to use Marshalling when I wrote this version. I’ll probably update this some day. If you want to get the asynchrounous part working, you’ll probably have to add in marshalling yourself :p)

making an Internet Explorer add-on

kiZZ kiZZ — Thu, 01 May 2008 13:37:02 +0000

I was recently doing a project to make an add-on for Internet Explorer. How do I put it? I must say that was an extremely frustrating experience. The documentation is severely lacking, if not non-existent. One page from MSDN (which states the psarray is a BSTR is even dead wrong).

I have never been more motivated to contribute to the scene by writing my own “Internet Explorer add-on writing survival guide” sort of thing, but then I realized Pete has beat me to it by opening an entire [bho-wiki][wiki].

Nevertheless, I’m writing this article so more people can find these scarce information on IE add-on. On the other hand, here are some thoughts I had when writing the IE add-on. If you’re about to write an IE add-on, these would probably be the “survival guides” you’re using (as there aren’t many).

1. An IE add-on is a Browser Helper Object (BHO)

To save you some time searching in vain, try using BHO in your Google searches. This would give you programming-related information instead of where to download the latest IE add-on product.

2. Use C#

C++ just plain sucks, in terms of writing an IE add-on. Don’t get me wrong here, the language is marvelous, but Microsoft’s support just seems broken and unfinished. The whole Win32 hacking, COM stuffs involved when using C++ just makes your life 10 times more difficult.

The interface for C# is much more usable, and you won’t have to deal with all the bullshits when programming in the outdated (and most probably unmaintained) Win32 API.

Another major issue with C++ is Visual C++ 2008 Express. It doesn’t come with ATL, which means a ridiculous amount of grunt work dealing with pointers, releasing and stuffs (as illustrated below, as I happened to use VC++ Express). VC# doesn’t have this problem and it even has a number of libraries built-in (such as regular expressions).

Example 1: getElementById from BHO

Using C++

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35

IHTMLCollection * elements;
HRESULT hr = document->get_all ( &elements );
assert ( SUCCEEDED ( hr ) );

long numElements;
hr = elements->get_length ( &numElements );
assert ( SUCCEEDED ( hr ) );

IHTMLElement * element;

for ( int i = 0; i < numElements; i++ )
{
VARIANT index;
index.vt = VT_I4;
index.intVal = i;

IDispatch * elementDispatch;

hr = elements->item ( index, index, &elementDispatch );
assert ( SUCCEEDED ( hr ) );

hr = elementDispatch->QueryInterface ( IID_IHTMLElement, (void **)&element );
assert ( SUCCEEDED ( hr ) );

elementDispatch->Release();

// After all this hard work, we have finally obtained our element!
// GetAttribute is a custom function, I didn't even bother to
// list out it's content here
if ( GetAttribute ( element, L"id" ) == id ) break;
else { element->Release(); element = NULL; }
}

elements->Release();
return element;

Using C#

1	document.getElementById(id);

Example 2: execute some Javascript

Using C++

1
2
3
4
5
6
7
8

BSTR code = SysAllocString ( L"alert ( 'Hello World' );" );
BSTR language = SysAllocString ( L"javascript" );
VARIANT result = {0};

window->execScript ( code, language, &result );

SysFreeString ( code );
SysFreeString ( language );

Using C#

1	window.execScript ( "alert ( 'Hello World' )", "javascript" );

3. Use jQuery for DOM manipulation

jQuery is a client side JavaScript library to simplify common tasks (the “write less, do more” library as they say). I found it delightfully pleasant to use. It has some powerful functions like CSS selectors that would be a pain in the ass to implement in BHO. With jQuery, you can do stuff like:

1
2
3
4
5

// Find all the <a> elements with class "bookmark" and
// target attribute "_blank" and hide
// them smoothly using a slow slide up animation

$('a.bookmark[@target="_blank"]').slideUp("slow");

To use jQuery, simply use execScript to load the jQuery file on document complete.

4. About XmlHttpRequest

Fetching external Web pages using client side JavaScript will give you a permission denied error because of the cross-domain policies of most modern browsers. XmlHttpRequest must be done in the BHO. Sven offers an example code to use IXmlHttpRequest, which is very valuable considering the lack of official documentation. Pete has rolled his own CPeteHttpRequest class using WinHttp.

(Strangely enough, both of the above are coded in C++. I have yet to come across an equally efficient way to do XmlHttpRequest in C#. Please feel free to leave a comment if you have good ideas)

the kiZZ notes » BHO

K2HttpRequest: asynchronous HTTP request using IWebBrowser2

making an Internet Explorer add-on