MSXML2.XMLHTTP.send hangs...

[shockley]

I'm using the procedure below in a loop to download thousands of files
everyday. Once in a while--say, 1 time in 10,000, the oXHTTP.send command
hangs. When this happens, the only thing to do is close the application
with ctrl/alt/del and start the
whole thing again. Is there no way around these manual restarts (other than
the very complex solution I've come up with using a scripting program to
monitor progress)--seems like there should be a built-in error mechanism for
when the command hangs?

Shockley



Sub saveImg(sURL, sPath)
Dim oXHTTP As New MSXML2.XMLHTTP
Dim oStream As New ADODB.Stream

oXHTTP.Open "GET", sURL, False
oXHTTP.send

oStream.Type = adTypeBinary
oStream.Open
oStream.Write oXHTTP.responseBody
oStream.SaveToFile sPath, adSaveCreateOverWrite
oStream.Close

Set oXHTTP = Nothing
Set oStream = Nothing

End Sub

[Rowland Shaw]

Post by shockley
I'm using the procedure below in a loop to download thousands of files
everyday. Once in a while--say, 1 time in 10,000, the oXHTTP.send command
hangs. When this happens, the only thing to do is close the application
with ctrl/alt/del and start the whole thing again.

The bug may be with what you're connecting *to*. Can you reproduce by just
requesting a straightforward HTML file? I've seen something like this
before, where the XMLHTTP object was deadlocking, but the fault was
identified (with much assistance from MS PSS) as actually lying in the
server code, and how it was closing it's connections.

Have you tried much packet sniffing or other diagnostics etc?

[ghanashyam]

shockley,

It seems the problem is with server side
implemntation.You may be do better diagnosis using the
ethernet sniffer for HTTP packets.
But the XmlHttp(MSXML2.dll) and ServerXmlHttp(MSXML3.dll)
are not good utility to use for HTTP post/get commands.
The reasons are as below..
1) XmlHttp(MSXML2.dll) is not serversafe,leaks memory and
handle if used in a server appln.
2) Both the objects are singlethreaded(Apartment),if you
are using these objects in a multithreaded environment
then all your HTTP requests are getting serialized.
3)These HTTP related objects are having huge dependency
i,e they need complete IE installation to work correctly.

It is better not to use this object in your design from
beginning.There are couple of Open source free software
libraries are available on net which you can use for HTTP
communication.

Ghanashyam

-----Original Message-----

Post by shockley
I'm using the procedure below in a loop to download

thousands of files

Post by shockley
everyday. Once in a while--say, 1 time in 10,000, the

oXHTTP.send command

Post by shockley
hangs. When this happens, the only thing to do is

close the application

Post by shockley
with ctrl/alt/del and start the whole thing again.

The bug may be with what you're connecting *to*. Can you

reproduce by just

requesting a straightforward HTML file? I've seen

something like this

before, where the XMLHTTP object was deadlocking, but

the fault was

identified (with much assistance from MS PSS) as

actually lying in the

server code, and how it was closing it's connections.
Have you tried much packet sniffing or other diagnostics

etc?

.

[shockley]

ghanashyam,

Thanks so much for the comments. It all sounds very interesting, but also
very much beyond my level of understanding. But I will keep these comments
in mind as I go forward and inevitably come to understand at least bits and
pieces. If you would, I'd appreciate a link or two to some of the open
source libraries you refer to.

Kind Regards,
Shockley

Post by ghanashyam
shockley,
It seems the problem is with server side
implemntation.You may be do better diagnosis using the
ethernet sniffer for HTTP packets.
But the XmlHttp(MSXML2.dll) and ServerXmlHttp(MSXML3.dll)
are not good utility to use for HTTP post/get commands.
The reasons are as below..
1) XmlHttp(MSXML2.dll) is not serversafe,leaks memory and
handle if used in a server appln.
2) Both the objects are singlethreaded(Apartment),if you
are using these objects in a multithreaded environment
then all your HTTP requests are getting serialized.
3)These HTTP related objects are having huge dependency
i,e they need complete IE installation to work correctly.
It is better not to use this object in your design from
beginning.There are couple of Open source free software
libraries are available on net which you can use for HTTP
communication.
Ghanashyam

-----Original Message-----

Post by shockley
I'm using the procedure below in a loop to download

thousands of files

Post by shockley
everyday. Once in a while--say, 1 time in 10,000, the

oXHTTP.send command

Post by shockley
hangs. When this happens, the only thing to do is

close the application

Post by shockley
with ctrl/alt/del and start the whole thing again.

The bug may be with what you're connecting *to*. Can you

reproduce by just

requesting a straightforward HTML file? I've seen

something like this

before, where the XMLHTTP object was deadlocking, but

the fault was

identified (with much assistance from MS PSS) as

actually lying in the

server code, and how it was closing it's connections.
Have you tried much packet sniffing or other diagnostics

etc?

.

[shockley]

Can you reproduce by just requesting a straightforward HTML file?<

Do you mean something like:

Dim ie As New InternetExplorer
ie.navigate sURL

No, I haven't tried it, and I wonder why not! I have no practical reason for
using the msxml routine, other than that it is the only one I have tried so
far. Yours seems like a good substitute if I can figure out the syntax to
open the file and save it. The above method, as with a method like
"Workbooks.Open" allows for error trapping and continuation of the
application in the case of the file not being found.

I also found this method on the MS website:

lResult = URLDownloadToFile _
(0, ms_URL, MyPath, 0, 0) _
'API Lib "urlmon"

It appears to include the capability to monitor progress of command
execution, so this may work for me. I'll need to find an API reference book
to understand the command syntax.

Have you tried much packet sniffing or other diagnostics etc?<

I think you presume too much computer know-how on my part!

Thanks for the suggestions!
Shockley

Post by shockley
I'm using the procedure below in a loop to download thousands of files
everyday. Once in a while--say, 1 time in 10,000, the oXHTTP.send command
hangs. When this happens, the only thing to do is close the application
with ctrl/alt/del and start the whole thing again.

The bug may be with what you're connecting *to*. Can you reproduce by just
requesting a straightforward HTML file? I've seen something like this
before, where the XMLHTTP object was deadlocking, but the fault was
identified (with much assistance from MS PSS) as actually lying in the
server code, and how it was closing it's connections.
Have you tried much packet sniffing or other diagnostics etc?

[Rowland Shaw]

Post by shockley

Can you reproduce by just requesting a straightforward HTML file?<

Dim ie As New InternetExplorer
ie.navigate sURL

No, I mean requesting a static HTML or XML file -- so the sURL passed into
your original function points at a static file that will not change.

What is the nature of the destination sURL in your example? are you
requesting a file generated dynamically (eg from CGI, ASP, ISAPI, etc)?

[shockley]

I mean requesting a static HTML or XML file <

No, haven't tried this. What would this show?

What is the nature of the destination sURL in your example? are you

requesting a file generated dynamically (eg from CGI, ASP, ISAPI, etc)?<

I don't believe you would call these dynamically generated files. I'm
getting stock charts (gif and png files) and stock options tables (html
files). They are updated once a minute or once every 5 minutes during
trading hours and then after market close they stay the same. Here are the 3
different sites and examples of the files I'm downloading:

options:
http://finance.yahoo.com/q/os?s=ENER&m=2003-10-17

png charts:
http://ichart.yahoo.com/z?s=MSFT&a=v&t=1d&z=l&q=b

gif charts:
http://chart.bigcharts.com/custom/alliance/chart.asp?symb=MSFT&time=1da&freq
=1min&style=340&size=4

Post by shockley

Can you reproduce by just requesting a straightforward HTML file?<

Dim ie As New InternetExplorer
ie.navigate sURL

No, I mean requesting a static HTML or XML file -- so the sURL passed into
your original function points at a static file that will not change.
What is the nature of the destination sURL in your example? are you
requesting a file generated dynamically (eg from CGI, ASP, ISAPI, etc)?

[Rowland Shaw]

Post by shockley

I mean requesting a static HTML or XML file <

No, haven't tried this. What would this show?

That the fault wasn't down to a fault in the way the remote end of the
operation closed the socket (which I've seen before)

Post by shockley

What is the nature of the destination sURL in your example? are you

requesting a file generated dynamically (eg from CGI, ASP, ISAPI, etc)?<
I don't believe you would call these dynamically generated files. I'm
getting stock charts (gif and png files) and stock options tables (html
files). They are updated once a minute or once every 5 minutes during
trading hours and then after market close they stay the same.

They are dynamic in the sense that they are generated on demand.

With such remote sources, I suspect that network traffic may cause the
occasional "blip" in connectivity -- it could be that causing your
problems.

It's nigh on impossible to make suggestions with such a wide and diverse
system, dealing with third party systems, in addition to your own.

[shockley]

Post by Rowland Shaw
That the fault wasn't down to a fault in the way the remote end of the

operation closed the socket (which I've seen before)<

I could try this-but my guess is that the problem is more likely network
traffic related, as you suggest, since some days are good and some days are
bad.

I will probably find a different type of command to download the files that
lets me monitor its success and to time-out as desired. With
"MSXML2.XMLHTTP.send" command, for instance, if I unplug the cable to my dsl
modem, will just hang, whereas "ie.Navigate" will return a "server not
found" page, which lets me decide what to do next.

I must say, though, that it amazes me that the internet works as well as it
does and that I'm able to reliably get so much information from the net on a
daily basis!

Many thanks for your help, and any further suggestions are appreciated.

Regards,
Shockley

Post by Rowland Shaw

Post by shockley

I mean requesting a static HTML or XML file <

No, haven't tried this. What would this show?

That the fault wasn't down to a fault in the way the remote end of the
operation closed the socket (which I've seen before)

Post by shockley

What is the nature of the destination sURL in your example? are you

requesting a file generated dynamically (eg from CGI, ASP, ISAPI, etc)?<
I don't believe you would call these dynamically generated files. I'm
getting stock charts (gif and png files) and stock options tables (html
files). They are updated once a minute or once every 5 minutes during
trading hours and then after market close they stay the same.

They are dynamic in the sense that they are generated on demand.
With such remote sources, I suspect that network traffic may cause the
occasional "blip" in connectivity -- it could be that causing your
problems.
It's nigh on impossible to make suggestions with such a wide and diverse
system, dealing with third party systems, in addition to your own.