Skip to main content

VBA in Excel to automate IE for crawling a web page 


If you want to open a website and go through the results of a webpage using VBA 
you can achieve it by first including a reference to Microsoft HTML Object Library 
in your VBA editor.


The following snippet of code should be a good starting point of how you can achieve the same


Sub GoToWebSiteAndPlayAround()

Dim appIE As Object ' InternetExplorer.Application
Dim sURL As String

Application.ScreenUpdating = False
Set appIE = CreateObject("InternetExplorer.Application")

'URL with the search term 'Cancer' at Science
sURL = "http://www.google.com?q=vba" 'this URL to be replaced by your target web page

With appIE
    .navigate sURL
    ' uncomment the line below if you want to watch the code execute, or for debugging
    '.Visible = True
End With

' loop until the page finishes loading
Do While appIE.readyState <> 4
    DoEvents
Loop

'Get info from HTML by ID and Name
Dim outerDiv, innerSpan, requiredtext, HTMLDoc
Dim spanCollection, outerSpan
   Set HTMLDoc = appIE.document
   Set outerDiv = HTMLDoc.getElementById("outerDivClassName")
        Set spanCollection = outerDiv.getElementsByTagName("SPAN")
       
        For Each outerSpan In spanCollection
            If outerSpan.className = "outerSpanClassName" Then
                requiredtext = outerSpan.innerHTML
                Debug.Print requiredtext
            End If
        Next

Application.ScreenUpdating = True
appIE.Quit
End Sub
Labels:

Comments

  1. Jon DiorioFebruary 17, 2014 at 7:06 PM

    Hi Hitesh... what if you want to determine the final URL for the page?

    ReplyDelete
  2. hiteshgoldeneyeFebruary 17, 2014 at 9:10 PM

    Hi Jon,

    In case the url is redirected you can wait for the required url /web page title to appear. Please have a look at this link for more help http://vba-corner.livejournal.com/4623.html. Let me know if this resolves your requirement

    ReplyDelete

Post a Comment

Popular posts from this blog

Handy Tips on working on Microsoft Word or Outlook

I have complied a small set of handy tips for working on Microsoft Word or Microsoft Outlook – 1. To change case (toggle between upper case, lower case and camel case),  select the text and press Shift+F3 2. While editing documents, it is not uncommon to delete words, phrases, and the like. Different people take different approaches to the task. For instance, some people just select the text and press Delete, while others may simply hold down the Delete or Backspace keys until the unwanted characters disappear. If you are in the latter group, and you spend a lot of time pressing Delete or Backspace, you may be interested in a handy shortcut provided by Word. All you need to do is hold down the Ctrl key to speed up your editing tasks. Here is a list of what you can achieve with this Ctrl+ <-- (left arrow key)        Goto Previous word Ctrl+ --> (right arrow key)      Goto Next word Ctrl+Delete     ...
4 comments
Read more

Hyper-V VM 100% disk utilization

I was facing a persistent issue of 100% disk utilization on starting any VM in my Hyper-V. The below link helped me to resolve it https://social.technet.microsoft.com/Forums/office/en-US/2b184f22-ea8c-4bc3-9f64-8b46eeaeef61/hyperv-host-disk-usage-at-100?forum=w8itprovirt
Post a Comment
Read more

A simple customization in MPOS (Blank Operation) with AX7

Hi All, I recently had to add a Blank Operation to Modern POS (MPOS) to open a url from MPOS. Blank Operation as you may already be aware enable you to extend Microsoft Dynamics Retail for POS by adding custom logic that can be triggered from the Retail POS Register buttons. The way to implement Blank Operations in MPOS  is different from Enterprise POS as MPOS is a modern app as compared to EPOS which is a windows forms based app. So lets explore a very simple customization i.e. we want to open a URL on triggering a button from MPOS. 1. We would need to start with AX to add a button to the layout of MPOS. If you do not want to disturb the standard layouts its better to copy one of the existing layouts and then modify it using the designer. Please note the designer only opens in Internet Explorer so it will save you time by not trying to open it in other browsers e.g Chrome 2. Next we need to add this layout to the Store where we intend to use it...
Post a Comment
Read more