How-to Download a Webpage or a File Using Curl

A while back, I played around with curl to send e-mails:

and was surprised at how relatively easy it was to use.

I have previous posted a few ways to download content from the web, such as:

but decided to play around with curl on this since I knew that this was supposed to be another one of its strengths.

The Code

Below is the function I devised to either download whole webpages or individual files on a server.

'---------------------------------------------------------------------------------------
' Procedure : curl_downloadFile
' Author    : Daniel Pineault, CARDA Consultants Inc.
' Website   : http://www.cardaconsultants.com
' Purpose   : Download a file using curl, overwrites existing files without prompting!
' Copyright : The following is release as Attribution-ShareAlike 4.0 International
'             (CC BY-SA 4.0) - https://creativecommons.org/licenses/by-sa/4.0/
' Req'd Refs: None required
' References: Requires curl, but this is part of Windows 10, 11 or can be downloaded
'
' Input Variables:
' ~~~~~~~~~~~~~~~~
' sURL                  : URL of the file to download
' sOutputFolder         : Path where to save the file to
' sOutputFileName       : Filename to be given to the downloaded file
' bCreateFolderIfMissing: Should the folder be created if it does not already exist
' bOpenOutputFolder     : Should Windows Explorer be launch to the folder once the
'                         operation has completed
'
' Usage:
' ~~~~~~
' curl_downloadFile("https://www.google.com/someimage.jpg", "C:\Temp\", , True, True)
'
' curl_downloadFile("https://www.google.com", "C:\Temp\", "google.html", , True)
'
' Revision History:
' Rev       Date(yyyy-mm-dd)        Description
' **************************************************************************************
' 1         2024-08-22              Initial public release
'---------------------------------------------------------------------------------------
Function curl_downloadFile(ByVal sURL As String, _
                           ByVal sOutputFolder As String, _
                           Optional sOutputFileName As String, _
                           Optional bCreateFolderIfMissing As Boolean = False, _
                           Optional bOpenOutputFolder As Boolean = False) As Boolean
    On Error GoTo Error_Handler
    Dim sCmd                  As String
    Dim sDownloadedFile       As String

    If sURL = "" Then
        MsgBox "You must supply a valid URL of a file to download.", _
               vbCritical Or vbOKOnly, "Operation Aborted"
        GoTo Error_Handler_Exit
    End If
    If sOutputFolder = "" Then
        MsgBox "You must supply a valid Output Folder of a file to be downloaded to.", _
               vbCritical Or vbOKOnly, "Operation Aborted"
        GoTo Error_Handler_Exit
    Else
        If Not GA_FolderExist(sOutputFolder) Then
            If bCreateFolderIfMissing Then
                MyMkDir sOutputFolder 'Should use a recursive solution, just for illustrative purposes
            Else
                MsgBox "The specified Output Folder does not exist and must be created first or the command changed.", _
                       vbCritical Or vbOKOnly, "Operation Aborted"
                GoTo Error_Handler_Exit
            End If
        End If
    End If
    
    ' Build the output file folder/name
    ' **************************************************
    If sOutputFileName = "" Then
        sDownloadedFile = sOutputFolder & GetFileName(Replace(sURL, "/", "\"))
    Else
        sDownloadedFile = sOutputFolder & sOutputFileName
    End If
    
    ' Might want to check if the file exists before proceeding any further
    ' If GA_FileExist(sDownloadedFile) ...
    
    ' Build the curl command
    ' -m 5 => allow 5 seconds for the command to run before stopping
    ' **************************************************
    sCmd = "cd """ & sOutputFolder & """ && curl -m 5"
    
    If sOutputFileName = "" Then
           sCmd = sCmd & " -LO """ & sURL & """"
    Else
           sCmd = sCmd & " -L """ & sURL & """ -o """ & sOutputFolder & sOutputFileName & """"
    End If
    If bOpenOutputFolder Then
        'If bOpenOutputFolder Then sCmd = sCmd & " && start ."
        sCmd = sCmd & " && explorer /select, """ & sDownloadedFile & """"
    End If
    'Debug.Print sCmd
    ' Run the curl command
    ' **************************************************
    ' Production  => Hide window from user / Close cmd prompt once operation completed
    CreateObject("Wscript.Shell").Run "cmd /c " & sCmd, 0, True
    ' Development => Display window to user / Do not close cmd prompt
    'CreateObject("Wscript.Shell").Run "cmd /k " & sCmd, 1, True
    
    ' See if the file was successfully downloaded
    ' **************************************************
    If GA_FileExist(sDownloadedFile) Then curl_downloadFile = True

Error_Handler_Exit:
    On Error Resume Next
    Exit Function

Error_Handler:
    MsgBox "The following error has occurred" & vbCrLf & vbCrLf & _
           "Error Source: curl_downloadFile" & vbCrLf & _
           "Error Number: " & Err.Number & vbCrLf & _
           "Error Description: " & Err.Description & _
           Switch(Erl = 0, "", Erl <> 0, vbCrLf & "Line No: " & Erl) _
           , vbOKOnly + vbCritical, "An Error has Occurred!"
    Resume Error_Handler_Exit
End Function

 

Required Helper Functions

Now the above requires a few helper function for extract file names, validating file/folders exists and these can all be found on my blog by using the following links:

 

Usage Examples

The above it pretty much self-explanatory, but below are a couple quick examples of how it can be used.

Downloading an image

curl_downloadFile "https://www.google.com/images/branding/googlelogo/2x/googlelogo_light_color_272x92dp.png", "C:\Temp\", , ,True

Downloading an image and renaming it

curl_downloadFile "https://www.google.com/images/branding/googlelogo/2x/googlelogo_light_color_272x92dp.png", "C:\Temp\", "google.png", ,True

Downloading a Webpage

curl_downloadFile "https://www.google.com", "C:\Temp\", "google.html", , True

In all of the above examples, the last True input argument is to make VBA open Windows Explorer to show you the file. Per the function header, simply switch that to False should you not wish to have this behavior.
 

Conclusion

As you can see by reviewing the code, it is pretty simple and all come down to building a curl command similar to:

curl -m 5 -L "https://www.google.com/images/branding/googlelogo/2x/googlelogo_light_color_272x92dp.png" -o "C:\Temp\myPic.png"

The rest is all just checks, massaging and error handling.

The more I use curl, the more I like it. It is truly very powerful and it’s already part of Windows, so nothing to install. I hope you give it a whirl for yourself.