Как выполнить массовую загрузку файлов в Powershell с URL-адресами с двумя переменными числами?
Мне нужно загрузить большое количество (сотни тысяч) файлов с веб-сайта http://www.siaapm.cultura.mg.gov.br/acervo/memorial_imprensa/, а шаблон URL — http://www.siaapm. .cultura.mg.gov.br/acervo/memorial_imprensa/1/1.jpg. Поэтому мне нужно массовое решение, которое включает два диапазона номеров: папка и файл. Как это можно сделать в Windows PowerShell?
Итак, вот что я попробовал:
Все решения, которые я нашел до сих пор, касались изменений только в имени файла, а не в папке.
Download Multiple Files from http using Powershell with proper names
I have searched for something similar and I keep running across the FTP download answers. This is helpful information, but ultimately proving to be difficult to translate. I have found a powershell script and it works, but I am wondering if it can be tweaked for my needs. I don’t have much experience with powershell scripting, but I’m trying to learn.
The need is this. I need to download and install a series of files to a remote machine, unattended. The files are distributed via email via tinyurls. I currently throw those into a .txt file, then have a powershell script read the list and download each file.
Requirements of the project and why I have turned to powershell (and not other utilities), is that these are very specialized machines. The only tools available are ones that are baked into Windows 7 embedded.
The difficulties I run into are: The files download one at the time. I would like to grab as many downloads at the same time that the web server will allow. (usually 6)
The current script creates file names based off the tinyurl. I need the actual file name from the webserver.
How to Download a File with PowerShell
PowerShell is a great scripting language to write all kinds of scripts. But did you know that you can also download a file with PowerShell? You can use PowerShell to download single or multiple files from the internet.
There are a couple of methods when it comes to downloading files with PowerShell. We can download files from any URL with PowerShell, local network shares, and from behind credential protected websites.
In this article, we are going to start with the most straightforward method to download a single file and we are also going to take a look at other (faster) methods to download a file with PowerShell.
Powershell Download File from URL
We are going to start with the most common way to download a file from an URL with PowerShell. For this, we will be using the Invoke-WebRequest cmdlet. To download a file we need to know the source URL and give up a destination for the file that we want to download.
The parameter -OutFile is required. You don’t need to enter the full path, but a file name is required.
Authentication with Invoke-WebRequest
Some online resources require you to log in before you can access/download the files. With the Invoke-WebRequest cmdlet, we can provide the credentials that are needed for downloading the files.
If you are creating a script that will need to run automatically, then you will need to store the credentials in the script itself. I recommend creating a secure string password and store it in a text file on the computer that is running the script. It still won’t be super secure, but it’s a little bit better than using a plaintext password in your script.
Download files faster with Start-BitsTransfer in PowerShell
The Invoke-WebRequest method is available in all PowerShell versions and can also be used on Linux machines. But the downside is that it’s a bit slow. With Invoke-WebRequest, the file is buffered in the memory first, before it’s written to the disk.
A faster and better way is to use the Start-BitsTransfer cmdlet in PowerShell. This cmdlet allows you to queue files, set priority (useful for bandwidth limitation), can run in the background and download multiple files asynchronous.
This is the most basic method of downloading a file with BitsTransfer, you only need a source and destination.
By default, the download jobs run in the foreground consuming the maximum bandwidth available. You can change this by setting the priority of the job:
- Foreground – Default
- High
- Normal
- Low
Only the idle network bandwidth is used when you set the priority to high, normal, or low.
Another option is to run the download job asynchronous, allowing you to start multiple download jobs at the same time. When you use this method, make sure that you complete the download job when it’s finish.
As you can see I have downloaded the same bin file as before. But if we look in the destination folder we only see a .tmp file.
You will need to run Complete-BitsTransfer to finish the download job.
Downloading Multiple Files with PowerShell
To download multiple files with PowerShell we first need to know which files are available. We can use the Invoke-WebRequest cmdlet first to get the content from the webpage.
If you take a look at the content of http://speed.transip.nl then you will see a list of binary files that we can download.
First, we are going to scrape the website
This will return not only the content of the webpage but also other properties, like Links and InputFields.
This is a pretty simple webpage, but let’s say we only want the files that start with the name “random”. We can filter the links with a simple like query and select only the href property from each link.
So we now have the links for all random binary files. All we need to do is download each one of them.
When you need to download multiple files it’s better to use the Start-BitsTransfer cmdlet. It allows you to download multiple files simultaneously in the background with the parameter -Asynchronous
Other advantages of the BitsTransfer cmdlet is it can handle connection interruptions and is aware of your network bandwidth usage.
We can start all the download jobs by using the parameter – Asynchronous . Without it, the BitsTransfer cmdlet downloads the first file completely before starting the next download while putting your script on hold in the meantime.
You can use the Get-BitTransfer cmdlet to show the progress of the download. If you want to stop the download job then use the Remove-BitTransfer cmdlet. You can stop a single job based on its JobId or all jobs with:
Complete the BitsTransfer download
It’s important to complete the transferred file when using BitsTransfer in combination with the Asynchronous parameter. When using Asynchronous it creates a temp file during the download process. But to actually use the file you will need to run the following cmdlet:
PowerShell Download file from Server
We won’t be using the Invoke-WebRequest to download files from a local network source, like a server or NAS, with PowerShell. Instead, we can simply use the Copy-Item cmd to download a file from a server.
The Copy-Items cmdlet takes a source and destination, just like the Invoke-WebRequest cmdlet.
If you want to know more about the Copy-Item cmdlet, then you should read this article where I explain more about the cmdlet and alternatives.
Powershell Download Zip File
The method to download zip files is pretty much the same as a normal file. But I wanted to show you how that downloads and extracts the zip file. This way you can immediately process the files inside the zip file without manual interaction.
I am going to use this sample csv on GitHub which we can download in a zip file. We have to set a destination for the zip file itself and a path where we want to extract the files to.
The Invoke-WebRequest downloads the zip file just like any other file.
The next step is to extract the zip file automatically in the desired location. For this we are going to use a COM object. With the COM object we can extract the zip file and copy the content to the desired location.
Wrapping Up
Downloading files with PowerShell is pretty easy when you have the exact URL of the source file. When you need to scrape a website first then it can be a little bit more work to set up properly.
Try to use the Start-BitsTransfer cmdlet for downloading files and set the priority to normal when using it in an autonouse script. BitsTransfer has more option when it comes to retries, resuming and bandwidth control then Invoke-WebRequest.
If you have any questions about how you can download a file with PowerShell, then drop a comment below.
The Overnight Admin
I’m a Windows PowerShell Enthusiast trying to spread all the great things PowerShell can do with the world!
Pages
- Home
- PowerShell Tips and tricks
- PowerShell Functions
- PowerShell Links
Monday, August 11, 2014
Download all images from a web page with PowerShell
Web Scraping also known as Screen Scraping, Web Data Extraction, or Web Harvesting is a technique used to extract large amounts of data from one or multiple web sites.
Most websites don’t offer the functionality to save the data from their site onto your computer. Typically the only option is to Right Click > Save As, which can become a very tedious task very quickly. Being able to scrap a site of its content could most certainly have it’s uses, such as; perhaps you want to download Wikipidia(which I heard is only 14GBs with no pictures), or your really into something like PowerShell, you could search Google for all images with powershell in the name and then download them to you computer.[Next Upcoming Post]
In the below function I scrape my web sites homepage for all of it’s images, this means that my computer will do a search of my homepage [http://www.matthewkerfooot.com] for all image files and then download them to my local machine.
The Invoke-WebRequest cmdlet used above will filter my webpage for all image files via select src . Which would give us a list of all of the image paths that we will be downloading later.
The only difference from web scraping and web browsing is that when you are scraping it is usually automated and you are also saving the data, not just viewing it.
function Get-WebPageImages
<
<#
.CREATED BY:
Matt Kerfoot
.CREATED ON:
08/11/2014
.Synopsis
Downloads all available images from the specified $URL (A mandatory Variable)
.DESCRIPTION
This funciton will download all images from a specific web page and save them to your desktop by default.
Requires PSv3+
.EXAMPLE
PS C:\> Get-WebPageImages -Url http://www.matthewkerfoot.com -outputpath c:\
#>
[ CmdletBinding () ]
Param ( [ Parameter (Mandatory = $false ,
ValueFromPipelineByPropertyName = $true ,
Position = 0 ) ]
$Url = «http://www.TheOvernightAdmin.com» ,
$OutputPath = «$env:USERPROFILE\Desktop\»
)
$iwr = Invoke-WebRequest -Uri $Url
$images = ( $iwr ) . Images | select src