Installing Selenium WebDriver with Python and PyCharm From Scratch (on Windows)
Who wants to try some Automation? Who doesn’t?! As of today the Automation (Test Automation) is the key to success. So let me show you how to install Selenium with Python and PyCharm really quick.
Here’s my video guide for these who prefer to watch:
Here are setup instructions in text:
1. Installing Python
Go to python.org, click on “Downloads” and download the latest Python version for Windows (edit: Python 3.9 doesn’t support Windows 7, so get 3.8.x). Run the file after download is complete, check “Add Python to PATH” checkbox and click “Install Now”. “Setup was successful” message will confirm the end of installation.
2. Installing Selenium WebDriver
Go to the command line: open Start Menu, type “cmd” and hit Enter. Type “pip install -U selenium” command, press Enter. You should get “Successfully installed selenium” message shortly.
3. Getting Drivers
To make selenium control a browser you need a specific driver for that browser. Let’s download drivers for Google Chrome and Mozilla Firefox for example.
Go to selenium.dev, click on “Downloads”, scroll down till “Platforms Supported by Selenium” section and open “Browsers” menu. Click on “documentation” link under Firefox, then “geckodriver releases” link. Download a Windows 32-bit or 64-bit driver, depending on your system type (see the end of this story to learn how to find your OS type).
Go back to Selenium Downloads page and click on “documentation” link under Chrome. Open the “current stable release” link and download “chromedriver_win32.zip” file.
Extract both drivers to their own folder on drive C:\ (for example “c:\se”): that might save you a trouble dealing with PATH errors later.
4. Installing PyCharm
PyCharm is Integrated Development Environment (IDE), or, simply put, it’s the code editor which makes Python programming more comfortable and efficient.
Go to jetbrains.com and click on “Tools” menu item, then on “PyCharm”. Scroll down and click on “DOWNLOAD PYCHARM NOW” button. Click on “Download” button under “Community”. When download completes run the file, click “Next” button, then “Next” again. Check “Add launchers to the PATH” and “Create Associations” checkboxes (or check all checkboxes if you want). Click “Next”, “Next” once more. Reboot the computer after setup is done: this is necessary step to update system PATHS.
Run PyCharm: open Start Menu, type “pycharm” and press Enter. Go with default settings if you asked. When PyCharm is done loading click on “Create New Project”. Give a name to your project and click on “Create” button.
Now let’s add the selenium package to our PyCharm project. Click “File”, then “Settings”. In the Setting window choose your Project and click on “Project Interpreter”. After that click on “plus” icon on the right, type “selenium” into the search field. Choose “selenium” from the results list and click “Install Package” button. Wait until the package is installed, you’ll see a message when it’s ready.
5. Testing Selenium with Chrome and Firefox
First make sure that Chrome and Firefox are both updated to the last version.
In PyCharm right click your project, select “New” and click on “Python File”. Enter a name for the file and hit Enter. Paste the code from below, just edit the path to the Chrome Driver file. Open context menu (right click) and click on “Run”. Selenium should make Chrome browser open “yahoo.com” site.
Now try the following code to test Selenium on Firefox (remember to change the driver’s path):
That’s it! Hope your setup went smooth and pleasant. I wrote two examples of automating tasks with Selenium and Python here (need to be updated, yes):
Install browser drivers
Through WebDriver, Selenium supports all major browsers on the market such as Chrome/Chromium, Firefox, Internet Explorer, Edge, and Safari. Where possible, WebDriver drives the browser using the browser’s built-in support for automation.
Since all the driver implementations except for Internet Explorer are provided by the browser vendors themselves, they are not included in the standard Selenium distribution. This section explains the basic requirements for getting started with the different browsers.
Read about more advanced options for starting a driver in our driver configuration documentation.
Four Ways to Use Drivers
1. Selenium Manager (Beta)
Selenium Manager helps you to get a working environment to run Selenium out of the box. Beta 1 of Selenium Manager will configure the drivers for Chrome, Firefox, and Edge if they are not found on the PATH . No extra configuration is needed. Future releases of Selenium Manager will eventually even download browsers if necessary.
Read more at the blog announcement for Selenium Manager .
2. Driver Management Software
Most machines automatically update the browser, but the driver does not. To make sure you get the correct driver for your browser, there are many third party libraries to assist you.
Install a Selenium library
First you need to install the Selenium bindings for your automation project. The installation process for libraries depends on the language you choose to use. Make sure you check the Selenium downloads page to make sure you are using the latest version.
Requirements by language
View the minimum supported Java version here.
Installation of Selenium libraries for Java is accomplished using a build tool.
Maven
Specify the dependency in the project’s pom.xml file:
Gradle
Specify the dependency in the project build.gradle file as testImplementation :
The minimum supported Python version for each Selenium version can be found in Supported Python Versions on PyPi
There are a couple different ways to install Selenium.
Download
Alternatively you can download the PyPI source archive (selenium-x.x.x.tar.gz) and install it using setup.py:
Require in project
To use it in a project, add it to the requirements.txt file:
A list of all supported frameworks for each version of Selenium is available on Nuget
There are a few options for installing Selenium.
Packet Manager
.NET CLI
CSProj
in the project’s csproj file, specify the dependency as a PackageReference in ItemGroup :
Additional considerations
Further items of note for using Visual Studio Code (vscode) and C#
Install the compatible .NET SDK as per the section above. Also install the vscode extensions (Ctrl-Shift-X) for C# and NuGet. Follow the instruction here to create and run the “Hello World” console project using C#. You may also create a NUnit starter project using the command line dotnet new NUnit . Make sure the file %appdata%\NuGet\nuget.config is configured properly as some developers reported that it will be empty due to some issues. If nuget.config is empty, or not configured properly, then .NET builds will fail for Selenium Projects. Add the following section to the file nuget.config if it is empty:
For more info about nuget.config click here. You may have to customize nuget.config to meet you needs.
Now, go back to vscode, press Ctrl-Shift-P, and type “NuGet Add Package”, and enter the required Selenium packages such as Selenium.WebDriver . Press Enter and select the version. Now you can use the examples in the documentation related to C# with vscode.
You can see the minimum required version of Ruby for any given Selenium version on rubygems.org
Selenium can be installed two different ways.
Install manually
Add to project’s gemfile
You can find the minimum required version of Node for any given version of Selenium in the Node Support Policy section on npmjs
Selenium Setup
This article contains notes on installing, coding, and running Selenium.
-
Run in Google Cloud a Docker image contaning Selenium and associated software. built using Ansible scripts that created the image.
Add Tesseract to extract text from pictures (OCR = Optical Character Recognition)
Some misconceptions
When we mention “Selenium” and “performance testing” in the same sentence, the first thing that most people say is “you can only run a few users on a machine”.
But I’m not talking about running several browser instances on each single machine.
I’m talking about running Selenium once while another program converts what goes back and forth over the network into a script.
The secret sauce is …
Running example
Selenium has no GUI. It runs as a console
However, reports are produced by TestNG, a plug-in to Selenium.
Docker images
Docker images containing Selenium server:
Ansible
Ansible task files to establish Selenium server:
- https://github.com/arknoll/ansible-role-selenium
- https://github.com/quarkslab/ansible-selenium-server
- https://mtlynch.io/testing-ansible-selenium/
Run using Maven after GitHub
- Install Maven.
- Install Selenium.
- Install the various browsers Selenium will control (Chrome, Firefox, etc.).
- Navigate to or create a folder to hold a new folder to be created by Git.
Clone from GitHub a repository containing sample tests:
git clone https://github.com/wilsonmar/Selenium-samples
cd Selenium-samples
Select a folder:
cd Python-soup
Look at the root layer of the repository. These files are there for use with Eclipse IDE:
- .classpath
- .project
- .settings
- .metadata
Some prefer to add them in .gitignore so they are not in the repo.
.gitignore .DS_Store
.DS_Store files should be ignored. They are created by MacOS. There is an entry for it in the .gitignore file so they are not stored in GitHub.
Maven
Invoke Maven to download dependencies and run Selenium:
mvn clean verify -Pbrowser-phantomjs
Maven creates a folder named target to receive downloads before starting Selenium.
verify -Pbrowser-phantomjs specifies use of the PhantomJS headless browser. Alternately, other browsers:
mvn clean verify -Pbrowser-chrome
mvn clean verify -Pbrowser-firefox
mvn clean verify -Pbrowser-edge
mvn clean verify -Pbrowser-internet-explorer
mvn clean verify -Pbrowser-opera
PROTIP: Several separate runs are needed to test on several browsers.
At the end of the run, you should see:
View Sample Selenium scripts
If you’re not using an IDE, use the Atom text editor to open a folder list:
atom .
View the pom.xml file.
The browsers handled by Selenium are identifined by a profile with a property within the pom.xml file which also specifies to Maven what dependencies to download and how to run Selenium.
PROTIP: If you work within enterprise firewalls, change the external URLs to internal ones, which may be managed within Nexus or Artifactory servers.
Dive into folder to view:
src/test/resources/webdrivermanager.properties .
Here is where Maven knows to download drivers.
PROTIP: The LATEST is specified. But a specific version would ensure that all drivers downloaded are the ones previously tested to work with each other. Specific versions can be specified with java invocation:
Dive into the src folder.
Notice there is a main and a test folder. Under each is a folder path:
java/selenium/utils
At the end of that path under main contains a TestUtils.java file which defines generic Java utility functions such as randomBetween, isDuplicatePresent, isAllEquals.
Selenium is all about testing, so the end of the path under test contains many more java files to control the browser.
test/java/selenium/configurations/TestConfig.java makes use of properties browser.name and base_url retrieved by variables in
main/java/selenium/configurations/TypedProperties.java .
PROTIP: Properties controlling a specific test are defined in properties files rather than hard-coded into code so that different properties can be used during a run by temporarily replacing the file.
SeleniumTestWrapper
Tests are driven by a wrapper which App-specific test code extend:
Look in SeleniumTestWrapper.java defined as a abstract class which are extended by other code.
Code in the file controls agent strings and cookies that browsers automatically send back to servers.
The code also manages the screen dimensions of the browser window.
These enable app-specific test code to focus on business. Under the Annotations folder are files that define code generation by the Java compiler.
Annotations
Each Java compiler annotation: @Before, @Rule, and @After is defined within code imported:
The annotation code imported add additional functionality such as logging. These decorators also provide metadata (data about data).
Instead of using Java inheritance, Java frameworks Spring and Hibernate use AOP (Aspect oriented programming) to provide a mechanism to inject code for preProcessing and postProcessing for an event. A “hook” in code before and after a method execution for consumer code in those places.
App-specific Tests
Edit the file defined to test an app:
The file contains annotations:
These annotations are defined by imports:
The earlier version, 2.53 and below, used these libraries:
The earlier 2.x code was:
QUESTION: How can we read the code in these annotation libraries?
Page Objects
The core driving code refers to definitions within the pageobjects folder:
The Page Object design pattern separates locator definitions in a separate java file so that various tests only need to refer to a reference rather than reduntantly specifying the way to locate objects on each page. This reduces maintenance over time as the website HTML changes. Change the locator technique in one place and all tests are good again.
The basic rule is that tests don’t declare variables (to manage state on their own), manipulate the DOM directly, nor create objects (using the “new” constructor keyword).
Webdriver Install
There are Web Driver programs for each combination of operating system (macOS, Windows, Linux, etc.), internet browser (Chrome, Firefox, etc.) and each version (78, 79, etc.). This means the Web Driver you install today would likely be obsolete when a new version of the browser is automatically installed on your machine.
PROTIP: There are two ways to install Selenium Web Driver. One is to manually install for whatever is your current version. The other is to use a Webdriver package manager.
Webdriver Manager
- Checks for the latest version of the WebDriver binary
- Downloads the WebDriver binary if it’s not present on your system
- Exports the required WebDriver Java environment variables needed by Selenium
Its automation makes use of Maven utility commonly used by Java.
The WebDriver can download files from an open source repository:
Selenium3Hello1
Obtain Selenium to test a sample call to Google Search:
Selenium3Hello1 is used to verify whether the Selenium core install works. It doesn’t use any browser driver.
Selenium3GoogleSearch1 works on multiple browsers.
Look into the folder:
NOTE: It is not best practice, but the sample scripts in my GitHub contains binary files copied from binary repositories.
File Sea.jpg is required only by the sample program from Neotys.
PROTIP: For team/production coding, place photos in a folder such as pics .
The file is placed at the project’s root folder to make it easy to specify its path.
File chromedriver.exe and other browser driver files are at the root of each script folder.
The more correct way is to specify the files (and their specific versions) in a pom.xml file that point to the location of those external dependencies, and then have each user of the repository to run Maven to obtain those files.
Nevertheless, the drivers are included so you can get going quickly.
Invoke sample command
On Windows:
It is assumed that the Java program is within the PATH which the operating system looks for executables.
View the sample command file Selenium3Chrome1.bat
PROTIP: Windows command prompt (cmd.exe) allows the umlaut ^ (Shift + 6) character to indicate line continuation.
Run the sample Selenium Java program on a Windows machine:
Selenium3Usahidi1NeoloadChrome1.bat
On Macs & Linux
View the sample command file Selenium3Chrome1.sh
Notice there is no “.exe” in the driver.
PROTIP: Bash shell scripts use the back-slash character (above the Enter/return key) to indicate line continuation.
chmod +X *.sh
Run the sample Selenium Java program:
Selenium3Usahidi1NeoloadChrome1.sh
Coding
Drivers for browsers go into the Reference folder.
Implicit waits. Don’t use them. Especially with explicit waits.
Cross platform
Windows vs. Mac vs. Linux
Cross browser
Obtain jars and drivers
There are the ways to assemble what Selenium needs:
a). Copy a working Selenium project (as shown above) that already has the files needed. Then edit it for your own uses. This is the quickest and simplest way.
b). Copy set of jars from an in-house binary respository (such as Nexus or Artifactory).
d). Manual download from websites, which you’ll need to do when a new version comes along.
e). Build from source.
PROTIP: Download all the latest install files at one sitting to test compatibility of the set. Assemble them together in a single folder for copying into each Selenium project folder so each can stand alone when distributed.
This explains the driver operation.
Maven
Manual Download
If you’re using a sample script, skip this.
PROTIP: Download files for all operating systems so the scripts begin as cross-system capable (works on Windows, Mac, Linux).
Under the “Selenium Standalone Server” section heading, click the link next to “Download version”:
Click Save of file named with the same version number, such as:
selenium-server-standalone-3.5.0.jar
to the Downloads folder.
Java bindings
Under the “Selenium Client & WebDriver Language Bindings” section heading, click the link click the Download link associated with the programming language you use, such as Java.
Click Save of file named with the same version number:
selenium-java-3.5.0.zip
to the Downloads folder.
Windows Internet Explorer driver
On Windows only, under “The Internet Explorer Driver Server” section heading, click the link 64-bit Windows IE
Click Save of file named with the same version number, such as:
IEDriverServer_x64_3.5.0.zip
to the Downloads folder.
Mac Safari WebDriver
This section is only applicable on an Apple Mac.
Under the “Safari” section heading in the Selenium webpage, the SafariDriver.safariextz file is deprecated. So no need to download it.
Starting with Safari 10 that comes with OS X El Capitan and macOS Sierra, in 2016, WebKit running Java 1.8+ supports the new W3C WebDriver browser automation API. So Selenium automatically launches the driver without further configuration.
However, it needs to be enabled because it’s off by default.
Click “Develop” on the menu bar and select “Allow Remote Automation”.
Authorize safaridriver to launch the webdriverd service which hosts the local web server.
Open a Terminal window to run:
cd /usr/bin
./safaridriver -p 5678
BLAH: The response asks for a port number.
Complete the authentication prompt.
To verify, run the Selenium script:
Eclipse IDE
-
Within Eclipse, right-click on your project name to select New, Folder.
Type in name “lib” (for library). Finish.
Server into lib folder
Click OK to the pop-up dialog for Copy files.
Selenium Java into lib folder
Within the Downloads folder, unzip by double-clicking on file
selenium-java-3.5.0.zip.
Dive into the folders to drag
selenium-java-3.5.0.jar
to the lib folder.
Click OK to the pop-up dialog for Copy files.
Eclipse folder
Click on the door icon and a Finder window opens up.
new References folder for Browser drivers
Select all the files.
NOTE: Previous versions did not include JUnit and others.
Choose Copy Files. OK.
Establish lib as References
To make Eclipse recognize the jar files:
Highlight the two files in the lib folder.
Click first file. Hold Cntrl while clicking the second file.
Right-click for Build Path. Add to Build Path.
A Referenced Libraries item should appear under Package Explorer.
Download Chrome driver
Open Chrome browser, finish work on all windows as you’ll need to relaunch by the time this is done.
Notice “Version 79.0.3945.79” or whatever it is what you do this.
Right-Click the file for your operating system, such as:
Click “Save” and wait for the download to finish.
Use your mouse to drag to move the chromedriver file to that folder.
Try your Selenium WebDriver code.
Use Gecko Firefox driver
The “Latest release” is shown at the top. Later releases are below.
Mozilla also calls it the “Marionette Proxy”.
Scroll down to the Downloads section and click on the file for your operating system.
geckodriver-v0.18.0-macos.tar.gz for Mac (1.31 MB)
geckodriver-v0.18.0-win64.zip for Windows
Unzip the macos file for the geckodriver executable.
Unzip the win64 file for the geckodriver.exe executable.
Restart your system and try your Selenium WebDriver code as shown in video.
Headless drivers
Ghost Driver or PhantomJS turns Selenium “headless”, accessing the DOM.
HTML Unit from https://selenium-release.storage.googleapis.com/index.html selenium-html-runner-3.5.0.jar
Junit into lib folder
Click the text to the right of “Looking for the latest version?”, such as Download junit-4.10.jar (253.2 kB).
This downloads file junit-4.10.jar (from 2011).
Drag from within the Downloads folder file
junit-4.10.jar
and drop it within the lib folder when the mouse turns into a “+” sign.
New Class
IntelliJ IDE
Java Coding
## TODO: Random number
Add-on functionality
TestNG has more in-built annotations than JUnit, making testing easier.
TestNG requires a download from http://testng.org/
Its @DataProvider and parameters enables data-driven testing.
JUnit does not generate a HTML reports. But TestNG generates an XSLT report.
- https://www.youtube.com/watch?v=OTtFSnZY4f8
VIDEO on emitting industry-standard logs.
VIDEO on Advanced topics.
- https://www.youtube.com/watch?v=0UQ9pAlY3qg
Read CSV files
To get your Selenium Java code to read CSV files:
Specify the dependency in Maven, Ant, etc. or
Download from https://mvnrepository.com/artifact/net.sf.opencsv/opencsv/
Add in your Java code:
Read Excel files
To get your Selenium Java code to read Excel files:
Download Apache POI — the Java library for Microsoft documents from
https://poi.apache.org/download.html
the binary distribution for the latest stable version, such as
poi-bin-3.16-20170419.tar.gz
Unzip using RAR so you don’t extract
Select the file under the HTTP heading.
Rock Stars
Simon Stewart (@sha98c, blog.rocketpoweredjetpants.com) invented WebDriver, now Lead Committer
- State of the Union at SeleniumConf Austin 5 April 2017. says the original vision was:
FIT Test > FIT Fixture > Page Objects > WebDriver
“Definitely do not use FIT”
“We test so that when we release software, we are confident it works, as early as possible.”
- Automation Best Practices 14 Jul 2016
Video Tutorials
- Selenium WebDriver Eclipse Java Project Setup: For the absolute beginner on Windows with Java 1.6 [13:58]
https://www.dropbox.com/s/inuirqwhlr3w7zf/slides.pdf?dl=0 Dave Hoeffer
CA DevTest
SauceLab Cloud
Open an account at SauceLab.com.
set environment variables SAUCE_USERNAME SAUCE_ACCESS_KEY
SELENIUM_BROWSER SELENIUM_VERSION SELENIUM_PLATFORM
Python Web scraping
This is based on Pluralsight’s 1h 7m video course “Scraping Dynamic Web Pages with Python and Selenium” by Pratheerth Padman covers use of Python invoked by Jupyter Notebook. Python has library Beautiful Soup (to scrape HTML and XML from web pages) and Selenium 2.0 WebDriver (to emulate keyboard and mouse movements based on JSON commands).
-
Install Selenium for Python3 (covered above)
Install Python with virtualvenv and Anaconda.
Download my repository containing Jupyter Notebooks:
Open a Terminal and navigate to that folder. See https://wilsonmar.github.io/jupyter
Call Selenium from Python
In Jupyter Notebook, run file demo_mod2.ipynb file which opens a hard-coded web page, then quit.
To open browser options and set arguments (as if manually in Settings”) before opening:
Run file demo_mod3 -2.ipynb file which manipulates iframes:
To handle popups:
Run file demo_mod4.ipynb in Jupyter to use the soup class to scrape quotes attributed to Soccer player Wayne Rooney from premierleague.com.
NOTE: There is an error in this script.
Other info
NOTE: All “Selenium3” require Java 1.8+.
PROTIP: A number is included with each component to provide for version control, since everything changes all the time in IT.