Allure Report using Annotations

Allure Reporting framework works fine with any test framework like TestNG, JUnit etc. Below example covers the implementation of Allure Reports in Selenium using TestNG, Java and Maven.

Let us first try to understand allure annotations before using them in our selenium example.

@Epic
@Features
@Stories/@Story
@Severity(SeverityLevel.BLOCKER)
@Description("In this cool test we will check cool thing")
@Step
@Attachment
@Link

These annotations are used in the same way like we use TestNG annotations or JUnit annotations.

@Severity In allure, we can define any @Test with @Severity annotation with any of these values like BLOCKER, CRITICAL, NORMAL, MINOR, TRIVIAL. By looking at this, we can understand the Severity of test if Failed.

 @Severity(SeverityLevel.BLOCKER)

@Description("What a test does to be included here")
We can add detailed description for each test method/class using @Description annotation

Allure severity annotation example

@Step - We can annotate any method with any visibility modifier (public, private, protected) with @Step annotation. Ex - @Step("Entering {0} and {1}")

In our example, steps are defined at our page classes which has logic for tests. We can also pass parameters with {} syntax. For example login(username, password), {0} is first parameter – username, {1} is second parameter – password.

Allure annotations step code with parameters

Allure annotations step with parameters

@Epic - Epics are useful as placeholders for large requirements for your product or project. Epic will be divided into smaller user stories at the appropriate time.

A User Story can be split to smaller Tasks, and can be part of some larger Feature and Epic.

In agile, tests can also be classified by Epic, stories and features. If you’re using this then you can annotate your test with @Epic, @Stories and @Features annotations.

For a better understanding of the difference between epic, feature and user stories, we have shown you a below picture with each activity in Agile.

Allure annotations agile

@Attachment - An attachment is simply a method annotated with @Attachment that returns either a String or byte[], which should be added to the report. We can have failure screenshot as attachment.

Allure attachment failure screenshot

Allure report screenshot attachment

@Links - We can link tests to some resources such as TMS (test management system) or bug tracker. This always helpful to link test cases to the test method.

Let us look at below allure report example along with allure annotations using Selenium, TestNG and Maven.

Step 1 : Create simple maven project and add dependencies to your pom.xml like below :-

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
  <modelVersion>4.0.0</modelVersion>
  <groupId>com.test</groupId>
  <artifactId>AllureExample</artifactId>
  <version>0.0.1-SNAPSHOT</version>
  
    <properties>
		<jre.level>1.8</jre.level>
		<jdk.level>1.8</jdk.level>
		<aspectj.version>1.9.2</aspectj.version>
	</properties>

	<build>
		<plugins>
			<!-- Compiler plug-in -->
			<plugin>
				<groupId>org.apache.maven.plugins</groupId>
				<artifactId>maven-compiler-plugin</artifactId>
				<version>3.5.1</version>
				<configuration>
					<source>${jdk.level}</source>
					<target>${jdk.level}</target>
				</configuration>
			</plugin>
			<!-- Added Surefire Plugin configuration to execute tests -->
			<plugin>
				<groupId>org.apache.maven.plugins</groupId>
				<artifactId>maven-surefire-plugin</artifactId>
				<version>2.19.1</version>
				<configuration>
					<testFailureIgnore>true</testFailureIgnore>
					<argLine>
						-javaagent:"${settings.localRepository}/org/aspectj/aspectjweaver/${aspectj.version}/aspectjweaver-${aspectj.version}.jar"
					</argLine>
					<suiteXmlFiles>
						<suiteXmlFile>testng.xml</suiteXmlFile>
					</suiteXmlFiles>
				</configuration>
				<dependencies>
					<dependency>
						<groupId>org.aspectj</groupId>
						<artifactId>aspectjweaver</artifactId>
						<version>${aspectj.version}</version>
					</dependency>
				</dependencies>
			</plugin>
		</plugins>
	</build>
	
		<dependencies>
		<!-- https://mvnrepository.com/artifact/org.testng/testng -->
		<dependency>
			<groupId>org.testng</groupId>
			<artifactId>testng</artifactId>
			<version>6.14.3</version>
		</dependency>
		<!-- https://mvnrepository.com/artifact/io.qameta.allure/allure-testng -->
		<dependency>
			<groupId>io.qameta.allure</groupId>
			<artifactId>allure-testng</artifactId>
			<version>2.8.1</version>
		</dependency>
		<dependency>
			<groupId>org.seleniumhq.selenium</groupId>
			<artifactId>selenium-server</artifactId>
			<version>3.14.0</version>
		</dependency>

		<dependency>
			<groupId>io.github.bonigarcia</groupId>
			<artifactId>webdrivermanager</artifactId>
			<version>3.0.0</version>
		</dependency>
	</dependencies>

	<reporting>
		<excludeDefaults>true</excludeDefaults>
		<plugins>
			<plugin>
				<groupId>io.qameta.allure</groupId>
				<artifactId>allure-maven</artifactId>
				<version>2.9</version>
			</plugin>
		</plugins>
	</reporting>
  
</project>

Step 2 : - Create Basepage.java and add code below :

package com.pages;

import org.openqa.selenium.By;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.WebElement;

import io.qameta.allure.Step;

public class BasePage {
	protected WebDriver driver;
	private By signInButton = By.linkText("Sign in");

	public BasePage(WebDriver driver) {
		this.driver = driver;
	}

	@Step("Clicking on Sign in button")
	public SignInPage clickSignInBtn() {
		System.out.println("clicking on sign in button");
		WebElement signInBtnElement = driver.findElement(signInButton);
		if (signInBtnElement.isDisplayed() || signInBtnElement.isEnabled())
			signInBtnElement.click();
		else
			System.out.println("Element not found");
		return new SignInPage(driver);
	}

	public void clickImagesLink() {
		// It should have a logic to click on images link
		// And it should navigate to google images page
	}

	@Step("Get page Title")
	public String getPageTitle() {
		String title = driver.getTitle();
		return title;
	}

	@Step("Verify base page Title {0}")
	public boolean verifyBasePageTitle(String title) {
		return getPageTitle().contains(title);
	}
}

Step 3 : - Now create CreateAccountPage.java and add below code : -

package com.pages;

import org.openqa.selenium.By;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.WebElement;

public class CreateAccountPage {

	private WebDriver driver;
	private By headerPageTxt = By.cssSelector(".signuponepage.main.content.clearfix>h1");

	public CreateAccountPage(WebDriver driver) {
		this.driver = driver;
	}

	public String getPageTitle() {
		String title = driver.getTitle();
		return title;
	}

	public boolean verifyPageTitle() {
		String pageTitle = "Create your Google Account";
		return getPageTitle().contains(pageTitle);
	}

	public boolean verifyCreateAccountPageText() {
		WebElement element = driver.findElement(headerPageTxt);
		String pageText = "Create your Google Account";
		return element.getText().contains(pageText);
	}

	public void createAccount() {
		// need to write steps for creating an account
	}
}

Step 4 : - Now Create an other class SignInPage.java and add below code : -

package com.pages;

import org.openqa.selenium.By;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.WebElement;

import io.qameta.allure.Step;

public class SignInPage {

	private WebDriver driver;

	private By headerPageSubText = By.id("headingSubtext");
	private By createAccountLink = By.cssSelector(".daaWTb content[class='CwaK9'] span.RveJvd");
	private By emailTextBox = By.id("identifierId");
	private By passwordTextBox = By.name("password");
	private By nextBtn = By.id("identifierNext");
	private By loginBtn = By.id("signIn");
	private By errorMsgTxt = By.id("errormsg_0_Passwd");
	private By headText = By.id("headingText");

	public SignInPage(WebDriver driver) {
		this.driver = driver;
	}

	public String getSignInPageTitle() {
		String pageTitle = driver.getTitle();
		return pageTitle;
	}

	public boolean verifySignInPageTitle() {
		String expectedTitle = "Sign in - Google Accounts";
		return getSignInPageTitle().contains(expectedTitle);
	}

	public boolean verifySignInPageText() {
		WebElement headtext = driver.findElement(headText);
		String text1 = headtext.getText();
		
		WebElement subText = driver.findElement(headerPageSubText);
		String pageSubText = subText.getText();
		
		String conpHeaderText = text1 + pageSubText;
		String expectedPageText = "Sign in with your Google Account";
		return conpHeaderText.equalsIgnoreCase(expectedPageText);
	}

	public CreateAccountPage clickonCreateAnAccount() {
		WebElement element = driver.findElement(createAccountLink);
		if (element.isDisplayed() || element.isEnabled())
			element.click();
		return new CreateAccountPage(driver);
	}

	@Step("Verify sign with uname {0 and password {1}")
	public boolean verifySignIn(String uname, String pwd) {
		enterUserName(uname);
		enterPassword(pwd);
		clickOnSignIn();
		return getErrorMessage().contains("incorrect");
	}

	@Step("Enter username {0}")
	public void enterUserName(String userName) {
		WebElement emailTxtBox = driver.findElement(emailTextBox);
		if (emailTxtBox.isDisplayed())
			emailTxtBox.sendKeys(userName);
	}

	@Step("Enter password {0}")
	public void enterPassword(String password) {
		WebElement passwordTxtBox = driver.findElement(passwordTextBox);
		if (passwordTxtBox.isDisplayed())
			passwordTxtBox.sendKeys(password);
	}

	public void clickNextBtn() {
		WebElement nxtBtn = driver.findElement(nextBtn);
		nxtBtn.click();
	}

	public void clickOnSignIn() {
		WebElement signInBtn = driver.findElement(loginBtn);
		if (signInBtn.isDisplayed())
			signInBtn.click();
	}

	public String getErrorMessage() {
		String strErrorMsg = null;
		WebElement errorMsg = driver.findElement(errorMsgTxt);
		if (errorMsg.isDisplayed() && errorMsg.isEnabled())
			strErrorMsg = errorMsg.getText();
		return strErrorMsg;
	}
}

Step 5 : - We will now create test classes like below :-

First create TestBase.java to initiate our browser. It can be chrome/firefox browser. We will pass the parameter from testng.xml file.

package com.test;

import org.openqa.selenium.WebDriver;
import org.openqa.selenium.chrome.ChromeDriver;
import org.openqa.selenium.firefox.FirefoxDriver;
import org.testng.annotations.AfterClass;
import org.testng.annotations.BeforeClass;
import org.testng.annotations.Parameters;

import io.github.bonigarcia.wdm.WebDriverManager;

public class TestBase {

	private static WebDriver driver;

	public WebDriver getDriver() {
		return driver;
	}

	private void setDriver(String browserType, String appURL) {
		switch (browserType) {
		case "chrome":
			driver = initChromeDriver(appURL);
			break;
		case "firefox":
			driver = initFirefoxDriver(appURL);
			break;
		default:
			System.out.println("browser : " + browserType + " is invalid, Launching Firefox as browser of choice..");
			driver = initFirefoxDriver(appURL);
		}
	}

	private static WebDriver initChromeDriver(String appURL) {
		System.out.println("Launching google chrome with new profile..");
		WebDriverManager.chromedriver().setup();
		driver = new ChromeDriver();
		driver.manage().window().maximize();
		driver.navigate().to(appURL);
		return driver;
	}

	private static WebDriver initFirefoxDriver(String appURL) {
		System.out.println("Launching Firefox browser..");
		WebDriverManager.firefoxdriver().setup();
		driver = new FirefoxDriver();
		driver.navigate().to(appURL);
		return driver;
	}

	@Parameters({ "browserType", "appURL" })
	@BeforeClass
	public void initializeTestBaseSetup(String browserType, String appURL) {
		try {
			setDriver(browserType, appURL);

		} catch (Exception e) {
			System.out.println("Error....." + e.getStackTrace());
		}
	}

	@AfterClass
	public void tearDown() {
		driver.quit();
	}
}

Step 6 : - Now create BasePageTest.java and add below code : -

package com.test;

import org.openqa.selenium.WebDriver;
import org.testng.Assert;
import org.testng.annotations.BeforeClass;
import org.testng.annotations.Test;

import com.pages.BasePage;

import io.qameta.allure.Description;
import io.qameta.allure.Severity;
import io.qameta.allure.SeverityLevel;
import io.qameta.allure.Story;


public class BasePageTest extends TestBase {

	private WebDriver driver;

	@BeforeClass
	public void setUp() {
		driver = getDriver();
	}

	@Severity(SeverityLevel.NORMAL)
	@Description("Verify home page title")
	@Story("Test verify base page title")
	@Test
	public void verifyHomePage() {
		System.out.println("Home page test...");
		BasePage basePage = new BasePage(driver);
		Assert.assertTrue(basePage.verifyBasePageTitle("Google"), "Home page title doesn't match");
	}

	@Severity(SeverityLevel.TRIVIAL)
	@Description("Verify home page test 1")
	@Story("Test verify base Test1")
	@Test
	public void baseTest1() {

		System.out.println("Hey im in test1 test");
	
	}

	@Severity(SeverityLevel.TRIVIAL)
	@Description("Verify home page test 2")
	@Story("Test verify base Test")
	@Test
	public void baseTest2() throws InterruptedException {
		Assert.assertTrue(true, "Simple test ");
	}

	@Severity(SeverityLevel.TRIVIAL)
	@Description("Verify home page test 3")
	@Story("Test verify base Test")
	@Test
	public void baseTest3() {
		Assert.assertTrue(false, "This is expected failure");
	}

}

Step 7 : - Now create CreateAccountTest.java to verify create account feature and add below code : -

package com.test;

import org.openqa.selenium.WebDriver;
import org.testng.Assert;
import org.testng.annotations.BeforeClass;
import org.testng.annotations.Test;

import com.pages.BasePage;
import com.pages.CreateAccountPage;
import com.pages.SignInPage;

import io.qameta.allure.Description;
import io.qameta.allure.Severity;
import io.qameta.allure.SeverityLevel;
import io.qameta.allure.Story;

public class CreateAccountTest extends TestBase {
	
	private WebDriver driver;
	private SignInPage signInPage;
	private BasePage basePage;
	private CreateAccountPage createAccountPage;

	@BeforeClass
	public void setUp() {
		driver = getDriver();
	}

	@Severity(SeverityLevel.BLOCKER)
    @Description("Create account test")
	@Story("Test create account verify")
	@Test
	public void verifyCreateAnAccountPage() {
		System.out.println("Create An Account page test...");
		basePage = new BasePage(driver);
		signInPage = basePage.clickSignInBtn();
		createAccountPage = signInPage.clickonCreateAnAccount();
		Assert.assertTrue(createAccountPage.verifyPageTitle(), "Page title not matching");
		Assert.assertTrue(createAccountPage.verifyCreateAccountPageText(), "Page text not matching");
	}

	@Severity(SeverityLevel.NORMAL)
    @Description("Create account test")
	@Story("Test create account test1")
	@Test
	public void createAccountExample1() {
		Assert.assertTrue(false, "This is expected failure");
	}

	@Story("Test create account test2")
	@Test
	public void createAccountExample2() {
		Assert.assertTrue(false, "This is expected failure");
	}
}

Step 8 : - Create SignInPageTest.java and add below code : -

package com.test;

import org.openqa.selenium.WebDriver;
import org.testng.Assert;
import org.testng.annotations.BeforeClass;
import org.testng.annotations.Test;

import com.pages.BasePage;
import com.pages.SignInPage;

import io.qameta.allure.Description;
import io.qameta.allure.Story;
import io.qameta.allure.Severity;
import io.qameta.allure.SeverityLevel;

public class SignInPageTest extends TestBase {

	private WebDriver driver;
	private SignInPage signInPage;
	private BasePage basePage;

	@BeforeClass
	public void setUp() {
		//Get driver
		driver = getDriver();
	}

	@Severity(SeverityLevel.BLOCKER)
    @Description("Verify Sign In functionality")
	@Story("Test verify sign in")
	@Test
	public void verifySignInFunction() {
		System.out.println("Sign In functionality details...");
		basePage = new BasePage(driver);
		signInPage = basePage.clickSignInBtn();
		Assert.assertTrue(signInPage.verifySignInPageTitle(), "Sign In page title doesn't match");
		Assert.assertTrue(signInPage.verifySignInPageText(), "Page text not matching");
		Assert.assertTrue(signInPage.verifySignIn("usertest", "p123"), "Unable to sign in");
	}

	@Story("Test verify sign in test1")
	@Test
	public void SignInPageTestOne() {
		Assert.assertTrue(true, "This is simple test");
	}

	@Story("Test verify sign in test2")
	@Test
	public void SignInPageTestTwo() {
		Assert.assertTrue(false, "This is expected failure");
	}

	@Story("Test verify sign in test3")
	@Test
	public void SignInPageTestThree() {
		Assert.assertTrue(true, "This is not expected failure");
	}
}

Step 9 : - To execute the above tests, we need testng.xml file. Create testng.xml file and below code :-

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd" >
<suite name="Test Suite" verbose="2">

	<parameter name="browserType" value="chrome" />
	<parameter name="appURL" value="https://google.com" />
	<test name="User Tests">
		<classes>
			<class name="com.test.BasePageTest" />
			<class name="com.test.SignInPageTest" />
			<class name="com.test.CreateAccountTest" />
		</classes>
	</test>
</suite>

After adding the classes and testng.xml file, your project should look something like below :-

Allure maven project example

We should also take the screenshot on test failure and attach to allure report using @Attachment annotation . We use testng test listener to capture screenshots on failure.

	public void onTestFailure(ITestResult result) {
		System.out.println("*** Test execution " + result.getMethod().getMethodName() + " failed...");
		System.out.println(result.getMethod().getMethodName() + " failed!");

		ITestContext context = result.getTestContext();
		WebDriver driver = (WebDriver) context.getAttribute("driver");

		// attach screenshots to report
		saveFailureScreenShot(driver);
	}
	
	
	@Attachment
	public byte[] saveFailureScreenShot(WebDriver driver) {
		return ((TakesScreenshot)driver).getScreenshotAs(OutputType.BYTES);
	}

Once we run tests using mvn clean test, we can generate allure reports by running the below command as the XML files are generated during the maven build process.

To generate html report and automatically open it in a web browser, run the following command:

allure serve

The above open your report in default browser. You can also run allure generate to generate allure-report folder on your root project directory. But to open the html report, we need Apache/webserver.

The default page would be the 'Overview' page with dashboards and widgets.

1. Overview page has several default widgets representing basic characteristics of your project and test environment.

2. Categories - It shows total defects. If the test is assertion failure, it reports as 'Product defect' and if a test has failed because of some exception, it shows the report as 'Test defect'.

3. Suites - Displays information based on suite executed

4. Graphs - Results are represented using pie chart and bar charts.

5. Timeline - Displays timeline of executed tests

6. Behaviors - information on results aggregated according to stories and features.

7. Packages - information on test executed in the package

Allure report overview

Allure also shows historical data (trends) of the tests, But the right way to use history features is using one of Allure CI plugins like Jenkins/TeamCity/Bamboo.

Selenium Tutorials: 

Comments

Test body is missing for passed tests? Any idea on this?

Add new comment