Selenium's headless mode allows running browser tests without a visible browser window by configuring the browser driver with the headless option. This approach enables faster and more efficient test execution as rendering the browser window is unnecessary.

Earlier to execute your Selenium tests in headless mode, you had to utilize the ChromeOptions as shown below.

ChromeOptions options = new ChromeOptions();
options.setHeadless(true);

By using setHeadless() method, you will accomplish the same result.

Finally, you must supply the options as a parameter when creating an instance of the ChromeDriver.

ChromeOptions options = new ChromeOptions();
options.setHeadless(true);
WebDriver driver = new ChromeDriver(options);

The traditional --headless, and since version 96, Chrome has a new headless mode that allows users to get the full browser functionality (even run extensions). Between versions 96 to 108 it was --headless=chrome, after version 109 --headless=new.

Usage for Chrome 96 to Chrome 108:

options.addArguments("--headless=chrome") 

For Chrome 109 and above, the '--headless=new' flag will now allow you to get the full functionality of Chrome in the new headless mode, and you can even run extensions in it. (For Chrome versions 96 through 108, use --headless=chrome)

Using --headless=new should bring a better experience when using headless with Selenium.

Usage for Chrome 109 and above : -

Here's an example of how to enable headless mode in Java using Selenium and Chrome (version 109 and above):

import org.openqa.selenium.WebDriver;
import org.openqa.selenium.chrome.ChromeDriver;
import org.openqa.selenium.chrome.ChromeOptions;

public class HeadlessExampleSelenium {

    public static void main(String[] args) {
        ChromeOptions options = new ChromeOptions();
        options.addArguments("--headless=new")

        WebDriver driver = new ChromeDriver(options);

        // Perform your tests here

        driver.quit();
    }
}

In this example, we create a ChromeOptions instance and set the headless option by passing arguments '--headless=new' . Then, we create a new ChromeDriver instance using these options, and our tests will run in headless mode without showing a visible browser window.
Finally, we quit the driver to clean up the resources used.

Please refer these below links for more details -

The Chromium developers recently added a 2nd headless mode (in 2021). See https://bugs.chromium.org/p/chromium/issues/detail?id=706008#c36

They later renamed the option in 2023 for Chrome 109 -> https://github.com/chromium/chromium/commit/e9c516118e2e1923757ecb13e6d9...
sourceStackoverflow