Web application crawl scripts

To ensure that DAST Essentials or Dynamic Analysis can scan all areas of the target URLs of your web application, you can create crawl scripts (login scripts) that the scanners use to access these areas. You record the script with the Selenium IDE, save it as a file, then upload the script file in the Veracode Platform or using the REST API.

Crawl scripts are not supported for analyses of API specifications or Postman Collections.

Upload a crawl script for Dynamic Analysis

Upload a recorded crawl script using Selenium commands to a Dynamic Analysis scan for a web application.

To upload a crawl script for a DAST Essentials scan of a web application, see Configure login scripts.

Before you begin:

• Ensure your script is an HTML or SIDE (JSON format) file, and that the file size does not exceed 5 MB.
• Review the Selenium script best practices.
• See the example Selenium scripts.

To complete this task:

1. In the Veracode Platform, go to the URL configuration page for the scan for which you want to provide your script.
2. In the Crawl Script section, select Use a crawl script to augment the scan of this URL.
3. To navigate to and select your crawl script file, select Choose File.
4. If you want to limit the scan to only what the crawl script dictates, select Scan only what is specified in the crawl script.
5. Select Save. The analysis runs the crawl script during prescan and returns information about any commands that might fail during the URL scan.

Supported Selenium commands

The following table lists the Selenium commands that Veracode supports in sequence scripts. You can upload these scripts to Dynamic Analysis or DAST Essentials for use in a web application scan. Your version of Selenium IDE might not support all listed commands. Review the best practices before running your scan.

For more information about these commands, see the Selenium documentation.

Command	Description
assertAlert, assertPrompt, or assertConfirmation	These commands have identical implementations. They switch focus to the current alert box on the page and determine if the user-provided text matches the alert text. If it does not match or there is no alert box on the page, these commands fail.
assertChecked	This command succeeds when you select the targeted checkbox or radio button and fails when when you do not select them.
assertElementPresent	This command always uses the timeout period of 30 seconds. If you cannot locate the targeted element within that amount of time, the command times out and fails.
assertText or verifyText	assertText and verifyText have identical implementations.
assertTextPresent or verifyTextPresent	Veracode checks the provided text against the HTML source of the currently rendered page, not the original source. Therefore, HTML elements you add to the page through JavaScript can be matched with this command only if the JavaScript code mutating the DOM finishes executing before this command is executed. This command requires the scan engine to fully serialize the page currently loaded in the browser. It can be extremely time-consuming for complex pages.
check or uncheck	check can operate on checkbox and radio button types of form inputs, but uncheck can only operate on checkboxes.
click or clickAndWait	These commands have identical implementations and functionality.
clickAt
deleteAllCookies
doubleClick or doubleClickAndWait	doubleClick and doubleClickAndWait have identical implementations and functionality.
fireEvent or focus	focus fails if the element to be focused is not focusable, such as an <img> or <br /> element. Elements that are not focusable might have focus events bound to them. In this case, you can use the fireEvent command to force an event that you want to fire to focus on elements that are not focusable.
keyUp
keyDown
keyPress
mouseDown or mouseUp	mouseDown and mouseUp are always fired together when the simulated mouse left button clicks on an element.
mouseOver or mouseMove	mouseOver and mouseMove are always fired together when the simulated mouse moves on an element.
mouseOut
open or close	If you must manage multiple windows, we recommend explicitly selecting them by name or title before attempting to execute any commands. If more than one window is still open after you close the originally selected window, selection of the new main window is unpredictable.
pause	There is no need for long, duplicated pause commands.
refresh
runScript
select
selectAndWait	This command fails if the targeted dropdown menu does not load within the timeout period, which is 60 seconds by default.
selectFrame
selectPopUp	When there is only one window currently open, this command fails because the only open window must be the main window.
selectWindow	If there is only one window currently open, this command always succeeds.
submit
type
typeKeys or sendKeys	typeKeys and sendKeys have identical implementations.
verifyHtmlSource	Veracode checks the provided text against the HTML source of the currently rendered page, not the original source. Therefore, HTML elements you add to the page through JavaScript can be matched with this command only if the JavaScript code mutating the DOM finishes executing before you execute the command. This command requires the scan engine to fully serialize the page currently loaded in the browser. This process can be extremely time-consuming for complex pages.
waitForElementToLoad, waitForTitle, waitForTextPresent, or waitForElementPresent	You can provide an optional timeout value in the value element of these Selenium IDE commands. If the value element does not have a value, the default timeout is 30 seconds. If the scan engine cannot locate the element within that amount of time, the command times out and fails.
waitForFrameToLoad	Support for this command only includes the name and ID string locators.When this command returns successfully, it automatically switches to the loaded frame, as if the selectFrame command was called. In place of this command, a generic pause is often a better solution.
waitForPageToLoad	You can provide an optional timeout period in the target element of the command. The default value is 30 seconds. The maximum value is 60 seconds. If the page does not load within the configured timeout period, the command fails.

Selenium script best practices

This section provides best practices for recording and optimizing your crawl scripts.

General

• To ensure that a webpage finishes loading before interacting with it, such as signing in, include a wait command, such as waitForPageToLoad, at the beginning of the script.
• Always include a command in the script that validates successful sign in to the application. For example, waitForElementVisible. See Targeting elements and the example scripts.
• To find text that is visible on a webpage before successfully signing in, include a text verification command, such as assertTextPresent or verifyTextPresent, at the end of the script.
• Where possible, avoid including pauses in the script.
• Where possible, ensure each target in the script refers to the ID parameters in the HTML of each webpage. For example, "target": "id=username". See Targeting elements and the example scripts.

See the example Selenium scripts.

When to use crawl scripts

You do not need a crawl script to perform a complete Dynamic Analysis or DAST Essentials scan of a web application. Create crawl scripts only if you need to access features or functionality in your web application that require specific inputs in a multistep process.

For example, a website might display different features depending on the region a user selects from a dropdown menu, or it might restrict access to certain features based on the data a user enters into a specific area of the application. You might also want to provide a crawl script to Dynamic Analysis or DAST Essentials if the results show that certain pages or functionality were not crawled.

A Dynamic Analysis or DAST Essentials scan can use heuristics to navigate an application. However, to fully understand the business logic around how data is exchanged between the database and the user interface, you must use crawl scripts or augment the analysis with Manual Penetration Testing.

Remove pre-populated values

• Cache and cookies: Always clear your browser cache and cookies. If Dynamic Analysis or DAST Essentials is scanning a web application for the first time, it cannot use any information saved in cache, cookies, or local storage.
• Input fields: if the browser autopopulates any of the input fields, such as a username, Selenium does not record them in the script.

Targeting elements

• Input fields: the scan engine typically runs events on an input field while filling in the field. Adding additional mouse selections to the script, such as onFocus and onBlur events, is unnecessary and might cause problems.
• Multiple methods: if a step fails when the script executes, use an alternate method to identify targets.
• Elements with long waits: any command that targets an element, such as a mouse click or type command, automatically has an implicit waitForElementPresent process that runs with a 30-second timeout. If an element takes longer than 30 seconds to load, you might need to add additional waits. We recommend adding these additional waits sparingly as they can increase your scan duration or cause other scan issues.

Recording sequence

When recording your script, we recommend you use the following sequence in this order:

1. Open the webpage.
2. Wait for the sign in page to load.
3. Enter the username and password. If the username and password fields are on different pages, after entering the username, include a step, such as selecting Next on the page with the username field, to go to the next page, then enter the password.
4. Select Submit.
5. Wait for the next page to load and validate that a specific element is visible.

Example Selenium scripts

The following Selenium scripts demonstrate how to record a script.

Use the script examples as templates

To use these scripts as a template:

1. Copy all JSON from the example.
2. Paste the JSON into a text file.
3. Save the text file with the .side extension.
4. Open the SIDE file in the Selenium IDE or a text editor.
5. Update the commands and values or add commands.

Example script 1

The following script demonstrates signing in to a web application where the username and password fields are on the same page.

The script commands run in the following order:

• Opens the URL for the web application.
• Waits for the sign-in page to load.
• Enters the username.
• Enters the password.
• Selects the sign-in button to sign in.
• Waits for an element to load, which indicates successful sign in.

{
  "id": "b66324ad-1350-4277-85f1-c48b3800a868",
  "version": "2.0",
  "name": "Login",
  "url": "<appURL>",
  "tests": [{
    "id": "577b18f4-5b2f-49e8-9fc3-e320cd8f00fa",
    "name": "Login",
    "commands": [{
      "id": "03d62bf6-8784-48cc-b319-30e1c809ab11",
      "comment": "",
      "command": "open",
      "target": "</pathToLogin>",
      "targets": [],
      "value": ""
    }, {
      "id": "81b9dc4d-7c78-49f9-bc4c-9e2186625169",
      "comment": "",
      "command": "waitForElementVisible",
      "target": "id=<usernameField>",
      "targets": [],
      "value": "30000"
    }, {
      "id": "53c5ed70-6548-4e39-b456-d0e5502945d4",
      "comment": "",
      "command": "type",
      "target": "id=<usernameField>",
      "targets": [],
      "value": "<username>"
    }, {
      "id": "1b55d2a4-e472-48e6-822d-a5b54a2986d9",
      "comment": "",
      "command": "type",
      "target": "id=<passwordField>",
      "targets": [],
      "value": "<password>"
    }, {
      "id": "da24efdf-cdd3-4858-b9f8-52e0666b6d73",
      "comment": "",
      "command": "click",
      "target": "id=<loginButton>",
      "targets": [],
      "value": ""
    }, {
      "id": "566c3651-194c-4621-bc87-7f5cb64356fe",
      "comment": "",
      "command": "waitForElementVisible",
      "target": "id=<elementOnSuccessfulLoginPage>",
      "targets": [],
      "value": "30000"
    }]
  }],
  "suites": [{
    "id": "e17f2325-4ac8-46d3-8f25-7b087a41d356",
    "name": "Default Suite",
    "persistSession": false,
    "parallel": false,
    "timeout": 300,
    "tests": ["577b18f4-5b2f-49e8-9fc3-e320cd8f00fa"]
  }],
  "urls": ["<appUrl>"],
  "plugins": []
}

Example script 2

The following script demonstrates signing in to a web application where the username and password fields are on separate pages.

The script commands run in the following order:

• Opens the URL for the web application.
• Waits for the sign-in page to load.
• Enters the username.
• Selects the sign-in button to open the password page.
• Enters the password.
• Selects the sign-in button to sign in.
• Waits for an element to load, which indicates successful sign in.

{
  "id": "b66324ad-1350-4277-85f1-c48b3800a868",
  "version": "2.0",
  "name": "Login",
  "url": "<appUrl>",
  "tests": [{
    "id": "577b18f4-5b2f-49e8-9fc3-e320cd8f00fa",
    "name": "Login",
    "commands": [{
      "id": "03d62bf6-8784-48cc-b319-30e1c809ab11",
      "comment": "",
      "command": "open",
      "target": "</pathToLogin>",
      "targets": [],
      "value": ""
    }, {
      "id": "81b9dc4d-7c78-49f9-bc4c-9e2186625169",
      "comment": "",
      "command": "waitForElementVisible",
      "target": "id=<usernameField>",
      "targets": [],
      "value": "30000"
    }, {
      "id": "53c5ed70-6548-4e39-b456-d0e5502945d4",
      "comment": "",
      "command": "type",
      "target": "id=<usernameField>",
      "targets": [],
      "value": "<username>"
    }, {
      "id": "d2ba43a8-e109-415a-a2be-0b19e9ffef75",
      "comment": "",
      "command": "click",
      "target": "id=<loginButton>",
      "targets": [],
      "value": ""
    }, {
      "id": "4f9104fd-0f7f-4aeb-885f-b27b1eca4363",
      "comment": "",
      "command": "waitForElementVisible",
      "target": "id=<passwordField>",
      "targets": [],
      "value": "30000"
    }, {
      "id": "1b55d2a4-e472-48e6-822d-a5b54a2986d9",
      "comment": "",
      "command": "type",
      "target": "id=<passwordField>",
      "targets": [],
      "value": "<password>"
    }, {
      "id": "da24efdf-cdd3-4858-b9f8-52e0666b6d73",
      "comment": "",
      "command": "click",
      "target": "id=<loginButton>",
      "targets": [],
      "value": ""
    }, {
      "id": "566c3651-194c-4621-bc87-7f5cb64356fe",
      "comment": "",
      "command": "waitForElementVisible",
      "target": "id=<elementOnSuccessfulLoginPage>",
      "targets": [],
      "value": "30000"
    }]
  }],
  "suites": [{
    "id": "e17f2325-4ac8-46d3-8f25-7b087a41d356",
    "name": "Default Suite",
    "persistSession": false,
    "parallel": false,
    "timeout": 300,
    "tests": ["577b18f4-5b2f-49e8-9fc3-e320cd8f00fa"]
  }],
  "urls": ["<appUrl>"],
  "plugins": []
}

Example script 3

The following script demonstrates signing in to a web application with multifactor authentication (MFA) enabled using a TOTP scanner variable.

The script commands run in the following order:

• Opens the URL for the web application.
• Waits for the sign-in page to load.
• Enters the username.
• Enters the password.
• Selects the button to sign in.
• Waits for the TOTP field for two-step authentication to load.
• Enters the TOTP seed from the specified scanner variable.
• Selects the submit button to sign in with the TOTP seed.
• Waits for an element to load, which indicates successful sign in.

{
  "id": "68c1e8b0-d794-4b37-a7c6-793eb8a434dc",
  "version": "2.0",
  "name": "Login_TOTP",
  "url": "<appURL>",
  "tests": [{
    "id": "d084a794-2ca2-48ce-bb5c-a4638eeb1f94",
    "name": "Login",
    "commands": [{
      "id": "8b2550d6-ee29-4230-9ccd-f44798a2407a",
      "comment": "",
      "command": "open",
      "target": "<loginPath>",
      "targets": [],
      "value": ""
    }, {
      "id": "439e6617-c4d8-49d7-8073-54f638aafc21",
      "comment": "",
      "command": "waitForElementPresent",
      "target": "id=<usernameField>",
      "targets": [],
      "value": "30000"
    }, {
      "id": "3c0db2ff-db10-4fcd-8d43-a2c7438e8e1e",
      "comment": "",
      "command": "type",
      "target": "id=<usernameField>",
      "targets": [],
      "value": "<username>"
    }, {
      "id": "ee4e1cc9-f9ff-4119-ad3e-b3d3cda39824",
      "comment": "",
      "command": "type",
      "target": "id=<passwordField>",
      "targets": [],
      "value": "<password>"
    }, {
      "id": "55f3d219-a84f-471d-a13e-41638ad396dc",
      "comment": "",
      "command": "click",
      "target": "id=<loginButton>",
      "targets": [],
      "value": ""
    }, {
      "id": "9840bd4c-52b8-4df8-9b8c-dc7c3bf71b6e",
      "comment": "",
      "command": "waitForElementPresent",
      "target": "id=<totpcodeField>",
      "targets": [],
      "value": "30000"
    }, {
      "id": "14b8ff3c-05e8-4dbb-a824-2d6dcf39988d",
      "comment": "",
      "command": "type",
      "target": "id=<totpcodeField>",
      "targets": [],
      "value": "${TOTP_SECRET}"
    }, {
      "id": "7c0231b6-84f6-4229-9d0b-928f8df5a378",
      "comment": "",
      "command": "click",
      "target": "id=submit",
      "targets": [],
      "value": ""
    }, {
      "id": "5218f117-7594-45f9-b0fe-31ac08144ddf",
      "comment": "",
      "command": "waitForElementPresent",
      "target": "id=<elementOnSuccessfulLoginPage>",
      "targets": [],
      "value": "30000"
    }]
  }],
  "suites": [{
    "id": "34ea279e-bdb6-4c97-b562-1c06635c7898",
    "name": "Default Suite",
    "persistSession": false,
    "parallel": false,
    "timeout": 300,
    "tests": ["d084a794-2ca2-48ce-bb5c-a4638eeb1f94"]
  }],
  "urls": ["<appURL>"],
  "plugins": []
}

Configure scanner variables

You create scanner variables in the Veracode Platform to define information that you can reference in your login scripts for web application scans. The variables consist of a reference key and value. You typically create scanner variables that define URL sign in credentials you want to keep safe and reuse in multiple scripts.

You can also configure and reference variables with the REST API.

If you use scriptable request modification (SRM) authentication for API scans, you could reference the variables in your SRM scripts.

Create a scanner variable for Dynamic Analysis

Before you begin:

If you want to create a variable that defines a time-based one-time password (TOTP) secret for signing in to a URL that requires multifactor authentication (MFA), ensure you have the TOTP secret.

To complete this task:

1. In the Veracode Platform, configure an analysis for a web application.
2. On the Configure or Edit Configure page, in the URLs to Scan table, locate a URL, then select Configure .
3. In the Configure window, expand Scanner Variables.
4. For Reference Key, enter the variable key. For example, My_password_key.
5. For Value, enter the variable value. When the login script runs the command that references the variable, it passes this value.
6. To indicate that the variable defines a TOTP secret for multifactor authentication, select TOTP seed.
7. Optionally, to add additional variables, select Add Variable.

Create a scanner variable for DAST Essentials

Before you begin:

If you want to create a variable that defines a time-based one-time password (TOTP) secret for signing in to a URL that requires multifactor authentication (MFA), make sure you have the TOTP secret.

note

Multifactor authentication (MFA) is not supported by AI-assisted login.

To complete this task:

1. Sign in to the Veracode Platform.
2. Select Scans and Analysis > DAST Essentials.
3. Locate the target you want to configure. In the Actions column, select Configure.
4. Select the Authentication tab.
5. In the Enterprise mode authentication settings section, under Scanner variables, add a reference key and value. For Reference Key, enter the variable key (for example, My_password_key). For Value, enter the variable value. When the login script runs the command that references the variable, it passes this value.
6. To indicate that the variable defines a TOTP secret for multifactor authentication, select TOTP seed.
7. To use the variable with AI to complete the login form, select the Utilize in AI-Assisted Login checkbox. Before selecting this checkbox, you must enable AI-Login in form-based authentication. AI-assisted login uses generative AI to configure variables and complete complex login forms.
8. Optionally, select Add Another to create additional variables.

Reference a variable in a login script

Before you begin:

• You have created a scanner variable.
• You have a login script. To create a script from a template, you can use the example scripts.

To complete this task:

1. Open your login script in the Selenium IDE or a code editor.

2. Add a command or edit an existing command in which you want to reference the scanner variable.

3. For the command value, enter the scanner variable reference key as ${<My_variable_key>}, where My_variable_key is your reference key name. This must match the Reference Key field under Scanner Variables in the Veracode Platform.

   For example, in this example script, the referenced variable ${TOTP_SECRET} defines the TOTP secret for a URL that requires multifactor authentication. That command uses the variable to type the TOTP secret on the sign in page of a web application.

4. Save your Selenium project or login script.

5. When the login script is complete, you can upload it to the Veracode Platform.

Obtain a TOTP secret

Before you begin:

• You have installed and configured a multifactor authentication tool for the URLs you want to scan.

To complete this task:

1. Sign in to your web application. A window opens and displays your authentication code or a QR code.
2. Locate the TOTP secret. For help with locating the TOTP secret, see the documentation for your authentication tool.
3. Copy the TOTP secret and add it to your scanner variable or save it to a secure, temporary location.