Automation API

The automation object is available in the browser.webfuseSession namespace in every content script of an extension. If you want to call it from a background script or a popup, you need to send a message to the content script and then call the method of the automation API.

Note

For consistency with pythonic automation methodology, documentation defaults to snake case. Properties on the Automation API, however, are available through both snake case and camel case. For instance, left_click() is synonymous with leftClick().

Caution

Automation works right in the content of a proxied page. The Automation API is thus naturally accessible from the content script, which can access the respective page’s window scope. For ease of use, the API is also accessible from background and popup component scripts. In that case, automation is happening in the currently active tab. To use the Automation API from a non-content script, at least one content script must hence be present in the extension.

Targeting

By default, automation actions work on the element below the virtual mouse pointer. A target can optionally be specified in different ways: Either via CSS selector (selector: string), or absolute point coordinates ([x: number, y: number]).

type Target = Element | string | [number, number];  // Element reference, CSS selector or point coordinate

Note

Targeting is able to pierce shadow root and iframe boundaries.

mouse_move()

Move the virtual mouse pointer.

browser.webfuseSession.automation.mouse_move(
    target: Target,
    persistent: boolean = false
): Promise<void>

Parameters

target

• Mouse pointer target.

persistent

• Whether to keep the mouse pointer on screen (hides after some time by default).

Returns

A promise that resolves once the mouse was moved.

scroll()

Scrolls the deepest scrollable element under the target by the given amount in the given direction.

browser.webfuseSession.automation.scroll(
    target: Target,
    direction: 'vertical' | 'horizontal',
    amount: number
): Promise<void>

Parameters

target

• Scroll(able) target.

direction

• The direction to scroll.

amount

• The amount of pixels to scroll.

Returns

A promise that resolves once scroll ended.

Example

await browser.webfuseSession.automation.scroll(100, 'down', '#scrollable');

left_click()

Perform a left (primary) mouse button click.

browser.webfuseSession.automation.left_click(
    target: Target,
    moveMouse: boolean = false
): Promise<void>

Parameters

target

• Click target.

[moveMouse]

• Whether to optionally move the virtual mouse pointer to the target center before the click.

Returns

A promise that resolves once click was performed.

Example

await browser.webfuseSession.automation.left_click([100, 250]);

middle_click()

Perform a middle (wheel) mouse button click.

browser.webfuseSession.automation.middle_click(
    target: Target,
    moveMouse: boolean = false
): Promise<void>

Parameters

target

• Click target.

[moveMouse]

• Whether to optionally move the virtual mouse pointer to the target center before the click.

Returns

A promise that resolves once click was performed.

right_click()

Perform a right (secondary) mouse button click.

browser.webfuseSession.automation.right_click(
    target: Target,
    moveMouse: boolean = false
): Promise<void>

Parameters

target

• Click target.

[moveMouse]

• Whether to optionally move the virtual mouse pointer to the target center before the click.

Returns

A promise that resolves once click was performed.

type()

browser.webfuseSession.automation.type(
    target: Target,
    text: string,
    moveMouse: boolean = false,
    overwrite: boolean = false,
    timePerChar: number = 100
): Promise<void>

Type text to an element. Typing is natural, i.e. as if a human presses a sequence of keys.

Parameters

target

• Typing target.

text

• Text to type.

[moveMouse]

• Whether to optionally imply a virtual mouse pointer movement into the center of the target.

[overwrite]

• Whether to overwrite the current value of the target element.

[timePerChar]

• Average time to type a character in milliseconds.

Returns

A promise that resolves once text was typed.

key_press()

browser.webfuseSession.automation.key_press(
    target: Target,
    key: "a" | "b" | ... | "Y" | "Z",
    options?: {
        altKey?: boolean;
        ctrlKey?: boolean;
        metaKey?: boolean;
        shiftKey?: boolean;
    }
): Promise<void>

Press a key on an element.

Parameters

target

• Key press target.

key

• Key to press.

[options]

• Booleans to hold down a secondary during the press: alt, ctrl, meta, or shift.

Returns

A promise that resolves once key was pressed.

setSelection()

browser.webfuseSession.automation.setSelection(target: Target, text: string, occurrence: number = 0): Promise<void>

Parameters

target

• Text content selection target.

text

• Text to select (empty text also removes any existing selection).

[occurrence]

• The index of occurrence if text repeats in the target (e.g., 1 for the second occurrence).

Returns

A promise that resolves once the selection was applied.

getSelection()

browser.webfuseSession.automation.getSelection(): Promise<string>

Returns

A promise that resolves with the currently selected text (or empty string if nothing is selected).

wait()

browser.webfuseSession.automation.wait(ms: number): Promise<void>

Parameters

ms

• The amount of milliseconds to wait.

Returns

A promise that resolves once the given time passed.

take_dom_snapshot()

browser.webfuseSession.automation.take_dom_snapshot(options?: {
    rootSelector?: string;
    crossframe?: boolean;
    crossshadow?: boolean;
    revealMaskedElements?: boolean;
    modifier?: "downsample" | {
        name: string;
        params?: unknown[];
    };
}): Promise<void>

Serialize the DOM for various processing purposes, such as for LLM input.

Parameters

[options]

• DOM snapshot options:

• [rootSelector] Selector of the element to designate as the snapshot root (documentElement by default).

• [crossframe] Webfuse Exclusive Whether to include iframe subtrees (false by default).

• [crossshadow] Webfuse Exclusive Whether to include shadow DOM subtrees (true by default).

• [revealMaskedElements] Whether to include masked elements (false by default).

• [modifier] Snpashot modifier (Identity by default).

  • name Modifier name.
  • [params] Modifier parameter record.

Note

Cross-frame snapshots (crossframe = true) will have all iframe contents inlined, e.g.:

<html>
  <head></head>
  <body>
    <h1>Parent</h1>
    <iframe src="/child">
      <html>
        <h1>Child</h1>
      </html>
    </iframe>
  </body>
</html>

Caution

Cross-frame snapshots allow LLMs, provided with the snapshot, to target elements within pages embedded via iframe by CSS selectors, e.g.:

div > iframe body form#login

All actuation methods on the Automation API (left_click(), type(), …) do work with cross-frame selectors. Such access to descendant frames is Webfuse-exclusive behavior.

For clean scoping, frame boundaries are not crossed implicitly (e.g., div iframe body ≠ div body).

Note

Cross-shadow snapshots (crossshadow = true) will have all shadow root contents inlined, e.g.:

<div>
  <custom-element>
    <shadow-root>
      <strong>Shadow</strong>
      <p>
        <slot></slot>
      </p>
    </shadow-root>
    <b>Slotted</b>
  </custom-element>
</div>

Caution

Cross-shadow snapshots allow LLMs, provided with the snapshot, to target elements within shadow DOMs by CSS selectors, e.g.:

div > custom-element shadow-root p

All actuation methods on the Automation API (left_click(), type(), …) do work with cross-shadow selectors. In order to cross a shadow-root, it must explicitly be contained in the respective CSS selector (shadow-root). This syntax is not valid according to the CSS selector specification. Passing a cross-shadow (pseudo) selector to, e.g., document.querySelector() would fail.

Without an explicit shadow root pseudo selector, descendants refer to slotted elements.

Returns

A promise that resolves to the snapshot.

Modifiers

accessibility-tree

Translate the DOM to an accessibility tree representation.

const domSnapshot = await browser.webfuseSession
    .automation
    .take_dom_snapshot({
        modifier: {
            name: 'accessibility-tree'
        }
    })

Example

<form
    role="form"
    aria-describedby="recipe-hint"
    aria-labelledby="recipe-form-title">
    <div
        role="group"
        aria-labelledby="checkbox-group">
        <h3 id="checkbox-group">Recipe Preferences</h3>
        <label for="notifications"
            aria-describedby="notifications-description">
            <input type="checkbox" id="notifications"
                name="notifications"
                aria-label="Enable recipe update notifications">
            Receive recipe updates
        </label>
        <p id="notifications-description">I would like to receive updates.</p>
    </div>
    <button type="button" onclick="PASTA.showRecipes()"
        aria-controls="recipe-results"
        aria-label="Show pasta recipes for selected type" role="button"
        aria-live="assertive">
        Show Recipes
    </button>
</form>

{
  "role": "RootWebArea",
  "source": "html",
  "children": [
    {
      "name": "Recipe Preferences",
      "properties": {
        "level": 3
      },
      "role": "heading",
      "source": "#checkbox-group"
    },
    {
      "children": [
        {
          "name": "Enable recipe update notifications",
          "properties": {
            "aria-label": "Enable recipe update notifications"
          },
          "role": "checkbox",
          "source": "#notifications",
          "states": {
            "checked": false
          }
        }
      ],
      "properties": {
        "aria-describedby": "notifications-description"
      },
      "role": "generic",
      "source": "html > body > section > form > div > label",
      "description": "I would like to receive updates."
    }
  ]
}

Tip

Read more about our accessibility tree implementation here: webfuse-com/accessibility-tree.

downsampled recommended

Reduce the overall DOM below about 32K (2^15) estimated LLM input tokens. The resulting DOM can be considered a low resolution variant, which retains the majority of inherent UI features.

Caution

The downsampled modifier enables options assignUniqueIDs and keepUnknownElements (see below).

const domSnapshot = await browser.webfuseSession
    .automation
    .take_dom_snapshot({
        rootSelector: '#app',
        modifier: 'downsample',
    })

D2Snap

Applies the D2Snap algorithm to the DOM. This will reduce its size, while retaining a majority of UI features. The algorithm was developed in order to mitigate the prevalent DOM token size disadvantage.

const domSnapshot = await browser.webfuseSession
    .automation
    .take_dom_snapshot({
        modifier: {
            name: 'D2Snap',
            params: {
                hierarchyRatio: 0.4, textRatio: 0.6, attributeRatio: 0.8,
                // or
                // k: 0.4, l: 0.6, m: 0.8
                options: {
                    assignUniqueIDs: false,
                    keepUnknownElements: false,
                    skipMarkdownTranslation: false,
                }
            }
        }
    })

[modifier.params.options]

• [assignUniqueIDs] Whether to add a unique data attribute data-uid to every element in the DOM in order to allow identification of equivalent elements across the original and the downsampled DOM. For example, <button class="btn btn-primary" data-uid="27">Click here!</button>.
• [keepUnknownElements] Whether to keep unknown (custom) elements in the downsampled DOM.
• [skipMarkdownTranslation] Whether to skip content HTML to Markdown translation.

Example

<section class="container" tabindex="3" required="true" type="example">
  <div class="mx-auto" data-topic="products" required="false">
    <h1>Our Pizza</h1>
    <div>
      <div class="shadow-lg">
        <h2>Margherita</h2>
        <p>
          A simple classic: mozzarela, tomatoes and basil.
          An everyday choice!
        </p>
        <button type="button">Add</button>
      </div>
      <div class="shadow-lg">
        <h2>Capricciosa</h2>
        <p>
          A rich taste: mozzarella, ham, mushrooms, artichokes, and olives.
          A true favourite!
          </p>
        <button type="button">Add</button>
      </div>
    </div>
  </div>
</section>

<!-- k = .4, l = .6, m = .8 -->
<section>
  # Our Pizza
  <div>
    ## Margherita
    A simple classic:
    <button>Add</button>
    ## Capricciosa
    A rich taste:
    <button>Add</button>
  </div>
</section>

Tip

Read more about our D2Snap implementation here: webfuse-com/D2Snap.

Tip

Downsampling bases on the adaptive D2Snap algorithm. Adaptive downsampling might fail for certain DOMs and token limits. In such case, we recommend to use the one of the explicit D2Snap modifiers with custom parameters instead. D2Snap was proposed in Thassilo M. Schiepanski & Nicholas Piël. (2025). Beyond Pixels: Exploring DOM Downsampling for LLM-Based Web Agents.

AdaptiveD2Snap

Applies the AdaptiveD2Snap algorithm to the DOM. This is an adaptive version of the D2Snap algorithm that does not require explicit parameters.

const domSnapshot = await browser.webfuseSession
    .automation
    .take_dom_snapshot({
        modifier: {
            name: 'AdaptiveD2Snap',
            params: {
                maxTokens: 32768,
                maxIterations: 3,
                options: {
                    assignUniqueIDs: false,
                    skipMarkdownTranslation: false,
                }
            }
        }
    })

Note

'AdaptiveD2Snap' options are the same as for 'D2Snap'.

take_gui_snapshot()

Serialize the GUI for various processing purposes, such as for LLM input. Serialized GUI corresponds to a screenshot. Hence, this is an alias of webfuseSession.takeScreenshot().

browser.webfuseSession.automation.take_gui_snapshot(): Promise<ImageBitmap>