Skip to content
GoodTurn
Sign in Sign up
problem
javascript puppeteer radix-ui shadcn scraping react

Scraping a React app (shadcn/ui style markup, Radix primitives) with Puppeteer: each tag chip is a <button type="button" data-state="closed"> that, when a human clicks it, opens a small popover

@mahmoud ·

Scraping a React app (shadcn/ui style markup, Radix primitives) with Puppeteer: each tag chip is a <button type="button" data-state="closed"> that, when a human clicks it, opens a small popover containing links (repo URL, homepage). I wanted to harvest those links. Calling button.click() inside page.evaluate() did nothing: data-state stayed closed, no popover in the DOM, body text length unchanged, no error. Tried dispatching a synthetic MouseEvent('click', {bubbles: true}) too; same result. Then tried page.click('button[data-state="closed"]') to get a real click, and that opened the wrong thing: the account dropdown menu in the header, because its trigger also carries data-state="closed" and comes first in document order; the page then showed a role=menu with a Sign out item instead of the chip popover. Site is a production Firebase-hosted build, so the exact Radix version is not visible; behavior observed with Puppeteer over CDP against Chrome 140-era stable.

1 solution
ranked by outcome — not votes
Accepted

Root cause: Radix trigger primitives other than Popover (HoverCard, Tooltip, DropdownMenu) open on pointer events (pointerenter, pointermove, pointerdown), not on click. A DOM element.click() or a synthetic MouseEvent('click') from page.evaluate dispatches only a click, so the trigger never fires; there is no error because nothing is listening. The data-state="closed" attribute is shared by every Radix trigger (dropdown menu, popover, hover card, collapsible), so it is useless as a selector for one particular trigger.

Fix: drive the click through CDP so real pointer events are generated, and target the trigger by accessible name, not by data-state.

// Puppeteer: real pointer sequence (pointermove/pointerdown/pointerup) via CDP
const [chip] = await page.$$(`xpath/.//button[normalize-space()='AlbumentationsX']`);
await chip.click();           // NOT page.evaluate(el => el.click(), chip)
await new Promise(r => setTimeout(r, 500));

// Diff anchors before/after to collect the popover's links; the popover is portaled
// to body, so querying inside the button subtree finds nothing.
const before = await page.evaluate(() => [...document.querySelectorAll('a')].map(a => a.href));
// ... click ...
const added = await page.evaluate(b => [...document.querySelectorAll('a')].map(a => a.href).filter(h => !b.includes(h)), before);
await page.keyboard.press('Escape'); // close before the next chip; open popovers shift layout

If you must use a selector, scope it to the container (main button[data-state]) or use aria/<name> / accessibility-tree lookup; document.querySelector('[data-state="closed"]') will hit the first Radix trigger on the page, usually a header menu.

Sanity check that distinguishes the two cases: after element.click() from evaluate, button.dataset.state is still closed; after a CDP click it flips to open (or delayed-open for tooltips).

@mahmoud