Taking a screenshot¶
To take a screenshot of a web page and write it to datasette-io.png
run this:
shot-scraper https://datasette.io/
If a file called datasette-io.png
already exists the filename datasette-io.1.png
will be used instead.
You can use the -o
option to specify a filename:
shot-scraper https://datasette.io/ -o datasette.png
Use -o -
to write the PNG image to standard output:
shot-scraper https://datasette.io/ -o - > datasette.png
If you omit the protocol http://
will be added automatically, and any redirects will be followed:
shot-scraper datasette.io -o datasette.png
Adjusting the browser width and height¶
The browser window used to take the screenshots defaults to 1280px wide and 780px tall.
You can adjust these with the --width
and --height
options (-w
and -h
for short):
shot-scraper https://datasette.io/ -o small.png --width 400 --height 800
If you provide both options, the resulting screenshot will be of that size. If you omit --height
a full page length screenshot will be produced (the default).
Screenshotting a specific area with CSS selectors¶
To take a screenshot of a specific element on the page, use --selector
or -s
with its CSS selector:
shot-scraper https://simonwillison.net/ -s '#bighead'
This produces simonwillison-net.png
containing this image:

When using --selector
the height and width, if provided, will set the size of the browser window when the page is loaded but the resulting screenshot will still be the same dimensions as the element on the page.
You can pass --selector
multiple times. The resulting screenshot will cover the smallest area of the page that contains all of the elements you specified, for example:
shot-scraper https://simonwillison.net/ \
-s '#bighead' -s .overband \
-o bighead-multi-selector.png
To capture a rectangle around every element that matches a CSS selector, use --selector-all
:
shot-scraper https://simonwillison.net/ \
--selector-all '.day' \
-o just-the-day-boxes.png
You can add --padding 20
to add 20px of padding around the elements when the shot is taken.
Specifying elements using JavaScript filters¶
The --js-selector
and --js-selector-all
options can be used to use JavaScript expressions to select elements that cannot be targetted just using CSS selectors.
The options should be passed JavaScript expression that operates on the el
variable, returning true
if that element should be included in the screenshot selection.
To take a screenshot of the first paragraph on the page that contains the text “shot-scraper” you could run the following:
shot-scraper https://github.com/simonw/shot-scraper \
--js-selector 'el.tagName == "P" && el.innerText.includes("shot-scraper")'
The el.tagName == "P"
part is needed here because otherwise the <html>
element on the page will be the first to match the expression.
The generated JavaScript that will be executed on the page looks like this:
Array.from(document.getElementsByTagName('*')).find(
el => el.tagName == "P" && el.innerText.includes("shot-scraper")
).classList.add("js-selector-a1f5ba0fc4a4317e58a3bd11a0f16b96");
The --js-selector-all
option will select all matching elements, in a similar fashion to the --selector-all
option described above.
Waiting for a delay¶
Sometimes a page will not have completely loaded before a screenshot is taken. You can use --wait X
to wait the specified number of milliseconds after the page load event has fired before taking the screenshot:
shot-scraper https://simonwillison.net/ --wait 2000
Waiting until a specific condition¶
In addition to waiting a specific amount of time, you can also wait until a JavaScript expression returns true using the --wait-for expression
option.
This example takes the screenshot the moment a <div>
with an ID of content
becomes available in the DOM:
shot-scraper https://.../ \
--wait-for 'document.querySelector("div#content")'
Here’s an example that waits for a specific element to become available (in this case a cookie consent overlay) and then removes it before the screenshot is taken:
shot-scraper -h 800 'https://www.spiegel.de/international/' \
--wait-for "() => {
const div = document.querySelector('[id^="sp_message_container"]');
if (div) {
div.remove();
return true;
}
}"
If the expression takes longer than 30 seconds to resolve shot-scraper
will exit with an error.
Executing custom JavaScript¶
You can use custom JavaScript to modify the page after it has loaded (after the ‘onload’ event has fired) but before the screenshot is taken using the --javascript
option:
shot-scraper https://simonwillison.net/ \
-o simonwillison-pink.png \
--javascript "document.body.style.backgroundColor = 'pink';"
Using JPEGs instead of PNGs¶
Screenshots default to PNG. You can save as a JPEG by specifying a -o
filename that ends with .jpg
.
You can also use --quality X
to save as a JPEG with the specified quality, in order to reduce the filesize. 80 is a good value to use here:
shot-scraper https://simonwillison.net/ \
-h 800 -o simonwillison.jpg --quality 80
% ls -lah simonwillison.jpg
-rw-r--r--@ 1 simon staff 168K Mar 9 13:53 simonwillison.jpg
Device scale factor¶
The --scale-factor
option sets a specific device scale factor, which effectively simulates different device pixel ratios. This setting is useful for testing high-definition displays or emulating screens with various pixel densities.
For example, setting --scale-factor 3
results in an image with a CSS pixel ratio of 3, which is ideal for emulating a high-resolution display, such as Apple iPhone 12 screens.
To take a screenshot with a scale factor of 3 (tripled resolution), run the following command:
shot-scraper https://simonwillison.net/ -o simon.png \
--width 390 --height 844 --scale-factor 3
This will multiply both the width and height of the screenshot by 3, resulting in an image that is 1170px wide and 2532px high, matching the iPhone 12’s screen.
The --scale-factor
option takes a positive float as input. For example, setting --scale-factor 2.625
simulates the Google Pixel 6’s CSS pixel ratio.
Retina images¶
The --retina
option is a shortcut to set a device scale factor of 2. This means that an image will have its resolution effectively doubled, emulating the display of images on retina or higher pixel density screens.
shot-scraper https://simonwillison.net/ -o simon.png \
--width 400 --height 600 --retina
This example will produce an image that is 800px wide and 1200px high.
Note: The --retina
option should not be used in conjunction with the --scale-factor
flag as they are mutually exclusive. If both are provided, the command will raise an error to prevent conflicts.
Transparent background¶
The --omit-background
option instructs the browser to ignore the default background, allowing for the capture of a page with a transparent background. Does not work with JPG images or when quality
is set.
shot-scraper https://simonwillison.net/ -o simon.png \
--width 400 --height 600 --omit-background
Interacting with the page¶
Sometimes it’s useful to be able to manually interact with a page before the screenshot is captured.
Add the --interactive
option to open a browser window that you can interact with. Then hit <enter>
in the terminal when you are ready to take the shot and close the window.
shot-scraper https://simonwillison.net/ -o after-interaction.png \
--height 800 --interactive
This will output:
Hit <enter> to take the shot and close the browser window:
# And after you hit <enter>...
Screenshot of 'https://simonwillison.net/' written to 'after-interaction.png'
Logging all requests¶
It can sometimes be useful to see a list of all of the requests that the browser made while it was rendering a page.
Use --log-requests
to output newline-delimited JSON representing each request, including requests for images and other assets.
Pass -
to output the list to standard output, or use a filename to write to a file on disk.
The output looks like this:
% shot-scraper http://datasette.io/ --log-requests -
{"method": "GET", "url": "http://datasette.io/", "status": 302, "size": null, "timing": {"startTime": 1663211674984.7068, "domainLookupStart": 0.698, "domainLookupEnd": 1.897, "connectStart": 1.897, "secureConnectionStart": -1, "connectEnd": 18.726, "requestStart": 18.86, "responseStart": 99.75, "responseEnd": 101.75000000162981}}
{"method": "GET", "url": "https://datasette.io/", "status": 200, "size": 34592, "timing": {"startTime": 1663211675085.51, "domainLookupStart": 0.187, "domainLookupEnd": 0.197, "connectStart": 0.197, "secureConnectionStart": 15.719, "connectEnd": 63.854, "requestStart": 64.098, "responseStart": 390.231, "responseEnd": 399.268}}
{"method": "GET", "url": "https://datasette.io/static/site.css", "status": 200, "size": 3952, "timing": {"startTime": 1663211675486.027, "domainLookupStart": -1, "domainLookupEnd": -1, "connectStart": -1, "secureConnectionStart": -1, "connectEnd": -1, "requestStart": 0.408, "responseStart": 99.407, "responseEnd": 100.433}}
...
Note that the size
field here will be the size of the response in bytes, but in some circumstances this will not be available and it will be returned as "size": null
.
Browser arguments¶
Additional arguments to pass to the browser instance. The list of Chromium flags can be found here.
For example, to remove font render hinting:
shot-scraper https://simonwillison.net/ -o no-hinting.png \
--height 800 --browser-arg "--font-render-hinting=none"
To add multiple arguments, add --browser-arg
for each argument:
shot-scraper https://simonwillison.net/ -o no-hinting-no-gpu.png \
--height 800 --browser-arg "--font-render-hinting=none" --browser-arg "--disable-gpu"
Taking screenshots of local HTML files¶
You can pass the path to an HTML file on disk to take a screenshot of that rendered file:
shot-scraper index.html -o index.png
CSS and images referenced from that file using relative paths will also be included.
Tips for executing JavaScript¶
If you are using the --javascript
option to execute code, that code will be executed after the page load event has fired but before the screenshot is taken.
You can use that code to do things like hide or remove specific page elements, click on links to open menus, or even add annotations to the page such as this pink arrow example.
This code hides any element with a [data-ad-rendered]
attribute and the element with id="ensNotifyBanner"
:
document.querySelectorAll(
'[data-ad-rendered],#ensNotifyBanner'
).forEach(el => el.style.display = 'none')
You can execute that like so:
shot-scraper https://www.latimes.com/ -o latimes.png --javascript "
document.querySelectorAll(
'[data-ad-rendered],#ensNotifyBanner'
).forEach(el => el.style.display = 'none')
"
In some cases you may need to add a pause that executes during your custom JavaScript before the screenshot is taken - for example if you click on a button that triggers a short fading animation.
You can do that using the following pattern:
new Promise(takeShot => {
// Your code goes here
// ...
setTimeout(() => {
// Resolving the promise takes the shot
takeShot();
}, 1000);
});
If your custom code defines a Promise
, shot-scraper
will wait for that promise to complete before taking the screenshot. Here the screenshot does not occur until the takeShot()
function is called.
If you see errors relating to CSP headers such as “Failed to fetch dynamically imported module” you can work around them using the –bypass-csp option.
Viewing console.log() output¶
Almost all of the shot-scraper
commands accept a --log-console
option, which will cause them to output any calls to console.log()
to standard error while the command is running.
This includes both console.log()
calls in the existing page JavaScript, as well as any calls to that method that you include in your own custom JavaScript.
For example, running --log-console
while taking a screenshot of the Facebook homepage will show this warning message, which Facebook logs to the developer tools console to help protect people from social engineering attacks:
% shot-scraper shot facebook.com --log-console
.d8888b. 888 888
d88P Y88b 888 888
Y88b. 888 888 This is a browser feature intended for
"Y888b. 888888 .d88b. 88888b. 888 developers. If someone told you to copy-paste
"Y88b. 888 d88""88b 888 "88b 888 something here to enable a Facebook feature
"888 888 888 888 888 888 Y8P or "hack" someone's account, it is a
Y88b d88P Y88b. Y88..88P 888 d88P scam and will give them access to your
"Y8888P" "Y888 "Y88P" 88888P" 888 Facebook account.
888
888
888
See https://www.facebook.com/selfxss for more information.
Screenshot of 'http://facebook.com' written to 'facebook-com.png'
shot-scraper shot --help
¶
Full --help
for this command:
Usage: shot-scraper shot [OPTIONS] URL
Take a single screenshot of a page or portion of a page.
Usage:
shot-scraper www.example.com
This will write the screenshot to www-example-com.png
Use "-o" to write to a specific file:
shot-scraper https://www.example.com/ -o example.png
You can also pass a path to a local file on disk:
shot-scraper index.html -o index.png
Using "-o -" will output to standard out:
shot-scraper https://www.example.com/ -o - > example.png
Use -s to take a screenshot of one area of the page, identified using one or
more CSS selectors:
shot-scraper https://simonwillison.net -s '#bighead'
Options:
-a, --auth FILENAME Path to JSON authentication context file
-w, --width INTEGER Width of browser window, defaults to 1280
-h, --height INTEGER Height of browser window and shot - defaults
to the full height of the page
-o, --output FILE
-s, --selector TEXT Take shot of first element matching this CSS
selector
--selector-all TEXT Take shot of all elements matching this CSS
selector
--js-selector TEXT Take shot of first element matching this JS
(el) expression
--js-selector-all TEXT Take shot of all elements matching this JS
(el) expression
-p, --padding INTEGER When using selectors, add this much padding in
pixels
-j, --javascript TEXT Execute this JS prior to taking the shot
--scale-factor FLOAT Device scale factor. Cannot be used together
with '--retina'.
--retina Use device scale factor of 2. Cannot be used
together with '--scale-factor'.
--omit-background Omit the default browser background from the
shot, making it possible take advantage of
transparency. Does not work with JPEGs or when
using --quality.
--quality INTEGER Save as JPEG with this quality, e.g. 80
--wait INTEGER Wait this many milliseconds before taking the
screenshot
--wait-for TEXT Wait until this JS expression returns true
--timeout INTEGER Wait this many milliseconds before failing
-i, --interactive Interact with the page in a browser before
taking the shot
--devtools Interact mode with developer tools
--log-requests FILENAME Log details of all requests to this file
--log-console Write console.log() to stderr
-b, --browser [chromium|firefox|webkit|chrome|chrome-beta]
Which browser to use
--browser-arg TEXT Additional arguments to pass to the browser
--user-agent TEXT User-Agent header to use
--reduced-motion Emulate 'prefers-reduced-motion' media feature
--fail Fail with an error code if a page returns an
HTTP error
--skip Skip pages that return HTTP errors
--bypass-csp Bypass Content-Security-Policy
--silent Do not output any messages
--auth-password TEXT Password for HTTP Basic authentication
--auth-username TEXT Username for HTTP Basic authentication
--help Show this message and exit.