Scrape: scrape javascript product page?

Hi everybody,

I am trying to create a sensor based on text on a website; the actual product I want to check is this one.

When the product is in stock, it will display Online verfügbar in this css selector div.flex:nth-child(4) > div:nth-child(1) > div:nth-child(2) > div:nth-child(1) > span:nth-child(1).

When I curl the URL, or when I show source code in browser, all I get is this

<!DOCTYPE html>
<html lang="de" dir="ltr">
<head>
	<meta charset="UTF-8">
	<title>dm-drogerie markt - dauerhaft günstig online kaufen</title>
	<meta name="viewport" content="width=device-width, initial-scale=1">
	<link rel="manifest" href="/manifest.json">
    <link rel="icon" type="image/png" sizes="32x32" href="/favicon-32x32.png">
    <link rel="icon" type="image/png" sizes="16x16" href="/favicon-16x16.png">
    <link rel="apple-touch-icon" sizes="180x180" href="/apple-touch-icon.png">
    <link rel="mask-icon" href="/safari-pinned-tab.svg" color="#002878">
    <link rel="apple-touch-startup-image"
    	  media="(width: 414px) and (height: 896px) and (-webkit-device-pixel-ratio: 3)"
    	  href="/screen/splashscreen_1242x2688.png">
    <link rel="apple-touch-startup-image"
    	  media="(width: 414px) and (height: 896px) and (-webkit-device-pixel-ratio: 2)"
    	  href="/screen/splashscreen_828x1792.png">
    <link rel="apple-touch-startup-image"
    	  media="(width: 375px) and (height: 812px) and (-webkit-device-pixel-ratio: 3)"
    	  href="/screen/splashscreen_1125x2436.png">
    <link rel="apple-touch-startup-image"
    	  media="(width: 414px) and (height: 736px) and (-webkit-device-pixel-ratio: 3)"
    	  href="/screen/splashscreen_1242x2208.png">
    <link rel="apple-touch-startup-image"
    	  media="(width: 375px) and (height: 667px) and (-webkit-device-pixel-ratio: 2)"
    	  href="/screen/splashscreen_750x1334.png">
    <link rel="apple-touch-startup-image"
    	  media="(width: 1024px) and (height: 1366px) and (-webkit-device-pixel-ratio: 2)"
    	  href="/screen/splashscreen_2048x2732.png">
    <link rel="apple-touch-startup-image"
    	  media="(width: 834px) and (height: 1194px) and (-webkit-device-pixel-ratio: 2)"
    	  href="/screen/splashscreen_1668x2388.png">
    <link rel="apple-touch-startup-image"
    	  media="(width: 834px) and (height: 1112px) and (-webkit-device-pixel-ratio: 2)"
    	  href="/screen/splashscreen_1668x2224.png">
    <link rel="apple-touch-startup-image"
    	  media="(width: 768px) and (height: 1024px) and (-webkit-device-pixel-ratio: 2)"
    	  href="/screen/splashscreen_1536x2048.png">

	<meta name="apple-mobile-web-app-title" content="dm">
	<meta name="apple-mobile-web-app-capable" content="yes">
	<meta name="apple-mobile-web-app-status-bar-style" content="default">
		<style>
	#initial-logo {
		z-index: -9999;
		align-items: center;
		flex-direction: column;
		display: flex;
		padding-top: 30vh;
	    position: absolute;
	    width: 100%;
	}
	
	#first-paint-logo {
		width: 4.375rem;
	}
	</style>

    <link rel="preconnect" href="https://assets.dm.de">
    <link rel="preconnect" href="https://assets.dm.de" crossorigin>
    <link rel="preconnect" href="https://exc.mm.dm.de">
    <link rel="preconnect" href="https://del.mm.dm.de">
    <link rel="preconnect" href="https://media.dm-static.com">
				<link rel="stylesheet" type="text/css" href="https://assets.dm.de/js-libraries/2025.107.7272/css/dm-base.min.css">
		<link rel="stylesheet" type="text/css" href="https://assets.dm.de/design-system/12.52.11/design-system_dm.min.css">
		<link rel="stylesheet" type="text/css" href="https://assets.dm.de/design-system/12.52.11/theme.min.css">
		<link rel="stylesheet" type="text/css" href="https://assets.dm.de/stoerungsbanner/2024.1009.693/css/stoerungsbanner.min.css">
		<link rel="stylesheet" type="text/css" href="https://assets.dm.de/content-ui/1.1252.0/css/content-full.min.css">
		<link rel="stylesheet" type="text/css" href="https://assets.dm.de/product/2025.121.44645/product-dm.min.css">
		<link rel="stylesheet" type="text/css" href="https://assets.dm.de/productsearch/2025.117.17367/css/productsearch-dm.min.css">
		<link rel="stylesheet" type="text/css" href="https://assets.dm.de/storeavailability/2025.103.59031/availability-dm.min.css">
	
	<script src="/scripts/head.js"></script>
</head>
<body>
	<div id="initial-logo">
		<div id="logowrapper">
			<img id="first-paint-logo" alt="dm Logo" src="data:image/svg+xml;base64,PHN2ZyB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciIGRhdGEtZG1pZD0iRG1CcmFuZEljb24iIHZpZXdCb3g9IjAgMCAyMCAxMy41NCIgd2lkdGg9IjQ4IiBoZWlnaHQ9IjEwMCUiIHJvbGU9ImltZyIgYXJpYS1sYWJlbGxlZGJ5PSJEbUJyYW5kSWNvblRpdGxlIiBwb2ludGVyLWV2ZW50cz0ibm9uZSIgZGF0YS1kZXNpZ25zeXN0ZW09InRydWUiPjx0aXRsZSBpZD0iRG1CcmFuZEljb25UaXRsZSI+ZG0tTWFya2VubG9nbzwvdGl0bGU+PHBhdGggZmlsbD0iI0ZGRkZGRiIgZD0iTTE5Ljg4LDcuOTVsLTAuOTksMC4xN2MtMC4xLTAuMjItMC41MS0wLjg3LTAuNTEtMC44N3MtMC44MiwwLjEzLTAuOTUsMC4xNWwwLjQzLTIuNjIgYzAuMjgtMS4zNi0wLjE2LTIuNTktMS42OC0yLjU5Yy0wLjU5LDAtMS4wNywwLjI4LTEuMzUsMC40NWMtMC4yMi0wLjI2LTAuNjMtMC40NS0xLjA1LTAuNDVjLTAuMjcsMC0wLjg5LDAuMDUtMS40NiwwLjQ3bDAuMDctMC4zOSBjLTAuMzMtMC4wMy0xLjYxLDAtMS45NSwwLjAzbDAuNDMtMi4yNUMxMC4yNiwwLDguNjgtMC4wOCw2Ljk2LDAuMTdMNi41MiwyLjM0Yy0xLjcsMC40OC0yLjc4LDIuMDUtMi43OCw0LjIgYzAsMC40MiwwLjA2LDAuNzksMC4xNiwxLjFDMi45NCw3Ljc2LDEuMiw4LjA3LDAuMDMsOC43MWMwLjI2LDAuMjUsMC42NiwwLjYyLDAuOTQsMC45NkwwLDEwLjAyYzAuODksMS4wNCwxLjU4LDIsMi4xNywzLjUyIGMxLjI2LTAuNjYsMy4wMS0xLjY0LDYuMTgtMS42NGMxLjQ5LDAsNS44OCwxLjEzLDguOTMsMS4xM2MwLjk2LDAsMS43Mi0wLjA5LDIuNDctMC40N0MxOS45MSwxMS41OCwyMC4xNCw5LjU3LDE5Ljg4LDcuOTUiPjwvcGF0aD48cGF0aCBmaWxsPSIjRTMwNjEzIiBkPSJNMTguNSw5LjczYy0wLjc4LDAuNTMtMi4zMywxLjItNC43NCwwLjZjLTAuMjMtMC4wNi0wLjQ1LTAuMTItMC42Ni0wLjE4Yy0xLjUyLTAuNjYtMy4wNi0xLjM2LTMuNjgtMS44MyBjMS43NCwwLjI3LDQuMzQsMC41NCw4LjU0LTAuMTRDMTcuOTYsOC4xOCwxOC4zNiw4LjgxLDE4LjUsOS43MyBNMi4xOSwxMC4xNmwtMC43MywwLjI2YzAsMCwwLjg0LDEsMC45OCwxLjMybDAuMjEsMC41NyBjMCwwLDIuNjUtMS40Niw2LjUtMS4yNGMwLjg3LDAuMDUsMS42NCwwLjMsMS42NCwwLjNzLTEuNTctMS4wMi0xLjg4LTEuMTJDNy40LDkuODgsNS45Niw5Ljc2LDUuMTEsOS43MyBjLTEuNDYsMC4zOS0yLjcxLDEuMDUtMi43MSwxLjA1UzIuMjksMTAuNDcsMi4xOSwxMC4xNiI+PC9wYXRoPjxwYXRoIGZpbGw9IiNGRUM3MDAiIGQ9Ik0xOS4wOSw5LjI0YzAsMC0wLjE5LDAuMjMtMC41OSwwLjQ5Yy0wLjc4LDAuNTMtMi4zMywxLjItNC43NCwwLjZjLTAuMjMtMC4wNi0wLjQ1LTAuMTItMC42Ni0wLjE4IGMtMi4yLTAuNjQtMy41LTEuNDctNi40My0xLjgzQzQuNjgsOC4zNywyLjY2LDguNiwxLjU1LDkuMDRjMCwwLDAuMywwLjMxLDAuMzksMC40NGMwLjA1LDAuMSwwLjE1LDAuMzksMC4yNSwwLjY5IGMwLjExLDAuMzIsMC4yMSwwLjYyLDAuMjEsMC42MnMxLjI1LTAuNjYsMi43MS0xLjA1QzUuOCw5LjU1LDYuNTMsOS40NCw3LjIsOS40OGMyLjI0LDAuMTQsNC4wNSwxLjMsNi4wNiwyLjI0IGMxLjM5LDAuMzksNC43OSwwLjY5LDUuNzUsMC4yNkMxOSwxMS45NiwxOS4xOSwxMS4wMywxOS4wOSw5LjI0Ij48L3BhdGg+PHBhdGggZmlsbD0iIzFGMzY4NSIgZD0iTTcuNDIsNC4xNGMtMC45MSwwLTEuNDMsMS4yNy0xLjQzLDIuMTdjMCwwLjM3LDAuMDksMC41MywwLjI4LDAuNTNjMC41MSwwLDEuMzgtMS40OCwxLjU2LTIuNEw3Ljg4LDQuMiBDNy43NSw0LjE3LDcuNjEsNC4xNCw3LjQyLDQuMTQgTTguNSw3Ljk2SDcuMTdjMC4wNi0wLjM1LDAuMTctMC43MywwLjM2LTEuMjVINy41MUM3LjA5LDcuNDEsNi41LDguMDgsNS43NSw4LjA4IGMtMC43NywwLTEuMTYtMC41LTEuMTYtMS41NGMwLTEuODMsMC45NS0zLjUsMy4wNy0zLjVjMC4xMiwwLDAuMjQsMC4wMSwwLjQyLDAuMDRMOC4zLDEuOTdWMS45NmMtMC4yMS0wLjA0LTAuNTMtMC4wOC0wLjgyLTAuMSBsMC4xOS0wLjkxYzAuNjctMC4wNywxLjM2LTAuMTIsMi4xNy0wLjA5TDguNSw3Ljk2eiBNMTYuNDcsNy45NmgtMS4zMkwxNS42Myw1YzAuMS0wLjQ4LDAuMDMtMC42Ny0wLjE5LTAuNjcgYy0wLjQ1LDAtMS4yNSwxLjE4LTEuNDQsMi4zNGwtMC4yMSwxLjI5aC0xLjMyTDEyLjk2LDVjMC4xLTAuNDgsMC4wMy0wLjY3LTAuMTktMC42N2MtMC40NSwwLTEuMjMsMS4xNy0xLjQ1LDIuMzFsLTAuMjUsMS4zMkg5Ljc1IGwwLjcxLTMuNzJWNC4yMWMtMC4yMS0wLjA0LTAuNTMtMC4wOC0wLjgyLTAuMUw5LjgzLDMuMmMwLjctMC4wOCwxLjM2LTAuMTIsMi4wOS0wLjA5Yy0wLjA4LDAuMzgtMC4yMSwwLjgyLTAuMzcsMS4yNWgwLjAyIGMwLjQ0LTAuNzUsMC45MS0xLjMyLDEuNzYtMS4zMmMwLjU3LDAsMC45NSwwLjMsMC45NSwwLjk2YzAsMC4xMi0wLjAzLDAuMjQtMC4wOCwwLjQ0bDAuMDEsMC4wMWMwLjQ0LTAuOCwwLjk5LTEuNCwxLjg0LTEuNCBjMC45NCwwLDEuMTQsMC42OCwwLjk2LDEuNTlMMTYuNDcsNy45NnoiPjwvcGF0aD48L3N2Zz4&#x3D;">
		</div>
	</div>

	<div id="app"></div>
						<script src="https://assets.dm.de/js-libraries/2025.107.7272/js/libraries-dm.min.js"></script>
						<script src="https://assets.dm.de/design-system/12.52.11/design-system_globals.min.js"></script>
						<script src="https://assets.dm.de/design-system/12.52.11/design-system_dm.min.js"></script>
						<script src="https://assets.dm.de/om/api/2025.121.6365/om-api.min.js"></script>
						<script src="https://assets.dm.de/stoerungsbanner/2024.1009.693/js/stoerungsbanner.min.js" async ></script>
						<script src="https://assets.dm.de/composing/2025.116.22911/js/composing-dm.min.js"></script>
						<script src="https://assets.dm.de/credit/2025.116.7335/credit-checker.min.js"></script>
						<script src="https://assets.dm.de/om/form/2024.1120.5552/formuiInitializer.min.js"></script>
						<script src="https://assets.dm.de/om/components/2025.114.6924/om-components-next.min.js"></script>
						<script src="https://assets.dm.de/search/2025.121.15554/js/search-dm.min.js"></script>
						<script src="https://assets.dm.de/storefinder/1.3626.0/storefinderInitializer.min.js"></script>
						<script src="https://assets.dm.de/storefinder/1.3626.0/storePersistence.min.js"></script>
						<script src="https://assets.dm.de/web-performance/2025.117.1051/js/web-performance.min.js"></script>
						<script src="https://assets.dm.de/authentication-api/prod/dmnext-authentication-api.min.js"></script>
						<script src="https://assets.dm.de/content-ui/1.1252.0/content-full.min.js"></script>
						<script src="https://assets.dm.de/flexmodule-odt/1.130.0/odt-flex.min.js"></script>
						<script src="https://assets.dm.de/product/2025.121.44645/product-dm.min.js"></script>
						<script src="https://assets.dm.de/om/review-ui/2025.108.3963/review-ui.min.js"></script>
						<script src="https://assets.dm.de/productsearch/2025.117.17367/js/productsearch-dm.min.js"></script>
						<script src="https://assets.dm.de/retail-media/0.20241218.14626/js/retail-media-dm.min.js"></script>
						<script src="https://assets.dm.de/shoppinglist/2025.118.13019/shoppinglist.min.js"></script>
						<script src="https://assets.dm.de/accountwidget/2025.117.35955/accountWidget.min.js"></script>
						<script src="https://assets.dm.de/storeavailability/2025.103.59031/availability-dm.min.js"></script>
						<script src="https://assets.dm.de/chatbot/2025.116.11240/js/chatbot.min.js"></script>
						<script src="https://assets.dm.de/cart/2025.120.56889/js/cart-dm.min.js"></script>
						<script src="https://assets.dm.de/recoweb/2024.1128.42787/js/recommendation.min.js"></script>
			<script src="/scripts/main.js"></script>
	<script type="application/ld+json" >
	{
	  "@context": "https://schema.org/",
	  "@type": "WebSite",
	  "name": "dm-drogerie markt",
	  "url": "https://www.dm.de/",
	  "potentialAction": {
	    "@type": "SearchAction",
	    "target": "https://www.dm.de/search?query={search_term_string}",
	    "query-input": "required name=search_term_string"
	  }
	}
	</script>
</body>
</html>

The shop (dm.de) does not have a public API.

Is it still possible to query via scrape whether the product is currently available or sold out? This particular item is often not available, so I wanted to create an alert when the product status in Home Assistant would change from “nicht verfügbar” (unavailable) to “online verfügbar” (available online).

Thank you in advance for your help :slight_smile:

If you have a scraper that implements a complete browser engine that can/will execute JavaScript, yes. If you do not, well then no.

You would need something like Scrapy implemented as a HASS integration and I don’t think that is available. That would suck up a whole lot of performance and probably not suitable for low powered servers like a Pi.

You could of course reverse engineer the JavaScript and see what requests it sends. Or if whatever data you need is packed directly into one of the script files already, then you could scrape that instead. I did something similar just recently for extracting data out of the YouTube history feed.

Is there a reason you want to use scrape?

The product data seems to be dynamically loaded from this URL:

https://products.dm.de/product/DE/products/detail/gtin/4066447355987

which appears to give a JSON response regardless of browser ID or headers,
so using the RESTful integration:

rest:
  - resource: https://products.dm.de/product/DE/products/detail/gtin/4066447355987
    scan_interval: 3600
    sensor:
      - name: Kokoswasser details
        value_template: "{{ now() }}"
        json_attributes_path: $.price
        json_attributes:
          - notDeliverableToStore
          - priceCurrencyIso
          - isSellout
          - price

That gives:

I suspect the isSellout being false is what you’re looking for. If it is, consider using this for the value_template instead:

value_template: "{{ 'Available' if value_json['price']['isSellout'] == 'false' else 'Not available' }}"
1 Like

Thank you so much!

There wasn’t any reason for this, I simply thought it was the way to go. Tbh I had forgotten about RESTful.

Could you please tell me how you’ve found this particular URL that provided all the json data? While I can adapt from it (gtin seems to be the product number that is at the very end of the original URL I posted) it would be nice to know this kind of stuff for the next time some store tries to make it harder to get data like this :wink:

F12 DevTools in your browser, watch the Network tab as you reload, look for likely resources.

It’s a bit of an art and guesswork, but start with xhr responses that are from the domain itself and aren’t framework files (e.g. jQuery etc). Look at the responses for probable data.

There’s often a bit more work to do with headers to be able to read the resource, but this one was straightforwards.

1 Like