Scrape Sensor Platform .. faster testing?

I am trying to scrape a download number value from a public website but the trial and error approach with always restarting home assistant is quite cumbersome to nail down the exact value.

Is there some easier test platform like the developer tools in home assistant to try if the sensor syntax is correct before restarting everytime?

So far I have this:

  #Grab PrusaPrinter Download Counter
  - platform: scrape
    resource: https://www.prusaprinters.org/social/101152-pixelwave/about
    name: Prusa Download Counter
    select: 'strong[_ngcontent-prusa-front-app-c181]'
    value_template: '{{ value }}'

But it throws “unknown” …

I think you can just reload the sensors.
But that still might be a bit slow process.

I generally find Node red better for these purposes as you can use regex or substring to get the value.
Not ideal when it’s HTML but I know the limitations.

…tried reloading all of them. But unfortunately only restart works.

Node Red I have not touched yet …

Looking at the source i’d say it’s impossible to scrape.
It’s not a html value it’s a javascript or jqurey or similar.

Damn … its that “971” value:

I know. But look at the source instead.
It’s more or less blank

Just send a POST to “https://www.prusaprinters.org/graphql/” with JSON payload:

{"operationName":"UserProfileSocial","variables":{"id":"101152"},"query":"query UserProfileSocial($id: ID!) {\n  user(id: $id) {\n    ...AvatarUserFragment\n    hashId\n    publicUsername\n    email\n    place\n    offersPrintOnDemand\n    offersTechSupport\n    offersTechService\n    offersShowAndTell\n    isFollowedByMe\n    downloadCount\n    printsCount\n    eduProjectsCount\n    userGcodesCount\n    followersCount\n    followingCount\n    makesCount\n    collectionsCount\n    eventsCount\n    rating\n    isPublicSpace\n    publicSpaceUrl\n    isTeacher\n    promotionLinks {\n      status\n      url\n      __typename\n    }\n    badgesProfileLevel {\n      profileLevel\n      __typename\n    }\n    badgesSelection {\n      order\n      userBadge {\n        id\n        name\n        description\n        levelsCount\n        currentBadgeDisplayName\n        currentBadgeLevel {\n          name\n          image\n          __typename\n        }\n        __typename\n      }\n      __typename\n    }\n    coordinates {\n      lat\n      lng\n      __typename\n    }\n    locationSharing\n    bio\n    donationLinks {\n      id\n      title\n      url\n      __typename\n    }\n    socialLinks {\n      id\n      socialType\n      url\n      __typename\n    }\n    printers {\n      id\n      name\n      __typename\n    }\n    languages {\n      id\n      name\n      __typename\n    }\n    techService {\n      techServiceBio\n      techServicePrinters {\n        id\n        name\n        __typename\n      }\n      techServiceRatingAvg\n      techServiceRatingCount\n      __typename\n    }\n    __typename\n  }\n}\n\nfragment AvatarUserFragment on UserType {\n  id\n  publicUsername\n  avatarFilePath\n  slug\n  badgesProfileLevel {\n    profileLevel\n    __typename\n  }\n  __typename\n}\n"}

you’ll get the value in the response JSON

{
    "data": {
        "user": {
[...]
            "downloadCount": 971,

How do I integrate this into Home Assistant as a sensor output?

1 Like

I put this in config:

  - platform: rest
    resource: https://www.prusaprinters.org/graphql/
    method: POST
    payload: '{"operationName":"UserProfileSocial","variables":{"id":"101152"},"query":"query UserProfileSocial($id: ID!) {\n  user(id: $id) {\n    ...AvatarUserFragment\n    hashId\n    publicUsername\n    email\n    place\n    offersPrintOnDemand\n    offersTechSupport\n    offersTechService\n    offersShowAndTell\n    isFollowedByMe\n    downloadCount\n    printsCount\n    eduProjectsCount\n    userGcodesCount\n    followersCount\n    followingCount\n    makesCount\n    collectionsCount\n    eventsCount\n    rating\n    isPublicSpace\n    publicSpaceUrl\n    isTeacher\n    promotionLinks {\n      status\n      url\n      __typename\n    }\n    badgesProfileLevel {\n      profileLevel\n      __typename\n    }\n    badgesSelection {\n      order\n      userBadge {\n        id\n        name\n        description\n        levelsCount\n        currentBadgeDisplayName\n        currentBadgeLevel {\n          name\n          image\n          __typename\n        }\n        __typename\n      }\n      __typename\n    }\n    coordinates {\n      lat\n      lng\n      __typename\n    }\n    locationSharing\n    bio\n    donationLinks {\n      id\n      title\n      url\n      __typename\n    }\n    socialLinks {\n      id\n      socialType\n      url\n      __typename\n    }\n    printers {\n      id\n      name\n      __typename\n    }\n    languages {\n      id\n      name\n      __typename\n    }\n    techService {\n      techServiceBio\n      techServicePrinters {\n        id\n        name\n        __typename\n      }\n      techServiceRatingAvg\n      techServiceRatingCount\n      __typename\n    }\n    __typename\n  }\n}\n\nfragment AvatarUserFragment on UserType {\n  id\n  publicUsername\n  avatarFilePath\n  slug\n  badgesProfileLevel {\n    profileLevel\n    __typename\n  }\n  __typename\n}\n"}'
 

But sensor output is this:

They require the content-type

  - platform: rest
    resource: https://www.prusaprinters.org/graphql/
    method: POST
    payload: '{"operationName":"UserProfileSocial","variables":{"id":"101152"},"query":"query UserProfileSocial($id: ID!) {\n  user(id: $id) {\n    ...AvatarUserFragment\n    hashId\n    publicUsername\n    email\n    place\n    offersPrintOnDemand\n    offersTechSupport\n    offersTechService\n    offersShowAndTell\n    isFollowedByMe\n    downloadCount\n    printsCount\n    eduProjectsCount\n    userGcodesCount\n    followersCount\n    followingCount\n    makesCount\n    collectionsCount\n    eventsCount\n    rating\n    isPublicSpace\n    publicSpaceUrl\n    isTeacher\n    promotionLinks {\n      status\n      url\n      __typename\n    }\n    badgesProfileLevel {\n      profileLevel\n      __typename\n    }\n    badgesSelection {\n      order\n      userBadge {\n        id\n        name\n        description\n        levelsCount\n        currentBadgeDisplayName\n        currentBadgeLevel {\n          name\n          image\n          __typename\n        }\n        __typename\n      }\n      __typename\n    }\n    coordinates {\n      lat\n      lng\n      __typename\n    }\n    locationSharing\n    bio\n    donationLinks {\n      id\n      title\n      url\n      __typename\n    }\n    socialLinks {\n      id\n      socialType\n      url\n      __typename\n    }\n    printers {\n      id\n      name\n      __typename\n    }\n    languages {\n      id\n      name\n      __typename\n    }\n    techService {\n      techServiceBio\n      techServicePrinters {\n        id\n        name\n        __typename\n      }\n      techServiceRatingAvg\n      techServiceRatingCount\n      __typename\n    }\n    __typename\n  }\n}\n\nfragment AvatarUserFragment on UserType {\n  id\n  publicUsername\n  avatarFilePath\n  slug\n  badgesProfileLevel {\n    profileLevel\n    __typename\n  }\n  __typename\n}\n"}'
    value_template: "{{ value_json.data.user.downloadCount }}"
    headers:
      Content-Type: application/json
1 Like

Awesome! Works … outputs exactly the number. Thanks a lot for the quick reply.

Sorry to res an old thread, but with the domain name change to printables, it seems this sensor no longer works, is there a work around for this does anyone know?

Thanks! I just wanted to post the same question … I tried the obvious to change the query request from “resource: Printables” to “resource: Printables” but it did not work …

Ok found the solution replace: “Printables”

… with the current API location: “https://api.printables.com/graphql/​”

Tested and works again for me! :slight_smile:

Amazing! I was almost there myself, cant believe I didnt try api.printab… I feel stupid now! Thanks for the update though!

1 Like

… the query request does not seem to work anymore?

Ok … found out the fix myself. One topic in the JSON query seems to be outdated so you just have to remove “currentBadgeDisplayName\n”. Works again!

  #Grab PrusaPrinter Download Counter
  - platform: rest
    resource: https://api.printables.com/graphql/
    method: POST
    payload: '{"operationName":"UserProfileSocial","variables":{"id":"101152"},"query":"query UserProfileSocial($id: ID!) {\n  user(id: $id) {\n    ...AvatarUserFragment\n    hashId\n    publicUsername\n    email\n    place\n    offersPrintOnDemand\n    offersTechSupport\n    offersTechService\n    offersShowAndTell\n    isFollowedByMe\n    downloadCount\n    printsCount\n    eduProjectsCount\n    userGcodesCount\n    followersCount\n    followingCount\n    makesCount\n    collectionsCount\n    eventsCount\n    rating\n    isPublicSpace\n    publicSpaceUrl\n    isTeacher\n    promotionLinks {\n      status\n      url\n      __typename\n    }\n    badgesProfileLevel {\n      profileLevel\n      __typename\n    }\n    badgesSelection {\n      order\n      userBadge {\n        id\n        name\n        description\n        levelsCount\n        currentBadgeLevel {\n          name\n          image\n          __typename\n        }\n        __typename\n      }\n      __typename\n    }\n    coordinates {\n      lat\n      lng\n      __typename\n    }\n    locationSharing\n    bio\n    donationLinks {\n      id\n      title\n      url\n      __typename\n    }\n    socialLinks {\n      id\n      socialType\n      url\n      __typename\n    }\n    printers {\n      id\n      name\n      __typename\n    }\n    languages {\n      id\n      name\n      __typename\n    }\n    techService {\n      techServiceBio\n      techServicePrinters {\n        id\n        name\n        __typename\n      }\n      techServiceRatingAvg\n      techServiceRatingCount\n      __typename\n    }\n    __typename\n  }\n}\n\nfragment AvatarUserFragment on UserType {\n  id\n  publicUsername\n  avatarFilePath\n  slug\n  badgesProfileLevel {\n    profileLevel\n    __typename\n  }\n  __typename\n}\n"}'
    name: Prusa Download Counter
    value_template: "{{ value_json.data.user.downloadCount }}"
    headers:
      Content-Type: application/json

A shorter query for just the download count:

  #Grab PrusaPrinter Download Counter
  - platform: rest
    resource: https://api.printables.com/graphql/
    method: POST
    payload: '{"operationName":"UserProfileSocial","variables":{"id": "101152"},"query":"query UserProfileSocial($id: ID!) {\n  user(id: $id) {\n    publicUsername\n    downloadCount\n}}" }' 
    name: Prusa Download Counter
    value_template: "{{ value_json.data.user.downloadCount }}"
    headers:
      Content-Type: application/json

Came here to make the same finding! Thanks again!