403 Forbidden when attempting to get a json response

Configuration
1

Has anyone had their restful sensor return a 403 forbidden but in chrome/firefox/etc it brings up the json just fine?

DEBUG (MainThread) [httpx._client] HTTP Request: GET https://coinminerz.com/api/worker_stats?RGThB11CZ7X4hYY6mpTr5sT4K7jgxfDF6t "HTTP/1.1 403 Forbidden"

But if you navigate there via browser it shows up just fine
https://coinminerz.com/api/worker_stats?RGThB11CZ7X4hYY6mpTr5sT4K7jgxfDF6t

Is it something in the header that the server is seeing that causing it to reject the request?

2

Show your rest sensor config.

3

Yes, it filters on user-agent.
Add a dummy one

headers:
  user-agent: "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:96.0) Gecko/20100101 Firefox/96.0"
4

My rest sensor config


  - platform: rest
    resource: https://coinminerz.com/api/worker_stats?RGThB11CZ7X4hYY6mpTr5sT4K7jgxfDF6t
    name: avn_unpaid
    headers:
      user-agent: "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:96.0) Gecko/20100101 Firefox/96.0"
    value_template: "{{ value_json.immature }}"

  - platform: rest
    resource: https://coinminerz.com/api/worker_stats?RGThB11CZ7X4hYY6mpTr5sT4K7jgxfDF6t
    name: avn_paid
    headers:
      user-agent: "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:96.0) Gecko/20100101 Firefox/96.0"    
    value_template: "{{ value_json.paid }}"

I just checked the logs again, I think I might be out of luck it looks like I am hitting a captcha, Unless there is a way around it

<!--[if lt IE 7]> <html class="no-js ie6 oldie" lang="en-US"> <![endif]-->
<!--[if IE 7]>    <html class="no-js ie7 oldie" lang="en-US"> <![endif]-->
<!--[if IE 8]>    <html class="no-js ie8 oldie" lang="en-US"> <![endif]-->
<!--[if gt IE 8]><!--> <html class="no-js" lang="en-US"> <!--<![endif]-->
<head>

<title>Please Wait... | Cloudflare</title>
  
<meta name="captcha-bypass" id="captcha-bypass" />
<meta charset="UTF-8" />
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
<meta http-equiv="X-UA-Compatible" content="IE=Edge,chrome=1" />
<meta name="robots" content="noindex, nofollow" />
<meta name="viewport" content="width=device-width,initial-scale=1" />
<link rel="stylesheet" id="cf_styles-css" href="/cdn-cgi/styles/cf.errors.css" type="text/css" media="screen,projection" />
<!--[if lt IE 9]><link rel="stylesheet" id='cf_styles-ie-css' href="/cdn-cgi/styles/cf.errors.ie.css" type="text/css" media="screen,projection" /><![endif]-->
<style type="text/css">body{margin:0;padding:0}</style>


<!--[if gte IE 10]><!-->
<script>
  if (!navigator.cookieEnabled) {
    window.addEventListener('DOMContentLoaded', function () {
      var cookieEl = document.getElementById('cookie-alert');
      cookieEl.style.display = 'block';
    })
  }
</script>
<!--<![endif]-->


  
    <script type="text/javascript">
    //<![CDATA[
    (function(){
      window._cf_chl_opt={
        cvId: "2",
        cType: "managed",
        cNounce: "1216",
        cRay: "6da274e4cace643e",
        cHash: "af9bcef72da5517",
        cPMDTk: "FsnloiiUFvKtiJg2I60pGjruR3c4xLu.M0Sg0gkNWsI-1644298128-0-gaNycGzNB30",
        cUPMDTk: "\/api\/worker_stats?RGThB11CZ7X4hYY6mpTr5sT4K7jgxfDF6t&__cf_chl_tk=FsnloiiUFvKtiJg2I60pGjruR3c4xLu.M0Sg0gkNWsI-1644298128-0-gaNycGzNB30",
        cFPWv: "g",
        cTTimeMs: "1000",
        cLt: "n",
        cRq: {
          ru: "aHR0cHM6Ly9jb2lubWluZXJ6LmNvbS9hcGkvd29ya2VyX3N0YXRzP1JHVGhCMTFDWjdYNGhZWTZtcFRyNXNUNEs3amd4ZkRGNnQ=",
          ra: "TW96aWxsYS81LjAgKFdpbmRvd3MgTlQgMTAuMDsgV2luNjQ7IHg2NDsgcnY6OTYuMCkgR2Vja28vMjAxMDAxMDEgRmlyZWZveC85Ni4w",
          rm: "R0VU",
          d: "t2f2jeyJBS5I/SMp1ra1fILNKIe41Gmp3DrS/dUOWwv1f60jNtrCTfM6UIl2vu/2iag+6iu3ta54fT+VZeMhUTu/A3V5dptGtSNSuULn9kMzl9Cz5IlKVmXaMvxAvPtzUNNFFADxYq7H+GskYks+w7KdDuLcjVi1a8hC0NOUp6YXyuAfBvYraH0nwA0A619d/B9LsWBccc9tGFo5yjgn8ydhE5KXDveWzNrAr93a0Jjd+xBVOILxg+YJht0yqg0+Z6aBvL6OdDO8HyZ++UCneGKDu/rP7IRNYEkoSLogLm1Pe88DArD3DUDJf7nYC5q/HCf5T6rpWU+J+wZQHyP0TsI5IIBJIDOMTYjU0RN4a0z4X9DaEQrPe79x3K2fiMVt1pE+ON+xdyJni/e+dd51m5PLfoRaSEPmJAO+4jY6DGKkIm8Iw5S88wGLoNDWCpJQi1xRQA1wdtReZvkVjC8PrG94+Y7Rig2h3uh5mGghqoaaU7AYYm8gGC3aW9XYnLuh/UoWPjIcrqFykDCqgd3FRZeAs9zm+wWjmK7uvNy1HIeZlzxV2ktpTmiBqYjfUumCJEPGC3dU780Zl6Y0rKtoPH8NwPg8fgmkkVvd7Uo0gNvk6OZYAFsI3nAPGN6MJLthb1uzL2ovSbg7X55XvbhovDcA9Z/h1MI4apx3X/ug9Oo=",
          t: "MTY0NDI5ODEyOC4xMzUwMDA=",
          m: "N2UtRAdoERk5zWvzHUthmqmckNtA3W5OAYOaensVkHY=",
          i1: "L8gGxKkTX5aAKt8qwQPXjQ==",
          i2: "vRd6ntOV7CmR/LgnlJRnVg==",
          zh: "iQsHfq2LIaFqTMuPDRVc9NkdRpWrIz/pFmJ5F2SNAZ8=",
          uh: "3TCITkidaJ5bpdDNeF7kYfCSp65T/cpzQfavD8w74W0=",
          hh: "opw7nm6hMBuI5A7Ci/GGO0s1Wj0sHd53p9uHT77QHPU=",
        }
      };
    }());
    //]]>
    </script>
  

<style type="text/css">
  #cf-wrapper #spinner {width:69px; margin:  auto;}
  #cf-wrapper #cf-please-wait{text-align:center}
  .attribution {margin-top: 32px;}
  .bubbles { background-color: #f58220; width:20px; height: 20px; margin:2px; border-radius:100%; display:inline-block; }
  #cf-wrapper #challenge-form { padding-top:25px; padding-bottom:25px; }
  #cf-hcaptcha-container { text-align:center;}
  #cf-hcaptcha-container iframe { display: inline-block;}
  @keyframes fader     { 0% {opacity: 0.2;} 50% {opacity: 1.0;} 100% {opacity: 0.2;} }
  #cf-wrapper #cf-bubbles { width:69px; }
  @-webkit-keyframes fader { 0% {opacity: 0.2;} 50% {opacity: 1.0;} 100% {opacity: 0.2;} }
  #cf-bubbles > .bubbles { animation: fader 1.6s infinite;}
  #cf-bubbles > .bubbles:nth-child(2) { animation-delay: .2s;}
  #cf-bubbles > .bubbles:nth-child(3) { animation-delay: .4s;}
</style>
</head>
<body>
  <div id="cf-wrapper">
    <div class="cf-alert cf-alert-error cf-cookie-error" id="cookie-alert" data-translate="enable_cookies">Please enable cookies.</div>
    <div id="cf-error-details" class="cf-error-details-wrapper">
      <div class="cf-wrapper cf-header cf-error-overview">
      
        <h1 data-translate="managed_challenge_headline">Please wait...</h1>
        <h2 class="cf-subheadline"><span data-translate="managed_checking_msg">We are checking your browser...</span> coinminerz.com</h2>
      
      </div>
      
      <div class="cf-section cf-highlight cf-captcha-container">
        <div class="cf-wrapper">
          <div class="cf-columns two">
            <div class="cf-column">
            
              <div class="cf-highlight-inverse cf-form-stacked">
                <form class="challenge-form managed-form" id="challenge-form" action="/api/worker_stats?RGThB11CZ7X4hYY6mpTr5sT4K7jgxfDF6t&amp;__cf_chl_f_tk=FsnloiiUFvKtiJg2I60pGjruR3c4xLu.M0Sg0gkNWsI-1644298128-0-gaNycGzNB30" method="POST" enctype="application/x-www-form-urlencoded">
  
    <div id='cf-please-wait'>
      <div id='spinner'>
        <div id="cf-bubbles">
            <div class="bubbles"></div>
            <div class="bubbles"></div>
            <div class="bubbles"></div>
        </div>
      </div>
      <p data-translate="please_wait" id="cf-spinner-please-wait">Please stand by, while we are checking your browser...</p>
      <p data-translate="redirecting" id="cf-spinner-redirecting" style="display:none">Redirecting...</p>
      </div>
  
  <input type="hidden" name="md" value="uTU1yf27kpeP9xBY9iBxK9cjqG6x_8Bf1jntof3eblQ-1644298128-0-AZj11kS7Bdh7vbEq_LrAT18Of0QKPXFAU9Pq7-ZRAFHx-ADhP2N_zCO33ELFkb-YctiXTCRPKAQ0C-uWZZtBoJSlOBJ_A0uYxab8mRCL6gr9zlJ7DV1tgVXIW0peF7ssQqabxsKFA8kngGBGno-Qw8DtNN_15umR25z9TyW9faGemC7JXL8Y2tk84m1D4bIkKGc12ktiauffmub9QmlnwRoF_gqDR6DOEyv6b-k7lboCX-_NCWnuzamTlGDs_Tll151Alb_3eOqBrvWgoZ-Bf7IjyQITSw_PVg1GOkhl8mJhPsyarcdaotT_QSytctIDqjEO5LvccKTjV6WMMnD8QZzZR5WZBSuNGFIkVlbQAoSDreWoEy1PtwFKBq7sYgKPQZryYTBOmstESL4ts6uvo585rY5RoxZ8NXttrB1ZEbXZQGpmLx-rpNgfEiGYp5buQINd3i6YcrUj3K6zodjp9zwICdFwEv5zssHsgNQ3DtTlrN_HS7R1K0lAeTrxgGUSdKAXqpc4UgVG2BBo8rKeP1mi17F7VLOVeWwTP47CAqFSmiPdHYKkyUq4ANV3-KO5as6lsw6HzcEM-haLNihYpzOpEqgaODyZM-l3oFWTG7EGKVkow3v-kvU1bZWNqO4GB6RQX_9OuQ6tziCpmFKdufWcvth2M5GmQstvdkw9CedCREcUgogwjHGrM8pp2f1xaqtV2hekIq1i_Cf9C9B4Jf8_Bgx4byiZvlzwzZLNmI77Hs_x9de38nw7_kJTR9fOW-rdnWAXmlbspPWS8MRRGwRJCuVKsDb606Mtp_29Hjs01KJyRQaortFl5YdKzmkmhWezo5ufHDtmC0szLRyJAhMp7tUk0G0bVtcMYte05tHmNG-6k8JFRUoLbcJsZED6_tpVjeTao4zlkqUUOxtNHpohI_GWb5GwXE-wemtWu64nE3RN-OdR9ABOVhiXQuoWtpdM4CidNjyNokf2v0fQ2Vv3hTBGHrCJrnaD6HSFWj_b2hqVzTqp_nhVlr7p1El1cwhqO_0fC39FHChXaD-prZhNII4DoYWNy8HiybM6nWIv" />
  <input type="hidden" name="r" value="cZCgVq12jsUUTHjTy_uee2mbBi8RAB1Z8JXJcOLkoa0-1644298128-0-AdhWo+RU89UPbLI+tmJn0kwCQiJSODPwfJSquwFUcvh+suI6g6raM2tDAm/hWRafZy8Bn/gI/WiP9nQp9nFm03686KSwFmhMDtvv9nVgZlSAyL1Z5rRmA5fd/AI7iXog6yP49Uz0cUY1nyeFEUV5JEiFkkPkdhkcO4S4TDGQSUzHqFqKDmWW+ftn7fioAaZPKUYs12Nf2ScaREMXROF0yqmLFMiB2imtQ8c1BIbriTNHssc6leUnm4ggUFXKGHrfXOdpUeOGcKqukkbdRs30cd/HAApPO6rFnYKBa5Im6/ioicCXhktAwXwhX+0mK029h2DhAPRyB26K9IX16SfYEAqpvZJAYHX5mZbnycJ39Exf33OSZadndjvy9s6SwQw+56jKAGV63pc3JAlJ7eor4ZDCdQwxUQdjkn84ntg1i7e3Sf1dyFFiFgmt6OpfR8wfDdDGaWkznQlT7Gr7K3K15g0qybfz4bsQQ9PzOtgO/FFINIga6g+CgjSbSKpiscnl8dB/SO/GNN7Hm/F3kBiTNqTNU7MlE3vB/8WwZTCDGZ7xtPvq4JRm2H4jzRHwlDDR6TGv4HIv3AAXFeE1g+sY/Y11zuHrQQVp/IOjMem/khCZyXIs0RJSTVtJkQs1Qpkeo7nhQhS/A8Zq8WQ64p2KxHnMfVX7EGEm0RrgMlQQNqL9cZeZseQ4NoRkdpnPUr31j2OBJj+3MfxPByQby0a+2q5ViF5aWD6QcRxBvzsxxg4C7KIoA97ETcL9AJJ8hLOWD6v1M/5ILkwfra/fDeA5IVe3OqfM3l53A+AsM46qehIDuuK/dQykE7H/Eql/spVEoI676GWEQhAijirRBZ1MgoVxCj7KypBXPa1GBVzqmrEAkPUm8spH38JDUW7DITdsOLUccTku05KWRtcMy2D489UPGDhzdiJ1YoOty8X8qNEdEYy9Lp1SqJOOt6KSc7sEEkyLW+Ax6meJuv9aR+bdV9YQf0Ku6+gDgRnw1NPbVjqnReSR2lYROogWVMzduNaaCz53a8HXBsLuI066n+Vb6puDHa0M9jZoBRHK8ysnc3BnjhxAZEDb6+4Nb/DIQcvP71C9HcDvriKZGsB9rlgDU1LeKiUyW8U2WthlWiBdrYYhURtffG0oZd/zoJ5OJT72MSDBBvJMo6wGkuHxp1rYXiIBayrUal+eWC/y4HkmYlTh1H2o+65FKz6Tm3ck3/8kYoJ56iN51U5BDJdl82BR6a7EzAO52tBzCGOTf//UkZGbLPuhm3ChinMJCYxsPbAzrAqOTY46B2UaaswpV1PG690CgGWVVqrx01KxQtVsPjeWjqmRbJSFyqwZW6itebxHAm9uErbHg3M8jA+a59ZBpBE94PAF3QQuVKok3OvzlK6F5+qpn8xfQ7Ph6RghR21zlmEA4ctQBw6yWjTvn/mY7KA8M9xJToT70NBspYHKzyZGZYrOUxu6/WEsXV7/6UqrFnrHhiw0tMwFL58D6i5PClQNkTlcMkDKzlNAR0rFabJBVZHohtm6ixEcmMI6XxWpKTvkf56XbjypqfT/YnECPNTYA/lVE+ps6OPH0Wq7XZWOLbfq4x7s+xD0xK98CRh9qN1OA/TTx7APBSKCwlcdtMyqjuOr6VWvPH39C2V99xVEw2hZH1HHjtHb6hgzeMLYdrmsk8vhRul61DVwH/dHitp8OwVWNSgnYc7csWqzRIf08R6o10Ena+1oi6MeAeAdtYZLWs56LhmitnBxkr1lu5TDvfm6v5MfNCmUFc8Izj2hOfOXlnFpJZXgdaV81ob1omPChh/RdXFJ99cDN7eND4PB8KeF8bacC/YqWOvlz5HbDvStoGsl72Um3cZqA77eEQ==">
  <input type="hidden" name="cf_captcha_kind" value="h">
  <input type="hidden" name="vc" value="396a00dac71ffba952cbe86280b1f0d9">
  
  <noscript id="cf-captcha-bookmark" class="cf-captcha-info">
  <h1 data-translate="turn_on_js" style="color:#bd2426;">Please turn JavaScript on and reload the page.</h1>
  </noscript>
    <div id="no-cookie-warning" class="cookie-warning" data-translate="turn_on_cookies" style="display:none">
      <p data-translate="turn_on_cookies" style="color:#bd2426;">Please enable Cookies and reload the page.</p>
    </div>
  <script type="text/javascript">
  //<![CDATA[
    var a = function() {try{return !!window.addEventListener} catch(e) {return !1} },
      b = function(b, c) {a() ? document.addEventListener("DOMContentLoaded", b, c) : document.attachEvent("onreadystatechange", b)};
      b(function(){
        var cookiesEnabled=(navigator.cookieEnabled)? true : false;
        if(!cookiesEnabled){
          var q = document.getElementById('no-cookie-warning');q.style.display = 'block';
        }
      });
  //]]>
  </script>
  <div id="trk_captcha_js" style="background-image:url('/cdn-cgi/images/trace/captcha/nojs/h/transparent.gif?ray=6da274e4cace643e')"></div>
</form>
  
  <script type="text/javascript">
    //<![CDATA[
    (function(){
        var isIE = /(MSIE|Trident\/|Edge\/)/i.test(window.navigator.userAgent);
        var trkjs = isIE ? new Image() : document.createElement('img');
        trkjs.setAttribute("src", "/cdn-cgi/images/trace/managed/js/transparent.gif?ray=6da274e4cace643e");
        trkjs.id = "trk_managed_js";
        trkjs.setAttribute("alt", "");
        document.body.appendChild(trkjs);
        var cpo=document.createElement('script');
        cpo.type='text/javascript';
        cpo.src="/cdn-cgi/challenge-platform/h/g/orchestrate/managed/v1?ray=6da274e4cace643e";
        
        window._cf_chl_opt.cOgUQuery = location.search === '' && location.href.indexOf('?') !== -1 ? '?' : location.search;
        window._cf_chl_opt.cOgUHash = location.hash === '' && location.href.indexOf('#') !== -1 ? '#' : location.hash;
        if (window._cf_chl_opt.cUPMDTk && window.history && window.history.replaceState) {
          var ogU = location.pathname + window._cf_chl_opt.cOgUQuery + window._cf_chl_opt.cOgUHash;
          history.replaceState(null, null, "\/api\/worker_stats?RGThB11CZ7X4hYY6mpTr5sT4K7jgxfDF6t&__cf_chl_rt_tk=FsnloiiUFvKtiJg2I60pGjruR3c4xLu.M0Sg0gkNWsI-1644298128-0-gaNycGzNB30" + window._cf_chl_opt.cOgUHash);
          cpo.onload = function() {
            history.replaceState(null, null, ogU);
          };
        }
        
        document.getElementsByTagName('head')[0].appendChild(cpo);
    }());
    //]]>
    </script>
  


              </div>
            </div>

            <div class="cf-column">
              <div class="cf-screenshot-container">
              
                <span class="cf-no-screenshot"></span>
              
              </div>
            </div>
          </div>
        </div>
      </div>

      <div class="cf-section cf-wrapper">
        <div class="cf-columns two">
          <div class="cf-column">
            <h2 data-translate="why_captcha_headline">Why do I have to complete a CAPTCHA?</h2>
            
            <p data-translate="why_captcha_detail">Completing the CAPTCHA proves you are a human and gives you temporary access to the web property.</p>
          </div>

          <div class="cf-column">
            <h2 data-translate="resolve_captcha_headline">What can I do to prevent this in the future?</h2>
            

            <p data-translate="resolve_captcha_antivirus">If you are on a personal connection, like at home, you can run an anti-virus scan on your device to make sure it is not infected with malware.</p>

            <p data-translate="resolve_captcha_network">If you are at an office or shared network, you can ask the network administrator to run a scan across the network looking for misconfigured or infected devices.</p>
            
              
              <p data-translate="resolve_captcha_privacy_pass"> Another way to prevent getting this page in the future is to use Privacy Pass. You may need to download version 2.0 now from the <a rel="noopener noreferrer" href="https://addons.mozilla.org/en-US/firefox/addon/privacy-pass/">Firefox Add-ons Store</a>.</p>
              
            
          </div>
        </div>
      </div>
      

      <div class="cf-error-footer cf-wrapper w-240 lg:w-full py-10 sm:py-4 sm:px-8 mx-auto text-center sm:text-left border-solid border-0 border-t border-gray-300">
  <p class="text-13">
    <span class="cf-footer-item sm:block sm:mb-1">Cloudflare Ray ID: <strong class="font-semibold">6da274e4cace643e</strong></span>
    <span class="cf-footer-separator sm:hidden">&bull;</span>
    <span class="cf-footer-item sm:block sm:mb-1"><span>Your IP</span>: 130.51.200.132</span>
    <span class="cf-footer-separator sm:hidden">&bull;</span>
    <span class="cf-footer-item sm:block sm:mb-1"><span>Performance &amp; security by</span> <a rel="noopener noreferrer" href="https://www.cloudflare.com/5xx-error-landing" id="brand_link" target="_blank">Cloudflare</a></span>
    
  </p>
</div><!-- /.error-footer -->


    </div>
  </div>

  <script type="text/javascript">
  window._cf_translation = {};
  
  
</script>


</body>
</html>
5

Yep, looks like cloudflare flagged your IP address (or range of).
But you should have the same through browser, do you?

6

Its weird, it just takes me there, no captcha or restrictions
nocaptcha