Is there a way to auto-reboot when miner goes offline?

OS setup and Linux
1

I wonder if there is a script that if the rig cannot communicate through API (or is just offline) it initiates reboot or shutdown & boot in 30s. Sometimes it gets kinda stuck, the rig is working but not mining and looks offline and not all rigs are always physically accessible.

Temporary solution: I decided to buy smart plugs (15 usd here), so when it goes offline, I will shut down from android app. Mobo has a feature for power cuts to reboot automatically when power is back.

1 Like
2

I have very similar approach with Woox smart plugs and IFTTT. So I ping the HiveOS API every 2 minutes, if the rig is offline for 4 or more minutes I send command automatically to IFTTT and that resets the plug which resets the serverā€¦ not the best solution, but at least something.

I tried to set up the watchdog, got 2 USB watchdogs but was not able to make them work fully reliably.

3

How did you set this up? Could you walk me through doing the same thing, what are you using to ping HiveOS API? and then how to send that to IFTTT?

4

Hello my friend, could you explain how configure the IFTTT for ping and restart rig?

Or does anyone have any other solution? I also have a smart plug and I reset the rig remotely, but for that I need to know itā€™s locked, so I lose hours of mining.

5

@dncz explains his setup here: Rig total freeze, requires power reset - #9 by dncz
I started to replicate his setup, but realized my smart plugs do not work with IFFFT. One of my rigs has been crashing in the middle of the night and not restarting, so I wrote a simple cron script. Iā€™m not entirely sure what happens to the system when it freezes up, so I donā€™t know whether the cron script will successfully run, but I guess Iā€™ll find out. Here are the steps:

  1. Login to hiveos.farm (on a browser), click on your username in the top right corner, scroll down, and click on ā€œGenerate new Personal API-token.ā€ Once itā€™s generated, click on ā€œShowā€ and copy the 220 character API key and save it somewhere.
  2. Go to hiveos.farm and click on your farm, then copy and save your farm id from the url. Select each worker from the drop down list, then copy and save your worker id from the url. The format is: https://the.hiveos.farm/farms/{$farmid}/workers/{$workerid}/
    You can test whether the api is working with the following command in Terminal (replace {$apikey} and {$farmid} with that of your own api key and farm id):
    curl -s -w '\n%{http_code}' -H 'Content-Type: application/json' -H 'Authorization: Bearer {$apikey}' https://api2.hiveos.farm/api/v2/farms/{$farmid}
  3. SSH into your rig and create a reboot script. For instance: mkdir -p /home/user/scripts ; nano /home/user/scripts/worker1reboot.sh
    Paste the following code into your script, replacing {$API-Key}, {$farmid}, and {$workerid} with that of your own api key, farm id, and worker id:

#!/bin/bash
worker_status=$( curl -s -w '\n%{http_code}' -H 'Content-Type: application/json' -H 'Authorization: Bearer {$apikey}' https://api2.hiveos.farm/api/v2/farms/{$farmid}/workers/{$workerid} | grep -o '"stats":{"online":[a-z]*' | sed 's/[ ":,{]//g;s/stats//;s/online//' )
datetime=$(date '+%Y-%m-%d_%H:%M:%S')
if [[ "$worker_status" = 'false' ]] ; then
echo "cron script initiated at $datetime" >> /home/user/worker1reboot.log
sleep 300
if [[ "$worker_status" = 'false' ]] ; then
echo "cron script restarted worker1 at $datetime" >> /home/user/worker1reboot.log
sudo reboot
fi
elif [[ "$worker_status" != 'true' ]] && [[ "$worker_status" != 'false' ]] ; then
echo "cron script initiated at $datetime, but was unable to check the online status of worker1" >> /home/user/worker1reboot.log
fi

  1. Save the script by pressing ctrl+x, then y, then enter. Then make the script executable:
    chmod +x /home/user/scripts/worker1reboot.sh
  2. Create a new crontab with crontab -e. Scroll to the bottom of the window and press enter. Paste this on the last line:
    */5 * * * * /home/user/scripts/worker1reboot.sh
    Assuming youā€™re editing the crontab with Internal File Editor, which is the default editor for hiveos, press enter again so thereā€™s another blank line on the bottom, otherwise the editor will complain about it. Then press FN+F10 and then y to save your changes. The cron script will run every five minutes and check the online status of your worker. If the status shows offline, it will wait for five minutes and check the status again. If itā€™s still offline, it will reboot the rig.
1 Like
6

Iā€™m hoping this resolves the offline issue Iā€™ve been having, thank you. I have one slight modification. when I use crontab -e, the changes disappear after I exit crontab. I found a solution, and that is to use the following command:
sudo nano /hive/etc/crontab.root
then I add
*/5 * * * * /home/user/scripts/worker1reboot.sh
to the file, press control +x to exit, yes to save the changes, and from there, the script will be included in crontab.
Iā€™m waiting to confirm this resolves the offline issue, but Iā€™m hopeful. Thanks for your help!

7

Can you put a screenshot of this?? I dont really know how can i do it.

Thanks.

8

I wrote a script that will set everything up for you, as long as youā€™re using Teamredminer or lolminer (or both in a dual mining setup). I canā€™t confirm that all of the checks work, but in theory they should. I also set it up as a timer/service instead of a cron job, so I could have it wait until the rig has been booted for 5 minutes to start running checks. Any feedback would be appreciated. These random freezes are really annoyingā€¦ canā€™t believe hiveos hasnā€™t fixed it yet.
Follow the instructions here: Bash script to reboot rig that is offline or has a dead gpu

9

Solution to the freezing problem is here:

1 Like
10

This topic was automatically closed 416 days after the last reply. New replies are no longer allowed.