Server crash when performing automatic backup?

Hello everyone, I struggle with a recurrent crash on my server, here is what happen:

My YunoHost server


Hardware: Raspberry Pi at home
YunoHost version: YunoHost 4.3.2.2 (stable).
I have access to my server : through the webadmin
Are you in a special context or did you perform some particular tweaking on your YunoHost instance ? : Installation on SD card but most of the data like Nextcloud and Synapse are on a USB HDD.

Description of my issue

Since 2 or 3 weeks in a row, my server crash when the automatic backup performed by the app Archivist is executed. 2 Apps logs failed backup operation: Nextcloud and Synapse, the others apps seems to backup correctly. (FreshRSS and Wallabag)

Here is the 2 fulls logs:

The errors in each case seems to be related to '[Errno 5] Input/output error'')]'
I don’t know if it helps.

I don’t see on the logs at what time exactly the serveur crash but on the morning when I try to access to my server it doesn’t respond and I have to reboot it by unplug - plug the power adapter.

Also I have read on this thread that the cause may be due to the backup destination being on the same HDD that the actual data, can it be the cause even if the backup perfomed well until then?

Last information, after checking my backup files:

  • The last Nextcloud file (4.5Go) has been made the 26/09 (I didn’t know It was this old).
  • The last Synapse (2.8Go) has been made the 17/10.

So maybe the backup operation and the crash are not entirely related since it didn’t crashed until recently.

I think that’s it, thanks for any help, have a good day!

1 Like

You should quickly test your hard drive with smartmontools to be sure your HDD is not dying.

You should do a copy of your disk if you can and buy a “spare” HDD to replace this one if needed.

Ok, thanks for your answer, i’ll make a copy of the disk. I tried smartmontools as you suggested but it tells me that my disk doesn’t have SMART capacity which is weird.

Here is the result:

sudo smartctl -i /dev/sda1

=== START OF INFORMATION SECTION ===
Vendor: TOSHIBA
Product: External USB 3.0
Revision: 5438
Compliance: SPC-4
User Capacity: 1,000,204,883,968 bytes [1.00 TB]
Logical block size: 512 bytes
Serial number: 20180708012742F
Device type: disk
Local Time is: Sun Nov 21 20:17:34 2021 UTC
SMART support is: Unavailable - device lacks SMART capability.

I tried the -T permissive argument with no success.


I also ran hdparm in order to have more information on my HDD and it seems to have SMART features:

hdparm /dev/sda1

/dev/sda1:
 multcount     = 16 (on)
 readonly      =  0 (off)
 readahead     = 256 (on)
 geometry      = 121601/255/63, sectors = 1953523116, start = 2048
root@cloud:/var/log# hdparm -I /dev/sda1

/dev/sda1:

ATA device, with non-removable media
	Model Number:       TOSHIBA MQ04UBF100                      
	Serial Number:      7884TXVJT
	Firmware Revision:  JU002U  
	Transport:          Serial, ATA8-AST, SATA 1.0a, SATA II Extensions, SATA Rev 2.5, SATA Rev 2.6, SATA Rev 3.0
Standards:
	Used: unknown (minor revision code 0x006d) 
	Supported: 10 9 8 7 6 5 
	Likely used: 10
Configuration:
	Logical		max	current
	cylinders	16383	16383
	heads		16	16
	sectors/track	63	63
	--
	CHS current addressable sectors:    16514064
	LBA    user addressable sectors:   268435455
	LBA48  user addressable sectors:  1953525168
	Logical  Sector size:                   512 bytes
	Physical Sector size:                  4096 bytes
	Logical Sector-0 offset:                  0 bytes
	device size with M = 1024*1024:      953869 MBytes
	device size with M = 1000*1000:     1000204 MBytes (1000 GB)
	cache/buffer size  = unknown
	Form Factor: 2.5 inch
	Nominal Media Rotation Rate: 5400
Capabilities:
	LBA, IORDY(can be disabled)
	Queue depth: 32
	Standby timer values: spec'd by Standard, no device specific minimum
	R/W multiple sector transfer: Max = 16	Current = 16
	Advanced power management level: 128
	DMA: sdma0 sdma1 sdma2 mdma0 mdma1 mdma2 udma0 udma1 udma2 udma3 udma4 *udma5 
	     Cycle time: min=120ns recommended=120ns
	PIO: pio0 pio1 pio2 pio3 pio4 
	     Cycle time: no flow control=120ns  IORDY flow control=120ns
Commands/features:
	Enabled	Supported:
	    	SMART feature set
	    	Security Mode feature set
	   *	Power Management feature set
	   *	Write cache
	   *	Look-ahead
	   *	Host Protected Area feature set
	   *	WRITE_BUFFER command
	   *	READ_BUFFER command
	   *	NOP cmd
	   *	DOWNLOAD_MICROCODE
	   *	Advanced Power Management feature set
	    	SET_MAX security extension
	   *	48-bit Address feature set
	   *	Device Configuration Overlay feature set
	   *	Mandatory FLUSH_CACHE
	   *	FLUSH_CACHE_EXT
	   *	SMART error logging
	   *	SMART self-test
	   *	General Purpose Logging feature set
	   *	WRITE_{DMA|MULTIPLE}_FUA_EXT
	   *	64-bit World wide name
	   *	IDLE_IMMEDIATE with UNLOAD
	    	Write-Read-Verify feature set
	   *	WRITE_UNCORRECTABLE_EXT command
	   *	{READ,WRITE}_DMA_EXT_GPL commands
	   *	Segmented DOWNLOAD_MICROCODE
	    	unknown 119[6]
	    	unknown 119[9]
	   *	Gen1 signaling speed (1.5Gb/s)
	   *	Gen2 signaling speed (3.0Gb/s)
	   *	Native Command Queueing (NCQ)
	   *	Host-initiated interface power management
	   *	Phy event counters
	   *	Idle-Unload when NCQ is active
	   *	Host automatic Partial to Slumber transitions
	   *	Device automatic Partial to Slumber transitions
	   *	READ_LOG_DMA_EXT equivalent to READ_LOG_EXT
	    	DMA Setup Auto-Activate optimization
	    	Device-initiated interface power management
	   *	Software settings preservation
	   *	SMART Command Transport (SCT) feature set
	   *	SCT Write Same (AC2)
	   *	SCT Error Recovery Control (AC3)
	   *	SCT Features Control (AC4)
	   *	SCT Data Tables (AC5)
	   *	reserved 69[1]
	   *	reserved 69[3]
	   *	DOWNLOAD MICROCODE DMA command
Security: 
	Master password revision code = 65534
		supported
	not	enabled
	not	locked
	not	frozen
	not	expired: security count
		supported: enhanced erase
	190min for SECURITY ERASE UNIT. 190min for ENHANCED SECURITY ERASE UNIT.
Logical Unit WWN Device Identifier: 0000000000000000
	NAA		: 0
	IEEE OUI	: 000000
	Unique ID	: 000000000
Checksum: correct

I don’t know how to check the health of my disk any further for now.

Be sure to have a copy elsewhere of what’s in the disk.
If you have any windows pc try

Otherwise try to connect it via sata connector.

Thanks for your reply, unfortunately I have no access to any windows PC right now (my main computer is on Mac OS).

Meanwhile I managed to make smartmontools works by adding -d -sat argument. Here is what I got after activating SMART and ran sudo smartctl -a -d sat /dev/sda1 :

root@cloud:~# sudo smartctl -a  -d sat /dev/sda1
smartctl 6.6 2017-11-05 r4594 [armv7l-linux-5.10.63-v7+] (local build)
Copyright (C) 2002-17, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Device Model:     TOSHIBA MQ04UBF100
Serial Number:    7884TXVJT
LU WWN Device Id: 0 000000 000000000
Firmware Version: JU002U
User Capacity:    1,000,204,886,016 bytes [1.00 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    5400 rpm
Form Factor:      2.5 inches
Device is:        Not in smartctl database [for details use: -P showall]
ATA Version is:   ACS-3 T13/2161-D revision 5
SATA Version is:  SATA 3.3, 3.0 Gb/s (current: 3.0 Gb/s)
Local Time is:    Mon Nov 22 00:10:52 2021 UTC
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART Status not supported: Incomplete response, ATA output registers missing
SMART overall-health self-assessment test result: PASSED
Warning: This result is based on an Attribute check.

General SMART Values:
Offline data collection status:  (0x00)	Offline data collection activity
					was never started.
					Auto Offline Data Collection: Disabled.
Self-test execution status:      (   0)	The previous self-test routine completed
					without error or no self-test has ever 
					been run.
Total time to complete Offline 
data collection: 		(  120) seconds.
Offline data collection
capabilities: 			 (0x5b) SMART execute Offline immediate.
					Auto Offline data collection on/off support.
					Suspend Offline collection upon new
					command.
					Offline surface scan supported.
					Self-test supported.
					No Conveyance Self-test supported.
					Selective Self-test supported.
SMART capabilities:            (0x0003)	Saves SMART data before entering
					power-saving mode.
					Supports SMART auto save timer.
Error logging capability:        (0x01)	Error logging supported.
					General Purpose Logging supported.
Short self-test routine 
recommended polling time: 	 (   2) minutes.
Extended self-test routine
recommended polling time: 	 ( 175) minutes.
SCT capabilities: 	       (0x003d)	SCT Status supported.
					SCT Error Recovery Control supported.
					SCT Feature Control supported.
					SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x000b   100   100   050    Pre-fail  Always       -       0
  2 Throughput_Performance  0x0005   100   100   050    Pre-fail  Offline      -       0
  3 Spin_Up_Time            0x0027   100   100   001    Pre-fail  Always       -       2564
  4 Start_Stop_Count        0x0032   100   100   000    Old_age   Always       -       51372
  5 Reallocated_Sector_Ct   0x0033   100   100   050    Pre-fail  Always       -       9800
  7 Seek_Error_Rate         0x000b   100   100   050    Pre-fail  Always       -       0
  8 Seek_Time_Performance   0x0005   100   100   050    Pre-fail  Offline      -       0
  9 Power_On_Hours          0x0032   037   037   000    Old_age   Always       -       25556
 10 Spin_Retry_Count        0x0033   253   100   030    Pre-fail  Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       85
191 G-Sense_Error_Rate      0x0032   100   100   000    Old_age   Always       -       8
192 Power-Off_Retract_Count 0x0032   100   100   000    Old_age   Always       -       29
193 Load_Cycle_Count        0x0032   085   085   000    Old_age   Always       -       156979
194 Temperature_Celsius     0x0022   100   100   000    Old_age   Always       -       32 (Min/Max 11/70)
196 Reallocated_Event_Count 0x0032   100   100   000    Old_age   Always       -       9800
197 Current_Pending_Sector  0x0032   100   100   000    Old_age   Always       -       8
198 Offline_Uncorrectable   0x0030   100   100   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x0032   200   253   000    Old_age   Always       -       46
220 Disk_Shift              0x0002   100   100   000    Old_age   Always       -       0
222 Loaded_Hours            0x0032   096   096   000    Old_age   Always       -       1989
223 Load_Retry_Count        0x0032   100   100   000    Old_age   Always       -       0
224 Load_Friction           0x0022   100   100   000    Old_age   Always       -       0
226 Load-in_Time            0x0026   100   100   000    Old_age   Always       -       198
240 Head_Flying_Hours       0x0001   100   100   001    Pre-fail  Offline      -       0

SMART Error Log Version: 1
ATA Error Count: 495 (device log contains only the most recent five errors)
	CR = Command Register [HEX]
	FR = Features Register [HEX]
	SC = Sector Count Register [HEX]
	SN = Sector Number Register [HEX]
	CL = Cylinder Low Register [HEX]
	CH = Cylinder High Register [HEX]
	DH = Device/Head Register [HEX]
	DC = Device Command Register [HEX]
	ER = Error register [HEX]
	ST = Status register [HEX]
Powered_Up_Time is measured from power on, and printed as
DDd+hh:mm:SS.sss where DD=days, hh=hours, mm=minutes,
SS=sec, and sss=millisec. It "wraps" after 49.710 days.

Error 495 occurred at disk power-on lifetime: 25555 hours (1064 days + 19 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  40 51 08 60 b6 16 48  Error: UNC 8 sectors at LBA = 0x0816b660 = 135706208

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  25 00 08 60 b6 16 40 00      01:20:49.954  READ DMA EXT
  aa aa aa aa aa aa aa ff      01:20:49.939  [RESERVED]
  25 00 08 60 b6 16 40 00      01:20:49.489  READ DMA EXT
  35 00 f0 d0 0a dc 40 00      01:20:49.483  WRITE DMA EXT
  35 00 f0 e0 09 dc 40 00      01:20:49.479  WRITE DMA EXT

Error 494 occurred at disk power-on lifetime: 25555 hours (1064 days + 19 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  40 51 08 60 b6 16 48  Error: UNC 8 sectors at LBA = 0x0816b660 = 135706208

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  25 00 08 60 b6 16 40 00      01:20:49.489  READ DMA EXT
  35 00 f0 d0 0a dc 40 00      01:20:49.483  WRITE DMA EXT
  35 00 f0 e0 09 dc 40 00      01:20:49.479  WRITE DMA EXT
  35 00 f0 f0 08 dc 40 00      01:20:49.475  WRITE DMA EXT
  25 00 08 58 b6 16 40 00      01:20:49.379  READ DMA EXT

Error 493 occurred at disk power-on lifetime: 25555 hours (1064 days + 19 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  40 51 30 60 b6 16 48  Error: UNC 48 sectors at LBA = 0x0816b660 = 135706208

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  25 00 f0 a0 b5 16 40 00      01:20:15.672  READ DMA EXT
  35 00 f0 58 e1 db 40 00      01:20:15.668  WRITE DMA EXT
  25 00 10 90 b5 16 40 00      01:20:15.667  READ DMA EXT
  35 00 f0 68 e0 db 40 00      01:20:15.663  WRITE DMA EXT
  25 00 f0 a0 b4 16 40 00      01:20:15.109  READ DMA EXT

Error 492 occurred at disk power-on lifetime: 25555 hours (1064 days + 19 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  84 51 70 80 1f 91 40  Error: ICRC, ABRT 112 sectors at LBA = 0x00911f80 = 9510784

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  25 00 f0 00 1f 91 40 00      00:45:54.819  READ DMA EXT
  25 00 10 f0 1e 91 40 00      00:45:54.818  READ DMA EXT
  25 00 f0 00 1e 91 40 00      00:45:54.814  READ DMA EXT
  25 00 10 f0 1d 91 40 00      00:45:54.814  READ DMA EXT
  25 00 f0 00 1d 91 40 00      00:45:54.810  READ DMA EXT

Error 491 occurred at disk power-on lifetime: 25554 hours (1064 days + 18 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  40 51 48 78 85 04 48  Error: UNC 72 sectors at LBA = 0x08048578 = 134514040

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  25 00 f0 d0 84 04 40 00      00:14:08.700  READ DMA EXT
  25 00 10 c0 84 04 40 00      00:14:08.699  READ DMA EXT
  25 00 f0 d0 83 04 40 00      00:14:08.666  READ DMA EXT
  25 00 10 c0 83 04 40 00      00:14:08.665  READ DMA EXT
  25 00 f0 d0 82 04 40 00      00:14:08.647  READ DMA EXT

SMART Self-test log structure revision number 1
No self-tests have been logged.  [To run self-tests, use: smartctl -t]

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

root@cloud:~# 


Sounds pretty bad right? Does it mean I need to buy another HDD asap? if so, do you have any advice, this one was not that old, even if I don’t know the average life time for that kind of always on usage.

Thanks

Not good, seems dying.
You have a backup elsewhere I hope.
You can run a full test.

To run a shorttest

sudo smartctl -t short /dev/sdX

2 min after

sudo smartctl -l selftest /dev/sdX
1 Like

Ok so I just ran a short test as suggested, here is what I got:

=== START OF READ SMART DATA SECTION ===
SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Short offline       Completed without error       00%     25558         -

I think I need to try the long one, I will update this post once it’s done.

Ok so I tried 2 times, but the test seems to abort I don’t know why, the status is ‘Aborted by host’ while I didn’t do anything aside launching the long test command

sudo smartctl -t long -d sat /dev/sda1
waiting 175minutes then
sudo smartctl -l selftest -T permissive -d sat /dev/sda1

=== START OF READ SMART DATA SECTION ===
SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline    Aborted by host               90%     25607         -
# 2  Extended offline    Aborted by host               90%     25562         -
# 3  Short offline       Completed without error       00%     25558         -