215:
small because they stored tiny amounts of data. In modern disks the probability is much larger because they store much more data, whilst not being safer. That way, silent data corruption has not been a serious concern while storage devices remained relatively small and slow. In modern times and with the advent of larger drives and very fast RAID setups, users are capable of transferring 10 bits in a reasonably short time, thus easily reaching the data corruption thresholds.
100:
38:
108:
214:
One problem is that hard disk drive capacities have increased substantially, but their error rates remain unchanged. The data corruption rate has always been roughly constant in time, meaning that modern disks are not much safer than old disks. In old disks the probability of data corruption was very
133:
Data corruption can occur at any level in a system, from the host to the storage medium. Modern systems attempt to detect corruption at many layers and then recover or correct the corruption; this is almost always successful but very rarely the information arriving in the systems memory is corrupted
381:
is another method to reduce the likelihood of data corruption, as disk errors are caught and recovered from before multiple errors accumulate and overwhelm the number of parity bits. Instead of parity being checked on each read, the parity is checked during a regular scan of the disk, often done as
333:
Many errors are detected and corrected by the hard disk drives using the ECC codes which are stored on disk for each sector. If the disk drive detects multiple read errors on a sector it may make a copy of the failing sector on another part of the disk, by remapping the failed sector of the disk to
130:, results in the most dangerous errors as there is no indication that the data is incorrect. Detected data corruption may be permanent with the loss of data, or may be temporary when some part of the system is able to detect and correct the error; there is no data corruption in the latter case.
382:
a low priority background process. The "data scrubbing" operation activates a parity check. If a user simply runs a normal program that reads data from the disk, then the parity would not be checked unless parity-check-on-read was both supported and enabled on the disk subsystem.
90:
Some programs can give a suggestion to repair the file automatically (after the error), and some programs cannot repair it. It depends on the level of corruption, and the built-in functionality of the application to handle the error. There are various causes of the corruption.
141:. Environmental conditions can interfere with data transmission, especially when dealing with wireless transmission methods. Heavy clouds can block satellite transmissions. Wireless networks are susceptible to interference from devices such as microwave ovens.
210:
has acknowledged similar high data corruption rates in their systems. In 2021, faulty processor cores were identified as an additional cause in publications by Google and
Facebook; cores were found to be faulty at a rate of several in thousands of cores.
72:, thus the file might not be opened or might open with some of the data corrupted (or in some cases, completely corrupted, leaving the document unintelligible). The adjacent image is a corrupted image file in which most of the information has been lost.
193:
There are many error sources beyond the disk storage subsystem itself. For instance, cables might be slightly loose, the power supply might be unreliable, external vibrations such as a loud sound, the network might introduce undetected corruption,
114:
113:
110:
109:
115:
908:; Bairavasundaram, L., Goodson, G., Schroeder, B., Arpaci-Dusseau, A. C., Arpaci-Dusseau, R. H. 2008. An analysis of data corruption in the storage stack. In Proceedings of 6th Usenix Conference on File and Storage Technologies.
52:
that occur during writing, reading, storage, transmission, or processing, which introduce unintended changes to the original data. Computer, transmission, and storage systems use a number of measures to provide end-to-end
990:
112:
373:, compared with other data integrity approaches that do not span different layers in the storage stack and allow data corruption to occur while the data passes boundaries between the different layers.
544:
Silent Data
Corruption (SDC), sometimes referred to as Silent Data Error (SDE), is an industry-wide issue impacting not only long-protected memory, storage, and networking, but also computer CPUs.
230:
on more than 1.5 million HDDs over 41 months found more than 400,000 silent data corruptions, out of which more than 30,000 were not detected by the hardware RAID controller (only detected during
701:
253:, in which the system may run for a period of time with undetected initial error causing increasingly more problems until it is ultimately detected. For example, a failure affecting file system
369:, such file systems can also reconstruct corrupted data in a transparent way. This approach allows improved data integrity protection covering the entire data paths, which is usually known as
226:, which is a database software company specializing in large-scale data warehousing and analytics, faces silent corruption every 15 minutes. As another example, a real-life study performed by
64:
containing that data will produce unexpected results when accessed by the system or the related application. Results could range from a minor loss of data to a system crash. For example, if a
87:
with this payload method manages to alter files critical to the running of the computer's operating system software or physical hardware, the entire system may be rendered unusable.
727:
334:
a spare sector without the involvement of the operating system (though this may be delayed until the next write to the sector). This "silent correction" can be monitored using
385:
If appropriate mechanisms are employed to detect and remedy data corruption, data integrity can be maintained. This is particularly important in commercial applications (e.g.
979:
531:
335:
1049:
306:
for data across a set of hard disks and can reconstruct corrupted data upon the failure of a single or multiple disks, depending on the level of RAID implemented. Some
83:, usually by overwriting them with inoperative or garbage code, while a non-malicious virus may also unintentionally corrupt files when it accesses them. If a virus or
111:
338:
and tools available for most operating systems to automatically check the disk drive for impending failures by watching for deteriorating SMART parameters.
779:
741:
Hochschild, Peter H.; Turner, Paul Jack; Mogul, Jeffrey C.; Govindaraju, Rama
Krishna; Ranganathan, Parthasarathy; Culler, David E.; Vahdat, Amin (2021).
365:
checksumming to detect silent data corruption. In addition, if a corruption is detected and the file system uses integrated RAID mechanisms that provide
919:
665:
202:, etc. In 39,000 storage systems that were analyzed, firmware bugs accounted for 5–10% of storage failures. All in all, the error rates as observed by a
693:
1165:
1159:
510:
181:
829:
636:
389:), where an undetected error could either corrupt a database index or change data to drastically affect an account balance, and in the use of
126:
There are two types of data corruption associated with computer systems: undetected and detected. Undetected data corruption, also known as
299:
723:
257:
can result in multiple files being partially damaged or made completely inaccessible as the file system is used in its corrupted state.
557:
476:
765:
1018:
1041:
186:
Some errors go unnoticed, without being detected by the disk firmware or the host operating system; these errors are known as
1193:
954:
659:"Are Disks the Dominant Contributor for Storage Failures? A Comprehensive Study of Storage Subsystem Failure Characteristics"
461:
284:
266:
896:
862:
602:
279:
of data has an independently low probability of being changed, data corruption can generally be detected by the use of
1480:
157:
1130:
294:
If an uncorrectable data corruption is detected, procedures such as automatic retransmission or restoration from
742:
1277:
1273:
456:
84:
938:
David Fiala; Frank
Mueller; Christian Engelmann; Rolf Riesen; Kurt Ferreira; Ron Brightwell (November 2012).
1202:
506:
160:
or wear of the storage device fall into the former category, while software failure typically occurs due to
502:
1304:
1086:
1145:
1075:
1475:
1071:
884:
288:
1155:
658:
625:
137:
Data corruption during transmission has a variety of causes. Interruption of data transmission causes
1434:
833:
446:
350:
319:
149:
65:
1091:
1374:
1359:
1287:
498:
431:
80:
1186:
1096:
1045:
809:
771:
426:
99:
940:"Detection and Correction of Silent Data Corruption for Large-Scale High-Performance Computing"
1364:
1354:
1218:
980:"Rachet Up Reliability for Mission-Critical Applications: Intel Instruction Replay Technology"
761:
250:
41:
Photo data corruption; in this case, a result of a failed data recovery from a hard disk drive
1141:
Detection and
Correction of Silent Data Corruption for Large-Scale High-Performance Computing
246:
of data became permanently corrupted silently somewhere in the pathway from network to disk.
1314:
1263:
1248:
1228:
1213:
1150:
1104:
753:
416:
410:
394:
31:
565:
68:
is corrupted, when a person tries to open that file with a document editor they may get an
1444:
1379:
1369:
1339:
1282:
1243:
366:
310:
architectures employ various transparent checks to detect and mitigate data corruption in
272:
37:
1454:
1449:
1414:
1394:
1389:
1344:
1319:
1238:
421:
377:
231:
54:
1469:
1419:
1409:
1384:
1258:
1223:
1179:
775:
327:
69:
61:
49:
1429:
1424:
1404:
1399:
1329:
1324:
1299:
1292:
1268:
1100:
1014:
436:
342:
161:
1140:
939:
1349:
1309:
315:
1135:
803:
591:
451:
390:
303:
207:
199:
195:
169:
165:
153:
27:
Errors in computer data that introduce unintended changes to the original data
1136:
A Tunable, Software-based DRAM Error
Detection and Correction Library for HPC
888:
858:
206:
study on silent corruption are far higher than one in every 10 bits. Webshop
1439:
1334:
1108:
757:
311:
243:
239:
223:
145:
138:
1233:
466:
362:
280:
254:
17:
724:"Observations on Errors, Corrections, & Trust of Dependent Systems"
386:
76:
295:
227:
1166:
End-to-end Data
Protection in SAS and Fibre Channel Hard Disk Drives
986:
397:
data, where a small error can make an extensive dataset unusable.
346:
106:
98:
36:
562:
Oracle – Core Dumps of a Kernel Hacker's Brain – Eric Lowe's Blog
950:
471:
354:
235:
203:
1175:
1076:"End-to-end data integrity for file systems: a ZFS case study"
750:
Proceedings of the
Workshop on Hot Topics in Operating Systems
358:
307:
276:
219:
121:
Epilepsy warning: This video contains bright, flashing images.
30:"Corrupted" redirects here. For the Japanese metal band, see
1146:
End-to-end Data
Integrity for File Systems: A ZFS Case Study
927:. 8th Annual Workshop on Linux Clusters for Super Computing.
861:. Association for Computing Machinery. November 15, 2007.
144:
Hardware and software failure are the two main causes for
1131:
SoftECC: A System for
Software Memory Integrity Checking
1171:
222:
creator Jeff Bonwick stated that the fast database at
830:"Silent data corruption in disk arrays: A solution"
302:disk arrays have the ability to store and evaluate
1151:DRAM Errors in the Wild: A Large-Scale Field Study
1083:USENIX Conference on File and Storage Technologies
1015:"Read Error Severities and Error Management Logic"
859:"A Conversation with Jeff Bonwick and Bill Moore"
805:HotOS 2021: Cores That Don't Count (Fun Hardware)
79:may intentionally corrupt files as part of their
1042:"How I Use the Advanced Capabilities of Btrfs"
1040:Margaret Bierman; Lenz Grimmer (August 2012).
1187:
8:
238:over six months and involving about 97
60:In general, when data corruption occurs, a
1194:
1180:
1172:
687:
685:
103:Photo of an Atari 2600 with corrupted RAM.
1090:
889:"Keeping Bits Safe: How Hard Can It Be?"
182:Hard disk drive error rates and handling
490:
899:from the original on December 17, 2013
692:Bernd Panzer-Steindel (8 April 2007).
249:Silent data corruption may result in
134:and can cause unpredictable results.
7:
704:from the original on 27 October 2012
242:of data, found that about 128
1074:; Remzi H. Arpaci-Dusseau (2010).
639:from the original on 26 April 2012
326:technology, which is available on
298:can be applied. Certain levels of
271:When data corruption behaves as a
25:
1021:from the original on 7 April 2012
865:from the original on 16 July 2011
119:A video that has been corrupted.
1070:Yupu Zhang; Abhishek Rajimwale;
730:from the original on 2013-10-29.
605:from the original on 3 July 2012
234:). Another study, performed by
1052:from the original on 2014-01-02
996:from the original on 2016-02-02
960:from the original on 2014-11-07
812:from the original on 2021-12-22
785:from the original on 2021-06-03
671:from the original on 2022-01-25
513:from the original on 2010-12-26
590:bcantrill (31 December 2008).
556:Eric Lowe (16 November 2005).
462:List of data recovery software
267:Error detection and correction
1:
1158:, and an associated paper on
1156:A study on silent corruptions
477:Reed–Solomon error correction
624:jforonda (31 January 2007).
592:"Shouting in the Datacenter"
832:. NEC. 2009. Archived from
1497:
626:"Faulty FC port meets ZFS"
503:"Solar Storms: Fast Facts"
371:end-to-end data protection
264:
179:
29:
1209:
633:Blogger – Outside the Box
558:"ZFS saves the day(-ta)!"
198:and many other causes of
743:"Cores that don't count"
564:. Oracle. Archived from
532:"Silent Data Corruption"
457:Forward error correction
361:, use internal data and
324:Intel Instruction Replay
758:10.1145/3458336.3465297
507:Nature Publishing Group
978:Steve Bostian (2012).
413:, also called data rot
289:error correcting codes
188:silent data corruption
128:silent data corruption
123:
104:
42:
1072:Andrea Arpaci-Dusseau
885:David S. H. Rosenthal
320:instruction pipelines
118:
102:
57:, or lack of errors.
40:
1375:Protection (privacy)
447:Data Integrity Field
150:Background radiation
48:refers to errors in
887:(October 1, 2010).
499:Scientific American
432:Radiation hardening
407:Various resources:
283:, and can often be
1481:Product expiration
1046:Oracle Corporation
921:Silent corruptions
839:on 29 October 2013
571:on 5 February 2012
534:. Google Inc. 2023
427:Database integrity
251:cascading failures
200:soft memory errors
124:
105:
43:
1463:
1462:
1455:Wrangling/munging
1305:Format management
752:. pp. 9–16.
443:Countermeasures:
116:
16:(Redirected from
1488:
1196:
1189:
1182:
1173:
1119:
1118:
1116:
1115:
1094:
1080:
1067:
1061:
1060:
1058:
1057:
1037:
1031:
1030:
1028:
1026:
1011:
1005:
1004:
1002:
1001:
995:
984:
975:
969:
968:
966:
965:
959:
944:
935:
929:
928:
926:
915:
909:
907:
905:
904:
881:
875:
874:
872:
870:
855:
849:
848:
846:
844:
838:
826:
820:
819:
818:
817:
800:
794:
793:
791:
790:
784:
747:
738:
732:
731:
720:
714:
713:
711:
709:
689:
680:
679:
677:
676:
670:
663:
655:
649:
648:
646:
644:
630:
621:
615:
614:
612:
610:
596:
587:
581:
580:
578:
576:
570:
553:
547:
546:
541:
539:
528:
522:
521:
519:
518:
495:
417:Computer science
411:Data degradation
322:; an example is
196:cosmic radiation
139:information loss
117:
32:Corrupted (band)
21:
1496:
1495:
1491:
1490:
1489:
1487:
1486:
1485:
1466:
1465:
1464:
1459:
1435:Synchronization
1205:
1200:
1127:
1122:
1113:
1111:
1092:10.1.1.154.3979
1078:
1069:
1068:
1064:
1055:
1053:
1039:
1038:
1034:
1024:
1022:
1013:
1012:
1008:
999:
997:
993:
982:
977:
976:
972:
963:
961:
957:
942:
937:
936:
932:
924:
917:
916:
912:
902:
900:
883:
882:
878:
868:
866:
857:
856:
852:
842:
840:
836:
828:
827:
823:
815:
813:
802:
801:
797:
788:
786:
782:
768:
745:
740:
739:
735:
722:
721:
717:
707:
705:
691:
690:
683:
674:
672:
668:
661:
657:
656:
652:
642:
640:
628:
623:
622:
618:
608:
606:
594:
589:
588:
584:
574:
572:
568:
555:
554:
550:
537:
535:
530:
529:
525:
516:
514:
497:
496:
492:
488:
483:
403:
367:data redundancy
273:Poisson process
269:
263:
261:Countermeasures
218:As an example,
184:
178:
107:
97:
46:Data corruption
35:
28:
23:
22:
15:
12:
11:
5:
1494:
1492:
1484:
1483:
1478:
1468:
1467:
1461:
1460:
1458:
1457:
1452:
1447:
1442:
1437:
1432:
1427:
1422:
1417:
1412:
1407:
1402:
1397:
1392:
1387:
1382:
1377:
1372:
1367:
1362:
1360:Pre-processing
1357:
1352:
1347:
1342:
1337:
1332:
1327:
1322:
1317:
1312:
1307:
1302:
1297:
1296:
1295:
1290:
1285:
1271:
1266:
1261:
1256:
1251:
1246:
1241:
1236:
1231:
1226:
1221:
1216:
1210:
1207:
1206:
1201:
1199:
1198:
1191:
1184:
1176:
1170:
1169:
1163:
1160:data integrity
1153:
1148:
1143:
1138:
1133:
1126:
1125:External links
1123:
1121:
1120:
1062:
1032:
1006:
970:
930:
910:
876:
850:
821:
795:
766:
733:
715:
698:Data integrity
681:
650:
616:
582:
548:
523:
501:(2008-07-21).
489:
487:
484:
482:
481:
480:
479:
474:
469:
464:
459:
454:
449:
441:
440:
439:
434:
429:
424:
422:Data integrity
419:
414:
404:
402:
399:
378:Data scrubbing
287:by the use of
262:
259:
177:
174:
96:
93:
75:Some types of
55:data integrity
26:
24:
14:
13:
10:
9:
6:
4:
3:
2:
1493:
1482:
1479:
1477:
1474:
1473:
1471:
1456:
1453:
1451:
1448:
1446:
1443:
1441:
1438:
1436:
1433:
1431:
1428:
1426:
1423:
1421:
1418:
1416:
1413:
1411:
1408:
1406:
1403:
1401:
1398:
1396:
1393:
1391:
1388:
1386:
1383:
1381:
1378:
1376:
1373:
1371:
1368:
1366:
1363:
1361:
1358:
1356:
1353:
1351:
1348:
1346:
1343:
1341:
1338:
1336:
1333:
1331:
1328:
1326:
1323:
1321:
1318:
1316:
1313:
1311:
1308:
1306:
1303:
1301:
1298:
1294:
1291:
1289:
1286:
1284:
1281:
1280:
1279:
1275:
1272:
1270:
1267:
1265:
1262:
1260:
1257:
1255:
1252:
1250:
1247:
1245:
1242:
1240:
1237:
1235:
1232:
1230:
1227:
1225:
1222:
1220:
1217:
1215:
1212:
1211:
1208:
1204:
1197:
1192:
1190:
1185:
1183:
1178:
1177:
1174:
1167:
1164:
1161:
1157:
1154:
1152:
1149:
1147:
1144:
1142:
1139:
1137:
1134:
1132:
1129:
1128:
1124:
1110:
1106:
1102:
1098:
1093:
1088:
1084:
1077:
1073:
1066:
1063:
1051:
1047:
1043:
1036:
1033:
1020:
1016:
1010:
1007:
992:
988:
981:
974:
971:
956:
952:
948:
941:
934:
931:
923:
922:
914:
911:
898:
894:
890:
886:
880:
877:
864:
860:
854:
851:
835:
831:
825:
822:
811:
807:
806:
799:
796:
781:
777:
773:
769:
767:9781450384384
763:
759:
755:
751:
744:
737:
734:
729:
725:
719:
716:
703:
699:
695:
688:
686:
682:
667:
660:
654:
651:
638:
634:
627:
620:
617:
604:
600:
593:
586:
583:
567:
563:
559:
552:
549:
545:
533:
527:
524:
512:
508:
504:
500:
494:
491:
485:
478:
475:
473:
470:
468:
465:
463:
460:
458:
455:
453:
450:
448:
445:
444:
442:
438:
435:
433:
430:
428:
425:
423:
420:
418:
415:
412:
409:
408:
406:
405:
400:
398:
396:
392:
388:
383:
380:
379:
374:
372:
368:
364:
360:
356:
352:
348:
344:
339:
337:
331:
329:
328:Intel Itanium
325:
321:
317:
313:
309:
305:
301:
297:
292:
290:
286:
282:
278:
275:, where each
274:
268:
260:
258:
256:
252:
247:
245:
241:
237:
233:
229:
225:
221:
216:
212:
209:
205:
201:
197:
191:
189:
183:
175:
173:
171:
167:
164:in the code.
163:
159:
155:
151:
147:
142:
140:
135:
131:
129:
122:
101:
94:
92:
88:
86:
82:
78:
73:
71:
70:error message
67:
66:document file
63:
58:
56:
51:
50:computer data
47:
39:
33:
19:
1476:Data quality
1365:Preservation
1355:Philanthropy
1253:
1219:Augmentation
1162:(CERN, 2007)
1112:. Retrieved
1082:
1065:
1054:. Retrieved
1035:
1023:. Retrieved
1009:
998:. Retrieved
973:
962:. Retrieved
946:
933:
920:
918:Kelemen, P.
913:
901:. Retrieved
892:
879:
867:. Retrieved
853:
841:. Retrieved
834:the original
824:
814:, retrieved
804:
798:
787:. Retrieved
749:
736:
718:
706:. Retrieved
697:
673:. Retrieved
653:
641:. Retrieved
632:
619:
607:. Retrieved
598:
595:(Video file)
585:
573:. Retrieved
566:the original
561:
551:
543:
536:. Retrieved
526:
515:. Retrieved
493:
437:Software rot
384:
376:
375:
370:
343:file systems
340:
332:
330:processors.
323:
293:
270:
248:
217:
213:
192:
187:
185:
154:head crashes
143:
136:
132:
127:
125:
120:
89:
74:
59:
45:
44:
1425:Stewardship
1315:Integration
1264:Degradation
1249:Compression
1229:Archaeology
1214:Acquisition
869:14 December
843:14 December
694:"Draft 1.3"
538:January 30,
316:CPU buffers
304:parity bits
170:soft errors
168:cause most
166:Cosmic rays
1470:Categories
1445:Validation
1380:Publishing
1370:Processing
1340:Management
1254:Corruption
1244:Collection
1114:2014-08-12
1109:Q111972797
1056:2014-01-02
1000:2016-01-27
964:2015-01-26
903:2014-01-02
816:2021-06-02
789:2021-06-02
675:2014-01-18
664:. USENIX.
517:2009-12-08
486:References
452:ECC memory
395:compressed
345:, such as
336:S.M.A.R.T.
312:CPU caches
265:See also:
208:Amazon.com
180:See also:
1450:Warehouse
1415:Scrubbing
1395:Retention
1390:Reduction
1345:Migration
1320:Integrity
1288:Transform
1239:Cleansing
1087:CiteSeerX
893:ACM Queue
776:235311320
391:encrypted
285:corrected
281:checksums
244:megabytes
240:petabytes
232:scrubbing
224:Greenplum
172:in DRAM.
146:data loss
18:Corrupted
1420:Security
1410:Scraping
1385:Recovery
1259:Curation
1224:Analysis
1105:Wikidata
1050:Archived
1019:Archived
991:Archived
955:Archived
947:fiala.me
897:Archived
863:Archived
810:archived
780:Archived
728:Archived
702:Archived
700:. CERN.
666:Archived
637:Archived
603:Archived
511:Archived
467:Parchive
401:See also
363:metadata
255:metadata
95:Overview
81:payloads
1430:Storage
1405:Science
1400:Quality
1330:Lineage
1325:Library
1300:Farming
1283:Extract
1269:Editing
1101:5722163
1025:4 April
599:YouTube
387:banking
296:backups
291:(ECC).
77:malware
1350:Mining
1310:Fusion
1168:(HGST)
1107:
1099:
1089:
774:
764:
708:9 June
643:9 June
629:(Blog)
609:9 June
575:9 June
569:(Blog)
357:, and
351:HAMMER
228:NetApp
176:Silent
156:, and
85:trojan
1097:S2CID
1079:(PDF)
994:(PDF)
987:Intel
983:(PDF)
958:(PDF)
943:(PDF)
925:(PDF)
837:(PDF)
783:(PDF)
772:S2CID
746:(PDF)
669:(PDF)
662:(PDF)
347:Btrfs
341:Some
158:aging
1440:Type
1335:Loss
1293:Load
1203:Data
1027:2012
951:IEEE
871:2020
845:2020
762:ISBN
710:2012
645:2012
611:2012
577:2012
540:2023
472:RAID
355:ReFS
318:and
300:RAID
236:CERN
204:CERN
162:bugs
62:file
1278:ELT
1274:ETL
1234:Big
754:doi
393:or
359:ZFS
308:CPU
277:bit
220:ZFS
148:.
1472::
1103:.
1095:.
1085:.
1081:.
1048:.
1044:.
1017:.
989:.
985:.
953:.
949:.
945:.
895:.
891:.
808:,
778:.
770:.
760:.
748:.
726:.
696:.
684:^
635:.
631:.
601:.
597:.
560:.
542:.
509:.
505:.
353:,
349:,
314:,
190:.
152:,
1276:/
1195:e
1188:t
1181:v
1117:.
1059:.
1029:.
1003:.
967:.
906:.
873:.
847:.
792:.
756::
712:.
678:.
647:.
613:.
579:.
520:.
34:.
20:)
Text is available under the Creative Commons Attribution-ShareAlike License. Additional terms may apply.
