Knowledge (XXG)

Log-structured File System (BSD)

Source 📝

83:. Each summary block contains a pointer to the next summary block, linking segments into one long chain that LFS treats as a linear log. The segments do not necessarily have to be adjacent to each other on disk; for this reason, larger segment sizes (between 384KB and 1MB) are recommended because they amortize the cost of seeking between segments. 132:
Generally, garbage collection is delayed until there are no clean segments left; it can also be deferred for when the system is idle. Even then, only the least-dirty segments are picked for collection. This is intended to avoid the penalty of cleaning full segments when I/O bandwidth is most
71:
and directory formats are almost identical. This allows well-tested UFS file system code to be re-used; current implementations of LFS share the higher-level UFS code with the lower-level code for FFS, since both of these file systems share much in common with UFS.
103:
Unlike UFS, inodes in LFS do not have fixed locations. An inode map—a flat list of inode block locations—is used to track them. As with everything else, inode map blocks are also written to the log when they are changed.
207:
Matthews, Jeanna Neefe; Roselli, Drew; Costello, Adam M.; Wang, Randolph Y.; Anderson, Thomas E. (October 1997), "Improving the performance of log-structured file systems with adaptive methods",
129:
ones by copying live blocks from the dirty segment into the current segment and skipping the rest. The summary block in each segment contains a map to track live blocks.
172:
LFS becomes progressively less efficient as it nears maximum capacity, when the garbage collector has to run almost constantly to make clean segments available.
140:(usually scheduled about once every 30 seconds), LFS writes the last known block locations of the inode map and the number of the current segment to a 165:
in LFS, especially for slowly growing files or multiple simultaneous large writes. This inflicts a severe performance penalty, even though the
390: 365: 121: 1622: 1097: 1612: 783: 592: 226: 1234: 900: 582: 152:
work the same way—the file system simply reconstructs its state from the last checkpoint and resumes logging from there.
40: 55:
implementation was nonfunctional until work leading up to the 4.0 release made it viable again as a production file system.
679: 1617: 1033: 607: 1538: 1386: 1318: 612: 419: 176: 144:
at a fixed place on disk. There are two such regions; LFS alternates between them each checkpoint. Once written, a
1669: 1423: 1413: 1403: 795: 385: 358: 32: 17: 162: 1488: 1341: 1244: 1189: 1064: 920: 689: 1587: 1543: 1525: 1224: 1214: 669: 179:
or versioning, even though both features are trivial to implement in general on log-structured file systems.
1433: 1398: 1336: 815: 635: 499: 429: 1569: 1478: 1428: 1371: 1117: 1087: 1038: 990: 890: 863: 740: 630: 543: 434: 351: 1643: 1604: 1418: 1148: 1137: 1112: 1049: 753: 570: 560: 516: 481: 1648: 1594: 1533: 1122: 790: 731: 647: 258: 1638: 1500: 1463: 1376: 1102: 1092: 1077: 1018: 848: 511: 504: 491: 444: 306: 232: 148:
represents the last consistent snapshot of the file system. Recovery after a crash and normal
1453: 1393: 1229: 915: 885: 877: 746: 642: 617: 439: 222: 149: 1473: 1381: 1179: 829: 773: 462: 298: 212: 166: 64: 1582: 1495: 1003: 397: 36: 1483: 1468: 1408: 586: 457: 190:
Seltzer, Margo; Bostic, Keith; McKusick, Marshall Kirk; Staelin, Carl (January 1993),
119:
blocks, or blocks for which no newer copies exist further ahead in the log. The LFS
1663: 1513: 1458: 1132: 1072: 931: 209:
Proceedings of the sixteenth ACM symposium on Operating systems principles - SOSP '97
191: 325: 310: 236: 1577: 1285: 873: 840: 778: 758: 79:, only one of which is active at any one time. Each segment has a header called a 283: 1275: 374: 1553: 1351: 169:
for log-structured file systems assumes disk reads will mostly be cached away.
86:
Whenever a file or directory is changed, LFS writes to the head of this log:
1508: 1249: 1127: 858: 726: 721: 16:
This article is about the NetBSD file system. For the general concept, see
302: 217: 1326: 1270: 1239: 1028: 853: 709: 602: 555: 449: 1280: 1254: 1081: 714: 684: 575: 531: 48: 44: 1290: 1199: 1194: 998: 699: 52: 1548: 1346: 1331: 1308: 1303: 1298: 1209: 1204: 1142: 1023: 976: 971: 964: 959: 954: 949: 905: 895: 800: 768: 662: 657: 652: 548: 476: 424: 68: 1219: 1184: 1174: 1169: 1107: 1043: 1013: 1008: 944: 939: 910: 819: 810: 763: 704: 674: 565: 536: 526: 521: 486: 468: 347: 343: 107:
When a segment is filled, LFS goes on to fill the next free or
825: 805: 694: 596: 284:"The Design and Implementation of a Log-Structured Filesystem" 192:"An Implementation of a Log-Structured File System for UNIX" 282:
Rosenblum, Mendel; Ousterhout, John K (February 1992),
1631: 1603: 1568: 1524: 1442: 1364: 1317: 1263: 1158: 1063: 989: 929: 872: 839: 406: 324:Rosenblum, Mendel; Ousterhout, John K (June 1990), 63:Most of the on-disk format of LFS is borrowed from 35:(a concept originally proposed and implemented by 277: 275: 359: 8: 1446: 1162: 926: 412: 366: 352: 344: 216: 99:Inode map blocks updated to point at (3). 93:Indirect blocks updated to point to (1). 249: 333:Proceedings of the 1990 Summer Usenix 199:Proceedings of the 1993 Winter Usenix 7: 291:ACM Transactions on Computer Systems 111:segment. Segments are said to be 14: 96:Inodes updated to point to (2). 90:Any changed or new data blocks. 259:"NetBSD 4.0 Release CHANGELOG" 1: 257:Bouyer, Manuel (2007-12-15). 39:), originally developed for 31:) is an implementation of a 1539:Filesystem-level encryption 1686: 386:Comparison of file systems 125:turns dirty segments into 75:LFS divides the disk into 33:log-structured file system 25:Log-Structured File System 18:log-structured file system 15: 1449: 1165: 415: 381: 326:"The LFS Storage Manager" 163:file system fragmentation 1489:Extended file attributes 1190:Compact Disc File System 1588:Installable File System 67:. The indirect block, 43:. It was removed from 636:TiVo Media File System 500:Encrypting File System 631:Macintosh File System 303:10.1145/146941.146943 218:10.1145/268998.266700 1644:GUID Partition Table 991:Distributed parallel 739:Shared File System ( 211:, pp. 238–251, 161:There can be severe 1649:Apple Partition Map 1595:Virtual file system 1534:Access-control list 648:NetWare File System 175:LFS does not allow 1639:Master Boot Record 1464:Data deduplication 1103:Google File System 1019:Google File System 505:Extent File System 467:Byte File System ( 1670:Disk file systems 1657: 1656: 1564: 1563: 1454:Case preservation 1360: 1359: 1059: 1058: 985: 984: 747:Smart File System 142:checkpoint region 122:garbage collector 1677: 1474:Execute in place 1447: 1180:Boot File System 1163: 927: 463:Boot File System 413: 368: 361: 354: 345: 338: 336: 330: 321: 315: 313: 288: 279: 270: 268: 266: 265: 254: 239: 220: 201: 196: 167:design rationale 115:if they contain 1685: 1684: 1680: 1679: 1678: 1676: 1675: 1674: 1660: 1659: 1658: 1653: 1627: 1599: 1583:File system API 1560: 1520: 1496:File change log 1438: 1414:Record-oriented 1387:Self-certifying 1356: 1313: 1259: 1154: 1055: 981: 925: 868: 835: 408: 402: 398:Unix filesystem 377: 372: 342: 341: 328: 323: 322: 318: 286: 281: 280: 273: 263: 261: 256: 255: 251: 246: 229: 206: 194: 189: 186: 158: 61: 37:John Ousterhout 21: 12: 11: 5: 1683: 1681: 1673: 1672: 1662: 1661: 1655: 1654: 1652: 1651: 1646: 1641: 1635: 1633: 1629: 1628: 1626: 1625: 1623:Log-structured 1620: 1615: 1609: 1607: 1601: 1600: 1598: 1597: 1592: 1591: 1590: 1580: 1574: 1572: 1566: 1565: 1562: 1561: 1559: 1558: 1557: 1556: 1551: 1541: 1536: 1530: 1528: 1526:Access control 1522: 1521: 1519: 1518: 1517: 1516: 1511: 1503: 1498: 1493: 1492: 1491: 1484:File attribute 1481: 1476: 1471: 1469:Data scrubbing 1466: 1461: 1456: 1450: 1444: 1440: 1439: 1437: 1436: 1431: 1426: 1424:Steganographic 1421: 1416: 1411: 1406: 1404:Log-structured 1401: 1396: 1391: 1390: 1389: 1384: 1379: 1368: 1366: 1362: 1361: 1358: 1357: 1355: 1354: 1349: 1344: 1339: 1334: 1329: 1323: 1321: 1315: 1314: 1312: 1311: 1306: 1301: 1296: 1293: 1288: 1283: 1278: 1273: 1267: 1265: 1261: 1260: 1258: 1257: 1252: 1247: 1242: 1237: 1232: 1227: 1222: 1217: 1212: 1207: 1202: 1197: 1192: 1187: 1182: 1177: 1172: 1166: 1160: 1156: 1155: 1153: 1152: 1145: 1140: 1135: 1130: 1125: 1120: 1115: 1110: 1105: 1100: 1095: 1090: 1085: 1075: 1069: 1067: 1061: 1060: 1057: 1056: 1054: 1053: 1046: 1041: 1036: 1031: 1026: 1021: 1016: 1011: 1006: 1001: 995: 993: 987: 986: 983: 982: 980: 979: 974: 969: 968: 967: 957: 952: 947: 942: 936: 934: 924: 923: 918: 913: 908: 903: 898: 893: 888: 882: 880: 870: 869: 867: 866: 861: 856: 851: 845: 843: 837: 836: 834: 833: 823: 813: 808: 803: 798: 793: 788: 787: 786: 781: 771: 766: 761: 756: 751: 750: 749: 744: 734: 729: 727:Reliance Nitro 724: 719: 718: 717: 707: 702: 697: 692: 687: 682: 677: 672: 667: 666: 665: 655: 650: 645: 640: 639: 638: 633: 625: 620: 615: 610: 605: 600: 590: 587:Classic Mac OS 580: 579: 578: 568: 563: 558: 553: 552: 551: 541: 540: 539: 534: 529: 524: 514: 509: 508: 507: 502: 494: 489: 484: 479: 474: 473: 472: 465: 460: 458:Be File System 452: 447: 442: 437: 432: 427: 422: 416: 410: 404: 403: 401: 400: 395: 394: 393: 382: 379: 378: 373: 371: 370: 363: 356: 348: 340: 339: 316: 271: 248: 247: 245: 242: 241: 240: 228:978-0897919166 227: 203: 202: 185: 182: 181: 180: 173: 170: 157: 154: 101: 100: 97: 94: 91: 60: 57: 13: 10: 9: 6: 4: 3: 2: 1682: 1671: 1668: 1667: 1665: 1650: 1647: 1645: 1642: 1640: 1637: 1636: 1634: 1630: 1624: 1621: 1619: 1616: 1614: 1613:Cryptographic 1611: 1610: 1608: 1606: 1602: 1596: 1593: 1589: 1586: 1585: 1584: 1581: 1579: 1576: 1575: 1573: 1571: 1567: 1555: 1552: 1550: 1547: 1546: 1545: 1542: 1540: 1537: 1535: 1532: 1531: 1529: 1527: 1523: 1515: 1512: 1510: 1507: 1506: 1504: 1502: 1499: 1497: 1494: 1490: 1487: 1486: 1485: 1482: 1480: 1477: 1475: 1472: 1470: 1467: 1465: 1462: 1460: 1459:Copy-on-write 1457: 1455: 1452: 1451: 1448: 1445: 1441: 1435: 1432: 1430: 1427: 1425: 1422: 1420: 1417: 1415: 1412: 1410: 1407: 1405: 1402: 1400: 1397: 1395: 1392: 1388: 1385: 1383: 1380: 1378: 1375: 1374: 1373: 1370: 1369: 1367: 1363: 1353: 1350: 1348: 1345: 1343: 1340: 1338: 1335: 1333: 1330: 1328: 1325: 1324: 1322: 1320: 1316: 1310: 1307: 1305: 1302: 1300: 1297: 1294: 1292: 1289: 1287: 1284: 1282: 1279: 1277: 1274: 1272: 1269: 1268: 1266: 1262: 1256: 1253: 1251: 1248: 1246: 1243: 1241: 1238: 1236: 1233: 1231: 1228: 1226: 1223: 1221: 1218: 1216: 1213: 1211: 1208: 1206: 1203: 1201: 1198: 1196: 1193: 1191: 1188: 1186: 1183: 1181: 1178: 1176: 1173: 1171: 1168: 1167: 1164: 1161: 1157: 1151: 1150: 1146: 1144: 1141: 1139: 1136: 1134: 1131: 1129: 1126: 1124: 1121: 1119: 1116: 1114: 1111: 1109: 1106: 1104: 1101: 1099: 1096: 1094: 1091: 1089: 1086: 1083: 1079: 1076: 1074: 1071: 1070: 1068: 1066: 1062: 1052: 1051: 1047: 1045: 1042: 1040: 1037: 1035: 1032: 1030: 1027: 1025: 1022: 1020: 1017: 1015: 1012: 1010: 1007: 1005: 1002: 1000: 997: 996: 994: 992: 988: 978: 975: 973: 970: 966: 963: 962: 961: 958: 956: 953: 951: 948: 946: 943: 941: 938: 937: 935: 933: 932:wear leveling 928: 922: 919: 917: 914: 912: 909: 907: 904: 902: 899: 897: 894: 892: 889: 887: 884: 883: 881: 879: 875: 871: 865: 862: 860: 857: 855: 852: 850: 847: 846: 844: 842: 838: 831: 827: 824: 821: 817: 814: 812: 809: 807: 804: 802: 799: 797: 794: 792: 789: 785: 782: 780: 777: 776: 775: 772: 770: 767: 765: 762: 760: 757: 755: 752: 748: 745: 742: 738: 737: 735: 733: 730: 728: 725: 723: 720: 716: 713: 712: 711: 708: 706: 703: 701: 698: 696: 693: 691: 688: 686: 683: 681: 678: 676: 673: 671: 668: 664: 661: 660: 659: 656: 654: 651: 649: 646: 644: 641: 637: 634: 632: 629: 628: 626: 624: 621: 619: 616: 614: 611: 609: 606: 604: 601: 598: 594: 591: 588: 584: 581: 577: 574: 573: 572: 569: 567: 564: 562: 559: 557: 554: 550: 547: 546: 545: 542: 538: 535: 533: 530: 528: 525: 523: 520: 519: 518: 515: 513: 510: 506: 503: 501: 498: 497: 495: 493: 490: 488: 485: 483: 480: 478: 475: 470: 466: 464: 461: 459: 456: 455: 453: 451: 448: 446: 443: 441: 438: 436: 433: 431: 428: 426: 423: 421: 418: 417: 414: 411: 405: 399: 396: 392: 389: 388: 387: 384: 383: 380: 376: 369: 364: 362: 357: 355: 350: 349: 346: 334: 327: 320: 317: 312: 308: 304: 300: 296: 292: 285: 278: 276: 272: 260: 253: 250: 243: 238: 234: 230: 224: 219: 214: 210: 205: 204: 200: 193: 188: 187: 183: 178: 174: 171: 168: 164: 160: 159: 156:Disadvantages 155: 153: 151: 147: 143: 139: 134: 130: 128: 124: 123: 118: 114: 110: 105: 98: 95: 92: 89: 88: 87: 84: 82: 81:summary block 78: 73: 70: 66: 58: 56: 54: 50: 46: 42: 38: 34: 30: 26: 19: 1578:File manager 1147: 1048: 874:Flash memory 841:Optical disc 779:soft updates 759:Soup (Apple) 622: 409:non-rotating 375:File systems 332: 319: 297:(1): 26–52, 294: 290: 262:. Retrieved 252: 208: 198: 177:snapshotting 145: 141: 137: 135: 131: 126: 120: 116: 112: 108: 106: 102: 85: 80: 76: 74: 62: 28: 24: 22: 1544:Permissions 1159:Specialized 391:distributed 1570:Interfaces 1554:Sticky bit 1434:Versioning 1399:Journaling 1342:Rubberhose 1138:SMB (CIFS) 930:host-side 264:2019-02-04 244:References 146:checkpoint 138:checkpoint 1429:Synthetic 1372:Clustered 1319:Encrypted 1250:OverlayFS 859:ISO 13490 435:Amiga OFS 430:Amiga FFS 335:: 315–324 1664:Category 1514:Symbolic 1443:Features 1419:Semantic 1327:eCryptfs 1271:configfs 1240:SquashFS 1128:POHMELFS 1029:OrangeFS 854:ISO 9660 774:UFS/UFS2 722:Reliance 710:ReiserFS 556:Files-11 450:bcachefs 407:Disk and 311:28452863 237:11858758 150:mounting 133:needed. 77:segments 1632:Layouts 1618:Default 1281:debugfs 1255:UnionFS 1149:more... 1082:OpenAFS 1050:more... 715:Reiser4 685:OpenZFS 576:HAMMER2 532:ext3cow 512:Episode 49:OpenBSD 45:FreeBSD 1505:Links 1479:Extent 1409:Object 1377:Global 1295:specfs 1291:procfs 1286:kernfs 1264:Pseudo 1245:UMSDOS 1200:Davfs2 1195:cramfs 1133:Hadoop 1113:Lustre 999:BeeGFS 965:NILFS2 700:QNX4FS 663:NILFS2 571:HAMMER 561:Fossil 309:  235:  225:  59:Design 53:NetBSD 51:; the 1605:Lists 1549:Modes 1394:Flash 1365:Types 1347:SSHFS 1332:EncFS 1309:WinFS 1304:tmpfs 1299:sysfs 1276:devfs 1210:FTPFS 1205:EROFS 1143:SSHFS 1024:OCFS2 977:UBIFS 972:YAFFS 960:NILFS 955:LogFS 950:JFFS2 906:EROFS 896:exFAT 801:Xiafs 784:WAPBL 769:UBIFS 680:OneFS 658:NILFS 653:Next3 643:MINIX 549:exFAT 477:Btrfs 445:AthFS 425:AdvFS 329:(PDF) 307:S2CID 287:(PDF) 233:S2CID 195:(PDF) 184:Notes 136:At a 127:clean 113:dirty 109:clean 69:inode 1509:Hard 1501:Fork 1382:Grid 1235:MVFS 1230:NOVA 1225:LTFS 1220:Lnfs 1215:FUSE 1185:CDfs 1175:AXFS 1170:Aufs 1108:GPFS 1093:Coda 1044:Xsan 1034:PVFS 1014:GFS2 1009:CXFS 1004:Ceph 945:JFFS 940:CHFS 921:NVFS 911:F2FS 901:TFAT 886:APFS 876:and 820:z/OS 811:Xsan 796:WAFL 791:VxFS 764:Tux3 754:SNFS 736:SFS 705:ReFS 675:NTFS 627:MFS 613:HTFS 608:HPFS 603:HFS+ 566:GPFS 537:ext4 527:ext3 522:ext2 496:EFS 487:CXFS 482:CVFS 469:z/VM 454:BFS 440:APFS 420:ADFS 223:ISBN 117:live 47:and 27:(or 23:The 1352:ZFS 1337:EFS 1123:NFS 1118:NCP 1098:DFS 1088:AFP 1078:AFS 1065:NAS 1039:QFS 916:JFS 891:FAT 878:SSD 864:UDF 849:HSF 830:Sun 826:ZFS 816:zFS 806:XFS 732:RFS 695:QFS 690:PFS 670:NSS 623:LFS 618:JFS 597:MVS 593:HFS 583:HFS 544:FAT 517:ext 492:DFS 299:doi 213:doi 65:UFS 41:BSD 29:LFS 1666:: 1073:9P 741:VM 331:, 305:, 295:10 293:, 289:, 274:^ 231:, 221:, 197:, 1084:) 1080:( 832:) 828:( 822:) 818:( 743:) 599:) 595:( 589:) 585:( 471:) 367:e 360:t 353:v 337:. 314:. 301:: 269:. 267:. 215:: 20:.

Index

log-structured file system
log-structured file system
John Ousterhout
BSD
FreeBSD
OpenBSD
NetBSD
UFS
inode
garbage collector
mounting
file system fragmentation
design rationale
snapshotting
"An Implementation of a Log-Structured File System for UNIX"
doi
10.1145/268998.266700
ISBN
978-0897919166
S2CID
11858758
"NetBSD 4.0 Release CHANGELOG"


"The Design and Implementation of a Log-Structured Filesystem"
doi
10.1145/146941.146943
S2CID
28452863
"The LFS Storage Manager"

Text is available under the Creative Commons Attribution-ShareAlike License. Additional terms may apply.