This dissertation presents a new technique for disk storage management called a log-structured file system. The technique writes all file system changes in large sequential transfers to a log-like structure on disk. The key benefit is a high write performance that is independent of the workload. The large transfers also enable the efficient use of large disk arrays such as RAIDs. The technique minimizes the overhead of computing the redundancy information required by large RAIDs.
A log-structured file system achieves high write rates without sacrificing file retrieval performance. Files are read back from the log efficiently due to the indexing information that is maintained. The log structure also permits fast recovery from system crashes. Using a recovery system based on checkpoints and roll-forward the log-structured file system can quickly restore the disk to a consistent state.
An important focus of this dissertation is the technique used for free space management in a log-structured file system. The approach taken was to divide the disk into large segments to which the log was written. A segment cleaner mechanism exists to compress the live information from heavily fragmented segments. The mechanism reads in the fragmented segments, compacts the live data, and writes the data back to segments on disk. The dissertation includes a series of simulations that demonstrate the efficiency of a simple segment cleaning policy based on cost and benefit. The segment cleaner decides which segments to clean based on a function of the fraction alive in the segment and the age of the data in the segment.
I have implemented a prototype log-structured file system called Spite LFS; it outperforms current Unix file systems by an order of magnitude for small-file writes and matches or exceeds Unix performance for reads and large writes. Even when the overhead for cleaning is included, Sprite LFS can use 70% of the disk bandwidth for writing. Unix file systems typically can use only 5-10%.
Cited By
- Jaffer S, Mahdaviani K and Schroeder B (2022). Improving the Endurance of Next Generation SSD’s using WOM-v Codes, ACM Transactions on Storage, 18:4, (1-32), Online publication date: 30-Nov-2022.
- Desnoyers P (2014). Analytic Models of SSD Write Performance, ACM Transactions on Storage, 10:2, (1-25), Online publication date: 1-Mar-2014.
- Rumble S, Kejriwal A and Ousterhout J Log-structured memory for DRAM-based storage Proceedings of the 12th USENIX conference on File and Storage Technologies, (1-16)
- Desnoyers P Analytic modeling of SSD write performance Proceedings of the 5th Annual International Systems and Storage Conference, (1-10)
- Traeger A, Zadok E, Joukov N and Wright C (2008). A nine year study of file system and storage benchmarking, ACM Transactions on Storage (TOS), 4:2, (1-56), Online publication date: 11-May-2008.
- Ng W and Chen P (2001). The Design and Verification of the Rio File Cache, IEEE Transactions on Computers, 50:4, (322-337), Online publication date: 1-Apr-2001.
- Matthews J, Roselli D, Costello A, Wang R and Anderson T (1997). Improving the performance of log-structured file systems with adaptive methods, ACM SIGOPS Operating Systems Review, 31:5, (238-251), Online publication date: 1-Dec-1997.
- Matthews J, Roselli D, Costello A, Wang R and Anderson T Improving the performance of log-structured file systems with adaptive methods Proceedings of the sixteenth ACM symposium on Operating systems principles, (238-251)
- de Jonge W, Kaashoek M and Hsieh W The logical disk Proceedings of the fourteenth ACM symposium on Operating systems principles, (15-28)
- de Jonge W, Kaashoek M and Hsieh W (2019). The logical disk, ACM SIGOPS Operating Systems Review, 27:5, (15-28), Online publication date: 1-Dec-1993.
Index Terms
- The design and implementation of a log-structured file system
Recommendations
The design and implementation of a log-structured file system
This paper presents a new technique for disk storage management called a log-structured file system. A log-structured file system writes all modifications to disk sequentially in a log-like structure, thereby speeding up both file writing and crash ...
The design and implementation of a log-structured file system
SOSP '91: Proceedings of the thirteenth ACM symposium on Operating systems principlesThis paper presents a new technique for disk storage management called a log-structured file system. A log-structured file system writes all modifications to disk sequentially in a log-like structure, thereby speeding up both file writing and crash ...