File System Forensics

COMPREHENSIVE FORENSIC REFERENCE EXPLAINING HOW FILE SYSTEMS FUNCTION AND HOW FORENSIC TOOLS MIGHT WORK ON PARTICULAR FILE SYSTEMSFile System Forensics delivers comprehensive knowledge of how file systems function and, more importantly, how digital forensic tools might function in relation to specific file systems. It provides a step-by-step approach for file content and metadata recovery to allow the reader to manually recreate and validate results from file system forensic tools. The book includes a supporting website that shares all of the data (i.e. sample file systems) used for demonstration in the text and provides teaching resources such as instructor guides, extra material, and more. Written by a highly qualified associate professor and consultant in the field, File System Forensics includes information on:

* The necessary concepts required to understand file system forensics for anyone with basic computing experience
* File systems specific to Windows, Linux, and macOS, with coverage of FAT, ExFAT, and NTFS
* Advanced topics such as deleted file recovery, fragmented file recovery, searching for particular files, links, checkpoints, snapshots, and RAID
* Issues facing file system forensics today and various issues that might evolve in the field in the coming years

File System Forensics is an essential, up-to-date reference on the subject for graduate and senior undergraduate students in digital forensics, as well as digital forensic analysts and other law enforcement professionals. FERGUS TOOLAN, PHD, is an Associate Professor in the Norwegian Police University College. He has published over 30 peer-reviewed papers and supervised a number of master’s and PhD students throughout his career. Additionally, Dr. Toolan has provided consultancy services to a number of police services and other governmental organizations. He has taught a range of courses from introductory programming to advanced databases, and from computer hardware to discrete mathematics. Preface xvii

Acknowledgements xxi

PART I PRELIMINARIES 1

1 INTRODUCTION 3

1.1 What is Digital Forensics? 4

1.2 File System Forensics 5

1.3 Digital Forensic Principles 5

1.4 Digital Forensic Methodology 7

1.4.1 Preparation 8

1.4.2 Localisation/Preservation 8

1.4.3 Acquisition 8

1.4.4 Processing 9

1.4.5 Analysis 9

1.4.6 Reporting 9

1.4.7 Quality Assurance 10

1.4.8 Evidence Return 10

1.5 About This Book 10

1.5.1 Who Should Read This Book? 11

1.6 Book Structure 12

1.7 Summary 13

Exercises 13

Bibliography 14

2 LINUX AS A FORENSIC PLATFORM 17

2.1 Open-Source Software 17

2.1.1 Advantages of Open-Source Software 19

2.1.2 Open Source ≠ Free 20

2.2 Open-Source Software in Digital Forensics 20

2.3 What is Linux? 21

2.3.1 The Anatomy of the Linux OS 22

2.3.2 Linux Distributions 27

2.3.3 A (very) Brief History of Linux 28

2.4 Using Linux 29

2.4.1 User Accounts 30

2.4.2 Basic Linux Commands 32

2.4.2.1 Navigating the File System 32

2.4.2.2 Getting Help 34

2.4.2.3 Viewing/Editing Text Files 34

2.4.2.4 Managing Directories 35

2.4.2.5 Redirection and Pipes 35

2.5 Linux as a Forensic Platform 36

2.5.1 Commands for Digital Forensics 36

2.5.1.1 Hashing 36

2.5.1.2 Hex Viewers 38

2.5.1.3 Archiving/Compression 39

2.5.1.4 The file Command 40

2.5.1.5 The strings Command 40

2.5.1.6 Text Searching with (e)grep 41

2.6 Summary 42

Exercises/Discussion Topics 42

Bibliography 43

3 MATHEMATICAL PRELIMINARIES 45

3.1 Bits and Bytes 45

3.2 Number Systems 48

3.2.1 Notational Conventions 48

3.2.2 Decimal 48

3.2.3 Binary 49

3.2.4 Hexadecimal 50

3.2.5 Number Conversions 51

3.2.6 Number Conversion with Bash 51

3.2.7 Negative Numbers 53

3.2.8 Floating-Point Numbers 53

3.3 Representing Text 56

3.3.1 Ascii 56

3.3.2 Iso- 8859 57

3.3.3 Unicode 59

3.3.4 Utf- 8 60

3.3.5 Utf- 16 61

3.4 Representing Time 62

3.4.1 Unix Time 63

3.4.2 The Linux date Command 64

3.5 Endianness and Raw Data 64

3.6 Summary 66

Exercises 67

Bibliography 68

4 DISKS, PARTITIONS AND FILE SYSTEMS 69

4.1 Disk Storage 70

4.1.1 Traditional Rotational Hard Drives 71

4.1.1.1 Optical Media 72

4.1.2 Flash Drives 73

4.1.3 Solid-State Drives 73

4.2 Partitions 74

4.2.1 Creating Partitions/File Systems on Linux 74

4.2.1.1 Mounting File Systems on Linux 77

4.2.2 Master Boot Record 78

4.2.3 GUID Partition Table 80

4.3 File Systems 83

4.3.1 File System Concepts 83

4.3.2 Comparison of File Systems 86

4.4 Acquisition of File System Data 88

4.4.1 Logical vs Physical Acquisition 88

4.4.2 Acquisition Under Linux 88

4.4.2.1 The dd Family 89

4.4.2.2 Expert Witness Format (EWF) 90

4.4.2.3 guymager 91

4.5 Analysis of File Systems 92

4.5.1 The Sleuth Kit 92

4.5.1.1 Determine the Partition Layout 93

4.5.1.2 Determine the File System Type 93

4.5.1.3 List the Files 94

4.5.1.4 Recover File Metadata 95

4.5.1.5 Recover File Content 95

4.5.1.6 Other TSK Commands 95

4.5.2 Data Carving 96

4.6 Summary 97

Exercises 97

Bibliography 98

PART II WINDOWS FILE SYSTEMS 99

5 THE FAT FILE SYSTEM 101

5.1 On-Disk Structures 101

5.1.1 Layout 102

5.1.2 Volume Boot Record 102

5.1.3 File System Information (FSINFO) 102

5.1.4 File Allocation Table 104

5.1.5 Directory Entries 105

5.1.6 FAT Date and Time 108

5.1.7 Mapping Clusters to Sectors 109

5.2 Analysis of FAT 32 109

5.2.1 Creating FAT32 File Systems 109

5.2.2 Supplied FAT32 Image Files 110

5.2.3 FAT32 Manual Analysis 110

5.2.3.1 Process the VBR 111

5.2.3.2 Process the Root Directory 112

5.2.3.3 Process Sub-directories 113

5.2.3.4 Recover Metadata/Content 113

5.3 FAT32 Advanced Analysis 115

5.3.1 Deleted Files 116

5.3.2 The Volume Label 117

5.4 Summary 117

Exercises 118

Bibliography 118

6 THE EXFAT FILE SYSTEM 121

6.1 On-Disk Structures 121

6.1.1 Volume Boot Record 122

6.1.2 File Allocation Table 123

6.1.3 Directory Entries 125

6.1.3.1 Allocation Bitmap (Type: 0x81) 127

6.1.3.2 Up-Case Table (Type: 0x82) 128

6.1.3.3 Volume Label (Type: 0x83) 128

6.1.3.4 File (Type: 0x85) 129

6.1.3.5 Volume GUID (Type: 0xA0) 130

6.1.3.6 Stream Extension (Type: 0xC0) 130

6.1.3.7 Filename Extension 131

6.1.3.8 Other Directory Entries 132

6.2 Analysis of ExFAT 132

6.2.1 Creating ExFAT File Systems 132

6.2.2 Supplied ExFAT Image Files 132

6.2.3 ExFAT Manual Analysis 132

6.2.3.1 Step 1: Process the VBR 133

6.2.3.2 Step 2: Process the Root Directory 133

6.2.3.3 Step 3: Process Subdirectories 136

6.2.3.4 Step 4: Recover Metadata 137

6.2.3.5 Step 5: Recover Content 137

6.3 ExFAT Advanced Analysis 139

6.3.1 Long File Names 139

6.3.2 Deleted Files 140

6.3.3 Fragmented Files and Large Directories 141

6.4 Summary 142

Exercises 143

Bibliography 143

7 THE NTFS FILE SYSTEM 145

7.1 On-Disk Structures 146

7.1.1 $Boot 146

7.1.2 Indexes 147

7.1.3 Fixup Arrays 149

7.1.4 Time in NTFS 150

7.1.5 Master File Table 151

7.1.6 MFT Record Structure 152

7.1.6.1 MFT Record Header 152

7.1.6.2 Browsing Attributes 155

7.1.6.3 $STANDARD_INFORMATION (0x10) 155

7.1.6.4 $ATTRIBUTE_LIST (0x20) 156

7.1.6.5 $FILENAME (0x30) 156

7.1.6.6 $OBJECT_ID (0x40) 157

7.1.6.7 $SECURITY_DESCRIPTOR (0x50) 159

7.1.6.8 $VOLUME_NAME (0x60) 162

7.1.6.9 $VOLUME_INFORMATION (0x70) 162

7.1.6.10 $DATA (0x80) 163

7.1.6.11 $INDEX_ROOT (0x90) 163

7.1.6.12 $INDEX_ALLOCATION (0xA0) 165

7.1.6.13 $BITMAP (0xB0) 165

7.1.6.14 $REPARSE_POINT (0xC0) 166

7.1.6.15 $EA_INFORMATION (0xD0) and $EA (0xE0) 167

7.2 Analysis of NTFS 167

7.2.1 Creating NTFS File Systems 168

7.2.2 Supplied NTFS Image Files 168

7.2.3 NTFS Manual Analysis 168

7.2.3.1 Process $Boot 169

7.2.3.2 Recover $MFT 171

7.2.3.3 Process Directories 173

7.2.3.4 Recover File Metadata 177

7.2.3.5 Recover File Content 182

7.3 NTFS Advanced Analysis 185

7.3.1 Further File System Information 185

7.3.2 Deleted Files 186

7.3.3 Fragmented Files 187

7.3.4 Alternate Data Streams 190

7.3.5 Large MFT Records 191

7.4 Summary 194

Exercises 194

Bibliography 195

PART III LINUX FILE SYSTEMS 197

8 THE EXT2 FILE SYSTEM 199

8.1 On-Disk Structures 200

8.1.1 The Superblock 201

8.1.2 The Block Group Descriptor Table 204

8.1.3 The Inode Table 205

8.1.3.1 Mode/Permissions 207

8.1.3.2 Inode Flags 208

8.1.3.3 Block Pointers 208

8.1.4 The Data and Inode Bitmaps 209

8.1.5 Locating an Inode 209

8.2 Analysis of ext 2 210

8.2.1 Creating ext2 File Systems 210

8.2.2 Supplied ext2 Image Files 210

8.2.3 Ext2 Manual Analysis 211

8.2.3.1 Process the Superblock 211

8.2.3.2 Map the Block Groups 213

8.2.3.3 Process Root Directory Inode 216

8.2.3.4 Process the Root Directory 217

8.2.3.5 Process Directories 219

8.2.3.6 Process Files 219

8.3 Ext2 Advanced Analysis 222

8.3.1 Fragmented Files 222

8.3.2 Links 223

8.3.3 Deleted Files 225

8.4 Summary 226

Exercises 226

Bibliography 227

9 THE EXT3/EXT4 FILE SYSTEMS 229

9.1 Supplied Image Files 229

9.2 The ext3 File System 229

9.2.1 The Ext Journal 230

9.2.2 HTree Directory Indexing 237

9.3 The Ext4 File System 241

9.3.1 Large Inodes 241

9.3.1.1 Timestamps 241

9.3.2 Ext4 Data Storage 244

9.3.2.1 Extent-Based Storage 244

9.3.2.2 Inline Storage 248

9.3.2.3 Symbolic Links 248

9.3.3 File Deletion in Ext 4 249

9.3.4 Extended Attributes 252

9.3.5 Ext4 Block Group Descriptors 255

9.3.6 Flexible Block Groups 255

9.4 Summary 258

Exercises 259

Bibliography 260

10 THE XFS FILE SYSTEM 263

10.1 On-Disk Structures 264

10.1.1 Allocation Groups 264

10.1.2 Addressing 266

10.1.2.1 Inode Addressing 266

10.1.3 XFS B+ Trees 267

10.1.4 The Superblock 268

10.1.4.1 Locating Superblocks 268

10.1.5 XFS Signatures 271

10.1.6 XFS Inodes 271

10.1.7 Directories 273

10.1.8 Extents 274

10.1.9 Time in XFS 275

10.2 Analysis of XFS 275

10.2.1 Creating XFS File Systems 275

10.2.2 Supplied XFS Image Files 275

10.2.3 XFS Manual Analysis 276

10.2.3.1 Process the Superblock 276

10.2.3.2 Locate the Root Directory 277

10.2.3.3 Process the Root Directory 279

10.2.3.4 Process the Subdirectories 281

10.2.3.5 Recover File Content/Metadata 281

10.3 XFS Advanced Analysis 282

10.3.1 AG Free Space Management 283

10.3.1.1 AG Free List 285

10.3.2 AG Inode Management 286

10.3.3 Deleted Files 289

10.3.4 Extended Attributes 290

10.3.5 Links 291

10.3.6 The XFS Journal 292

10.4 Summary 300

Exercises 301

Bibliography 301

11 THE BTRFS FILE SYSTEM 303

11.1 On-Disk Structures 304

11.1.1 The Superblock 305

11.1.2 Btrfs Trees 305

11.1.3 Btrfs Tree Structure 307

11.1.3.1 Node Header Structure 307

11.1.3.2 Internal Node Structure 309

11.1.4 Btrfs Keys 309

11.1.5 Btrfs Items 310

11.1.6 Time in Btrfs 315

11.1.7 Logical and Physical Addressing 315

11.2 Analysis of Btrfs 317

11.2.1 Creating Btrfs File Systems 317

11.2.2 Supplied Btrfs Image Files 318

11.2.3 Btrfs Analysis Methodology 318

11.2.4 Manual Analysis of a Single Device File System 320

11.2.4.1 Process the Superblock 320

11.2.4.2 Process the CHUNK_ARRAY 321

11.2.4.3 Locate the CHUNK_TREE 322

11.2.4.4 Process the CHUNK_TREE 323

11.2.4.5 Locate the Root Tree 326

11.2.4.6 Locate the FS_TREE 327

11.2.4.7 Processing the FS_TREE 328

11.2.4.8 Process Directories 329

11.2.4.9 Recovering Metadata 335

11.2.4.10 Recovering File Contents 336

11.3 Btrfs Advanced Analysis 338

11.3.1 File Deletion 338

11.3.2 Analysis of Internal Nodes 342

11.3.3 Multiple Device Configuration 343

11.3.4 Subvolumes and Snapshots 346

11.4 Summary 350

Exercises 350

Bibliography 351

PART IV APPLE FILE SYSTEMS 353

12 THE HFS+ FILE SYSTEM 355

12.1 On-Disk Structures 355

12.1.1 Forks 357

12.1.2 Time in HFS+ 357

12.1.3 Volume Header 358

12.1.4 B-Trees 358

12.1.5 Catalog File 362

12.1.6 HFS+ Permissions 363

12.1.7 Text Encoding 365

12.1.8 Extents Overflow File 365

12.1.9 Allocation File 366

12.1.10 HFS+ Journal 367

12.2 Analysis of HFS+ 369

12.2.1 Creating HFS+ File Systems 369

12.2.2 Supplied HFS+ Image Files 370

12.2.3 HFS+ Manual Analysis 370

12.2.3.1 Process the Volume Header 370

12.2.3.2 Locate the Catalog File 371

12.2.3.3 Process the Catalog B-Tree 373

12.2.3.4 Gather Metadata 377

12.2.3.5 Recover File Content 377

12.3 HFS+ Advanced Analysis 380

12.3.1 Deleted Files 380

12.3.2 Index Nodes 381

12.3.3 Fragmented Files 383

12.3.4 Links 387

12.4 Summary 390

Exercises 391

Bibliography 391

13 THE APFS FILE SYSTEM 393

13.1 On-Disk Structures 394

13.1.1 Time in APFS 394

13.1.2 Objects 394

13.1.3 B-Trees 396

13.1.4 Containers and Volumes 399

13.1.5 Container Superblock 400

13.1.6 Volume Superblock 402

13.1.7 Object Maps 404

13.1.8 File-Related Structures 405

13.1.8.1 File System Keys 406

13.1.8.2 Inode 407

13.1.8.3 Directory Record 408

13.1.8.4 Extent 410

13.1.9 Checkpoints 410

13.1.10 Other APFS Structures 412

13.2 Analysis of APFS 412

13.2.1 Creating APFS File Systems 412

13.2.2 Supplied APFS Image Files 413

13.2.3 APFS Manual Analysis 413

13.2.3.1 Process the Container Superblock 414

13.2.3.2 Process the Container Object Map 415

13.2.3.3 Process the Volume Superblock 418

13.2.3.4 Process the Volume Object Map 418

13.2.3.5 Process the File System Tree 419

13.3 APFS Advanced Analysis 425

13.3.1 Deleted Files 425

13.3.2 Checkpoint Recovery 426

13.3.3 Multi-Level B-Trees 427

13.3.4 Multiple Volumes 429

13.3.5 Extended Attributes 430

13.3.6 Links 431

13.4 Summary 433

Exercises 433

Bibliography 434

PART V THE FUTURE 435

14 FUTURE CHALLENGES IN DIGITAL FORENSICS 437

14.1 Challenges in Digital Forensics 437

14.1.1 Data Volume 438

14.1.2 Multi-Source Correlation 439

14.1.3 New File Systems 440

14.1.4 Encryption 440

14.1.5 Cloud Storage 441

14.1.6 Lack of Resources 441

14.1.6.1 Human Resources 441

14.1.6.2 Software Resources 442

14.1.6.3 Hardware Resources 442

14.1.7 Tool Validation/Datasets 443

14.1.8 Lack of Standardisation 444

14.1.9 Legal/Scientific Challenges 444

14.1.10 Presentation of Evidence 445

14.1.11 Human Error/Bias 446

14.2 Where Do We Go from Here? 447

14.2.1 Training/Education 448

14.2.2 Free Open-Source Software (FOSS) 448

14.2.3 Triage 449

14.2.4 Artificial Intelligence (AI) 449

14.2.5 Live Data Forensics 450

14.2.6 Legal Solutions 451

14.2.7 Data Set Development/Tool Testing 452

14.2.8 Standardisation 452

14.2.9 Information Sharing 453

14.2.10 Virtualisation 453

14.3 Summary 454

Bibliography 454

Index 457