Foundations of Linux Debugging, Disassembling, and Reversing

Review topics ranging from Intel x64 assembly language instructions and writing programs in assembly language, to pointers, live debugging, and static binary analysis of compiled C and C++ code. This book is ideal for Linux desktop and cloud developers.

Using the latest version of Debian, you’ll focus on the foundations of the diagnostics of core memory dumps, live and postmortem debugging of Linux applications, services, and systems, memory forensics, malware, and vulnerability analysis. This requires an understanding of x64 Intel assembly language and how C and C++ compilers generate code, including memory layout and pointers.

This book provides the back­ground knowledge and practical foundations you’ll need in order to master internal Linux program structure and behavior. It consists of practical step-by-step exercises of increasing complexity with explanations and ample diagrams. You’ll also work with the GDB debugger and use it for disassembly and reversing.

By the end of the book, you will have a solid understanding of how Linux C and C++ compilers generate binary code. In addition, you will be able to analyze such code confidently, understand stack memory usage, and reconstruct original C/C++ code. Foundations of Linux Debugging, Disassembling, and Reversing is the perfect companion to Foundations of ARM64 Linux Debugging, Disassembling, and Reversing for readers interested in the cloud or cybersecurity.

WHAT YOU'LL LEARN

* Review the basics of x64 assembly language
* Examine the essential GDB debugger commands for debugging and binary analysis
* Study C and C++ compiler code generation with and without compiler optimizations
* Look at binary code disassembly and reversing patterns
* See how pointers in C and C++ are implemented and used




WHO THIS BOOK IS FOR

Software support and escalation engineers, cloud security engineers, site reliability engineers, DevSecOps, platform engineers, software testers, Linux C/C++ software engineers and security researchers without Intel x64 assembly language background, beginners learning Linux software reverse engineering techniques, and engineers coming from non-Linux environments.

Dmitry Vostokov is an internationally recognized expert, speaker, educator, scientist, inventor, and author. He is the founder of the pattern-oriented software diagnostics, forensics, and prognostics discipline (Systematic Software Diagnostics), and Software Diagnostics Institute (DA+TA: DumpAnalysis.org + TraceAnalysis.org). Vostokov has also authored books on software diagnostics, anomaly detection and analysis, software and memory forensics, root cause analysis and problem solving, memory dump analysis, debugging, software trace and log analysis, reverse engineering, and malware analysis. He has over 25 years of experience in software architecture, design, development, and maintenance in various industries, including leadership, technical, and people management roles. In his spare time, he presents various topics on Debugging.TV and explores Software Narratology, its further development as Narratology of Things and Diagnostics of Things (DoT), Software Pathology, and Quantum Software Diagnostics. His current interest areas are theoretical software diagnostics and its mathematical and computer science foundations, application of formal logic, artificial intelligence, machine learning, and data mining to diagnostics and anomaly detection, software diagnostics engineering and diagnostics-driven development, diagnostics workflow, and interaction. Recent interest areas also include cloud native computing, security, automation, functional programming, and applications of category theory to software development and big data. He is based out of Dublin, Ireland.

CHAPTER ONE - X64.1: MEMORY, REGISTERS, AND SIMPLE ARITHMETIC 11

Memory and Registers inside an Idealized Computer 11

Memory and Registers inside Intel 64-bit PC 12

“Arithmetic” Project: Memory Layout and Registers 13

“Arithmetic” Project: A Computer Program 14

“Arithmetic” Project: Assigning Numbers to Memory Locations 15Assigning Numbers to Registers 17

“Arithmetic” Project: Adding Numbers to Memory Cells 18

Incrementing/Decrementing Numbers in Memory and Registers 21

Multiplying Numbers 24

CHAPTER TWO - X64.2: CODE OPTIMIZATION 27

“Arithmetic” Project: C/C++ Program 27

Downloading GDB 28

GDB Disassembly Output – No Optimization 29

GDB Disassembly Output – Optimization 32

CHAPTER THREE - X64.3: NUMBER REPRESENTATIONS 33

Numbers and Their Representations 33

Decimal Representation (Base Ten) 34Ternary Representation (Base Three) 35

Binary Representation (Base Two) 36

Hexadecimal Representation (Base Sixteen) 37

Why are Hexadecimals Used? 38

CHAPTER FOUR - X64.4: POINTERS 41

A Definition 41

“Pointers” Project: Memory Layout and Registers 42

“Pointers” Project: Calculations 43

Using Pointers to Assign Numbers to Memory Cells 44

Adding Numbers Using Pointers 50

Incrementing Numbers Using Pointers 53

Multiplying Numbers Using Pointers 56

CHAPTER FIVE - X64.5: BYTES, WORDS, DOUBLE, AND QUAD WORDS 61

Using Hexadecimal Numbers 61

Byte Granularity 62

Bit Granularity 63

Memory Layout 64

CHAPTER SIX - X64.6: POINTERS TO MEMORY 67

Pointers Revisited 67

Addressing Types 68

Registers Revisited 73

NULL Pointers 74

Invalid Pointers 75

Variables as Pointers 76

Pointer Initialization 77

Initialized and Uninitialized Data 78

More Pseudo Notation 79

“MemoryPointers” Project: Memory Layout 80

CHAPTER SEVEN - X64.7: LOGICAL INSTRUCTIONS AND RIP 89

Instruction Format 89

Logical Shift Instructions 90

Logical Operations 91

Zeroing Memory or Registers 92

Instruction Pointer 93

Code Section 95

CHAPTER EIGHT - X64.8: RECONSTRUCTING A PROGRAM WITH POINTERS 97

Example of Disassembly Output: No Optimization 97

Reconstructing C/C++ Code: Part 1 99

Reconstructing C/C++ Code: Part 2 101

Reconstructing C/C++ Code: Part 3 103

Reconstructing C/C++ Code: C/C++ program 104

Example of Disassembly Output: Optimized Program 105

CHAPTER NINE - X64.9: MEMORY AND STACKS 107

Stack: A Definition 107

Stack Implementation in Memory 108

Things to Remember 110

PUSH Instruction 111

POP instruction 112

Register Review 113

Application Memory Simplified 115

Stack Overflow 116

Jumps 117

Calls 119

Call Stack 121

Exploring Stack in GDB 123

CHAPTER TEN - X64.10: FRAME POINTER AND LOCAL VARIABLES 127

Stack Usage 127

Register Review 128

Addressing Array Elements 129Stack Structure (No Function Parameters) 130

Function Prolog 131

Raw Stack (No Local Variables and Function Parameters) 132

Function Epilog 134

“Local Variables” Project 135

Disassembly of Optimized Executable 138

CHAPTER ELEVEN - X64.11: FUNCTION PARAMETERS 139

“FunctionParameters” Project 139

Stack Structure 140

Function Prolog and Epilog 142

Project Disassembled Code with Comments 144

Parameter Mismatch Problem 147

CHAPTER TWELVE - X64.12: MORE INSTRUCTIONS 149

CPU Flags Register 149

The Fast Way to Fill Memory 150

Testing for 0 152

TEST - Logical Compare 153

CMP – Compare Two Operands 154

TEST or CMP? 155

Conditional Jumps 156

The Structure of Registers 157

Function Return Value 158

Using Byte Registers 159

CHAPTER THIRTEEN - X64.13: FUNCTION POINTER PARAMETERS 161

“FunctionPointerParameters” Project 161

Commented Disassembly 162

CHAPTER FOURTEEN - X64.14: SUMMARY OF CODE DISASSEMBLY PATTERNS 169

Function Prolog / Epilog 169

LEA (Load Effective Address) 171

Passing Parameters 172

Accessing Parameters and Local Variables 173