Foundations of ARM64 Linux Debugging, Disassembling, and Reversing

Gain a solid understanding of how Linux C and C++ compilers generate binary code. This book explains the reversing and binary analysis of ARM64 architecture now used by major Linux cloud providers and covers topics ranging from writing programs in assembly language, live debugging, and static binary analysis of compiled C and C++ code. It is ideal for those working with embedded devices, including mobile phones and tablets.

Using the latest version of Red Hat, you'll look closely at the foundations of diagnostics of core memory dumps, live and postmortem debugging of Linux applications, services, and systems. You'll also work with the GDB debugger and use it for disassembly and reversing. This book uses practical step-by-step exercises of increasing complexity with explanations and many diagrams, including some necessary background topics. In addition, you will be able to analyze such code confidently, understand stack memory usage, and reconstruct original C/C++ code.

And as you'll see, memory forensics, malware, and vulnerability analysis, require an understanding of ARM64 assembly language and how C and C++ compilers generate code, including memory layout and pointers. This book provides the background knowledge and practical foundations you’ll need to understand internal Linux program structure and behavior.

Foundations of ARM64 Linux Debugging, Disassembling, and Reversing is the perfect companion to Foundations of Linux Debugging, Disassembling, and Reversing for readers interested in the cloud or cybersecurity.

WHAT YOU'LL LEARN

* Review the basics of ARM64 assembly language
* Examine the essential GDB debugger commands for debugging and binary analysis
* Study C and C++ compiler code generation with and without compiler optimizations


* Look at binary code disassembly and reversing patterns


* See how pointers in C and C++ are implemented and used

WHO THIS BOOK IS FOR

Software support and escalation engineers, cloud security engineers, site reliability engineers, DevSecOps, platform engineers, software testers, Linux C/C++ software engineers and security researchers without ARM64 assembly language background, and beginners learning Linux software reverse engineering techniques.

Dmitry Vostokov is an internationally recognized expert, speaker, educator, scientist, inventor, and author. He is the founder of the pattern-oriented software diagnostics, forensics, and prognostics discipline (Systematic Software Diagnostics), and Software Diagnostics Institute (DA+TA: DumpAnalysis.org + TraceAnalysis.org). Vostokov has also authored books on software diagnostics, anomaly detection and analysis, software and memory forensics, root cause analysis and problem solving, memory dump analysis, debugging, software trace and log analysis, reverse engineering, and malware analysis. He has over 25 years of experience in software architecture, design, development, and maintenance in various industries, including leadership, technical, and people management roles. In his spare time, he presents various topics on Debugging.TV and explores Software Narratology, its further development as Narratology of Things and Diagnostics of Things (DoT), Software Pathology, and Quantum Software Diagnostics. His current interest areas are theoretical software diagnostics and its mathematical and computer science foundations, application of formal logic, artificial intelligence, machine learning, and data mining to diagnostics and anomaly detection, software diagnostics engineering and diagnostics-driven development, diagnostics workflow, and interaction. Recent interest areas also include cloud native computing, security, automation, functional programming, and applications of category theory to software development and big data. He is based out of Dublin, Ireland.

CHAPTER 1 - A64.1: MEMORY, REGISTERS, AND SIMPLE ARITHMETIC 11

Memory and Registers inside an Idealized Computer 11

Memory and Registers inside ARM 64-bit Computer 12

“Arithmetic” Project: Memory Layout and Registers 13“Arithmetic” Project: A Computer Program 14

“Arithmetic” Project: Assigning Numbers to Memory Locations 15

Assigning Numbers to Registers 18

“Arithmetic” Project: Adding Numbers to Memory Cells 19

Incrementing/Decrementing Numbers in Memory and Registers 22Multiplying Numbers 25

CHAPTER 2 - A64.2: CODE OPTIMIZATION 29

“Arithmetic” Project: C/C++ Program 29

Downloading GDB 31

GDB Disassembly Output – No Optimization 32GDB Disassembly Output – Optimization 37

CHAPTER 3 - A64.3: NUMBER REPRESENTATIONS 39

Numbers and Their Representations 39

Decimal Representation (Base Ten) 40

Ternary Representation (Base Three) 41

Binary Representation (Base Two) 42

Hexadecimal Representation (Base Sixteen) 43

Why are Hexadecimals Used? 44

CHAPTER 4 - A64.4: POINTERS 47

A Definition 47

“Pointers” Project: Memory Layout and Registers 48

“Pointers” Project: Calculations 50

Using Pointers to Assign Numbers to Memory Cells 51

Adding Numbers Using Pointers 58

Incrementing Numbers Using Pointers 62

Multiplying Numbers Using Pointers 65

CHAPTER 5 - A64.5: BYTES, HALF WORDS, WORDS, AND DOUBLE WORDS 69

Using Hexadecimal Numbers 69

Byte Granularity 70

Bit Granularity 71Memory Layout 72

CHAPTER 6 - A64.6: POINTERS TO MEMORY 75

Pointers Revisited 75

Addressing Types 76

Registers Revisited 81

NULL Pointers 82

Invalid Pointers 83

Variables as Pointers 84

Pointer Initialization 85

Initialized and Uninitialized Data 86

More Pseudo Notation 87“MemoryPointers” Project: Memory Layout 88

CHAPTER 7 - A64.7: LOGICAL INSTRUCTIONS AND PC 99

Instruction Format 99

Logical Shift Instructions 100

Logical Operations 101

Zeroing Memory or Registers 102

Instruction Pointer 103

Code Section 105

CHAPTER 8 - A64.8: RECONSTRUCTING A PROGRAM WITH POINTERS 107

Example of Disassembly Output: No Optimization 107

Reconstructing C/C++ Code: Part 1 110

Reconstructing C/C++ Code: Part 2 112

Reconstructing C/C++ Code: Part 3 114

Reconstructing C/C++ Code: C/C++ program 116

Example of Disassembly Output: Optimized Program 117

CHAPTER 9 - A64.9: MEMORY AND STACKS 119

Stack: A Definition 119

Stack Implementation in Memory 120

Things to Remember 122

Stack Push Implementation 123Stack Pop Implementation 124

Register Review 125

Application Memory Simplified 126

Stack Overflow 127

Jumps 128

Calls 130Call Stack 131

Exploring Stack in GDB 133

CHAPTER 10 - A64.10: FRAME POINTER AND LOCAL VARIABLES 137

Stack Usage 137

Register Review 138Addressing Array Elements 139

Stack Structure (No Function Parameters) 140

Function Prolog 141

Raw Stack (No Local Variables and Function Parameters) 142

Function Epilog 144

“Local Variables” Project 145

Disassembly of Optimized Executable 148

CHAPTER 11- A64.11: FUNCTION PARAMETERS 149

“FunctionParameters” Project 149

Stack Structure 150

Function Prolog and Epilog 152

Project Disassembled Code with Comments 154

Parameter Mismatch Problem 158

CHAPTER 12 - A64.12: MORE INSTRUCTIONS 159

PSTATE Flags 159

Testing for 0 160

TST - Logical Compare 161

CMP – Compare Two Operands 162

TST or CMP? 163

Conditional Jumps 164

Function Return Value 165

CHAPTER 13 - A64.13: FUNCTION POINTER PARAMETERS 167

“FunctionPointerParameters” Project 167

Commented Disassembly 168

CHAPTER 14 - A64.14: SUMMARY OF CODE DISASSEMBLY PATTERNS 173

Function Prolog / Epilog 173

ADR (Address) 174

Passing Parameters 175

Accessing Saved Parameters and Local Variables 176