Identifying malicious code through reverse engineering / edited by Abhishek Singh ; with contributions by Baibhav Singh.

Format:

Book

Contributor:

Singh, Abhishek, 1976-

Singh, Baibhav

Louis A. Duhring Fund.

Series:

Advances in information security ; 44.

Advances in information security ; 44

Language:

English

Subjects (All):

Computer security.

Reverse engineering.

Computer viruses.

Software protection.

Physical Description:

x, 188 pages : illustrations ; 25 cm.

Place of Publication:

New York : Springer, [2009]

Summary:

Attacks take place everyday with computers connected to the internet, because of worms, viruses or due to vulnerable software. These attacks result in a loss of millions of dollars to businesses across the world.

Identifying Malicious Code through Reverse Engineering provides information on reverse engineering and concepts that can be used to identify the malicious patterns in vulnerable software. The malicious patterns are used to develop signatures to prevent vulnerability and block worms or viruses. This book also includes the latest exploits through various case studies.

Identifying Malicious Code through Reverse Engineering is designed for professionals composed of practitioners and researchers writing signatures to prevent virus and software vulnerabilities. This book is also suitable for advanced-level students in computer science and engineering studying information security, as a secondary textbook or reference.

Contents:

Assembly Language

1.0 Introduction 1

1.1 Registers 1

1.1.1 General Purpose Register 1

1.1.2 FLAGS Register 2

1.2 80x86 Instruction Format 3

1.2.1 Instruction Prefix 4

1.2.2 Lock and Repeat Prefixes 4

1.2.3 Segment Override Prefixes 4

1.2.4 Opcode 5

1.3 Instructions 7

1.3.1 Basic Instructions 7

1.3.2 Floating Point Instruction 10

1.4 Stack Setup 13

1.4.1 Passing Parameters in C to the Procedure 13

1.4.2 Local Data Space on the Stack 15

1.5 Calling Conventions 16

1.5.1 cdecl calling convention 16

1.5.2 fastcall calling convention 17

1.5.3 stdcall calling convention 17

1.5.4 thiscall 17

1.6 Data Constructs 17

1.6.1 Global Variables 18

1.6.2 Local Variables 18

1.6.3 Registers 19

1.6.4 Imported Variables 19

1.6.5 Thread Local Storage (TLS) 20

1.6.6 Executable Data Section 20

1.7 Representation of Arithmetic Operations in Assembly 21

1.7.1 Multiplication 22

1.7.2 Division 22

1.7.3 Modulo 24

1.8 Representation of Data Structure in Assembly 24

1.8.1 Representation of Array in Assembly 24

1.8.2 Representation of Linked List in Assembly 25

1.9 Virtual Function Call in Assembly 26

1.9.1 Representation of classes in Assembly 27

1.10 Conclusion 28

Fundamental of Windows

2.0 Introduction 29

2.1 Memory Management 29

2.1.1 Virtual Memory Management 29

2.1.1.1 Virtual Memory Management in Windows NT 32

2.1.1.2 Impact of Hooking 33

2.1.2 Segmented Memory Management 34

2.1.3 Paged Memory Management 36

2.2 Kernel Memory and User Memory 37

2.2.1 Kernel Memory Space 37

2.2.2 Section Object 38

2.3 Virtual Address Descriptor 39

2.3.1 User Mode Address Space 39

2.3.2 Memory Management in Windows 39

2.3.3 Objects and Handles 40

2.3.4 Named Objects 40

2.4 Processes and Threads 41

2.4.1 Context Switching 43

2.4.1.1 Context Switches and Mode Switches 43

2.4.2 Synchronization Objects 44

2.4.2.1 Critical Section 44

2.4.2.2 Mutex 44

2.4.2.3 Semaphore 45

2.4.2.4 Event 45

2.4.2.5 Metered Section 45

2.5 Process Initialization Sequence 46

2.5.1 Application Programming Interface 47

2.6 Reversing Windows NT 48

2.6.1 ExpEchoPoolCalls 49

2.6.2 ObpShowAllocAndFree 49

2.6.3 LpcpTraceMessages 49

2.6.4 MmDebug 49

2.6.5 NtGlobalFlag 49

2.6.6 SepDumpSD 50

2.6.7 CmLogLevel and CmLogSelect 50

2.7 Security Features in Vista 50

2.7.1 Address Space Layout Randomization (ASLR) 50

2.7.2 Stack Randomization 51

2.7.3 Heap Defenses 52

2.7.4 NX 54

2.7.5 /GS 55

2.7.6 Pointer Encoding 56

2.7.7 Cryptographic API in Windows Vista 58

2.7.8 Crypto-Agility 59

2.7.9 CryptoAgility in CNG 60

2.7.10 Algorithm Providers 62

2.7.11 Random Number Generator 63

2.7.12 Hash Functions 64

2.7.13 Symmetric Encryption 65

2.7.14 Asymmetric Encryption 67

2.7.15 Signatures and Verification 68

2.8 Conclusion 68

Portable Executable File Format

3.0 Introduction 69

3.1 PE file Format 69

3.2 Import Address Table 77

3.3 Executable and Linking Format 79

3.3.1 ELF Header 79

3.3.2 The Program Header Table 80

3.4 Conclusion 83

Reversing Binaries for Identifying Vulnerabilities

4.0 Introduction 85

4.1 Stack Overflow 85

4.1.1 CAN-2002-1123 Microsoft SQL Server 'Hello' Authentication Buffer Overflow" 88

4.1.2 CAN-2004-0399 Exim Buffer Overflow 88

4.1.3 Stack Checking 90

4.2 Off-by-One Overflow 90

4.2.1 OpenBSD 2.7 FTP Daemon Off-by-One 93

4.2.3 Non-Executable Memory 94

4.3 Heap Overflows 94

4.3.1 Heap Based Overflows 96

4.4 Integer Overflows 106

4.4.1 Types Integer Overflow 108

4.4.2 CAN-2004-0417 CVS Max dotdot Protocol Command Integer Overflow 111

4.5 Format String 112

4.5.1 Format String Vulnerability 113

4.5.2 Format String Denial of Service Attack 115

4.5.3 Format String Vulnerability Reading Attack 115

4.6 SEH Structure Exception Handler 116

4.6.1 Exploiting the SEH 119

4.7 Writing Exploits General Concepts 122

4.7.1 Stack Overflow Exploits 122

4.7.2 Injection Techniques 123

4.7.3 Optimizing the Injection Vector 123

4.8 The Location of the Payload 123

4.8.1 Direct Jump (Guessing Offsets) 124

4.8.2 Blind Return 124

4.8.3 Pop Return 124

4.8.4 No Operation Sled 125

4.8.5 Call Register 125

4.8.6 Push Return 126

4.8.7 Calculating Offset 126

4.9 Conclusion 126

Fundamental of Reverse Engineering

5.0 Introduction 127

5.1 Anti-Reversing Method 127

5.2.1 Anti Disassembly 128

5.2.1.1 Linear Sweep Disassembler 128

5.2.1.2 Recursive Traversal Disassembler 130

5.2.1.3 Evasion of Disassemble 131

5.2.2 Self Modifying Code 135

5.2.3 Virtual Machine Obfuscation 139

5.3 Anti Debugging Techniques 140

5.3.1 BreakPoints 142

5.3.1.1 Software Breakpoint 142

5.3.1.2 Hardware Breakpoint 143

5.3.1.3 Detecting Hardware BreakPoint 144

5.4 Virtual Machine Detection 145

5.4.1 Checking Fingerprint Inside Memory, File System and Registry 145

5.4.2 Checking System Tables 145

5.4.3 Checking Processor Instruction Set 146

5.5 Unpacking 147

5.5.1 Manual Unpacking of Software 148

5.5.1.1 Finding an Original Entry Point of an Executable 148

5.5.1.2 Taking Memory Dump 154

5.5.1.3 Import Table Reconstruction 156

5.5.1.4 Import Redirection and Code emulation 162

5.6 Conclusion 166.

Notes:

Includes index.

Local Notes:

Acquired for the Penn Libraries with assistance from the Louis A. Duhring Fund.

ISBN:

9780387098241

0387098240

9780387894683

0387894683

OCLC:

261924757

Publisher Number:

99935081774