Dissecting a Stealthy Malware: A Step-by-Step Reverse Engineering Guide

 Introduction

In today’s cybersecurity landscape, malware authors continue to refine their evasion techniques, making detection and analysis more challenging. In this post, we will take a real-world malware sample, analyze its behavior, and reverse-engineer its functionality. Whether you are a beginner in malware analysis or a seasoned professional, this guide will provide actionable insights to enhance your reverse engineering skills.

Step 1: Identifying the Malware Sample

Before diving into deep analysis, we need to collect information about the malware.

1.1 Collecting the Sample

• Source of the Sample: Malware can be found in phishing emails, malicious attachments, drive-by downloads, or suspicious executables flagged by endpoint security solutions.
• Hashing & Storage: Before analysis, calculate MD5, SHA1, and SHA256 hashes for identification and comparison. Use a dedicated malware repository like MalwareBazaar or VirusShare.

1.2 Preliminary Static Analysis

• Using VirusTotal: Upload the sample to VirusTotal to check existing detections and related metadata.
• File Type & Structure: Use the file command or PEStudio to determine if it’s a PE file, script, or packed binary.
• Analyzing PE Headers: Look at compilation timestamps, import tables, and suspicious section names using tools like CFF Explorer.

Step 2: Static Analysis Without Execution

Static analysis helps in extracting useful insights without running the malware.

2.1 Checking Strings

• Use strings (Linux) or BinText (Windows) to identify hardcoded URLs, commands, or suspicious file paths.
• Watch for obfuscation techniques like XOR encoding or base64-encoded payloads.

2.2 Inspecting Imports & Exports

• Tools like PEView and PEStudio help determine API calls related to process injection, network connections, or persistence mechanisms.
• Look for functions like CreateRemoteThread, WriteProcessMemory, or LoadLibrary that indicate possible code injection.

2.3 Identifying Packing & Obfuscation

• Run Detect It Easy (DIE) or PEiD to check for UPX or custom packers.
• If packed, use UPX -d to unpack standard UPX-packed files.

Step 3: Dynamic Analysis (Executing in a Safe Environment)

To observe runtime behavior, we execute the sample in a controlled environment.

3.1 Setting Up a Safe Environment

• Use FlareVM, REMnux, or Cuckoo Sandbox to analyze malware in an isolated VM.
• Disable internet access to prevent real-world impact but use tools like INetSim to fake network services.

3.2 Behavioral Monitoring

• Use Procmon to track file modifications and registry changes.
• Wireshark captures network traffic to identify command-and-control (C2) communications.
• Regshot compares registry states before and after execution to spot persistence mechanisms.

3.3 Identifying Persistence Mechanisms

• Look for malware creating autorun registry entries (HKLM\Software\Microsoft\Windows\CurrentVersion\Run).
• Check for scheduled tasks (schtasks /query) or service installations.

Step 4: Code Disassembly & Debugging

Code disassembly and debugging are essential techniques in reverse engineering and software analysis, used to understand the inner workings of a program, identify vulnerabilities, or analyze malware. Disassembly involves converting machine code into a human-readable assembly language using tools like IDA Pro, Ghidra, or Radare2, allowing analysts to examine program logic and behavior. Debugging, on the other hand, involves executing the program step-by-step using debuggers like GDB, WinDbg, or LLDB to inspect registers, memory, and function calls in real time. These techniques help security researchers, malware analysts, and software developers detect and fix issues, bypass protections, or gain insights into proprietary or malicious code. For deeper insights, we analyze the malware’s code. Breakpoints are handy to spot the perfect code area which the researcher really looking: https://www.edison-newworld.com/2024/05/setting-up-breakpoints-in-virtualalloc.html

4.1 Disassembly with Ghidra or IDA Pro

• Identify key functions, loops, and suspicious API calls.
• Use function cross-referencing to understand execution flow.

4.2 Debugging with x64dbg or OllyDbg

• Set breakpoints to inspect decryption routines and C2 communications.
• Look for anti-debugging techniques like IsDebuggerPresent or CheckRemoteDebuggerPresent.

4.3 Extracting Decrypted Payloads

• Identify encrypted sections and use memory dumping tools like Scylla to dump unpacked code.
• Analyze dumped binaries separately to uncover secondary payloads.

Step 5: Extracting IOCs (Indicators of Compromise)

After thorough analysis, extract key artifacts:

5.1 File Hashes & Artifacts

• Collect MD5, SHA256, and file paths for tracking across security platforms.
• Identify and extract dropped files from the system.

5.2 Network Indicators

• Capture C2 domains, IP addresses, and DNS queries.
• Use tools like Fakenet-NG to simulate network responses and observe malware behavior.

5.3 YARA Rules for Detection

• Write detection rules using YARA to classify similar malware samples.
• Example rule:

rule ExampleMalware {

    strings:

        $a = "malicious_string" nocase

    condition:

        $a

}

Real-World Case Study: Dissecting a VB6 RAT

Background

A recent malware sample written in VB6 was identified, using msvbvm60.dll for execution. Upon analysis, we found:

• API calls related to keylogging (GetAsyncKeyState).
• Registry-based persistence in HKCU\Software\Microsoft\Windows\CurrentVersion\Run.
• XOR-encrypted network communication to an external C2 server.

Conclusion

Reverse engineering malware is a critical skill for cybersecurity professionals. By following these structured steps, you can gain a deeper understanding of how modern threats operate and develop stronger detection and mitigation strategies.

Next Steps

🚀 Want to practice? Download sample malware from MalwareBazaar or VirusShare (safely in a VM).
💬 Have insights or questions? Drop them in the comments below!