Lab 11.2: Malware Detection by Memory Analysis

Learning Objectives

By the end of this lab, you will be able to:

• Identify in-memory malware techniques
• Use anomaly detection tools
• Understand process injection
• Analyze a basic memory dump
• Recognize memory compromise indicators

Estimated Duration

55 minutes

Prerequisites

• Lab 11.1 completed
• Isolated Windows 10/11 VM
• Process Explorer, Process Monitor
• Administrator rights

Professional Context

Detecting in-memory malware is critical because:

• "Fileless" malware does not write to disk
• Memory analysis reveals active behaviors
• It is an essential skill in incident response

---

WARNING

This lab uses techniques for educational purposes only. Execute only on isolated and test VMs.

---

Part 1: In-Memory Malware Techniques

Step 1.1: Overview

Technique	Description	Detection
Process Injection	Inject code into another process	Abnormal RWX regions
DLL Injection	Load a malicious DLL	Unsigned/unknown DLL
Process Hollowing	Replace a process's code	Image does not match disk
Reflective Loading	Load DLL without using the API	DLL without corresponding file
Fileless Malware	Code only in memory	Encoded scripts, suspicious PowerShell

Step 1.2: Key Indicators

Warning signs:

• RWX (Read-Write-Execute) memory regions
• System processes launched from wrong location
• Unsigned DLLs in sensitive processes
• Threads without associated module
• Abnormal network connections from legitimate processes

---

Part 2: Detection with Process Explorer

Step 2.1: Advanced Configuration

1. Launch Process Explorer as admin
2. Options > Verify Image Signatures
3. View > Select Columns > Add "Verified Signer"

Step 2.2: Identify Unsigned Processes

Processes in pink/purple have an unverified signature.

Processes that MUST be Microsoft signed:

• svchost.exe
• csrss.exe
• lsass.exe
• services.exe
• winlogon.exe
• explorer.exe

Question 1: Are all your system processes Microsoft signed?

Step 2.3: Verify Paths

Legitimate processes and their paths:
- svchost.exe    -> C:\Windows\System32\
- csrss.exe      -> C:\Windows\System32\
- lsass.exe      -> C:\Windows\System32\
- services.exe   -> C:\Windows\System32\
- explorer.exe   -> C:\Windows\

Verification script:

# Verify paths of critical processes
$criticalProcesses = @{
    "svchost" = "C:\Windows\System32\svchost.exe"
    "csrss" = "C:\Windows\System32\csrss.exe"
    "lsass" = "C:\Windows\System32\lsass.exe"
    "services" = "C:\Windows\System32\services.exe"
    "winlogon" = "C:\Windows\System32\winlogon.exe"
}

Write-Host "=== CRITICAL PROCESS VERIFICATION ===" -ForegroundColor Cyan

foreach ($proc in $criticalProcesses.Keys) {
    $running = Get-Process -Name $proc -ErrorAction SilentlyContinue
    
    foreach ($p in $running) {
        $expectedPath = $criticalProcesses[$proc]
        $actualPath = $p.Path
        
        if ($actualPath -ne $expectedPath) {
            Write-Host "[ALERT] $proc" -ForegroundColor Red
            Write-Host "  Expected: $expectedPath"
            Write-Host "  Found:  $actualPath"
        } else {
            Write-Host "[OK] $proc - $actualPath" -ForegroundColor Green
        }
    }
}

Question 2: Does the script detect any anomalies?

---

Part 3: DLL Injection Detection

Step 3.1: List a Process's DLLs

Process Explorer:

1. Select a process
2. View > Show Lower Pane
3. View > Lower Pane View > DLLs

PowerShell:

# List DLLs of a process
$proc = Get-Process notepad
$proc.Modules | Select-Object ModuleName, FileName, Size

# Look for suspicious DLLs (not in System32 or Program Files)
$proc.Modules | Where-Object {
    $_.FileName -notlike "C:\Windows\*" -and
    $_.FileName -notlike "C:\Program Files*"
} | Select-Object ModuleName, FileName

Step 3.2: Suspicious DLLs to Look For

Indicator	Description
DLL in Temp	Abnormal location
DLL in AppData	Potentially malicious
Unsigned DLL	Lack of verification
Random name	Possible malware
Very small size	Stub/loader

Step 3.3: DLL Detection Script

# Detection of suspicious DLLs
Write-Host "=== SUSPICIOUS DLL ANALYSIS ===" -ForegroundColor Cyan

$suspiciousPaths = @(
    "*\Temp\*",
    "*\AppData\Local\Temp\*",
    "*\Downloads\*",
    "*\Desktop\*"
)

Get-Process | Where-Object { $_.Modules } | ForEach-Object {
    $procName = $_.Name
    $_.Modules | ForEach-Object {
        $dll = $_
        foreach ($path in $suspiciousPaths) {
            if ($dll.FileName -like $path) {
                Write-Host "[SUSPECT] $procName loads: $($dll.FileName)" -ForegroundColor Red
            }
        }
    }
}

Write-Host "`nAnalysis complete." -ForegroundColor Green

Question 3: Are any suspicious DLLs loaded?

---

Part 4: Process Hollowing Detection

Step 4.1: Concept

Process Hollowing:

1. Create a legitimate process in suspended mode
2. Hollow out its memory
3. Inject malicious code
4. Resume execution

Result: The process appears legitimate but executes malicious code.

Step 4.2: Detection Indicators

Indicator	Description
Image ≠ Memory	Code in memory differs from file
Modified PEB	Altered Process Environment Block
Abnormal sections	Unexpected size/permissions

Step 4.3: Basic Verification

# Compare image size in memory vs on disk
# Requires advanced tools for true detection

Get-Process | Where-Object { $_.Path } | ForEach-Object {
    $proc = $_
    try {
        $fileSize = (Get-Item $proc.Path -ErrorAction Stop).Length
        $memorySize = $proc.WorkingSet64
        
        # Very high ratio may indicate an anomaly
        $ratio = [math]::Round($memorySize / $fileSize, 2)
        
        if ($ratio -gt 100) {
            Write-Host "[INFO] $($proc.Name) - Mem/file ratio: $ratio" -ForegroundColor Yellow
        }
    } catch {}
}

---

Part 5: Analysis with Process Monitor

Step 5.1: Capture Activity

procmon64.exe /accepteula

Step 5.2: Useful Filters for Detection

Create filters to detect suspicious behaviors:

1. Filter > Filter...
2. Add:
   • Operation is WriteFile then Include
   • Path contains \Temp\ then Include
   • Process Name is powershell.exe then Include

Step 5.3: Behaviors to Monitor

Operation	Suspicious if...
WriteFile	In Temp with .exe/.dll extension
CreateFile	Creation in system folders
RegSetValue	Modification of Run keys
CreateRemoteThread	In another process
VirtualAllocEx	Allocation in remote process

---

Part 6: Create a Memory Dump

Step 6.1: Dump a Process (Simple Method)

Task Manager:

1. Details tab
2. Right-click on the process
3. "Create dump file"

PowerShell:

# Use procdump (Sysinternals)
# procdump64.exe -ma <PID> dump.dmp

# Or via comsvcs.dll (note: detected by AVs)
# Method used by attackers to dump lsass

Step 6.2: Where Are Dumps Stored?

# Default location
$env:LOCALAPPDATA + "\Temp"

# List existing dumps
Get-ChildItem "$env:LOCALAPPDATA\Temp" -Filter "*.dmp" -ErrorAction SilentlyContinue

Step 6.3: Analyze a Dump (Basic)

For in-depth analysis, tools like Volatility are necessary.

With WinDbg (if installed):

windbg -z dump.dmp
!analyze -v
!process 0 0

---

Part 7: Complete Detection Script

# Memory anomaly detection script
# ========================================

param(
    [switch]$Verbose
)

Write-Host @"
=====================================
  MEMORY ANOMALY DETECTION
=====================================
"@ -ForegroundColor Cyan

$alerts = @()

# 1. Verify paths of critical processes
Write-Host "`n[1/5] Critical process verification..." -ForegroundColor Yellow

$criticalPaths = @{
    "svchost" = "C:\Windows\System32\svchost.exe"
    "lsass" = "C:\Windows\System32\lsass.exe"
    "csrss" = "C:\Windows\System32\csrss.exe"
    "services" = "C:\Windows\System32\services.exe"
}

foreach ($name in $criticalPaths.Keys) {
    Get-Process -Name $name -ErrorAction SilentlyContinue | ForEach-Object {
        if ($_.Path -and $_.Path -ne $criticalPaths[$name]) {
            $alerts += "[CRITICAL] $name from wrong path: $($_.Path)"
        }
    }
}

# 2. Look for DLLs in suspicious locations
Write-Host "[2/5] DLL analysis..." -ForegroundColor Yellow

$suspectDlls = @()
Get-Process | Where-Object { $_.Modules } | ForEach-Object {
    $_.Modules | Where-Object {
        $_.FileName -like "*\Temp\*" -or
        $_.FileName -like "*\Downloads\*"
    } | ForEach-Object {
        $alerts += "[SUSPECT] Suspicious DLL: $($_.FileName)"
    }
}

# 3. Processes with unusual network connections
Write-Host "[3/5] Connection analysis..." -ForegroundColor Yellow

$networkProcs = Get-NetTCPConnection -State Established -ErrorAction SilentlyContinue | 
    Where-Object { $_.RemoteAddress -notmatch "^(127\.|::1|0\.0\.0\.0)" }

foreach ($conn in $networkProcs) {
    $proc = Get-Process -Id $conn.OwningProcess -ErrorAction SilentlyContinue
    if ($proc.Name -in @("notepad", "calc", "mspaint")) {
        $alerts += "[SUSPECT] $($proc.Name) has a network connection: $($conn.RemoteAddress):$($conn.RemotePort)"
    }
}

# 4. PowerShell with connections
Write-Host "[4/5] PowerShell verification..." -ForegroundColor Yellow

$psConnections = Get-NetTCPConnection -OwningProcess (Get-Process -Name powershell -ErrorAction SilentlyContinue).Id -ErrorAction SilentlyContinue |
    Where-Object { $_.State -eq "Established" }

if ($psConnections) {
    $alerts += "[INFO] PowerShell has active network connections"
}

# 5. Processes with RWX memory (requires advanced tools)
Write-Host "[5/5] Verification complete." -ForegroundColor Yellow

# Report
Write-Host "`n========== REPORT ==========" -ForegroundColor Cyan

if ($alerts.Count -eq 0) {
    Write-Host "[OK] No major anomalies detected" -ForegroundColor Green
} else {
    Write-Host "Alerts detected: $($alerts.Count)" -ForegroundColor Red
    foreach ($alert in $alerts) {
        if ($alert -match "CRITICAL") {
            Write-Host "  $alert" -ForegroundColor Red
        } elseif ($alert -match "SUSPECT") {
            Write-Host "  $alert" -ForegroundColor Yellow
        } else {
            Write-Host "  $alert" -ForegroundColor Gray
        }
    }
}

Write-Host "`n=============================" -ForegroundColor Cyan

Question 4: Run the script. What alerts does it generate?

---

Deliverables

1. Screenshots:

   • Process Explorer with verified signatures
   • DLLs of a process
   • Detection script result

2. Answers to the 4 questions

3. Report (2 pages):

   • In-memory malware techniques
   • Detection methods used
   • Results of your VM analysis
   • Recommendations

---

Evaluation Criteria

Criterion	Points
Process Explorer	25%
DLL Detection	25%
Detection Script	30%
Report and Understanding	20%

---

Resources

• MITRE ATT&CK: Process Injection
• Volatility Framework
• Process Hollowing Explained

---

Lab developed as part of the Operating Systems course - MODULE 11