Nmap Architecture and Scanning Fundamentals
TCP/IP Mechanics Underlying Host Discovery and Port Scanning
Nmap's effectiveness rests on precise manipulation of protocol behaviors defined in RFC 793 (TCP), RFC 792 (ICMP), and RFC 768 (UDP). Host discovery and port scanning exploit the fact that a TCP/IP stack must respond to certain stimuli—even a refusal to communicate constitutes information.
TCP Three-Way Handshake and Scanning Implications
Client Server
| |
| -------- SYN --------> | [Scan probe: port open?]
| |
| <---- SYN/ACK -------- | [Port open: willing to establish]
| |
| -------- ACK --------> | [Full connect scan completes here]
| |
| <---- RST (or data) --- | [Nmap may RST to avoid half-open]
| |
| -------- SYN --------> |
| <---- RST/ACK -------- | [Port closed: actively refused]
| |
| -------- SYN --------> |
| (timeout / ICMP unreach) | [Port filtered: no response or blocked]
The handshake reveals three critical states from a single probe: open (SYN-ACK received), closed (RST-ACK received), and filtered (no response, or ICMP admin prohibited). Nmap's SYN scan (-sS) sends SYN and aborts before the final ACK—this "half-open" technique avoids logging in some application daemons and requires raw socket privileges.
For UDP, the absence of connection state means an open port typically returns no response, while a closed port returns ICMP Port Unreachable. UDP scanning is inherently slower and less reliable; many administrators mistakenly assume UDP exposure is benign.
⚠️ Authorized, defensive use only. Idle/zombie scans and certain fragmentation techniques described below are primarily used to test detection capabilities of your own monitoring infrastructure. Deploy only in lab environments or authorized detection-validation exercises.
Nmap Core Architecture
Nmap operates as a packet engine with four integrated subsystems:
| Subsystem | Function | Key Characteristic |
|---|---|---|
| Probe Engine | Crafts raw IP packets with specific TCP/UDP/ICMP flags, options, and payloads | Bypasses OS networking stack; full control of header fields |
| Response Analyzer | Parses returned packets against RFC-compliant expected behaviors | Distinguishes state by timing, flags, window size, and IP ID sequences |
| Timing System | Manulates probe rate, parallelization, and timeout calculations | Adapts to network conditions; critical for evasion and accuracy |
| Fingerprinting Engine | Corresponds probe responses against database of known OS/service signatures | Active probing reveals implementation quirks invisible to passive observation |
The timing system deserves particular attention. Nmap's -T0 through -T5 presets adjust inter-packet delay, timeout thresholds, and parallel probe count. -T4 (aggressive) assumes reliable LAN conditions; -T2 (polite) reduces collateral impact on fragile networks. Manual tuning via --min-rate, --max-retries, and --host-timeout permits precise control that presets cannot match.
Scan Types Taxonomy
Nmap implements distinct scan techniques by varying the probe's TCP flag composition:
| Technique | Flags Set | Response Interpretation | Typical Use Case |
|---|---|---|---|
| TCP Connect (-sT) | Full three-way handshake | Standard connect() result; no raw sockets needed | When SYN scan unavailable (unprivileged) or proxy traversal |
| SYN (-sS) | SYN only | Half-open; SYN-ACK = open, RST = closed, silence = filtered | Default, fastest, stealthiest privileged scan |
| UDP (-sU) | UDP payload to port | ICMP unreachable = closed; timeout = open or filtered | Service discovery on DNS, SNMP, VoIP infrastructure |
| ACK (-sA) | ACK only | RST = unfiltered (stateful firewall absent); timeout = filtered | Firewall rule mapping, not port state |
| Window (-sW) | ACK only (like -sA) | Window field in RST differentiates open/closed on certain systems | Rare; useful against specific legacy stacks |
| Maimon (-sM) | FIN/ACK | Open ports ignore per RFC 793; closed return RST | Obscure; bypasses some stateless packet filters |
| Idle/Zombie (-sI) | Spoofed from zombie host | IP ID sequence analysis on zombie reveals scan results | Anonymized scanning; requires predictable IP ID host |
The ACK and Window scans illustrate a crucial principle: not all scans determine whether a port accepts connections. These "firewall scans" map filtering rules. A port marked unfiltered merely means a probe reached the host; it says nothing about service availability. Conflating unfiltered with open is a common analytical failure.
State Machine: Beyond Open and Closed
Nmap reports six port states, and their semantics matter for accurate interpretation:
| State | Definition | Diagnostic Value |
|---|---|---|
| open | Service accepting connections | Target for version detection and vulnerability correlation |
| closed | No service bound; RST received | Confirms host reachable; firewall allows traffic |
| filtered | No response; probe dropped | Firewall or ACL intervening; requires evasion or alternative path |
| unfiltered | ACK scan response only; port state undetermined | Firewall absent; follow with SYN or Connect scan |
| open|filtered | UDP, IP proto, FIN/Null/Xmas scan ambiguity | Could not distinguish open from filtered; needs additional probing |
| closed|filtered | Maimon, ACK/Window ambiguity on certain systems | Rare; typically requires rescan with different technique |
A port marked filtered is not a clean result—it is an unanswered question. The absence of response may indicate a drop rule, rate limiting, or asymmetric routing. Security analysts must correlate with ICMP codes (type 3, code 1 = host unreachable; code 3 = port unreachable; code 13 = admin prohibited) and consider temporal variance across multiple probes.
OS Fingerprinting and Version Detection
Version detection (-sV) operates by sending service-specific probes and matching banner responses or behavioral quirks against nmap-service-probes. Probes are categorized by rarity; common probes run first, exotic ones only with --version-all. Detected versions are probabilistic—banners can be trivially forged, and implementations may emulate other services.
OS fingerprinting (-O) constructs a signature from:
- TCP initial sequence number (ISN) predictability
- IP ID generation algorithm (incremental, random, constant, timestamp-derived)
- TCP timestamp option behavior
- TCP window size and option ordering
- Explicit ICMP/TCP probe responses to malformed packets
The fingerprint is matched against nmap-os-db. Confidence levels reflect statistical proximity; a 95% match is not certainty. Virtualization, containerization, and deliberate OS spoofing degrade accuracy. Modern cloud infrastructure often returns generic Linux signatures that obscure the underlying kernel patch level.
Output Formats and Logging Granularity
Nmap provides multiple output formats, each serving distinct operational needs:
# Lab: Full SYN scan with all output formats for documentation
nmap -sS -O -sV -p- -T4 \
-oN scan-report-normal.txt \
-oX scan-report.xml \
-oG scan-report.gnmap \
-oS scan-report-skiddie.txt \
192.0.2.0/24
What it does: Performs comprehensive scan across all 65535 ports, with OS and version detection, writing four parallel output formats. When to use it: Baseline documentation, import to SIEM/reporting tools (XML), grep-based automation (gnmap), or human review (normal). Risks: Full port range (
-p-) with version detection generates significant traffic; intrusive on production. Expected output: Multiple files; XML suitable forndiffcomparison and parser ingestion.
| Format | Extension | Purpose | Trade-off |
|---|---|---|---|
| Normal (-oN) | .txt |
Human-readable with runtime metadata | Not machine-parseable; verbose |
| XML (-oX) | .xml |
Structured data; import to tools, databases, XSLT transforms | Verbose; requires parser; most complete |
| Grepable (-oG) | .gnmap |
One host per line; awk/grep/shell-script friendly |
Loses script output detail; flat structure |
| Script Kiddie (-oS) | .txt |
Leetspeak transformation | Useless except for ironic presentation |
The grepable format persists despite XML superiority because pipeline integration remains faster than DOM parsing for ad hoc extraction:
# Extract hosts with any open web ports from grepable output
awk '/Host: / && /80\/open|443\/open|8080\/open/' scan-report.gnmap
What it does: Filters grepable output for hosts with HTTP/HTTPS or alternate web ports open. When to use it: Quick triage without XML tooling. Risks: Grepable format omits NSE script output and OS fingerprint details. Expected output: Lines matching pattern, each containing full host status and port list.
Production variant (lower intensity, single output format for monitoring):
# Production: Service uptime check with rate limiting
nmap -sS -p 22,80,443,3389 --min-rate 50 --max-retries 2 \
-oG - 198.51.100.0/26 | tee daily-uptime.gnmap
What it does: Limited port check at 50 packets/second maximum, with reduced retry tolerance, streaming grepable output. When to use it: Routine service monitoring without impacting production. Risks: Even restrained scans may trigger IDS; coordinate with monitoring teams. Expected output: Single-line per host; suitable for cron-driven diff against prior runs.
Active vs. Passive Reconnaissance Contrast
| Dimension | Active (Nmap) | Passive |
|---|---|---|
| Packet generation | Probes sent to target | No direct interaction; observes existing traffic |
| Detectability | Logged by firewalls, IDS, host OS | Invisible to target |
| Completeness | Enumerates all responsive hosts/ports | Limited to communicating endpoints |
| Accuracy | Precise state determination; low false negative | Dependent on traffic volume; misses quiet hosts |
| Speed | Controlled by operator; can be rapid | Bounded by natural traffic patterns |
| Legal/contractual position | Requires explicit authorization | Ambiguous; monitor own networks only |
Nmap is fundamentally active. Each probe is a detectable event. Defensive operators should recognize that adversaries use identical techniques; Nmap's signature is well-catalogued in Snort/Suricata rules. The value of active scanning in defensive practice lies in baseline validation—confirming that your network's observed reality matches its documented state, and that your own detection infrastructure registers the activity it should.
Common Mistakes
| Mistake | Why it bites you |
|---|---|
Scanning without host discovery (-Pn default) |
Wastes hours probing blackhole routes or decommissioned subnets; always verify targets alive first |
Treating filtered as closed |
Misses exposed services behind stateful inspection; filtered ports may be accessible from alternative vantage points |
Running -A (aggressive: OS + version + traceroute + scripts) against everything |
Prohibitive time cost on large networks; version probes may crash fragile embedded services |
Ignoring --max-retries and --host-timeout |
Hanging scans on single unresponsive hosts delay entire job; timeouts prevent cascade failure |
| Trusting version banners blindly | Honeypots and deception frameworks serve forged banners; validate through behavioral testing |
Checklist: Pre-Scan Preparation
- [ ] Verify authorization scope: IP ranges, port boundaries, timing constraints
- [ ] Notify operational teams if production scanning; confirm monitoring exceptions
- [ ] Select scan technique matching privilege level and firewall posture
- [ ] Choose output format(s) based on downstream processing requirements
- [ ] Set timing template or manual rate limits appropriate to target resilience
- [ ] Document expected topology for deviation analysis post-scan