Operating System Fingerprinting and TCP/IP Stack Analysis

The Anatomy of nmap-os-db: Tests, Values, and Fuzzy Matching

Nmap's OS fingerprinting engine operates against a database of approximately 5,000+ fingerprints stored in nmap-os-db. Each fingerprint line follows a precise format: TEST(NAME)=VALUE. The test name encodes the probe type and target port state, while the value captures specific response characteristics.

Consider this excerpt from a Linux fingerprint:

SEQ(SP=101-105%GCD=1-6%ISR=104-108%TI=I%CI=I%II=I%SS=S%TS=7)
OPS(O1=M5B4ST11NW7%O2=M5B4ST11NW7%O3=M5B4NNT11NW7%O4=M5B4ST11NW7%O5=M5B4ST11NW7%O6=M5B4ST11)
WIN(W1=FE88%W2=FE88%W3=FE88%W4=FE88%W5=FE88%W6=FE88)
ECN(R=Y%DF=Y%T=3B-45%TG=40%W=FAF0%O=M5B4NNSNW7%CC=Y%Q=)
T1(R=Y%DF=Y%T=3B-45%TG=40%S=O%A=S+%F=AS%RD=0%Q=)

The SEQ line tests TCP sequence number predictability: SP (sequence number index), GCD (greatest common divisor), ISR (ISR counter), TI/CI/II (TCP/IP/ICMP ID generation), SS (shared sequence numbers), and TS (timestamp option algorithm). Values with ranges (101-105) permit natural variation; percent signs separate fields.

Nmap employs a weighted fuzzy matching algorithm rather than exact equality. Each test contributes to a subjectivity score—a running total of "penalty points" for deviations from perfect matches. A score of zero indicates identical behavior; higher scores indicate divergence. The engine sorts candidates by this score and reports only those below configurable thresholds. The default nmap-os-db format uses single-character codes for efficiency: R=Y means "response received: yes," DF=Y means "Don't Fragment bit set," S=O means "sequence number zero."

The Sixteen Core TCP/IP Stack Behavioral Tests

Nmap's OS detection sends a carefully orchestrated series of probes and measures sixteen distinct behavioral dimensions. These tests reveal implementation-specific quirks that differentiate operating systems at the kernel level.

Initial TTL (T/TT fields): The starting TTL value varies by OS—64 (Linux, macOS), 128 (Windows), 255 (Cisco IOS, Solaris). Nmap measures both the raw TTL (T) and guesses the initial value (TG). A responding packet with TTL 117 likely started at 128 and traversed 11 hops.

TCP Window Size (WIN/W fields): The advertised window in SYN-ACK responses often follows OS-specific patterns. Windows Server 2019 typically advertises 64240; Linux 5.x uses 29200 or 65535. Some systems vary window size by MSS, revealing RFC 1323 scaling implementation details.

Don't Fragment Bit (DF): Whether the target sets the DF bit in responses indicates path MTU discovery support and stack philosophy. Modern systems typically set DF; embedded devices may not.

IP ID Sequence Generation (TI/CI/II fields): The IP identification field generation algorithm reveals critical information. TI=I indicates incremental IDs (predictable, common in older systems); TI=RI means random increments; TI=Z indicates zero (some firewalls). This test distinguishes Windows (I or RI) from Linux (I in older kernels, more complex in newer).

TCP Options Ordering and Padding (O1-O6 fields): Perhaps the most distinctive fingerprint element. Each O field encodes options as type-length-value strings. M5B4 means MSS option with value 1460 (0x05B4); S means selective acknowledgment permitted; T means timestamp; N means NOP padding; W7 means window scale factor 7.

For example, O1=M5B4ST11NW7 decodes to: MSS 1460, SACK permitted, Timestamp with TSval and TSecr, NOP padding, Window scale 7. The ordering matters critically: Linux typically places MSS first; Windows often places it differently or omits SACK. Padding with NOPs to achieve four-byte alignment reveals compiler and stack implementation choices.

Additional tests include TCP explicit congestion notification (ECN), TCP timestamp behavior, acknowledgment number handling (T1-T7 tests against various port states), and UDP closed-port ICMP response analysis.

Active Versus Passive Fingerprinting: Methodology and Integration

Nmap's -O flag performs active fingerprinting—sending crafted probes and analyzing responses. This is detectable, intrusive, and often logged by intrusion detection systems. The alternative, passive fingerprinting, observes existing traffic without generating new packets.

Nmap integrates with p0f-style passive analysis primarily through the --osscan-limit tradeoff. When specified, Nmap restricts OS detection to targets with at least one open and one closed port, dramatically improving accuracy at the cost of skipping "difficult" targets. Without this flag, Nmap attempts OS detection against any responding host, potentially producing unreliable results.

Active fingerprinting's advantage is control: probes are precisely timed, target specific port states, and exercise edge-case behaviors. Passive fingerprinting's advantage is stealth but requires observing meaningful traffic (SYN exchanges, ICMP errors) and cannot probe closed ports directly. For red team operations, passive collection via p0f or packet capture analysis precedes any active confirmation.

The Open/Closed Port Requirement and --osscan-guess Implications

A critical, often misunderstood constraint: Nmap's OS detection requires at least one open and one closed port for full accuracy. This requirement exists because different tests target different port states. Sequence number tests require an open port to send SYN probes; ICMP port-unreachable behavior requires a closed UDP port; certain TCP option negotiations differ between open and closed responses.

When this ideal condition isn't met, Nmap applies penalties to the subjectivity score and may refuse to present results. Consider this scenario:

nmap -O --osscan-guess 192.168.1.50

Without --osscan-guess, Nmap outputs: "OS detection performed. Please report any incorrect results at https://nmap.org/submit/ . Nmap done: 1 IP address (1 host up) scanned in 12.34 seconds"—no OS identified. The engine found candidate matches but with insufficient confidence.

With --osscan-guess (or -A, which enables it implicitly), Nmap presents the closest matches regardless of threshold, prefixed with confidence percentages. Output resembles:

Aggressive OS guesses: Linux 5.4 - 5.10 (87%), Linux 5.0 - 5.5 (85%), 
                       Linux 4.15 - 5.8 (83%)
No exact OS matches for host (test conditions non-ideal).

The 87% confidence indicates strong but not definitive matching—likely due to missing closed-port tests or packet filtering. This flag should be used cautiously: guessed results in security assessments can lead to incorrect exploit selection or misdirected hardening efforts.

Subjectivity Scoring and CPE Standardization

Nmap's subjectivity scoring operates on an internal scale where lower scores indicate better matches. The engine computes this by comparing each test's observed value against the database entry, applying test-specific weights. TCP options and sequence number behavior carry higher weights than TTL because they're more distinctive.

When multiple fingerprints tie or nearly tie, Nmap reports all matches. The scoring system incorporates feature importance: window size tests differentiate Windows versions more reliably than Linux distributions; timestamp behavior distinguishes BSD variants.

Output standardization uses Common Platform Enumeration (CPE) format: cpe:/o:linux:linux_kernel:5.10. This enables machine parsing, vulnerability correlation, and asset management integration. Nmap's XML output (-oX) includes CPE data explicitly; grepable format (-oG) does not.

Evasion, Deception, and Counter-Fingerprinting

Adversaries actively disrupt OS fingerprinting through several techniques. IP Personality (a Linux iptables module) modifies outgoing packet characteristics to mimic alternative stacks—changing TTL, window size, TCP option ordering, and timestamp behavior. Configuration requires specifying a target personality:

iptables -t mangle -A POSTROUTING -o eth0 -j OSFAKE --os "Linux 2.4"

Scrubbing middleboxes—firewalls, load balancers, and proxies—normalize traffic, often producing "confused" fingerprints that don't match any known OS. Palo Alto Networks, Fortinet, and Cisco ASA devices in transparent mode frequently exhibit this behavior.

Deliberate stack inconsistencies represent advanced deception: a Linux host might use Windows-typical window sizes but Linux-typical TCP options, or randomize behaviors per-connection. Research tools like zscrambler implement this approach.

When encountering suspicious fingerprints, examine for internal contradictions: Does the TTL suggest Windows while TCP options suggest Linux? Are timestamp frequencies physically impossible? Nmap's -d (debug) output reveals raw test results for manual analysis.

IPv6 Limitations and Community Contribution

IPv6 OS fingerprinting remains significantly less mature than IPv4. The nmap-os-db contains far fewer IPv6 fingerprints, and certain tests (particularly IP ID sequence analysis) are less reliable due to IPv6's larger address space and different header structure. The -6 flag enables IPv6 scanning, but -O against IPv6 targets frequently produces "No OS matches" or low-confidence guesses.

Current workarounds include dual-stack correlation (fingerprinting IPv4 and inferring IPv6 identity), passive observation of IPv6 neighbor discovery behavior, and manual analysis of ICMPv6 responses. The Nmap development team actively solicits IPv6 fingerprint submissions.

To contribute new fingerprints, use --osscan-guess against known targets, capture the debug output (-d --packet-trace), and submit to https://nmap.org/submit/ following the template format. Include exact OS version, kernel version, network configuration (VM or bare metal), and any intermediate devices. High-quality submissions with multiple verification runs against diverse targets significantly expand database coverage.

Reliability guidance: Treat subjectivity scores below 100 as reliable; 100-250 as borderline requiring cross-validation; above 250 as actively suspect. Always correlate OS detection with service version scanning (-sV) and script results for defense in depth of your own assessment accuracy.