To start:
The best websites to review Meltdown and Spectre:
http://frankdenneman.nl/2018/01/05/explainer-spectre-meltdown-graham-sutherland/
https://meltdownattack.com/
Intel Announcement:
https://security-center.intel.com/advisory.aspx?intelid=INTEL-SA-00088&languageid=en-fr
Cisco Announcement:
https://tools.cisco.com/security/center/content/CiscoSecurityAdvisory/cisco-sa-20180104-cpusidechannel
Edit:12Jan2018 — added some more research URLs:
https://newsroom.intel.com/news/intel-security-issue-update-addressing-reboot-issues/
http://www.businessinsider.com/intels-telling-customers-not-to-install-its-fix-for-spectre-2018-1
https://www.theregister.co.uk/2018/01/12/intel_warns_meltdown_spectre_fixes_make_broadwells_haswells_unstable/
https://arstechnica.com/gadgets/2018/01/heres-how-and-why-the-spectre-and-meltdown-patches-will-hurt-performance/
This leads to a conversation about patches to the OS:
MS Windows KB:
https://support.microsoft.com/en-us/help/4072698/windows-server-guidance-to-protect-against-the-speculative-execution
MS SQL KB:
https://support.microsoft.com/en-us/help/4073225/guidance-for-sql-server
VMware KB:
https://kb.vmware.com/s/article/52264
https://www.vmware.com/us/security/advisories/VMSA-2018-0002.html
(VMware URL Updates – 20March2018)
https://blogs.vmware.com/security/2018/03/vmsa-2018-0004-3.html
https://www.vmware.com/security/advisories/VMSA-2018-0004.html
https://kb.vmware.com/s/article/52085
CVE-2017-5753, CVE-2017-5715, CVE-2017-5754
https://kb.vmware.com/s/article/52245
VMware Performance Impact for CVE-2017-5753, CVE-2017-5715, CVE-2017-5754
https://kb.vmware.com/s/article/52337
Red Hat KB:
https://access.redhat.com/security/vulnerabilities/speculativeexecution (click on Resolve tab)
https://access.redhat.com/articles/3311301
https://access.redhat.com/articles/3307751
The patch leads to a conversation about “CPU Performance” impact.
Linux based testing, (Kaiser patch):
https://medium.com/implodinggradients/meltdown-c24a9d5e254e
https://www.phoronix.com/scan.php?page=article&item=linux-415-x86pti&num=2
Red Hat specific testing:
https://access.redhat.com/node/3307751
Microsoft SQL Server specific:
https://www.brentozar.com/archive/2018/01/sql-server-patches-meltdown-spectre-attacks/
Cloud impact:
AWS:
https://www.theregister.co.uk/2018/01/04/amazon_ec2_intel_meltdown_performance_hit/
Azure:
https://azure.microsoft.com/en-us/blog/securing-azure-customers-from-cpu-vulnerability/
*** interesting note here, the article talks about a network impact.
Video Game Company that uses cloud reports CPU impact:
https://www.theverge.com/2018/1/6/16857878/meltdown-cpu-performance-issues-epic-games-fortnite
Vmware Performance impact:
https://blogs.vmware.com/management/2018/01/assess-performance-impact-spectre-meltdown-patches-using-vrealize-operations-manager.html
https://kb.vmware.com/s/article/52337
General hardware Testing from Blog Sites: (Be sure to check out all four pages.)
https://www.techspot.com/article/1556-meltdown-and-spectre-cpu-performance-windows/
http://www.datacenterknowledge.com/security/spectre-meltdown-hit-prem-windows-servers-hardest
Actual Performance KBs or Blogs from:
Microsoft:
https://cloudblogs.microsoft.com/microsoftsecure/2018/01/09/understanding-the-performance-impact-of-spectre-and-meltdown-mitigations-on-windows-systems
VMWare:
Assess performance impact of Spectre & Meltdown patches using vRealize Operations Manager
Some Scripting from VMWare blogs to assist with validation:
https://vinfrastructure.it/2018/01/meltdown-spectre-check-vsphere-environment/
https://www.ivobeerens.nl/2018/01/10/know-spectre-meltdown-vmware-environments/
https://www.virtuallyghetto.com/2018/01/verify-hypervisor-assisted-guest-mitigation-spectre-patches-using-powercli.html
https://github.com/lamw/vghetto-scripts/blob/master/powershell/VerifyESXiMicrocodePatch.ps1
*********************
Just to be 100% clear here,
As of this blog post (08Jan2018) not all research has been completed by all Application, OS, Hardware, and Software companies. We are still learning from Hardware vendors (Dell, HP, Cisco, Intel, Apple, etc…) that we need microcode updates, and firmware updates. I’m sure Cell phone companies will push carrier updates and Antivirus companies will patch to identify bad behavior.
Once we get a full vetting of our Hardware, then a vetting of our hypervisor stack, we can start to work on every OS type and patch level, and then the application patches. All it takes is that single rogue machine to thwart all of this patching and testing.
With all of this said, (and knowing that not ALL patches are released for the VMware vSphere suite as of today), I wanted to do some simple testing to see what the current patch + build differences would do to some simple Windows OS builds. We hear a ton of hype on the internet about massive 30% CPU impacts and while that may be true for some, I needed to make sure that wasn’t a global impact and easy to reproduce.
*********************
I did complete some quick lab testing:
I used four Cisco B200M4 blades with the same E5-2699 @ 2.2Ghz CPU to test.
Host1 – vSphere 6.0 – 5050593 – No Patch
Host2 – vSphere 6.0 – 6921384 – Security Patch
Host3 – vSphere 6.5 – 5969303 – No Patch
Host4 – vSphere 6.5 – 7388607 – Security Patch
I deployed four VMs to each host, each host will run four VMs – sixteen total VMs.
Windows 7 – 2vcpu 6GB-MEM
Windows 2008 – 4vCPU – 16GB-MEM
Windows 2012 – 4vCPU – 16GB-MEM
Windows 2016 – 4vCPU – 16GB-MEM
I used DRS Pinning rules to keep each VM set pinned to each host
I used two “quick” testing tools that are able to run under all four Windows OSs.
1 – CPU Mark from the PassMark suite.
– This tool is able to run a suite of tests (Interger Math, Compression, Floating Point, Extended Instructions, Encryption, Sorting, and Single Thread testing)
– I configured all sixteen servers to run all tests in “long test” mode with thirty iterations. The test ran the VM CPU to “near” 100% for two hours.
2 – 7-Zip benchmark tool.
– Under the tools menu for 7-zip is a tool call Benchmark. This tool will aggressivly max out the CPU and measure “MIPS” Millions of Instructions Per Second attempting to Compress and Decompress data.
– The test runs until stopped. I ran the test across all sixteen servers for two hours minimum and recorded the difference in “MIPS” per OS and ESXI host.
Vmware has a testing suite called VMMark but it is mostly Linux appliance based and the level of difficulty to configure is large. After spending a few hours, I was able to get it to fully deploy but it would never successfully complete a load test with valid results. The product even suggests sending the results to vmware for correct analysis.
I ran both tools over several hours across all four Windows OS and across all four vSphere ESXi patch versions.
Extremely summerized report of my findings:
First Application (CPU Mark):
– No known patterns between OS and ESXi version or patch level to show an easy to identify performance impact for just this OS.
Data numbers: (higher is better)
Key : OS – 6.0 No Patch – 6.0 Patched – 6.5 No Patch – 6.5 Patched
Windows 7 – 3763 – 3768 – 3761 – 3760
Windows 2008 – 7017 – 7026 – 7016 – 7011
Windows 2012 – 6987 – 6982 – 6984 – 6984
Windows 2016 – 7004 – 6997 – 6994 – 6990
*** Quick review for CPU Mark load testing:
The only test to show different results (out of the nine CPU tests in the single score) was Floating Point Math.
The other eight load tests show similar results across all ESXi hosts within the same Windows OS.
Results do show a slight decrease in performance Between 6.0 non patched and 6.5 patched.
In some cases, the vSphere 6.0 patched version performaned better than non patched 6.0 version
In no cases did the 6.5 patched perform better than non patched 6.5
I would suggest a 1% reduction in performace between current non patched 6.0 and patch 6.5 for any application that requires “Floating Point Math” operations.
Second Application (7-zip):
– No known patterns between OS and ESXi version or patch level to show an easy to identify performance impact.
I ran two tests:
Test 1:
7-zip 32MB setting 1:1 Core to thread :
Host1 – vSphere 6.0 – 5050593 – No Patch
Host2 – vSphere 6.0 – 6921384 – Security Patch
Host3 – vSphere 6.5 – 5969303 – No Patch
Host4 – vSphere 6.5 – 7388607 – Security Patch
Win7-Host1 – compress = 7238 – decompress = 6182
Win7-Host2 – compress = 7325 – decompress = 6182
Win7-Host3 – compress = 7238 – decompress = 5780
Win7-Host4 – compress = 7130 – decompress = 6222
Win2008-Host1 – compress = 13971 – decompress = 12241
Win2008-Host2 – compress = 14092 – decompress = 12282
Win2008-Host3 – compress = 13930 – decompress = 12282
Win2008-Host4 – compress = 13731 – decompress = 12323
Win2012-Host1 – compress = 14071 – decompress = 12341
Win2012-Host2 – compress = 14111 – decompress = 12303
Win2012-Host3 – compress = 14029 – decompress = 12344
Win2012-Host4 – compress = 14196 – decompress = 12303
Win2016-Host1 – compress = 13906 – decompress = 12261
Win2016-Host2 – compress = 13865 – decompress = 12303
Win2016-Host3 – compress = 13708 – decompress = 12223
Win2016-Host4 – compress = 13906 – decompress = 12144
Test 2:
7-Zip 192MB setting 1:1 Core to thread : 10 passes minimum
Win7-Host1 – Total score = 189% – 3477/6561 MIPS
Win7-Host2 – Total score = 189% – 3493/6569 MIPS
Win7-Host3 – Total score = 187% – 3444/6432 MIPS
Win7-Host4 – Total score = 189% – 3438/6467 MIPS
Win2008-Host1 – Total score = 370% – 3436/12653 MIPS
Win2008-Host2 – Total score = 370% – 3452/12712 MIPS
Win2008-Host3 – Total score = 371% – 3453/12733 MIPS
Win2008-Host4 – Total score = 370% – 3437/12644 MIPS
Win2012-Host1 – Total score = 375% – 3504/13058 MIPS
Win2012-Host2 – Total score = 373% – 3457/12821 MIPS
Win2012-Host3 – Total score = 370% – 3477/12818 MIPS
Win2012-Host4 – Total score = 377% – 3483/13060 MIPS
Win2016-Host1 – Total score = 372% – 3473/12828 MIPS
Win2016-Host2 – Total score = 371% – 3441/12674 MIPS
Win2016-Host3 – Total score = 371% – 3423/12638 MIPS
Win2016-Host4 – Total score = 369% – 3444/12654 MIPS
Load test summary:
No patterns found compared to the data from the first load gen tool (CPU Mark)
For Windows 7 the 6.0 patched version performed better
For Windows 2008 the 6.5 unpatched performed better but not by a lot.
For Windows 2012 the 6.5 patched version performed better.
For Windows 2016 the 6.0 unpatched performed better.
Overall notes and my thoughts:
I was unable to find a performance impact just from patches and version of vSphere based on only using MS Windows OS benchmark testing.
I would like to configure VMMark and find better results. Maybe for another day.
Windows 2008 and 2012 will not get a patch (for now)
Older versions of MSSQL will not get patched.
I think every application and Operating System will be impacted in different ways.
We have no way to know what Application specific impacts will exist because of different configurations per application instance.
We have no way to know what OS level impacts will exist because we cannot calcuate all possible application configurations across all versions and patch levels of MS Windows.
If an increase in CPU usage is identified, the amout of electricity used, per system, and electricity used to cool the datacenter will be large. (globaly)