How Allscripts Pro EHR can cripple your VDI infrastructure

As the administrator of a VMware Horizon View virtual desktop infrastructure (VDI) for a county hospital, I’m frequently looking for applications that abuse infrastructure resources by consuming excessive CPU, RAM, or Disk.

Allscripts Pro EHR and Allscripts Pro PM are two applications that often create unnecessary challenges in our environment.

  • Allscripts Pro PM requires Microsoft Data Execution Protection (DEP) to be disabled, which creates an unnecessary security risk
  • Allscripts Pro EHR has a way of launching several times, with each hidden process consuming an ever-increasing quantity of CPU resources

While the DEP issue bothers me the most, today I’m going to address how runaway CPU processes of Allscripts Pro EHR on enough virtual desktops can bring your entire VDI infrastructure to its knees. I’ll use Windows PowerShell to identify the problem desktops and stop those runaway processes.

Allscripts does not support VDI

To be fair, Allscripts does not support VDI. They only support deploying their client to end users via a Citrix XenApp farm or by traditional physical PCs. Absurd, I know.

For four years I have pleaded for Allscripts to support VDI, VMware Horizon View, and VMware ThinApp application virtualization solutions. Sadly, they refuse. I think I now understand why.

What happens to one, happens to many

Our VDI infrastructure consists of 10 physical ESXi hosts that each run 80 to 100 virtual desktops.

If a single application on a single virtual desktop consumes an excessive amount of CPU resources, it’s no big deal.

But when a couple hundred virtual desktops run the same poorly designed application that is consuming an excessive amount of CPU, the physical hosts begin to run out of available CPU resources and all desktops begin to suffer performance issues.

vdi-high-cpuYikes! Too many concurrent desktops with high CPU.

Using PowerShell to discover the evil application

All of our virtual desktops are linked-cloned and have a consistent naming convention. If I use the VMware vSphere Client to view all virtual machines in the cluster and sort by CPU, I can quickly identify the virtual desktops that are consuming the most CPU resources.

vdi-high-cpu-desktopsHmmm. Most of my problem VMs are in the two Allscripts desktop pools.

Looks like most of my CPU-consuming desktops are in our two Allscripts pools named VDI-ASPMLO and VDI-ASPMMO. I wonder what application is unique to those pools that might be causing the problem. I think I’ll use some PowerShell to figure it out.

This first PowerShell line creates a variable that includes the 200 virtual desktops that begin with the computer name “VDI-ASPM”.

PS C:> $TargetComputers = Get-ADComputer -Filter 'Name -Like "VDI-ASPM*"' | ForEach {$_.Name}

Next I want to retrieve all processes running on those computers that are consuming at least 1000 CPU resources, then sort them by amount of CPU consumed.

PS C:> Invoke-Command -ComputerName $TargetComputers -ScriptBlock { Get-Process | Where CPU -gt 1000 } -ErrorAction SilentlyContinue | Sort CPU -Descending | Format-Table -AutoSize

The result looks something like this:

   CPU(s)   Id ProcessName        PSComputerName
   ------   -- -----------        --------------
17,528.25 2292 EMR                VDI-ASPMMO-027
16,352.27 4740 EMR                VDI-ASPMMO-004
15,268.02  304 EMR                VDI-ASPMMO-047
14,148.84  420 EMR                VDI-ASPMLO-032
11,604.27 4180 EMR                VDI-ASPMLO-058
10,463.36 5844 EMR                VDI-ASPMMO-027
 8,339.34 3148 EMR                VDI-ASPMLO-061
 8,243.88 3084 EMR                VDI-ASPMLO-061
 8,065.75 5352 EMR                VDI-ASPMLO-104
 7,948.77 4300 EMR                VDI-ASPMLO-032
 7,924.20 2032 EMR                VDI-ASPMLO-061
 7,293.23 2368 LibreOffice Writer VDI-ASPMLO-027
 7,104.48 2636 EMR                VDI-ASPMMO-055
 7,085.19 5772 EMR                VDI-ASPMMO-042
 7,081.38 3728 EMR                VDI-ASPMLO-100
 7,052.19 1564 EMR                VDI-ASPMMO-024
 6,559.42 1360 EMR                VDI-ASPMLO-029
 6,391.80 5596 EMR                VDI-ASPMLO-029
 6,242.86  616 EMR                VDI-ASPMMO-106
 6,192.31 3604 EMR                VDI-ASPMLO-032
 6,062.73  332 EMR                VDI-ASPMLO-052
 5,875.50 2340 EMR                VDI-ASPMLO-052
 5,732.13 3924 EMR                VDI-ASPMLO-029
 5,659.23 2792 EMR                VDI-ASPMLO-052
 5,400.81  980 svchost            VDI-ASPMMO-002
 4,971.89 2504 EMR                VDI-ASPMLO-052
 4,946.58 3288 EMR                VDI-ASPMMO-106
 4,893.67  848 svchost            VDI-ASPMMO-002
...

I think I see a pattern. The EMR process belongs to the application Allscripts Pro EHR. It looks like there are many EMR processes that are consuming more than the normal amount of CPU. The average CPU process for the EMR application in our environment should be about 350 — not 1,000 to 17,000!

Why are there multiple EMR processes?

Is the user just doing something within EMR that is CPU intensive, or is something else going on?

To investigate, I first established a remote PowerShell session into one of the desktops that is consuming the most CPU resources: VDI-ASPMMO-027.

PS C:> Enter-PSSession VDI-ASPMMO-027

Now that I have a remote session, I wanted to take a closer look at the high-CPU consuming EMR process.

[VDI-ASPMMO-027]: PS C:> Get-Process | Where ProcessName -eq "EMR" | Sort CPU -Descending | Format-Table -AutoSize

Here was the surprising result.

Handles NPM(K)  PM(K) WS(K) VM(M)    CPU(s)   Id ProcessName
------- ------  ----- ----- -----    ------   -- -----------
    484     30 116928 47140   413 18,256.72 2292 EMR
    617     36 151188 67428   489 11,197.84 5844 EMR
    805     77 236652 61420   735    506.00 4272 EMR
    694     70 206676 61904   623    117.70 2336 EMR

For an unknown and unnecessary reason, there appear to be four EMR processes running on desktop VDI-ASPMMO-027.

If I use VMware vSphere, I can open a remote virtual console to view the user’s desktop. Here’s what I find.

user-taskbarIt’s frustrating that Allscripts would use the same icon for Pro PM and Pro EHR.

From left to right, we find several Internet Explorer tabs are open, one Misys Vision application open, one Allscripts Pro PM application open, one Allscripts Pro EHR application open, and one Allscripts Pro EHR Input Manager open. All of these processes can be accounted for, except for three of the CPU-intensive EMR processes.

It appears that the user is using only one EMR process and that three other EMR processes are consuming CPU resources but are not being used by the end user.

I’d like to kill EMR

Typically, the process that is consuming the most CPU resources is the process the end user is actively using. But I don’t think that is the case. I think the user is using the process (ID 2336) that is consuming the least amount of CPU resources.

To test, I’m going to kill all EMR processes except for the one that is consuming the least amount of resources.

If this PowerShell line returns all EMR processes sorted from least to first…

[VDI-ASPMMO-027]: PS C:> Get-Process | Where ProcessName -eq "EMR" | Sort CPU | Format-Table -AutoSize

…then adding the “-Skip 1” command will filter out the EMR process that is consuming the least amount of CPU resources (e.g. ID 2336).

[VDI-ASPMMO-027]: PS C:> Get-Process | Where ProcessName -eq "EMR" | Sort CPU | Select -Skip 1

Here is the result. Only three of the four EMR processes were returned.

Handles NPM(K)  PM(K) WS(K) VM(M)    CPU(s)   Id ProcessName
------- ------  ----- ----- -----    ------   -- -----------
    746     77 234000 52444   731    567.00 4272 EMR
    617     36 151188 67432   489 12,213.92 5844 EMR
    478     30 116932 47136   413 19,265.73 2292 EMR

All that’s left to do is bet that the user is using the EMR process that is consuming the least amount of CPU resources (e.g. ID 2336), and kill the others. I do this by piping the above result to the Stop-Process commmandlet.

[VDI-ASPMMO-027]: PS C:> Get-Process | Where ProcessName -eq "EMR" | Sort CPU | Select -Skip 1 | Stop-Process -Force

In each case, when I run this command while remotely viewing the user’s desktop, the end-user experiences no disruption in service. After killing the rogue Allscripts Pro EHR applications, performance for the end user dramatically improved.

Automatically kill rogue EMR processes

If you are not interested in manually finding and killing rogue EMR processes, here’s how to combine Group Policy, Scheduled Tasks, and PowerShell to kill Allscripts Pro EHR automatically.

Open Group Policy Management Editor and create a new policy named “PowerShell Stop-Process EMR if more than one is running” (or whatever you like). Navigate to Computer Configuration > Preferences > Control Panel Settings > Scheduled Tasks and create a new scheduled task named “PS Stop-Process EMR if multiple are running” that runs under the “NT AUTHORITY\System” account.

gp-st-general

Create a new Trigger that begins early and runs every five minutes. I suggest you should “Delay task for up to (random delay): 1 minute” to avoid all desktops from running this task at the same time.

gp-st-trigger

Under the Actions tab, I created two actions. Both are “Start a program” and both have Program/Script as “PowerShell”. My two Add Arguments are:

  • Get-Process | Where ProcessName -eq “EMR” | Sort CPU | Select -Skip 1 | Stop-Process -Force
  • Get-Process | Where ProcessName -eq “EMR” | Where CPU -gt 2000 | Stop-Process -Force

The first kills all but the least-consuming-CPU process. The second kills any EMR process that exceeds 2000 CPUs (which is way above average).

gp-st-actions

That’s it. All that is left to do is link your newly created Group Policy to the Active Directory Organizational Unit (OU) that contains the desktops you want to receive this policy.

In about 90 minutes, all desktops in the OUs you selected will have a new Scheduled Task that will run every 5 to 6 minutes that will kill all but the one EMR process that is consuming the least amount of CPU resources.

In our case, our ESXi hosts and end users were much happier to no longer have CPU resources needlessly consumed by a poorly designed application.