Patching the Citrix ADC Bug Doesn’t Mean You Weren’t Hacked

Source: Bleeping Computer / Ionut Ilascu

January 26: Citrix on Friday released the final patch for the critical vulnerability tracked as CVE-2019-19781 in its affected appliances. Many organizations are still at risk, though, as they continue to run Citrix servers without a fix or the advised mitigations.

This security flaw is as bad as it can be since it allows unauthenticated attackers to directly access a company’s local network from the internet and run code via directory traversal.

It affects the Citrix Application Delivery Controller (ADC), Citrix Gateway, and Citrix SD-WAN WANOP appliances.

Patching is Not Enough

Permanent fixes are now available for all supported versions of ADC, Gateway, and SD-WAN WANOP vulnerable to CVE-2019-19781. Customers can download them even if they don’t have an active maintenance contract:

Installing the patches is not enough, though. Victor Gevers of the GDI Foundation warns that even if a Citrix device is no longer vulnerable it does not necessarily mean that no harm was done before patching.

Researchers from FireEye found that one attacker that exploited CVE-2019-19781, secured the appliance, and left a backdoor behind for future access. They noticed that the threat actor dropped on the compromised Citrix servers a payload they call NOTROBIN, which gains persistence via a cron entry.

NOTROBIN and cryptocurrency miners are the typical payloads dropped after exploiting CVE-2019-19781, but ransomware operators also jumped in delivering a strain called Ragnarok, FireEye notes in a report today.

Through four volunteers, the GDI Foundation scanned the internet for vulnerable Citrix devices and validated the results to remove false positives, sinkholes, and honeypots.

More than 98,000 vulnerable Citrix endpoints were found and reported through the Dutch Institute for Vulnerability Disclosure (DIVD) to companies and Internet Service Providers when the owner could not be determined.

Gevers told BleepingComputer that about 11,800 wildcard TLS certificates to validate multiple sub-domains were exposed at some point, and some of them still are.

If an attacker steals a wildcard certificate, they can build phishing sites that impersonate a reputable entity such as governments, hospitals, universities, or companies.

“So after patching these organizations need to have this certificates revoked and get new ones “ – Victor Gevers

DIVD issued an alert on Wednesday about the poor combination of wildcard certificates with the Citrix vulnerability. They said that scan data from January 9-10 showed that there were more than 450 Citrix servers in the Netherlands using wildcard certificates.

Check yourself

Mass-scanning for vulnerable Citrix systems continues and if attackers are behind this, there is no doubt that they will take advantage of any opportunity that springs up.

If mitigations were applied on or after January 9 and Citrix ADC version 12.1 build 50.28 was used, DIVD recommends checking the type of TLS certificate, replacing them, and having them revoked by the issuing certificate authority.

Organizations with Citrix appliances can find if their servers fell for CVE-2019-19781 exploitation. A local scanner developed by Citrix and FireEye flags indicators of compromise gathered from incident response engagements related to this vulnerability.

The tool assesses one device at a time and looks at log sources and system forensic artifacts to find traces of compromise. However, the scanner does not ensure a system’s integrity since it relies on known signs.

The Numbers

CVE-2019-19781 was announced by Citrix on December 17, 2019, and six days later more details emerged from security company Positive Technologies. They estimated was that at least 80,000 companies in 158 countries were at risk.

Until exploits appeared in the public space on January 11, attackers did not shy away from scanning the web for vulnerable devices. Statistics collected by the GDI‌ Foundation show that close to 114,000 Citrix servers were vulnerable to this bug on that day, a slight drop from over 128,000 recorded on December 31, 2019.

Scanning and attack activity skyrocketed the day after the first exploit became public. Data collected by the SANS ISC honeypots in the 10 days that followed shows that on January 12 there 290,000 attack attempts from 532 IP addresses in 42 countries.

In total, over 550,000 compromise attempts were recorded, most of them originating from Russia.

At that point, Citrix had no patches ready and provided mitigation steps that worked for most products but were ineffective on some of them.

Daily scans from the GDI Foundation show that 10,099 devices were exposed and vulnerable today at 16:00 (GMT+1).

The stats are encouraging, especially in the healthcare sector. Gevers told us that there are more than 16,000 hospitals across the globe. On January 15, after exploits appeared, 662 of them had a vulnerable Citrix server. By Tuesday, the number dropped to 32.

It is important to note that the GDI Foundation is a non-profit organization that works with volunteers to alert and help vulnerable organizations. In the past four years, it reported one million security issues and data leaks. Of these, more than 926,000 were fixed.

Its portfolio of cleanup operations includes EternalBlue, Memcached, and plenty of MongoDB and Elasticsearch instances. Some entities did not forget the assistance and are prompt to respond to the Foundation’s reports. This trust, built on a bedrock of cyber trouble, reflects in how quick volunteers can solve a problem.

“And when we ask if we’re allowed to login in remotely to help fix an issue, the answer is becoming more often “yes.” Which we consider an honor but also something we still have to get used to.” – Victor Gevers

Gevers is aware that there’s plenty to be done in the future and says that they’re looking for local experts to help out and are trying to keep GDI Foundation staffed with volunteers 24/7.

Source: Bleeping Computer / Ionut Ilascu