memcached Reflection/Amplification Description and DDoS Attack Mitigation Recommendations

ASERT Threat Summary: memcached Reflection/Amplification Description and DDoS Attack Mitigation Recommendations

Date/Time: 27022018 2325UTC

Title/Number: memcached Reflection/Amplification Description and DDoS Attack Mitigation Recommendations - February 2018 - v1.4.

Severity: Critical

Distribution: TLP WHITE (see <https://www.us-cert.gov/tlp)

Categories: Availability

Authors: Roland Dobbins & Steinthor Bjarnason

Contributors: Keshav Prabhakar, Luan Nguyen, Kirill Kasavchenko, Tomas Sundstrom, Jason Lang, & Jonas Krogell.

Changes from previous version: Clarified packet-size classifiers and corrected values for same. ----

Description:

Arbor has observed a significant increase in the abuse of misconfigured memcached servers residing on Internet Data Center (IDC) networks as reflectors/amplifiers to launch high-volume UDP reflection/amplification attacks. As memcached servers typically have relatively high-bandwidth access links and reside on IDC networks with high-speed transit uplinks, the nature of memcached which lends itself to abuse in high-bandwidth reflection/amplification DDoS attacks, and the rapid rise in observed prevalence of these attacks, we have classified the severity of this ASERT Threat Summary as Critical.

memcached is an in-memory database caching system which is typically deployed in IDC, ‘cloud’, and Infrastructure-as-a-Service (IaaS) networks to improve the performance of database-driven Web sites and other Internet-facing services. Due to its nature as a form of organic caching middleware and its lack of access controls (unless specifically compiled with a rarely-used TLS authentication option), memcached should not be exposed to the public Internet. Unfortunately, there are many memcached deployments worldwide which have been deployed using the default insecure configuration, and without benefit of situationally-appropriate network access policies implemented as transit ACLs (tACLs) to shield memcached servers from abuse by attackers.

In 2010, a presentation at BlackHat USA indicated that there were many insecure memcached deployments Internet-wide which could be used to retrieve and possibly alter sensitive databases of Internet-facing services such as Web servers, e-commerce sites, etc. And in November of 2017, memcached was identified as a possible reflection/amplification vector by the China-based ‘360 Okee’ security research team.

We have observed a considerable uptick in memcached reflection/amplification attacks ranging in size from a few hundred mb/sec up to 500gb/sec and larger. The amplified attack traffic is sourced from UDP/11211, with a packet size of 1428 bytes (1442 bytes with layer-2 Ethernet framing included), and no fragmentation (memcached segments large responses at layer-7, as does ntp). The attacker typically ‘primes’ a given set of memcached reflectors/amplifiers with arbitrary-length key/value pairs, and then issues memcached queries for those key/value pairs, spoofing the IP addresses of targeted hosts/networks. Both the priming queries and the attack-stimulus queries can be directed from source ports of the attacker’s choice to UDP/11211 on abusable reflectors/amplifiers, meaning that the attacker has full control of which destination port is targeted on the destination hosts/networks.

It should also be noted that memcached priming queries can also be directed towards TCP/11211 on abusable memcached servers. TCP is not currently considered a high-risk memcached reflection/amplification transport as TCP queries cannot be reliably spoofed.

Arbor’s current assessment is that, as with most other DDoS attack methodologies, memcached DDoS attacks were initially - and for a very brief interval - employed manually by skilled attackers; they have subsequently been weaponized and made available to attackers of all skill levels via so-called ‘booter/stresser’ DDoS-for-hire botnets. The rapid increase in the prevalence of these attacks indicates that this relatively new attack vector was weaponized and broadly leveraged by attackers within a relatively short interval.

Due to the nature of both the memcached service/protocol implementation as well as the prevalence and high bandwidth typically available to memcached reflectors/amplifiers, it is critical that network operators take proactive measures to ensure they are prepared to detect, classify, traceback, and mitigate these attacks, as well as ensure that any memcached installations on their networks and/or networks of their end-customers cannot be exploited as reflectors/amplifiers ----

Collateral Impact: The potential collateral impact of memcached reflection/amplification DDoS attacks can be highly significant, as these attacks exhibit high reflection/amplification ratios and leverage server-class reflectors/amplifiers which typically feature high-bandwidth access-links and which reside in Internet Data Centers (IDCs) with high-speed upstream transit links. Outbound memcached reflection/amplification traffic, due to its high volume, can also have a negative impact on networks with populations of abusable memcached servers. memcached can also be leveraged for crossbound reflection/amplification attacks targeting services/servers residing within the same IDCs as the memcached reflectors/amplifiers. ----

Mitigating Factors: memcached reflection/amplification DDoS attacks can be successfully mitigated by implementing industry-standard Best Current Practices (BCPs) such as source-address validation/BCP38/BCP84; by leveraging network infrastructure functionality such as flowspec, transit ACLs (tACLs), and selective quality-of-service (QoS) policies; and by utilizing intelligent DDoS mitigation systems (IDMSes) such as Arbor SP/TMS and APS to defend the targets of these attacks, as well as to selectively prevent exploitable reflectors/amplifiers from being abused by attackers. ----

Recommended Actions: All relevant network infrastructure, host/application/service, and operational Best Current Practices (BCPs) should be implemented by network operators. In particular, state minimization is highly encouraged as a general operational principle to increase resilience in the face of attack. Situationally-appropriate network access policies should be implemented via transit ACLs (tACLs) on Internet Data Center (IDC) upstream transit links to block unauthorized network traffic destined for UDP/11211 and TCP/11211 from ingressing the IDC.

Network operators should export flow telemetry (e.g., NetFlow, IPFIX, s/Flow, cflowd/jflow, Netstream, et. al.) from their peering/transit/customer aggregation edges and Internet data center (IDC) distribution edges to Arbor SP, which provides the ability to detect, classify, and traceback DDoS attack traffic.

Given that intentional production use of memcached across the public Internet is vanishingly rare, traffic sourced from UDP/11211 may be safely rate-limited at peering/transit/customer aggregation edges by the application of situationally-appropriate QoS policies deployed on edge routers. Alternately, transit ACLs (tACLs) may be deployed at peering edges, customer aggregation edges, and Internet data center (IDC) distribution gateway edges to block network traffic sourced from UDP/11211. In either case, care should be exercised to avoid unnecessary overblocking, and Arbor SP should be utilized in order to determine the efficacy of QoS policies or tACLs implemented at network edges.

Arbor SP/TMS and APS IDMSes may be deployed in a situationally-appropriate manner to mitigate these attacks via multiple DDoS countermeasures, as well as flowspec for both attack mitigation and selective traffic diversion (SP/TMS).

TMSes and/or APSes may be used both to mitigate reflected/amplified DDoS attack traffic as well as to prevent memecached priming queries and attack-stimulation queries from reaching misconfigured - and thus exploitable - memcached servers located in IDC networks and on end-customer premise networks.

As always, network operators are strongly encouraged to implement source address validation/BCP38/BCP84 in order to prevent their networks and the networks of their end-customers from being leveraged in reflection/amplification DDoS attacks. It is also recommended that network operators scan their IDC networks, as well as those of their end-customers, in order to identify abusable memcached installations so that remediation can take place on a timely basis. ----

Applicable NETSCOUT Arbor Solutions: Arbor APS, Arbor SP, Arbor TMS. ----

References: http://memcached.org/

https://sensepost.com/blog/2010/blackhat-write-up-go-derper-and-mining-memcaches

https://github.com/sensepost/go-derper

https://www.anquanke.com/post/id/87233

https://www.apnic.net/wp-content/uploads/2017/01/SAVE-Factsheet-01-copy.pdf

https://tools.ietf.org/html/bcp38https://tools.ietf.org/html/bcp84