Poor DNS
Monday, 21 July 2008
See my updated blog entry: DNS Responses.
[UPDATE 2008-07-24: I have re-run the dig test against the wall of shame list. While it is wonderful news that many are now reporting as "GOOD", it is sad that it took a public disclosure and working exploit to cause them to apply the patch.]
[UPDATE 2008-07-25: I have received feedback from Verizon. The WHOIS information is inaccurate; the servers are owned and managed by Level3 and not Verizon.]
By now, we have all heard of the warning: a big bug exists in caching DNS servers. Patch now.
Dan Kaminsky clearly found something -- some type of big bug that has been lurking in DNS. Even without knowing the details, it is clear that the fix needs to be applied.
Many people in the community (myself included) have debated and speculated about whether Dan found something new, or a faster way to exploit a known problem, or simply rediscovered an ancient bug. The fact is, we will know in two weeks.
This topic has once again fueled the debate over full disclosure. One one hand, Dan has given the entire Internet notice of a defect, without providing details that could be exploited. Moreover, there is a fix, it is widely available, and strongly recommended. As Paul Vixie (creator of BIND and founder of the Internet Systems Consortium) wrote:
Moreover, there is a know attack coming: whether Dan's finding is new or old, it does not matter. He will be presenting his findings at Black Hat and Defcon. An ad hoc army of n00bs, kiddiez, anarchists, virus writers, phishers, and crackers stand ready to test the exploit shortly after they learn the details. As far as I can recall, this is the first time that there has ever been a serious threat announced, a fix available, and a known army of attackers sitting on the horizon, waiting for the weapon to become available. The attack is coming -- there is no reason not to implement the solution within the 30 day window before the conference.
Keep in mind, it is easy to tell if you are vulnerable. You can perform a simple DNS lookup in order to see if you are vulnerable:
With all of this in place, one would think that every major ISP would be rushing to apply the fix. However, this does not seem to be the case. With half of the 30-day warning period already past, a surprisingly large number of ISPs are still vulnerable. In fact, of the 60 DNS servers I tested, more than half of them were still vulnerable. Considering that many of the "safe" DNS servers were not vulnerable prior to this situation, this means that far fewer than half of the large ISPs have even reacted to the notice.
Here is the wall of shame so far. These are the DNS servers that I tested using dig and that returned a "POOR" result (indicating that they are vulnerable):
Verizon Level3, Adelphia, and Earthlink) are still identified as vulnerable. Consider this: The curious attacker will start by testing their own ISPs -- and since they likely use any one of the large ISPs, these servers will be hit first. And when the kiddiez learn of the exploit, they will go playing. They are certain to start with the lowest hanging fruit -- large companies that are vulnerable and support a huge number of users. That is this list. If a phisher were to poison the DNS server at any one of these providers (e.g., redirect queries for anybank.com to their phishing site), then the attackers could potentially intercept hundreds of thousands of bank accounts.
Sadly, there is no easy way to contact these companies. And complaining to their customer service won't help. Consider this: if these IT departments won't listen to warnings from CERT, the security community, and DNS vendors, then why will they listen to you?
Considering how well publicized the announcement has been, and the ample delay between announcement, patch and disclosure (all companies have had 30 days; some have had even longer), I certainly hope that any exploitation results in a class-action lawsuit based on gross incompetency.
However, not everything is bad. For completion, here are the DNS servers that have either applied the patch, or run DNS servers that are not vulnerable:
[UPDATE 2008-07-24: I have re-run the dig test against the wall of shame list. While it is wonderful news that many are now reporting as "GOOD", it is sad that it took a public disclosure and working exploit to cause them to apply the patch.]
[UPDATE 2008-07-25: I have received feedback from Verizon. The WHOIS information is inaccurate; the servers are owned and managed by Level3 and not Verizon.]
By now, we have all heard of the warning: a big bug exists in caching DNS servers. Patch now.
Dan Kaminsky clearly found something -- some type of big bug that has been lurking in DNS. Even without knowing the details, it is clear that the fix needs to be applied.
Many people in the community (myself included) have debated and speculated about whether Dan found something new, or a faster way to exploit a known problem, or simply rediscovered an ancient bug. The fact is, we will know in two weeks.
This topic has once again fueled the debate over full disclosure. One one hand, Dan has given the entire Internet notice of a defect, without providing details that could be exploited. Moreover, there is a fix, it is widely available, and strongly recommended. As Paul Vixie (creator of BIND and founder of the Internet Systems Consortium) wrote:
[T]ake the advisory seriously—we're not just a bunch of n00b alarmists, if we tell you your DNS house is on fire, and we hand you a fire hose, take it.
Moreover, there is a know attack coming: whether Dan's finding is new or old, it does not matter. He will be presenting his findings at Black Hat and Defcon. An ad hoc army of n00bs, kiddiez, anarchists, virus writers, phishers, and crackers stand ready to test the exploit shortly after they learn the details. As far as I can recall, this is the first time that there has ever been a serious threat announced, a fix available, and a known army of attackers sitting on the horizon, waiting for the weapon to become available. The attack is coming -- there is no reason not to implement the solution within the 30 day window before the conference.
Keep in mind, it is easy to tell if you are vulnerable. You can perform a simple DNS lookup in order to see if you are vulnerable:
dig @dnsserver porttest.dns-oarc.net in txt or go to Dan's site and run his "Check My DNS" script. The dig command will tell you if your system is "GOOD" or "POOR". Dan's system tells you if you are vulnerable or not.With all of this in place, one would think that every major ISP would be rushing to apply the fix. However, this does not seem to be the case. With half of the 30-day warning period already past, a surprisingly large number of ISPs are still vulnerable. In fact, of the 60 DNS servers I tested, more than half of them were still vulnerable. Considering that many of the "safe" DNS servers were not vulnerable prior to this situation, this means that far fewer than half of the large ISPs have even reacted to the notice.
Here is the wall of shame so far. These are the DNS servers that I tested using dig and that returned a "POOR" result (indicating that they are vulnerable):
- 4.2.2.1 --
Verizon (Level3)Level3 [Update: Reports 'FAIR' on 2008-07-24]
- 4.2.2.2 --
Verizon (Level3)Level3 [Update: Reports 'FAIR' on 2008-07-24]
- 4.2.2.3 --
Verizon (Level3)Level3 [Update: Reports 'FAIR' on 2008-07-24]
- 4.2.2.4 --
Verizon (Level3)Level3 [Update: Reports 'FAIR' on 2008-07-24]
- 4.2.2.5 --
Verizon (Level3)Level3 [Update: Reports 'FAIR' on 2008-07-24]
- 4.2.2.6 --
Verizon (Level3)Level3 [Update: Reports 'FAIR' on 2008-07-24]
- 24.113.32.29 -- Wave Broadband [Update: Still reports 'POOR' as of 2008-07-24]
- 24.113.32.30 -- Wave Broadband [Update: Still reports 'POOR' as of 2008-07-24]
- 24.48.217.226 -- Adelphia (
ComcastRoadRunner) [Update: Still reports 'POOR' as of 2008-07-24]
- 24.48.217.227 -- Adelphia (
ComcastRoadRunner) [Update: Still reports 'POOR' as of 2008-07-24]
- 67.21.13.2 -- Adelphia (
ComcastRoadRunner) [Update: Still reports 'POOR' as of 2008-07-24]
- 67.21.13.4 -- Adelphia (
ComcastRoadRunner) [Update: Still reports 'POOR' as of 2008-07-24]
68.168.1.42 -- Adelphia (Comcast)[Update: No longer returns results]
68.168.1.46 -- Adelphia (Comcast)[Update: No longer returns results]
- 68.87.64.196 -- Comcast [Update: Still reports 'POOR' as of 2008-07-24]
- 68.87.66.196 -- Comcast [Update: Still reports 'POOR' as of 2008-07-24]
- 68.87.85.98 -- Comcast [Update: Still reports 'POOR' as of 2008-07-24]
68.87.96.3 -- Comcast[Update: Reports 'GOOD' as of 2008-07-24]
68.87.96.4 -- Comcast[Update: Reports 'GOOD' as of 2008-07-24]
68.94.156.1 -- SBC/AT&T[Update: Reports 'GOOD' as of 2008-07-24]
68.94.157.1 -- SBC/AT&T[Update: Reports 'GOOD' as of 2008-07-24]
- 194.72.9.38 -- BTInternet [Update: Still reports 'POOR' as of 2008-07-24]
- 199.2.252.10 -- Sprintlink [Update: Still reports 'POOR' as of 2008-07-24]
- 202.188.1.5 -- Tmnet Streamyx [Update: Still reports 'POOR' as of 2008-07-24]
- 202.27.156.72 -- Xtra (New Zealand) [Update: Still reports 'POOR' as of 2008-07-24]
- 202.27.158.40 -- Xtra (New Zealand) [Update: Still reports 'POOR' as of 2008-07-24]
- 204.117.214.10 -- Sprintlink [Update: Still reports 'POOR' as of 2008-07-24]
- 204.97.212.10 -- Sprintlink [Update: Still reports 'POOR' as of 2008-07-24]
- 205.152.37.23 -- Bellsouth [Update: Still reports 'POOR' as of 2008-07-24]
207.69.188.186 -- Earthlink[Update: Reports 'GOOD' as of 2008-07-24]
207.69.188.187 -- Earthlink[Update: Reports 'GOOD' as of 2008-07-24]
209.55.0.110 -- Suddenlink[Update: Reports 'GOOD' as of 2008-07-24]
209.55.1.220 -- Suddenlink[Update: Reports 'GOOD' as of 2008-07-24]
Sadly, there is no easy way to contact these companies. And complaining to their customer service won't help. Consider this: if these IT departments won't listen to warnings from CERT, the security community, and DNS vendors, then why will they listen to you?
Considering how well publicized the announcement has been, and the ample delay between announcement, patch and disclosure (all companies have had 30 days; some have had even longer), I certainly hope that any exploitation results in a class-action lawsuit based on gross incompetency.
However, not everything is bad. For completion, here are the DNS servers that have either applied the patch, or run DNS servers that are not vulnerable:
- 64.81.111.2 -- Speakeasy
- 64.81.127.2 -- Speakeasy
- 64.81.159.2 -- Speakeasy
- 64.81.45.2 -- Speakeasy
- 64.81.79.2 -- Speakeasy
- 66.133.170.2 -- FrontierNet / Citlink / New North
- 66.133.191.35 -- FrontierNet / Citlink / New North
- 66.153.128.98 -- Horry Telephone Coop
- 66.153.162.98 -- Horry Telephone Coop
- 66.92.159.2 -- Speakeasy
- 66.92.224.2 -- Speakeasy
- 66.92.64.2 -- Speakeasy
- 66.93.87.2 -- Speakeasy
- 67.50.135.146 -- FrontierNet / Citlink / New North
- 68.10.16.25 -- Cox
- 68.10.16.30 -- Cox
- 68.12.16.25 -- Cox
- 68.12.16.30 -- Cox
- 68.2.16.30 -- Cox
- 68.9.16.30 -- Cox
- 170.215.126.3 -- FrontierNet / Citlink / New North
- 170.215.184.3 -- FrontierNet / Citlink / New North
- 170.215.255.114 -- FrontierNet / Citlink / New North
- 207.69.188.185 -- Earthlink
- 212.216.112.112 -- Alice (Italy)
- 212.216.172.62 -- Alice (Italy)
- 216.231.41.2 -- Speakeasy
- 216.254.95.2 -- Speakeasy
- 216.27.175.2 -- Speakeasy
- 216.67.192.3 -- FrontierNet / Citlink / New North
Slashdot posted a few links that seem to imply that the flaw has been revealed, or at least a plausible flaw has been revealed. Do you think this is the bug? And would this only affect non-SSL connections? What types of attacks would we see with a DNS-based bug?
And the same goes for anything from banks to webmail hosters to game sites and financial services, anything basically.
The basic bug btw is lack of trust in the DNS tree or by lack of that sufficient randomness in the queries.
(The latter is what causing the current fuss.)
Just for clarity:
In 2005, Comcast and Time Warner RoadRunner teamed up to acquire Adelphia.
http://www.iht.com/articles/2005/04/21/news/cable.php
I just re-ran the dig test 3 times (just to be sure). Two of the comcast are no longer responding. Three others still report 'POOR'. And two now report 'GOOD'.
It would not surprise me if Comcast patched the remaining servers in the next few days.
It is a pitty that it took a public disclosure and working exploit to make them perform the updates. While we don't like to think about it, this seems to demonstrate the need for full disclosure of exploits because large companies won't react under other situations.
DNS 1: 68.87.71.226
DNS 2: 68.87.73.242
http://www.robtex.com/dns/sys.gtei.net.html
I listed those servers as both Level3 and Verizon. As you pointed out, they are actually Level3 and gtei.net. However, the WHOIS record for gtei.net says Verizon and not Level3.
http://www.shootybangbang.com/whois.cgi?q=gtei.net
Since the domain owner is Verizon and the network owner is Level3, I listed them both.
As one user pointed out (http://www.neowin.net/forum/lofiversion/index.php/t598102.html)
"4.2.2.1 and 4.2.2.2 aren't run by any one entity; who they're run by depends on the user. 4.2.2.1 and 4.2.2.2 are standard IPs for the ISP DNS server physically nearest to the user."
There are also plenty of reports that associate the 4.2.2.x servers with Verizon:
http://www.broadbandreports.com/forum/r20763699-Other-RCN-DNS-Nameservers-Vulnerable-to-Poisoning-Techniques
Since ICANN requires WHOIS information for contacts to be accurate... Are you claiming that Verizon and/or Level3 are providing false WHOIS information about one of their services?
As I wrote to Anonymous:
I listed those servers as both Level3 and Verizon. As you pointed out, they are actually Level3 and gtei.net. However, the WHOIS record for gtei.net says Verizon and not Level3.
http://www.shootybangbang.com/whois.cgi?q=gtei.net
Since the domain owner is Verizon and the network owner is Level3, I listed them both.
In addition, since ICANN requires WHOIS information for contacts to be accurate... Are you claiming that Verizon and/or Level3 are providing false WHOIS information about this network service?
Thank You.
Thank you for the feedback. As soon as the WHOIS becomes updated with the correct information, I will correct my blog posting.
$dig @68.94.157.1 +short porttest.dns-oarc.net TXT
z.y.x.w.v.u.t.s.r.q.p.o.n.m.l.k.j.i.h.g.f.e.d.c.b.a.pt.dns-oarc.net.
"66.73.20.11 is POOR: 26 queries in 1.7 seconds from 1 ports with std dev 0.00"
$ dig @68.94.156.1 +short porttest.dns-oarc.net TXT
z.y.x.w.v.u.t.s.r.q.p.o.n.m.l.k.j.i.h.g.f.e.d.c.b.a.pt.dns-oarc.net.
"66.73.20.31 is POOR: 26 queries in 1.6 seconds from 1 ports with std dev 0.00"
Since I'm getting redirected to 66.73.20.11, I expect some sort of location-based traffic direction is going on. Whether you get to a fixed server or a unpatched one may be a matter of luck.
In an email exchange with the AT&T NOC, they stated that they planned to start patching next week. Better late than never?
Wow! Thanks for pointing this out!
Here's what I'm getting:
$ dig @68.94.157.1 +short porttest.dns-oarc.net TXT
z.y.x.w.v.u.t.s.r.q.p.o.n.m.l.k.j.i.h.g.f.e.d.c.b.a.pt.dns-oarc.net.
"65.68.49.26 is GOOD: 26 queries in 1.4 seconds from 26 ports with std dev 18277.22"
I'm redirected to a different address. It sure looks like the SBC/AT&T servers vary by region. (You should sell your home and move. Colorado is nice, and the DNS servers here work.
Meanwhile, I'm expecting to do an updated blog entry late today. I plan to include your finding. Thanks!
Appear to have only tested aldelphia unless this is not the full list.
Two of them are:
66.75.160.63 and
66.75.160.64
dig +short porttest.dns-oarc.net TXT @198.6.1.1
porttest.y.x.w.v.u.t.s.r.q.p.o.n.m.l.k.j.i.h.g.f.e.d.c.b.a.pt.dns-oarc.net.
"198.5.241.59 is POOR: 26 queries in 1.8 seconds from 19 ports with std dev 20"
$ dig +short porttest.dns-oarc.net TXT @198.6.1.2
porttest.y.x.w.v.u.t.s.r.q.p.o.n.m.l.k.j.i.h.g.f.e.d.c.b.a.pt.dns-oarc.net.
"137.39.110.185 is POOR: 26 queries in 1.8 seconds from 23 ports with std dev 17"
there a few more, in the bunch as well
Server: cache00.ns.uu.net
Address: 198.6.1.1#53
+----------------------------------------------------------------
Server: cache01.ns.uu.net
Address: 198.6.1.2#53
+----------------------------------------------------------------
Server: cache02.ns.uu.net
Address: 198.6.1.3#53
+----------------------------------------------------------------
Server: cache03.ns.uu.net
Address: 198.6.1.4#53
+----------------------------------------------------------------
Server: cache04.ns.uu.net
Address: 198.6.1.5#53
+----------------------------------------------------------------
Server: cache05.ns.uu.net
Address: 198.6.1.195#53
+----------------------------------------------------------------
Server: cache06.ns.uu.net
Address: 198.6.1.122#53
+----------------------------------------------------------------
Server: cache07.ns.uu.net
Address: 198.6.1.142#53
+----------------------------------------------------------------
Server: cache08.ns.uu.net
Address: 198.6.1.146#53
71.242.0.12 and 71.252.0.12
[Neal's Reply: Verizon uses a different type of DNS server that reports POOR even though it has taken other precautions and is actually protected against this type of attack.]
4.2.2.2 -- Level3 [Update: Reports 'GOOD' on 2008-08-09]
4.2.2.3 -- Level3 [Update: Reports 'GREAT' on 2008-08-09]
4.2.2.4 -- Level3 [Update: Reports 'GREAT' on 2008-08-09]
4.2.2.5 -- Level3 [Update: Reports 'GREAT' on 2008-08-09]
4.2.2.6 -- Level3 [Update: Reports 'GREAT' on 2008-08-09]
[Moderator: Please see the update linked at the top of this blog entry. http://www.hackerfactor.com/blog/index.php?/archives/205-DNS-Responses.html -Loris Kim]