Sinkholed

By Susam Pal on 03 Dec 2019

Introduction

On 26 Nov 2019 at 14:55 UTC, I logged into my server that hosts my website to perform a simple maintenance activity. Merely three minutes later, at 14:58 UTC, the domain name susam.in used to host this website was transferred to another registrant without any authorisation by me or without any notification sent to me. Since the DNS results for this domain name was cached on my system, I was unaware of this issue at that time. It would take me three days to realise that I had lost control of the domain name I had been using for my website for the last 12 years. This blog post documents when this happened, how this happened, and what it took to regain control of this domain name.

Contents

Domain Name Transfer

On 29 Nov 2019 at 19:00 UTC, when I visited my website hosted at https://susam.in/, I found that a zero-byte file was being served at this URL. My website was missing. In fact, the domain name resolved to an IPv4 address I was unfamiliar with. It did not resolve to the address of my Linode server anymore.

I checked the WHOIS records for this domain name. To my astonishment, I found that I was no longer the registrant of this domain. An entity named The Verden Public Prosecutor's Office was the new registrant of this domain. The WHOIS records showed that the domain name was transferred to this organisation on 26 Nov 2019 at 14:58 UTC, merely three minutes after I had performed my maintenance activity on the same day. Here is a snippet of the WHOIS records that I found: 

Domain Name: susam.in
Registry Domain ID: D2514002-IN
Registrar WHOIS Server:
Registrar URL:
Updated Date: 2019-11-26T14:58:00Z
Creation Date: 2007-05-15T07:19:26Z
Registry Expiry Date: 2020-05-15T07:19:26Z
Registrar: NIXI Special Projects
Registrar IANA ID: 700066
Registrar Abuse Contact Email:
Registrar Abuse Contact Phone:
Domain Status: clientTransferProhibited http://www.icann.org/epp#clientTransferProhibited
Domain Status: serverRenewProhibited http://www.icann.org/epp#serverRenewProhibited
Domain Status: serverDeleteProhibited http://www.icann.org/epp#serverDeleteProhibited
Domain Status: serverUpdateProhibited http://www.icann.org/epp#serverUpdateProhibited
Domain Status: serverTransferProhibited http://www.icann.org/epp#serverTransferProhibited
Registry Registrant ID:
Registrant Name:
Registrant Organization: The Verden Public Prosecutor's Office
Registrant Street:
Registrant Street:
Registrant Street:
Registrant City:
Registrant State/Province: Niedersachsen
...
Name Server: sc-c.sinkhole.shadowserver.org
Name Server: sc-d.sinkhole.shadowserver.org
Name Server: sc-a.sinkhole.shadowserver.org
Name Server: sc-b.sinkhole.shadowserver.org
...

The ellipsis denotes some records I have omitted for the sake of brevity. There were three things that stood out in these records:

  1. The registrar was changed from eNom, Inc. to NIXI Special Projects.
  2. The registrant was changed from Susam Pal to The Verden Public Prosecutor's Office.
  3. The name servers were changed from Linode's servers to Shadowserver's sinkholes.

Avalanche Botnet

On searching more about the new registrant on the web, I realised that it was a German criminal justice body that was involved in the takedown of the Avalanche malware-hosting network. It took a four-year concerted effort by INTERPOL, Europol, the Shadowserver Foundation, Eurojust, the Luneberg Police, and several other international organisations to finally destroy the Avalanche botnet on 30 Nov 2016. In this list of organisations, one name caught my attention immediately: The Shadowserver Foundation. The WHOIS name server records pointed to Shadowserver's sinkholes.

The fact that the domain name was transferred to another organisation merely three minutes after I had performed a simple maintenance activity got me worried. Was the domain name hijacked? Did my maintenance activity on the server have anything to do with it? What kind of attack one might have pulled off to hijack the domain name? I checked all the logs and there was no evidence that anyone other than me had logged into the server or executed any command or code on it. Further, a domain name transfer usually involves email notification and authorisation. None of that had happened. It increasingly looked like that the three minute interval between the maintenance activity and the domain name transfer was merely a coincidence.

More questions sprang up as I thought about it. The Avalanche botnet was destroyed in 2016. What has that got to do with the domain name being transferred in 2019? Did my server somehow become part of the Avalanche botnet? My server ran a minimal installation of the latest Debian GNU/Linux system. It was always kept up-to-date to minimise the risk of malware infection or security breach. It hosted a static website composed of static HTML files served with Nginx. I found no evidence of unauthorised access of my server while inspecting the logs. I could not find any malware on the system.

The presence of Shadowserver sinkhole name servers in the WHOIS records was a good clue. Sinkholing of a domain name can be done both malicously as well as constructively. In this case, it looked like the Shadowserver Foundation intended to sinkhole the domain name constructively, so that any malware client trying to connect to my server nefariously would end up connecting to a sinkhole address instead. My domain name was sinkholed! The question now was: Why was it sinkholed?

Support Ticket

On 29 Nov 2019 at 19:29 UTC, I submitted a support ticket to Namecheap to report this issue. At 21:05 UTC, I received a response from Namecheap support that they have contacted Enom, their upstream registrar, to discuss the issue. There was no estimate for when a resolution might be available.

At 21:21 UTC, I submitted a domain name transfer complaint to the Internet Corporation for Assigned Names and Numbers (ICANN). I was not expecting any response from ICANN because they do not have any contractual authority on a country code top-level domain (ccTLD).

At 21:23 UTC, I emailed National Internet Exchange of India (NIXI). NIXI is the ccTLD manager for .IN domain and they have authority on it. I found their contact details from the IANA Delegation Record for .IN. Again, I was not expecting a response from NIXI because they do not have any contractual relationship directly with me. They have a contractual relationship with Namecheap, so any communication from them would be received by Namecheap and Namecheap would have to pass that on to me.

At 21:30 UTC, ICANN responded and said that I should contact the ccTLD manager directly. Like I explained in the previous paragraph, I had already done that, so there was nothing more for me to do except wait for Namecheap to provide an update after their investigation. By the way, NIXI never replied to my email.

Tweets and Retweets

On 30 Nov 2019 at 07:30 UTC, I shared this issue on Twitter. I was hoping that someone who had been through a similar experience could offer some advice. In fact, soon after I posted the tweet, a kind person named Max from Germany generously offered to help by writing a letter in German addressed to the new registrant which was a German organisation. The reason for sinkholing my domain name was still unclear. I hoped that with enough number of retweets someone closer to the source of truth could shed some light on why and how this happened.

At 09:54 UTC, Richard Kirkendall, founder and CEO of Namecheap, responded to my tweet and informed that they were contacting NIXI regarding the issue. This seemed like a good step towards resolution. After all, the domain name was no longer under their upstream registrar named Enom. The domain name was now with NIXI as evident from the WHOIS records.

Several other users tweeted about my issue, added more information about what might have happened, and retweeted my tweet.

On 1 Dec 2019 at 11:48 UTC, Benedict Addis from the Shadowserver Foundation contacted me by email. He said that they had begun looking into this issue as soon as one of the tweets about this issue had referred to their organisation. He explained in his email that my domain name was sinkholed accidentally as part of their Avalanche operation. Although it is now three years since the initial takedown of the botnet, they still see over 3.5 million unique IP addresses connecting to their sinkholes everyday. Unfortunately, their operation inadvertently flagged my domain name as one of the domain names to be sinkholed because it matched the pattern of command and control (C2) domain names generated by a malware family named Nymaim, one of the malware families hosted on Avalanche. Although, they had validity checks to avoid sinkholing false-positives, my domain name unfortunately slipped through those checks. Benedict mentioned that he had just raised this issue with NIXI and requested them to return the domain name to me as soon as possible.

Domain Name Return

On 2 Dec 2019 at 04:00 UTC, when I looked up the WHOIS records for the domain name, I found that it had been returned to me already. At 08:37 UTC, Namecheap support responded to my support ticket to say that they had been informed that NIXI had returned the domain name to its original state. At 09:55 UTC, Juliya Zinovjeva, Domains Product Manager of Namecheap, commented on Twitter and confirmed the same thing.

Conclusion

Despite the successful resolution, it was still quite unsettling that a domain name could be transferred to another registrant and sinkholed for some perceived violation. I thought there would be more checks in place to confirm that a perceived violation was real before a domain name could be transferred. Losing a domain name I have been using actively for 12 years was an unpleasant experience. Losing a domain name accidentally should have been a lot harder than this. Benedict from the Shadowserver Foundation assured me that my domain name would be excluded from future sinkholing for this particular case. However, the possibility that this could happen again due to another unrelated operation by another organisation in future is disconcerting.

I also wondered if a domain name under a country code top-level domain (ccTLD) like .in is more susceptible to this kind of sinkholing than a domain name under a generic top-level domain (gTLD) like .com. I asked Benedict if it is worth migrating my website from .in to .com. He replied that in his personal opinion, NIXI runs an excellent, clean registry, and are very responsive in resolving issues when they arise. He also added that domain generation algorithms (DGAs) of malware are equally, and possibly more, problematic for .com domains. He advised against migrating my website.

Thanks to everyone who retweeted my tweet on this issue. Also, thanks to Richard Kirkendall (CEO of Namecheap), Namecheap support team, and Benedict Addis from the Shadowserver Foundation for contacting NIXI to resolve this issue.

Update on 06 Jan 2022: Nearly two years after this incident, I eventually moved this website to susam.net. The decision to do so was not influenced by this incident. I wanted this domain name since 2006 but it was unavailable back then. This domain name became available only very recently and I moved my website to this domain as soon as it became available.

Comments | #networking | #protocol | #technology