Susam's Networking Pages

Hacker News Hug of Deaf

2025-04-05T00:00:00Z

"It's essentially the Hacker News Hug of Deaf." – @TonyTrapp

About three years ago, I set up a tiny netcat loop on one of my Debian servers to accept arbitrary connections from the Hacker News (HN) community. The loop ran for 24 hours and did exactly three things whenever a client connected:

Send a simple ok message to the client.
Close the connection immediately.
Make my terminal beep four times.

That's it! It was a playful experiment in response to a thread about quirky, do-it-yourself alerting systems for friends and family. See this HN thread for the original discussion. Here is the exact command I ran on my server:

while true; do (echo ok | nc -q 1 -vlp 8000 2>&1; echo; date -u) | tee -a beeper.log; for i in 1 2 3 4; do printf '\a'; sleep 1; done & done

The nc command closes the connection immediately after sending the ok message and runs an inner for loop in a background shell that asynchronously prints the bell character to the terminal four times. Meanwhile, the outer while command loops back quickly to run a new nc process, thus making this one-liner script instantly ready to accept the next incoming connection.

Soon after I shared this, members of the HN community began connecting to the demo running on susam.net:8000. Anyone on the Internet could use any client of their choice to connect. Here's how I explained it in the HN thread:

Now anytime someone connects to port 8000 of my system by any means, I will hear 4 beeps! The other party can use whatever client they have to connect to port 8000 of my system, e.g. a web browser, nc HOST 8000, curl HOST:8000 or even ssh HOST -p 8000, irssi -c HOST -p 8000, etc.

In the next 24 hours, I received over 4761 connections, each one triggering four beeps. That's a total of 19 044 terminal beeps echoing throughout the day!

Number of connections received every hour since 31 Jan 2022 10:00 UTC

The data for the above graph is available at beeper.log. Now, 4761 isn't a huge number in the grand scheme of things, but it was still pretty cool to see people notice an obscure comment buried in a regular HN thread, act on it and make my terminal beep thousands of time.

At the end of the day, this was a fun experiment. Pointless, but fun! Computing isn't always about solving problems. Sometimes, it's also about exploring quirky ideas. The joy is in the exploration and having others join in made it even more enjoyable. Activities like this keep computing fun for me!

Update on 10 Apr 2025: I shared this article on Hacker News today and saw another surge in connections to my beeper loop.

Number of connections received every hour since 10 Apr 2025 10:00 UTC

The data for the above graph is available at beeper2.log. The data shows a total of 352 831 connections from 1396 unique client addresses over 14 hours. That amounts to a total of 1 411 324 beeps! Much of the traffic seems to have come from persistent client loops constantly connecting to my beeper loop. In particular, the client identified by the anonymised identifier C0276 made the largest number of connections by far, with 327 209 total connections. The second most active client, C0595, made only 6771 connections. There were 491 clients that connected exactly once. If you'd like to see the number of connections by each client, see beeperclient2.log.

In conclusion, the difference in the volume of connections between the earlier experiment and today's is striking. In the first round, three years ago, there were only 4761 connections from some readers of a comment thread. But in today's round, with this post being featured on the HN front page, there were 352 831 connections! It is fascinating to see how odd experiments like this can find so many participants within the HN community!

Pretty-Printing JSON Response with HTTP Headers

2024-04-14T00:00:00Z

Often while using curl with URLs that return a JSON response, I need to print the HTTP response headers along with the JSON response. Here is an example that shows how this can be done:

$ curl -sSi https://susam.net/code/lab/json/books.json
HTTP/1.1 200 OK
Server: nginx/1.18.0
Date: Sat, 05 Apr 2024 11:53:24 GMT
Content-Type: application/json
Content-Length: 172
Last-Modified: Sat, 05 Apr 2024 11:53:05 GMT
Connection: keep-alive
ETag: "67f119a1-ac"
Accept-Ranges: bytes

[
  {"title": "Gulliver's Travels", "author": "Jonathan Swift", "published": 1726},
  {"title": "Treasure Island", "author": "Robert Louis Stevenson", "published": 1883}
]

The above output is obtained using curl 7.77.0 (x86_64-apple-darwin21.0). The -i option is responsible for including the HTTP response headers. The -s and -S options are not too important for the current discussion but I usually happen to use them out of habit. The -s option suppresses the progress meter and error messages but the -S re-enables the display of error messages. This helps me avoid the progress meter in the output without having to lose visibility of any errors that may arise.

So far so good! But can we also have the JSON response pretty-printed with say jq? The above command prints both the HTTP headers and the response to the standard output, so piping the standard output to jq does not work. The jq command fails with an error as soon as it encounters the HTTP headers.

If, however, we manage to send the HTTP header and the response to different streams or files, then we could utilise jq to pretty-print the stream or file that contains the JSON response. Here is an example that shows how to do this:

$ curl -sSD head.txt -o out.json https://susam.net/code/lab/json/books.json && cat head.txt && jq . out.json
HTTP/1.1 200 OK
Server: nginx/1.18.0
Date: Sat, 05 Apr 2024 11:53:51 GMT
Content-Type: application/json
Content-Length: 172
Last-Modified: Sat, 05 Apr 2024 11:53:05 GMT
Connection: keep-alive
ETag: "67f119a1-ac"
Accept-Ranges: bytes

[
  {
    "title": "Gulliver's Travels",
    "author": "Jonathan Swift",
    "published": 1726
  },
  {
    "title": "Treasure Island",
    "author": "Robert Louis Stevenson",
    "published": 1883
  }
]

Alternatively, we can achieve this using a single command by printing the the HTTP headers to standard error. This ensures that only the JSON response is printed to standard output, which we can then pretty-print using jq. Here is an example:

$ curl -sSD /dev/stderr https://susam.net/code/lab/json/books.json | jq .
HTTP/1.1 200 OK
Server: nginx/1.18.0
Date: Sat, 05 Apr 2024 11:54:12 GMT
Content-Type: application/json
Content-Length: 172
Last-Modified: Sat, 05 Apr 2024 11:53:05 GMT
Connection: keep-alive
ETag: "67f119a1-ac"
Accept-Ranges: bytes

[
  {
    "title": "Gulliver's Travels",
    "author": "Jonathan Swift",
    "published": 1726
  },
  {
    "title": "Treasure Island",
    "author": "Robert Louis Stevenson",
    "published": 1883
  }
]

Read on website | #unix | #shell | #networking | #technology

Toy Traceroute With Ping

2022-02-01T00:00:00Z

Here is an example of a toy traceroute using ping on Linux:

$ for ttl in {1..30}; do ping -4 -c 1 -t $ttl example.com && break; done | grep -i from | nl -s ' ' -w 2
 1 From router1-lon.linode.com (212.111.33.229) icmp_seq=1 Time to live exceeded
 2 From if-0-1-0-0-0.gw1.lon1.gb.linode.com (109.74.207.10) icmp_seq=1 Time to live exceeded
 3 From be5787.rcr51.lon10.atlas.cogentco.com (204.68.252.58) icmp_seq=1 Time to live exceeded
 4 From be2589.ccr41.lon13.atlas.cogentco.com (154.54.59.37) icmp_seq=1 Time to live exceeded
 5 From be2099.ccr31.bos01.atlas.cogentco.com (154.54.82.34) icmp_seq=1 Time to live exceeded
 6 From verizondms.bos01.atlas.cogentco.com (154.54.11.54) icmp_seq=1 Time to live exceeded
 7 From ae-66.core1.bsa.edgecastcdn.net (152.195.233.129) icmp_seq=1 Time to live exceeded
 8 64 bytes from 93.184.216.34 (93.184.216.34): icmp_seq=1 ttl=57 time=63.0 ms

The output above was obtained on a Debian GNU/Linux 11.2 (bullseye) system. The above loop sends multiple ICMP echo requests with different Time to Live (TTL) values to reach the host for example.com. The TTL occurs as an 8-bit field in the Internet Protocol (IP) header of each packet. It is the 9th octet in an IPv4 header and the 8th octet in an IPv6 header.

When a router on the path to the destination receives a packet, it decrements the TTL value in the IP header by one. If the TTL value becomes 0 after the decrement operation, the router responds with a "time-to-live exceeded" ICMP message. Thus an echo request with TTL value set to 1 gets us an ICMP "time-to-live exceeded" message from the first router in the path, the next echo request with TTL value 2 gets us a similar ICMP message from the second router in the path and so on. The traceroute is complete when we receive a successful ICMP echo reply.

For comparison, here is the output of the actual traceroute command:

$ traceroute example.com
traceroute to example.com (93.184.216.34), 30 hops max, 60 byte packets
 1  router1-lon.linode.com (212.111.33.229)  0.602 ms  1.202 ms  1.326 ms
 2  if-0-1-0-1-0.gw1.lon1.gb.linode.com (109.74.207.14)  0.502 ms if-11-1-0-0-0.gw2.lon1.gb.linode.com (109.74.207.26)  0.401 ms if-11-0-0-0-0.gw2.lon1.gb.linode.com (109.74.207.30)  0.379 ms
 3  be5787.rcr51.lon10.atlas.cogentco.com (204.68.252.58)  0.573 ms  0.563 ms  0.566 ms
 4  be2589.ccr41.lon13.atlas.cogentco.com (154.54.59.37)  1.271 ms  1.311 ms ldn-bb1-link.ip.twelve99.net (62.115.122.188)  1.400 ms
 5  be2099.ccr31.bos01.atlas.cogentco.com (154.54.82.34)  63.511 ms  63.540 ms nyk-bb1-link.ip.twelve99.net (62.115.112.244)  73.397 ms
 6  nyk-b1-link.ip.twelve99.net (62.115.135.131)  70.113 ms verizondms.bos01.atlas.cogentco.com (154.54.11.54)  63.657 ms nyk-b1-link.ip.twelve99.net (62.115.135.131)  70.190 ms
 7  ae-66.core1.bsa.edgecastcdn.net (152.195.233.129)  63.535 ms edgecast-ic317659-nyk-b6.ip.twelve99-cust.net (62.115.147.199)  77.802 ms ae-66.core1.bsa.edgecastcdn.net (152.195.233.129)  63.582 ms
 8  93.184.216.34 (93.184.216.34)  62.895 ms ae-71.core1.nyb.edgecastcdn.net (152.195.69.139)  72.312 ms ae-70.core1.nyb.edgecastcdn.net (152.195.68.141)  69.419 ms
 9  93.184.216.34 (93.184.216.34)  70.827 ms  62.896 ms  73.342 ms

Each line in the output above corresponds to a hop. Each line shows one or more addresses. A maximum of three addresses can be seen in the result for each hop. That is because there are multiple paths to the destination and the traceroute command sends 3 UDP probe packets by default for each hop. In this manner, it ends up discovering multiple routes to the destination. It is worth noting here that by default traceroute sends UDP packets, not ICMP echo requests, with different TTL values as probe packets. But the route discovery mechanism remains the same. After sending each probe packet, it waits for ICMP "time-to-live exceeded messages" from the routers that fall in the path to the destination.

By comparing the two outputs above, we can see that the route found by the toy traceroute using ping is one of the several routes found by the traceroute command.

For those on macOS, the ping command options need to be modified as follows:

for ttl in {1..30}; do ping -c 1 -t 1 -m $ttl example.com && break; done | grep -i from | nl -s ' ' -w 2

On macOS, the -t option of the ping command specifies a timeout (not IP TTL) that prevents it from waiting for too long for a successful echo reply which we don't expect to receive. Further, on macOS, the -m option of the ping command specifies the IP TTL.

Read on website | #networking | #protocol | #shell | #technology

Simplicity of IRC

2022-01-09T00:00:00Z

During discussions with my friends and colleagues, whenever the topic of chat protocols comes up, I often remark how simple the Internet Relay Chat (IRC) protocol is and how this simplicity has fostered creativity in the lives of many young computer hobbyists growing up in the late 1990s and early 2000s. For many of us who were introduced to the Internet during that time, writing an IRC bot turned out to be one of our first few non-trivial hobby programming projects that involved network sockets, did something meaningful and served actual users.

Simplicity

The underlying payloads that IRC servers and clients exchange during an IRC session are quite simple to read manually and understand. While implementing IRC servers still involves significant work to keep track of users and channels as well as exchanging network state and messages between servers, implementing IRC clients can often be quite simple. With a convenient programming language, one can develop all kinds of fun tools and bots pretty quickly. Only creativity is the limit!

In the early days of IRC, it was quite common for someone with basic programming skills to write a simple IRC bot within a matter of hours. Such IRC bots typically responded to requests from users, answered frequently asked questions, hosted trivia quiz, etc. The simplicity of the protocol made it very enticing to write programs that could talk to IRC servers directly. In fact, many people chose to write the code to parse and create IRC payloads from scratch. Observing the TCP/IP packets with a packet analyser such as Wireshark or Tcpdump was all one needed to learn about the various payload formats. Additionally, back then RFC 1459 served as a good reference to learn the IRC specification.

As a result of the simplicity of the IRC protocol, sometimes when I wanted to join an IRC channel, say to seek some technical help, from a system without an IRC client installed, I would often just start a telnet, nc or openssl connection directly to my favourite IRC network and type out IRC protocol commands by hand to join channels and talk to users.

Session

To illustrate how simple the IRC protocol is, here is an example of a minimal IRC session that involves joining a channel and posting a message:

$ nc irc.libera.chat 6667
:strontium.libera.chat NOTICE * :*** Checking Ident
:strontium.libera.chat NOTICE * :*** Looking up your hostname...
:strontium.libera.chat NOTICE * :*** Couldn't look up your hostname
:strontium.libera.chat NOTICE * :*** No Ident response
NICK humpty
USER humpty humpty irc.libera.chat :Humpty Dumpty
:strontium.libera.chat 001 humpty :Welcome to the Libera.Chat Internet Relay Chat Network humpty
:strontium.libera.chat 002 humpty :Your host is strontium.libera.chat[204.225.96.123/6667], running version solanum-1.0-dev
:strontium.libera.chat 003 humpty :This server was created Sat Oct 30 2021 at 17:56:22 UTC
:strontium.libera.chat 004 humpty strontium.libera.chat solanum-1.0-dev DGQRSZaghilopsuwz CFILMPQSTbcefgijklmnopqrstuvz bkloveqjfI
:strontium.libera.chat 005 humpty MONITOR=100 CALLERID=g WHOX FNC ETRACE KNOCK SAFELIST ELIST=CMNTU CHANTYPES=# EXCEPTS INVEX CHANMODES=eIbq,k,flj,CFLMPQSTcgimnprstuz :are supported by this server
:strontium.libera.chat 005 humpty CHANLIMIT=#:250 PREFIX=(ov)@+ MAXLIST=bqeI:100 MODES=4 NETWORK=Libera.Chat STATUSMSG=@+ CASEMAPPING=rfc1459 NICKLEN=16 MAXNICKLEN=16 CHANNELLEN=50 TOPICLEN=390 DEAF=D :are supported by this server
:strontium.libera.chat 005 humpty TARGMAX=NAMES:1,LIST:1,KICK:1,WHOIS:1,PRIVMSG:4,NOTICE:4,ACCEPT:,MONITOR: EXTBAN=$,ajrxz :are supported by this server
:strontium.libera.chat 251 humpty :There are 66 users and 48644 invisible on 25 servers
:strontium.libera.chat 252 humpty 35 :IRC Operators online
:strontium.libera.chat 253 humpty 11 :unknown connection(s)
:strontium.libera.chat 254 humpty 21561 :channels formed
:strontium.libera.chat 255 humpty :I have 3117 clients and 1 servers
:strontium.libera.chat 265 humpty 3117 4559 :Current local users 3117, max 4559
:strontium.libera.chat 266 humpty 48710 50463 :Current global users 48710, max 50463
:strontium.libera.chat 250 humpty :Highest connection count: 4560 (4559 clients) (301752 connections received)
:strontium.libera.chat 375 humpty :- strontium.libera.chat Message of the Day -
:strontium.libera.chat 372 humpty :- Welcome to Libera Chat, the IRC network for
:strontium.libera.chat 372 humpty :- free & open-source software and peer directed projects.
:strontium.libera.chat 372 humpty :-
:strontium.libera.chat 372 humpty :- Use of Libera Chat is governed by our network policies.
:strontium.libera.chat 372 humpty :-
:strontium.libera.chat 372 humpty :- To reduce network abuses we perform open proxy checks
:strontium.libera.chat 372 humpty :- on hosts at connection time.
:strontium.libera.chat 372 humpty :-
:strontium.libera.chat 372 humpty :- Please visit us in #libera for questions and support.
:strontium.libera.chat 372 humpty :-
:strontium.libera.chat 372 humpty :- Website and documentation:  https://libera.chat
:strontium.libera.chat 372 humpty :- Webchat:                    https://web.libera.chat
:strontium.libera.chat 372 humpty :- Network policies:           https://libera.chat/policies
:strontium.libera.chat 372 humpty :- Email:                      support@libera.chat
:strontium.libera.chat 376 humpty :End of /MOTD command.
:humpty MODE humpty :+iw
JOIN #test
:humpty!~humpty@178.79.176.169 JOIN #test
:strontium.libera.chat 353 humpty = #test :humpty susam coolnickname ptl-tab edcragg
:strontium.libera.chat 366 humpty #test :End of /NAMES list.
PRIVMSG #test :Hello, World!
:susam!~susam@user/susam PRIVMSG #test :Hello, Humpty!
PART #test
:humpty!~humpty@178.79.176.169 PART #test
QUIT
:humpty!~humpty@178.79.176.169 QUIT :Client Quit
ERROR :Closing Link: 178.79.176.169 (Client Quit)

In the above session, the user connects to the Libera Chat network with the nickname humpty, joins a channel named #test and posts a message.

Note that the above session is not encrypted. By convention, IRC port 6667 is used for cleartext connections. A separate port, such as port 6697, is available for encrypted connections. Here is an example of an encrypted IRC session established with the OpenSSL command line tool:

$ openssl s_client -quiet -connect irc.libera.chat:6697 2> /dev/null
:strontium.libera.chat NOTICE * :*** Checking Ident
:strontium.libera.chat NOTICE * :*** Looking up your hostname...
:strontium.libera.chat NOTICE * :*** Couldn't look up your hostname
:strontium.libera.chat NOTICE * :*** No Ident response
NICK humpty
USER humpty humpty irc.libera.chat :Humpty Dumpty
:strontium.libera.chat 001 humpty :Welcome to the Libera.Chat Internet Relay Chat Network humpty
...

The ellipsis denotes lines omitted for the sake of brevity. The remainder of the session is quite similar to the first example in this post.

It is worth noting here that although the payload format of IRC protocol is quite simple, as one starts writing IRC clients, one would stumble upon several tiny details about the protocol that needs to be taken care of, e.g. authenticating to the network, responding to PING messages from the server to avoid ping timeouts, splitting messages into shorter messages so that the overall payload does not exceed the message length limit of 512 characters, etc. For a serious IRC client, relying on a suitable library that already solves these problems and implements the IRC specification accurately is of course going to be useful. But for a hobbyist who wants to understand the protocol and write some tools for fun, the textual nature of the IRC protocol and its simplicity offers a fertile ground for experimentation and creativity.

Join

In case you have never used IRC but this post has piqued your interest and you want to try it out, you probably don't want to be typing out IRC payloads by hand. You would want a good IRC client instead. Let me share some convenient ways to connect to the Libera Chat network. Say, you want to join the #python channel on Libera Chat network. Here are some ways to do it:

Join via web interface: web.libera.chat/#python.
Join using Irssi: On macOS, run brew install irssi to install it. On Debian, Ubuntu or a Debian-based Linux system, run sudo apt-get install irssi. Then enter irssi -c irc.libera.chat to connect to Libera Chat. Then within Irssi, type /join #python.

There are numerous other ways to join IRC networks. There are GUI desktop clients, web browser plugins, Emacs plugins, web-based services, bouncers, etc. that let users connect to IRC networks in various ways. On Libera Chat, there are various channels for open source projects (#emacs, #linux, etc.), communities around specific subjects (##math, #physics, etc.), programming languages (#c, #c++, #commonlisp, etc.). Type the /join command followed by a space and the channel name to join a channel and start posting and reading messages there. It is also possible to search for channels by channel names. For example, on Libera Chat, to search for all channels with 'python' in its name, enter the IRC command: /msg alis list python.

Although I have used Libera Chat in the examples above, there are plenty of other IRC networks too such as EFNet, DALNet, OFTC, etc. Libera Chat happens to be one of the very popular and active networks for open source projects and topic based communities. I use it everyday, so I chose it for the examples here. There are many tight-knit communities on Libera Chat. Some of my favourite ones are #commonlisp, #emacs, #python, etc. All of these have very nice and active communities with great attitudes towards beginners.

Sinkholed

2019-12-03T00:00:00Z

Introduction

On 26 Nov 2019 at 14:55 UTC, I logged into my server that hosts my website to perform a simple maintenance activity. Merely three minutes later, at 14:58 UTC, the domain name susam.in used to host this website was transferred to another registrant without any authorisation by me or without any notification sent to me. Since the DNS results for this domain name was cached on my system, I was unaware of this issue at that time. It would take me three days to realise that I had lost control of the domain name I had been using for my website for the last 12 years. This blog post documents when this happened, how this happened and what it took to regain control of this domain name.

Introduction
Domain Name Transfer
Avalanche Botnet
Support Ticket
Tweets and Retweets
Domain Name Return
Conclusion

Domain Name Transfer

On 29 Nov 2019 at 19:00 UTC, when I visited my website hosted at https://susam.in/, I found that a zero-byte file was being served at this URL. My website was missing. In fact, the domain name resolved to an IPv4 address I was unfamiliar with. It did not resolve to the address of my Linode server anymore.

I checked the WHOIS records for this domain name. To my astonishment, I found that I was no longer the registrant of this domain. An entity named The Verden Public Prosecutor's Office was the new registrant of this domain. The WHOIS records showed that the domain name was transferred to this organisation on 26 Nov 2019 at 14:58 UTC, merely three minutes after I had performed my maintenance activity on the same day. Here is a snippet of the WHOIS records that I found:

Domain Name: susam.in
Registry Domain ID: D2514002-IN
Registrar WHOIS Server:
Registrar URL:
Updated Date: 2019-11-26T14:58:00Z
Creation Date: 2007-05-15T07:19:26Z
Registry Expiry Date: 2020-05-15T07:19:26Z
Registrar: NIXI Special Projects
Registrar IANA ID: 700066
Registrar Abuse Contact Email:
Registrar Abuse Contact Phone:
Domain Status: clientTransferProhibited http://www.icann.org/epp#clientTransferProhibited
Domain Status: serverRenewProhibited http://www.icann.org/epp#serverRenewProhibited
Domain Status: serverDeleteProhibited http://www.icann.org/epp#serverDeleteProhibited
Domain Status: serverUpdateProhibited http://www.icann.org/epp#serverUpdateProhibited
Domain Status: serverTransferProhibited http://www.icann.org/epp#serverTransferProhibited
Registry Registrant ID:
Registrant Name:
Registrant Organization: The Verden Public Prosecutor's Office
Registrant Street:
Registrant Street:
Registrant Street:
Registrant City:
Registrant State/Province: Niedersachsen
...
Name Server: sc-c.sinkhole.shadowserver.org
Name Server: sc-d.sinkhole.shadowserver.org
Name Server: sc-a.sinkhole.shadowserver.org
Name Server: sc-b.sinkhole.shadowserver.org
...

The ellipsis denotes some records I have omitted for the sake of brevity. There were three things that stood out in these records:

The registrar was changed from eNom, Inc. to NIXI Special Projects.
The registrant was changed from Susam Pal to The Verden Public Prosecutor's Office.
The name servers were changed from Linode's servers to Shadowserver's sinkholes.

Avalanche Botnet

On searching more about the new registrant on the web, I realised that it was a German criminal justice body that was involved in the takedown of the Avalanche malware-hosting network. It took a four-year concerted effort by INTERPOL, Europol, the Shadowserver Foundation, Eurojust, the Luneberg Police and several other international organisations to finally destroy the Avalanche botnet on 30 Nov 2016. In this list of organisations, one name caught my attention immediately: The Shadowserver Foundation. The WHOIS name server records pointed to Shadowserver's sinkholes.

The fact that the domain name was transferred to another organisation merely three minutes after I had performed a simple maintenance activity got me worried. Was the domain name hijacked? Did my maintenance activity on the server have anything to do with it? What kind of attack one might have pulled off to hijack the domain name? I checked all the logs and there was no evidence that anyone other than me had logged into the server or executed any command or code on it. Further, a domain name transfer usually involves email notification and authorisation. None of that had happened. It increasingly looked like that the three minute interval between the maintenance activity and the domain name transfer was merely a coincidence.

More questions sprang up as I thought about it. The Avalanche botnet was destroyed in 2016. What has that got to do with the domain name being transferred in 2019? Did my server somehow become part of the Avalanche botnet? My server ran a minimal installation of the latest Debian GNU/Linux system. It was always kept up-to-date to minimise the risk of malware infection or security breach. It hosted a static website composed of static HTML files served with Nginx. I found no evidence of unauthorised access of my server while inspecting the logs. I could not find any malware on the system.

The presence of Shadowserver sinkhole name servers in the WHOIS records was a good clue. Sinkholing of a domain name can be done both malicously as well as constructively. In this case, it looked like the Shadowserver Foundation intended to sinkhole the domain name constructively, so that any malware client trying to connect to my server nefariously would end up connecting to a sinkhole address instead. My domain name was sinkholed! The question now was: Why was it sinkholed?

Support Ticket

On 29 Nov 2019 at 19:29 UTC, I submitted a support ticket to Namecheap to report this issue. At 21:05 UTC, I received a response from Namecheap support that they have contacted Enom, their upstream registrar, to discuss the issue. There was no estimate for when a resolution might be available.

At 21:21 UTC, I submitted a domain name transfer complaint to the Internet Corporation for Assigned Names and Numbers (ICANN). I was not expecting any response from ICANN because they do not have any contractual authority on a country code top-level domain (ccTLD).

At 21:23 UTC, I emailed National Internet Exchange of India (NIXI). NIXI is the ccTLD manager for .IN domain and they have authority on it. I found their contact details from the IANA Delegation Record for .IN. Again, I was not expecting a response from NIXI because they do not have any contractual relationship directly with me. They have a contractual relationship with Namecheap, so any communication from them would be received by Namecheap and Namecheap would have to pass that on to me.

At 21:30 UTC, ICANN responded and said that I should contact the ccTLD manager directly. Like I explained in the previous paragraph, I had already done that, so there was nothing more for me to do except wait for Namecheap to provide an update after their investigation. By the way, NIXI never replied to my email.

Tweets and Retweets

On 30 Nov 2019 at 07:30 UTC, I shared this issue on Twitter. I was hoping that someone who had been through a similar experience could offer some advice. In fact, soon after I posted the tweet, a kind person named Max from Germany generously offered to help by writing a letter in German addressed to the new registrant which was a German organisation. The reason for sinkholing my domain name was still unclear. I hoped that with enough number of retweets someone closer to the source of truth could shed some light on why and how this happened.

At 09:54 UTC, Richard Kirkendall, founder and CEO of Namecheap, responded to my tweet and informed that they were contacting NIXI regarding the issue. This seemed like a good step towards resolution. After all, the domain name was no longer under their upstream registrar named Enom. The domain name was now with NIXI as evident from the WHOIS records.

Several other users tweeted about my issue, added more information about what might have happened and retweeted my tweet.

On 1 Dec 2019 at 11:48 UTC, Benedict Addis from the Shadowserver Foundation contacted me by email. He said that they had begun looking into this issue as soon as one of the tweets about this issue had referred to their organisation. He explained in his email that my domain name was sinkholed accidentally as part of their Avalanche operation. Although it is now three years since the initial takedown of the botnet, they still see over 3.5 million unique IP addresses connecting to their sinkholes everyday. Unfortunately, their operation inadvertently flagged my domain name as one of the domain names to be sinkholed because it matched the pattern of command and control (C2) domain names generated by a malware family named Nymaim, one of the malware families hosted on Avalanche. Although, they had validity checks to avoid sinkholing false-positives, my domain name unfortunately slipped through those checks. Benedict mentioned that he had just raised this issue with NIXI and requested them to return the domain name to me as soon as possible.

Domain Name Return

On 2 Dec 2019 at 04:00 UTC, when I looked up the WHOIS records for the domain name, I found that it had been returned to me already. At 08:37 UTC, Namecheap support responded to my support ticket to say that they had been informed that NIXI had returned the domain name to its original state. At 09:55 UTC, Juliya Zinovjeva, Domains Product Manager of Namecheap, commented on Twitter and confirmed the same thing.

Conclusion

Despite the successful resolution, it was still quite unsettling that a domain name could be transferred to another registrant and sinkholed for some perceived violation. I thought there would be more checks in place to confirm that a perceived violation was real before a domain name could be transferred. Losing a domain name I have been using actively for 12 years was an unpleasant experience. Losing a domain name accidentally should have been a lot harder than this. Benedict from the Shadowserver Foundation assured me that my domain name would be excluded from future sinkholing for this particular case. However, the possibility that this could happen again due to another unrelated operation by another organisation in future is disconcerting.

I also wondered if a domain name under a country code top-level domain (ccTLD) like .in is more susceptible to this kind of sinkholing than a domain name under a generic top-level domain (gTLD) like .com. I asked Benedict if it is worth migrating my website from .in to .com. He replied that in his personal opinion, NIXI runs an excellent, clean registry and are very responsive in resolving issues when they arise. He also added that domain generation algorithms (DGAs) of malware are equally or possibly even more problematic for .com domains. He advised against migrating my website.

Thanks to everyone who retweeted my tweet on this issue. Also, thanks to Richard Kirkendall (CEO of Namecheap), Namecheap support team and Benedict Addis from the Shadowserver Foundation for contacting NIXI to resolve this issue.

Update on 06 Jan 2022: Nearly two years after this incident, I eventually moved this website to susam.net. The decision to do so was not influenced by this incident. I wanted this domain name since 2006 but it was unavailable back then. This domain name became available only very recently and I moved my website to this domain as soon as it became available.

Read on website | #networking | #protocol | #technology

File Transfer with SSH, Tee and Base64

2019-11-19T00:00:00Z

Computer servers deployed in a secure environment may allow SSH sessions but forbid SCP, SFTP and execution of remote commands without a login shell. Such restricted access is typically enforced with SSH gateways and firewalls. An SSH gateway provides controlled access to the remote system. A firewall can ensure that only an SSH gateway can connect to the remote system. Thus, users can be forced to connect to the remote system only via the SSH gateway which can now control what is allowed and what isn't.

Even if SCP, SFTP, port forwarding and remote command execution without a login shell are forbidden, as long as we get a login shell on our terminal and we can print data on the terminal, we are already able to transfer data from the remote system to our local system. The data is in the terminal. It is now only a matter of figuring out how to copy that data to a file.

Note: Various readers of this post notice that SCP or SFTP is not allowed and immediately begin suggesting me a solution similar to one of the following ones:

ssh HOST cat file > file
ssh HOST tar cf - file | tar xf -

Note that these solutions and other similar solutions are not going to work because the SSH gateway described in the previous two paragraphs forbids remote command execution without a login shell. It also blocks port forwarding, so any solution involving port forwarding is not going to work either.

Assuming that both the remote and local systems are Unix-like, the following steps show one way to accomplish copying a file from the remote system to our local system:

Connect to the remote system with ssh and pipe the output to tee to write the entire session to a text file on the local system.
```
ssh HOST | tee ssh.txt
```
This type of pipeline works as intended even while connecting to a remote system via a jumphost or an SSH gateway.
In the remote system, create a 10 MB file to serve as an example payload to be transferred.
```
head -c 10485760 /dev/urandom > /tmp/payload
```
You probably already have a meaningful payload that you want to copy, so in that case, you would skip this step.
Compute checksum on the file. This will be used later to verify that the entire file is transferred correctly.
```
sha1sum /tmp/payload
```
Print Base64 representation of the file.
```
base64 /tmp/payload
```
Depending on the Internet bandwidth, this can take a few seconds to a few minutes to complete.
End the SSH session.
```
exit
```
On the local system, extract the Base64 encoded payload and decode it. Assuming the shell prompt on the remote system ends with the dollar sign (i.e. $), the following command does this.
```
sed '1,/$ base64/d;/$ exit/,$d' ssh.txt | base64 --decode > payload
```
Extract the checksum computed on the original file.
```
grep -A 1 sha1sum ssh.txt
```
Compute checksum on the decoded payload.
```
sha1sum payload
```
Ensure that the checksum in this step matches the checksum in the previous step.

The steps above assume the use of the sha1sum command to compute checksum. If this command is unavailable, use sha1, shasum or something else that serves this purpose well. If you are worried about collision attacks, you might want sha256sum, sha256, shasum -a 256, etc. instead.

Read on website | #networking | #protocol | #technology

Zero Point Leet Seconds

2018-09-18T00:00:00Z

While calculating certain round-trip times, here is a number I came across that is surprisingly memorable: It takes light 0.1337 seconds to travel the length of the Earth's equator via vacuum or air. Let me repeat that. It takes light "zero point leet" seconds to travel once around the equator.

We are going to ignore practical considerations regarding how light would actually follow a curved path around the Earth's equator via vacuum or air. This allows us to find the minimum time that any signal must take in order to complete a round trip around the Earth. Despite the lack of practicality, this is an interesting result because it provides us a theoretical limit for the shortest time interval between sending a signal and receiving it after it has made a complete trip around the world.

The equatorial radius of the Earth is about 6378.137 km. The equatorial circumference of the Earth then is about 40075 km. The speed of light in vacuum is 299792.458 km/s by definition. The speed of light in air is about 299705 km/s. Therefore, it takes light about 40075/299792.458 seconds in vacuum and about 40075/299705 seconds in air to travel once around the equator. Both values can be written as 0.1337 seconds accurate up to 4 decimal places.

Read on website | #networking | #technology

AUTH CRAM-MD5

2011-11-07T00:00:00Z

Introduction

Last night, while I was setting up my SMTP server, I decided to dig deeper into CRAM-MD5 authentication mechanism. It is a challenge-response authentication mechanism and involves HMAC-MD5. We don't use SSL/TLS in the SMTP session examples below in order to show the underlying protocol in clear. In practice, any email program should be configured to use SSL/TLS while having a session with an SMTP server. We will first see a few examples of other authentication mechanisms before discussing the CRAM-MD5 mechanism.

AUTH PLAIN

Here is an example of a session that uses the PLAIN authentication mechanism:

$ telnet susam.in 25
Trying 106.187.41.241...
Connected to susam.in.
Escape character is '^]'.
220 tesseract.susam.in ESMTP Exim 4.72 Mon, 07 Nov 2011 20:27:56 +0530
EHLO nifty.localdomain
250-tesseract.susam.in Hello nifty.localdomain [122.167.80.194]
250-SIZE 52428800
250-PIPELINING
250-AUTH PLAIN LOGIN CRAM-MD5
250-STARTTLS
250 HELP
AUTH PLAIN AGFsaWNlAHdvbmRlcmxhbmQ=
235 Authentication succeeded
MAIL FROM:<alice@susam.in>
250 OK
RCPT TO:<example.recipient@gmail.com>
250 Accepted
DATA
354 Enter message, ending with "." on a line by itself
Date: Mon, 07 Nov 2011 20:28:00 +0530
From: Alice <alice@susam.in>
To: Example Recipient <example.recipient@gmail.com>
Subject: Test email

This is a test email.
.
250 OK id=1RNQef-0004e7-7s
QUIT
221 tesseract.susam.in closing connection
Connection closed by foreign host.

The string "AGFsaWNlAHdvbmRlcmxhbmQ=" in the AUTH PLAIN command is the the base64 encoding of the string "\0alice\0wonderland" where "\0" indicates a null character, "alice" is the sender's user name and "wonderland" is the sender's password. If an eavesdropper intercepts this traffic, he or she can easily find the user's password by simply decoding the base64 response sent by the client. Here is an example of decoding the base64 response with Python 2.7:

>>> 'AGFsaWNlAHdvbmRlcmxhbmQ='.decode('base64')
'\x00alice\x00wonderland'

This is also susceptible to replay attacks as the eavesdropper can use the AUTH PLAIN line containing the base64 encoded credentials to log into the server in future. This is why it is very important to secure the connection with SSL/TLS while having a session with the SMTP server.

Here is another example snippet that shows the LOGIN mechanism:

AUTH LOGIN
334 VXNlcm5hbWU6
YWxpY2U=
334 UGFzc3dvcmQ6
d29uZGVybGFuZA==
235 Authentication succeeded

Here are the base64 responses decoded with Python 2.7:

>>> 'VXNlcm5hbWU6'.decode('base64')
'Username:'
>>> 'YWxpY2U='.decode('base64')
'alice'
>>> 'UGFzc3dvcmQ6'.decode('base64')
'Password:'
>>> 'd29uZGVybGFuZA=='.decode('base64')
'wonderland'

If the session isn't encrypted, LOGIN authentication mechanism is susceptible to the same problems that PLAIN authentication mechanism is susceptible to.

AUTH CRAM-MD5

Let us take a look at the CRAM-MD5 authentication mechanism now. When the client selects the CRAM-MD5 authentication mechanism, the server sends a base64 encoded challenge like this:

AUTH CRAM-MD5
334 PDE3ODkzLjEzMjA2NzkxMjNAdGVzc2VyYWN0LnN1c2FtLmluPg==

An HMAC is calculated for this challenge with the password as the key and MD5 as the hash function. A string is formed by concatenating the user name, a space and the hexadecimal representation of the HMAC. The base64 encoding of this string is sent as the response by the client. The following statements in Python 2.7 show how a response can be formed for the above challenge:

>>> 'PDE3ODkzLjEzMjA2NzkxMjNAdGVzc2VyYWN0LnN1c2FtLmluPg=='.decode('base64')
'<17893.1320679123@tesseract.susam.in>'
>>> import hmac, hashlib
>>> hmac.new('wonderland', '<17893.1320679123@tesseract.susam.in>', hashlib.md5).hexdigest()
'64b2a43c1f6ed6806a980914e23e75f0'
>>> 'alice 64b2a43c1f6ed6806a980914e23e75f0'.encode('base64')
'YWxpY2UgNjRiMmE0M2MxZjZlZDY4MDZhOTgwOTE0ZTIzZTc1ZjA=\n'

Of course, this can be written as a small function:

import hmac, hashlib
def cram_md5_response(username, password, base64challenge):
    return (username + ' ' +
            hmac.new(password,
                     base64challenge.decode('base64'),
                     hashlib.md5).hexdigest()).encode('base64')

The following snippet shows the SMTP server accepting the client-response:

AUTH CRAM-MD5
334 PDE3ODkzLjEzMjA2NzkxMjNAdGVzc2VyYWN0LnN1c2FtLmluPg==
YWxpY2UgNjRiMmE0M2MxZjZlZDY4MDZhOTgwOTE0ZTIzZTc1ZjA=
235 Authentication succeeded

When the connection is not secured, CRAM-MD5 authentication mechanism is relatively more secure than the other two mechanisms because the password cannot be retrieved by decoding the base64 encoded response from the client. The password is used as the key to calculate the HMAC but the password itself is not present in the response. It prevents replay attacks too because the server sends an unpredictable challenge for every authentication. The response sent by the client for a certain challenge is invalid for another instance of authentication because the other instance would involve a different unpredictable challenge.

Timing With Curl

2010-07-10T00:00:00Z

Here is a command I use often while measuring why an HTTP request is taking too long:

curl -L -w "time_namelookup: %{time_namelookup}
time_connect: %{time_connect}
time_appconnect: %{time_appconnect}
time_pretransfer: %{time_pretransfer}
time_redirect: %{time_redirect}
time_starttransfer: %{time_starttransfer}
time_total: %{time_total}
" https://example.com/

Here is the same command written as a one-liner, so that I can copy it easily from this page with a triple-click whenever I need it in future:

curl -L -w "time_namelookup: %{time_namelookup}\ntime_connect: %{time_connect}\ntime_appconnect: %{time_appconnect}\ntime_pretransfer: %{time_pretransfer}\ntime_redirect: %{time_redirect}\ntime_starttransfer: %{time_starttransfer}\ntime_total: %{time_total}\n" https://example.com/

Here is how the output of the above command typically looks:

$ curl -L -w "namelookup: %{time_namelookup}\nconnect: %{time_connect}\nappconnect: %{time_appconnect}\npretransfer: %{time_pretransfer}\nstarttransfer: %{time_starttransfer}\ntotal: %{time_total}\n" https://example.com/
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
...
</html>
time_namelookup: 0.001403
time_connect: 0.245464
time_appconnect: 0.757656
time_pretransfer: 0.757823
time_redirect: 0.000000
time_starttransfer: 0.982111
time_total: 0.982326

In the output above, I have omitted most of the HTML output and replaced the omitted part with ellipsis for the sake of brevity.

The list below provides a description of each number in the output above. This information is picked straight from the manual page of curl 7.20.0. Here are the details:

time_namelookup: The time, in seconds, it took from the start until the name resolving was completed.
time_connect: The time, in seconds, it took from the start until the TCP connect to the remote host (or proxy) was completed.
time_appconnect: The time, in seconds, it took from the start until the SSL/SSH/etc connect/handshake to the remote host was completed. (Added in 7.19.0)
time_pretransfer: The time, in seconds, it took from the start until the file transfer was just about to begin. This includes all pre-transfer commands and negotiations that are specific to the particular protocol(s) involved.
time_redirect: The time, in seconds, it took for all redirection steps include name lookup, connect, pretransfer and transfer before the final transaction was started. time_redirect shows the complete execution time for multiple redirections. (Added in 7.12.3)
time_starttransfer: The time, in seconds, it took from the start until the first byte was just about to be transferred. This includes time_pretransfer and also the time the server needed to calculate the result.
time_total: The total time, in seconds, that the full operation lasted. The time will be displayed with millisecond resolution.

An important thing worth noting here is that the difference in the numbers for time_appconnect and time_connect time tells us how much time is spent in SSL/TLS handshake. For a cleartext connection without SSL/TLS, time_appconnect is reported as zero. Here is an example output that demonstrates this:

$ curl -L -w "time_namelookup: %{time_namelookup}\ntime_connect: %{time_connect}\ntime_appconnect: %{time_appconnect}\ntime_pretransfer: %{time_pretransfer}\ntime_redirect: %{time_redirect}\ntime_starttransfer: %{time_starttransfer}\ntime_total: %{time_total}\n" http://example.com/
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
...
</html>
time_namelookup: 0.001507
time_connect: 0.247032
time_appconnect: 0.000000
time_pretransfer: 0.247122
time_redirect: 0.000000
time_starttransfer: 0.512645
time_total: 0.512853

Also note that time_redirect is zero in both outputs above. That is because no redirection occurs while visiting example.com. Here is another example that shows how the output looks when a redirection occurs:

$ curl -L -w "time_namelookup: %{time_namelookup}\ntime_connect: %{time_connect}\ntime_appconnect: %{time_appconnect}\ntime_pretransfer: %{time_pretransfer}\ntime_redirect: %{time_redirect}\ntime_starttransfer: %{time_starttransfer}\ntime_total: %{time_total}\n" https://susam.net/blog
<!DOCTYPE HTML>
<html>
...
</html>
time_namelookup: 0.001886
time_connect: 0.152445
time_appconnect: 0.465326
time_pretransfer: 0.465413
time_redirect: 0.614289
time_starttransfer: 0.763997
time_total: 0.765413

When faced with a potential latency issue in web services, this is often one of the first commands I run several times from multiple clients because the results from this command help to get a quick sense of the layer that might be responsible for the latency issue.

Read on website | #networking | #technology | #how-to

WinPopup

2001-12-10T00:00:00Z

While browsing the C:\Windows directory of the Windows 98 system in our dorm room, I came across an interesting program named Winpopup.exe. It is a tiny little program that can be used to send messages from one Windows system to another on the same local area network (LAN).

Winpopup.exe in C:\Windows of Windows 98

Windows networking supports the notion of workgroups where one or more computers may logically belong to a common group. Computers belonging to the same workgroup can share resources such as files, printers, etc. with each other. To see the workgroup your computer belongs to, go to Start > Settings > Control Panel > Network > Identification and see the value of the field named "Workgroup". By default, this value is "WORKGROUP" but it can be changed to create smaller working groups of computers.

Apart from sending messages to a specific computer, WinPopup supports sending messages to an entire workgroup of computers too. An example of this is shown later in this post.

Workgroup configuration in Windows 98

To start WinPopup, go to My Computer > C: > Windows, then click on the link that says "Show Files", then scroll down to the bottom to find WinPopup.exe and finally double click on it to start it. Alternatively, you can also type win+r, type winpopup and type enter.

WinPopup running on Windows 98

To send a message, simply click on the envelope icon, select one of the radio buttons depending on whether you want to send a message to a specific computer or an entire workgroup, then type the name of the computer or workgroup you want to send your message to and then type the message to be sent.

Composing a message running on WinPopup running on Windows 98

When you are ready to send the message, just click on the OK button. If everything goes fine, a message box confirming that the message was successfully sent should appear.

Message sent successfully with WinPopup running on Windows 98

It is worth noting here that the recipient also needs to have WinPopup running in order to read messages successfully. I found this tool only a few days ago and I already find this tool to be very useful for communicating with other users of Windows systems.

Update on 30 Oct 2022: This article was imported into this website from an old intranet portal I used to run during my university days back in 2001-2005. While importing this article here, I took the liberty of adding a few screenshots taken from a Windows 98 system running in an emulator.

Read on website | #windows | #networking | #technology

Susam's Networking Pages

Hacker News Hug of Deaf

Pretty-Printing JSON Response with HTTP Headers

Toy Traceroute With Ping

Simplicity of IRC

Simplicity

Session

Join

Sinkholed

Introduction

Contents

Domain Name Transfer

Avalanche Botnet

Support Ticket

Tweets and Retweets

Domain Name Return

Conclusion

File Transfer with SSH, Tee and Base64

Zero Point Leet Seconds

AUTH CRAM-MD5

Introduction

AUTH PLAIN

AUTH CRAM-MD5

Further Reading

Timing With Curl

WinPopup