Susam's Unix Pages

Hacker News Hug of Deaf

2025-04-05T00:00:00Z

"It's essentially the Hacker News Hug of Deaf." – @TonyTrapp

About three years ago, I set up a tiny netcat loop on one of my Debian servers to accept arbitrary connections from the Hacker News (HN) community. The loop ran for 24 hours and did exactly three things whenever a client connected:

Send a simple ok message to the client.
Close the connection immediately.
Make my terminal beep four times.

That's it! It was a playful experiment in response to a thread about quirky, do-it-yourself alerting systems for friends and family. See this HN thread for the original discussion. Here is the exact command I ran on my server:

while true; do (echo ok | nc -q 1 -vlp 8000 2>&1; echo; date -u) | tee -a beeper.log; for i in 1 2 3 4; do printf '\a'; sleep 1; done & done

The nc command closes the connection immediately after sending the ok message and runs an inner for loop in a background shell that asynchronously prints the bell character to the terminal four times. Meanwhile, the outer while command loops back quickly to run a new nc process, thus making this one-liner script instantly ready to accept the next incoming connection.

Soon after I shared this, members of the HN community began connecting to the demo running on susam.net:8000. Anyone on the Internet could use any client of their choice to connect. Here's how I explained it in the HN thread:

Now anytime someone connects to port 8000 of my system by any means, I will hear 4 beeps! The other party can use whatever client they have to connect to port 8000 of my system, e.g. a web browser, nc HOST 8000, curl HOST:8000 or even ssh HOST -p 8000, irssi -c HOST -p 8000, etc.

In the next 24 hours, I received over 4761 connections, each one triggering four beeps. That's a total of 19 044 terminal beeps echoing throughout the day!

Number of connections received every hour since 31 Jan 2022 10:00 UTC

The data for the above graph is available at beeper.log. Now, 4761 isn't a huge number in the grand scheme of things, but it was still pretty cool to see people notice an obscure comment buried in a regular HN thread, act on it and make my terminal beep thousands of time.

At the end of the day, this was a fun experiment. Pointless, but fun! Computing isn't always about solving problems. Sometimes, it's also about exploring quirky ideas. The joy is in the exploration and having others join in made it even more enjoyable. Activities like this keep computing fun for me!

Update on 10 Apr 2025: I shared this article on Hacker News today and saw another surge in connections to my beeper loop.

Number of connections received every hour since 10 Apr 2025 10:00 UTC

The data for the above graph is available at beeper2.log. The data shows a total of 352 831 connections from 1396 unique client addresses over 14 hours. That amounts to a total of 1 411 324 beeps! Much of the traffic seems to have come from persistent client loops constantly connecting to my beeper loop. In particular, the client identified by the anonymised identifier C0276 made the largest number of connections by far, with 327 209 total connections. The second most active client, C0595, made only 6771 connections. There were 491 clients that connected exactly once. If you'd like to see the number of connections by each client, see beeperclient2.log.

In conclusion, the difference in the volume of connections between the earlier experiment and today's is striking. In the first round, three years ago, there were only 4761 connections from some readers of a comment thread. But in today's round, with this post being featured on the HN front page, there were 352 831 connections! It is fascinating to see how odd experiments like this can find so many participants within the HN community!

Pretty-Printing JSON Response with HTTP Headers

2024-04-14T00:00:00Z

Often while using curl with URLs that return a JSON response, I need to print the HTTP response headers along with the JSON response. Here is an example that shows how this can be done:

$ curl -sSi https://susam.net/code/lab/json/books.json
HTTP/1.1 200 OK
Server: nginx/1.18.0
Date: Sat, 05 Apr 2024 11:53:24 GMT
Content-Type: application/json
Content-Length: 172
Last-Modified: Sat, 05 Apr 2024 11:53:05 GMT
Connection: keep-alive
ETag: "67f119a1-ac"
Accept-Ranges: bytes

[
  {"title": "Gulliver's Travels", "author": "Jonathan Swift", "published": 1726},
  {"title": "Treasure Island", "author": "Robert Louis Stevenson", "published": 1883}
]

The above output is obtained using curl 7.77.0 (x86_64-apple-darwin21.0). The -i option is responsible for including the HTTP response headers. The -s and -S options are not too important for the current discussion but I usually happen to use them out of habit. The -s option suppresses the progress meter and error messages but the -S re-enables the display of error messages. This helps me avoid the progress meter in the output without having to lose visibility of any errors that may arise.

So far so good! But can we also have the JSON response pretty-printed with say jq? The above command prints both the HTTP headers and the response to the standard output, so piping the standard output to jq does not work. The jq command fails with an error as soon as it encounters the HTTP headers.

If, however, we manage to send the HTTP header and the response to different streams or files, then we could utilise jq to pretty-print the stream or file that contains the JSON response. Here is an example that shows how to do this:

$ curl -sSD head.txt -o out.json https://susam.net/code/lab/json/books.json && cat head.txt && jq . out.json
HTTP/1.1 200 OK
Server: nginx/1.18.0
Date: Sat, 05 Apr 2024 11:53:51 GMT
Content-Type: application/json
Content-Length: 172
Last-Modified: Sat, 05 Apr 2024 11:53:05 GMT
Connection: keep-alive
ETag: "67f119a1-ac"
Accept-Ranges: bytes

[
  {
    "title": "Gulliver's Travels",
    "author": "Jonathan Swift",
    "published": 1726
  },
  {
    "title": "Treasure Island",
    "author": "Robert Louis Stevenson",
    "published": 1883
  }
]

Alternatively, we can achieve this using a single command by printing the the HTTP headers to standard error. This ensures that only the JSON response is printed to standard output, which we can then pretty-print using jq. Here is an example:

$ curl -sSD /dev/stderr https://susam.net/code/lab/json/books.json | jq .
HTTP/1.1 200 OK
Server: nginx/1.18.0
Date: Sat, 05 Apr 2024 11:54:12 GMT
Content-Type: application/json
Content-Length: 172
Last-Modified: Sat, 05 Apr 2024 11:53:05 GMT
Connection: keep-alive
ETag: "67f119a1-ac"
Accept-Ranges: bytes

[
  {
    "title": "Gulliver's Travels",
    "author": "Jonathan Swift",
    "published": 1726
  },
  {
    "title": "Treasure Island",
    "author": "Robert Louis Stevenson",
    "published": 1883
  }
]

Read on website | #unix | #shell | #networking | #technology

Control, Escape and Meta Tricks

2023-06-16T00:00:00Z

Terminal Tricks

Open a Unix or Linux terminal emulator. If you have Terminal.app on macOS, ensure that the "Use Option as Meta Key" option is enabled in its "Preferences" section. Now type foo bar baz followed by meta+b (i.e. alt+b or option+b on modern keyboards). In a typical desktop environment with a typical and modern terminal emulator running a modern shell like Bash, Zsh, etc., the cursor should move backward by one word. Now type esc b. The cursor should move back again by one word. Finally type ctrl+[ b and the same thing should happen again. How are we able to perform the same operation in three different ways?

Note that if the desktop environment or the terminal emulator or the set of shell key bindings is configured differently, the results may vary. But we will assume that the typical defaults are in effect in the remainder of this post. To understand why these three key sequences yield the same result, it might be a good exercise to run the command cat and type the three key sequences again. The three key sequences we are talking about are:

meta+b (i.e. alt+b or option+b on modern keyboards)
esc b
ctrl+[ b

When we run cat and type the three key sequences mentioned above, the following output may appear:

$ cat
^[b^[b^[b

The output shows that the terminal sends the same input to cat each time: the escape character that appears as ^[ in the output and the character b. This becomes more apparent if instead of running cat, we run od -t d1 and type the three key sequences followed by ctrl+d enter:

$ od -t d1
^[b^[b^[b
0000000    27  98  27  98  27  98  10
0000007

Indeed decimal 27 is the code of the escape character. Similarly decimal 98 is the code of the character b.

Control Codes

Let us first discuss why typing ctrl+[ produces the escape character. The character [ has code 91 (binary 1011011) and holding the ctrl key while typing it results in a control code obtained by taking 91 (binary 1011011), keeping its five least significant bits and discarding the rest. We get the control code 27 (binary 11011) as the result. This is the code of the escape character. This explains why ctrl+[ produces the escape character and why the escape character is represented as ^[ while typing it into the standard input. The caret sign (^) here is a notation for the ctrl modifier.

The following table provides some more examples of control codes that can be obtained by typing the ctrl key along with some other key.

Key	Modified Character			Control Character
Key	Binary	Decimal	Character	Binary	Decimal	Character
`ctrl`+`@`	1000000	64	@	00000	0	Null
`ctrl`+`g`	1000111	71	G	00111	7	Bell
`ctrl`+`h`	1001000	72	H	01000	8	Backspace
`ctrl`+`i`	1001001	73	I	01001	9	Horizontal Tab
`ctrl`+`j`	1001010	74	I	01010	10	Line Feed
`ctrl`+`m`	1001101	77	M	01101	13	Carriage Return
`ctrl`+`[`	1011011	91	[	11011	27	Escape

This explains why typing ctrl+g in a modern terminal emulator produces an audible beep or a visual flash, why ctrl+h erases a character and so on. This also explains why we can type ctrl+[ in Vim to escape from insert mode to normal mode. While we can type escape with the esc key on the keyboard, we can do so with the ctrl+[ key too within the terminal.

The keen eyed may notice that the table above has lowercase letters in the first column but the second and third columns use the code of the corresponding uppercase letters. The lowercase letters in the first column is merely a notation I am using in this post to identify the keys on a keyboard. They don't actually mean lowercase characters. In fact, the very early keyboards only had uppercase letters and they simply toggled the 7th least significant bit of the modified character to obtain the control code.

By the way, an interesting thing worth noting here is that even if we do consider the code of the lowercase character and pick only its five least significant bits, we get the same control code as we would get if we started with the corresponding uppercase character. For example, consider the key sequence ctrl+g. The uppercase character G has the code 71 (decimal 1000111) and the lowercase character g has the code 103 (decimal 1100111). The five least significant bits are equal in both. So when we pick the five least significant bits and discard the rest, we get the same result, i.e. 7 (binary 111) which is the code of the bell character. This is due to the fact that the five least significant bits of the code of a lowercase character is exactly the same as that of the corresponding uppercase character. They differ only in their sixth least significant bit. This is only an interesting observation. It is of little significance though because like I mentioned earlier, the early keyboards only flipped a bit in the code of the uppercase characters when the ctrl modifier was applied.

In addition to what we have discussed so far, some terminal emulators also implement a few special rules such as the following:

Key	Modified Character	Resulting Character
`ctrl`+`space`	0100000	32	Space	0	0	Null
`ctrl`+`?`	0111111	63	?	1111111	127	Delete

Key

Modified Character

Resulting Character

Binary

Decimal

Character

Binary

Decimal

Character

One could argue that the first row shows a straightforward example of picking the least significiant five bits to arrive at the control code, so it is not really a special case. That is a fair point. However, in the history of computing, different systems have implemented slightly different methods to compute the resulting control codes from our input. Flipping the 7th least significant bit was one of the early methods. Turning off only the the 6th and the 7th least significant bits has been another method. Subtracting 64 from the character code has been yet another one. These different methods produce identical results for the first table but not so for the second table. For example, while turning off the 6th and 7th least significnat bits of 32 (the code of the space character) does give us 0 but merely flipping its 7th bit does not. Further, note that allowing ctrl+space to produce the null character is a bit redundant because ctrl+@ already did that right from the days of very early keyboards. The second entry above is also a special rule because we neither turn off bits nor subtract 64. Instead, we flip the 7th least significant bit which amounts to adding 64 to the code of the modified character. It is also the only control code that has its 6th and 7th least significant bits turned on.

Meta Key Sequences

The meta key no longer exists on modern keyboards. On modern keyboards, we use the alt or option key instead of meta. For example, when a shell's manual says that we need to type meta+b to move the cursor back by one word, what we really type on a modern keyboard is either alt+b or option+b. In fact, the cat and od experiments mentioned earlier show that when we type the modern alternative for the meta key along with another key, what most terminals really send to the underlying program is an escape control code (27) followed by the code of the modified character.

Meta Key Sequence	Escape Key Sequence	Control Key Sequence
`meta`+`b`	`esc` `b`	`ctrl`+`[` `b`
`meta`+`f`	`esc` `f`	`ctrl`+`[` `f`
`meta`+`/`	`esc` `/`	`ctrl`+`[` `/`
`meta`+`:`	`esc` `:`	`ctrl`+`[` `:`

Meta Key Sequence

Escape Key Sequence

Control Key Sequence

meta+b

esc b

ctrl+[ b

meta+f

esc f

ctrl+[ f

meta+/

esc /

ctrl+[ /

meta+:

esc :

ctrl+[ :

There are several layers of software involved between the keyboard input and the application running in the terminal and the exact behaviour of the alt or option key may vary depending on the configuration of each of these layers. Terminal configuration alone is a complex topic that can be discussed extensively. However, the behaviour described here is one of the popular defaults. Alternative behaviours exist but they generally produce similar effects for the user.

Awkward Vim Tricks

Most Vim users know that we can go from insert mode to command-line mode and search for patterns by typing either esc / or C-[ /. But what some people may find surprising is that we can also go from insert mode to searching patterns simply by typing meta+/. Yes, this can be verified by running Vim in a terminal emulator. While insert mode is active, type alt+/ or option+/ and the current mode should instantly switch to the command-line mode with the forward-slash (/) prompt waiting for our search pattern. This works only in a terminal emulator. It may not work in the graphical version of Vim. The table above illustrates why this works in a terminal emulator.

Similarly, in a Vim instance running within a terminal emulator, we can type meta+: to go directly from insert mode to command-line mode and enter Ex commands. We can type meta+0 to go directly from insert mode to the first character of a line or type meta+$ to go to the end of the line and so on. These are equivalent to typing esc 0, esc $, etc.

More interestingly, we can type meta+O to open a line above, meta+A to append text at the end of the line, meta+I to append text at the beginning of the line or meta+S to delete the current line while staying in insert mode! Since the Vim commands O, A, I and S leave us back in insert mode, we are able to perform an editing operation that involves leaving the insert mode, doing something interesting and returning to insert mode instantly using the the meta key combination. The following table summarises these observations:

Initial Mode	Meta Key Sequence	Equivalent To	Operation	Final State
Insert	`meta`+`/`	`esc` `/`	Enter command-line mode to search a pattern	Command-line
Insert	`meta`+`:`	`esc` `:`	Enter command-line mode to enter Ex command	Command-line
Insert	`meta`+`0`	`esc` `0`	Move to the first character of the line	Normal
Insert	`meta`+`$`	`esc` `$`	Move to the end of the line	Normal
Insert	`meta`+`O`	`esc` `O`	Begin a new line above and insert text	Insert
Insert	`meta`+`A`	`esc` `A`	Append text to the end of line	Insert
Insert	`meta`+`I`	`esc` `I`	Insert text before the first non-blank in line	Insert
Insert	`meta`+`S`	`esc` `S`	Delete line and insert text	Insert

Initial Mode

Meta Key Sequence

Equivalent To

Operation

Final State

Insert

meta+/

esc /

Enter command-line mode to search a pattern

Command-line

Insert

meta+:

esc :

Enter command-line mode to enter Ex command

Command-line

Insert

meta+0

esc 0

Move to the first character of the line

Normal

Insert

meta+$

esc $

Move to the end of the line

Normal

Insert

meta+O

esc O

Begin a new line above and insert text

Insert

meta+A

esc A

Append text to the end of line

Insert

meta+I

esc I

Insert text before the first non-blank in line

Insert

meta+S

esc S

Delete line and insert text

Insert

There is no good reason to use Vim like this but it works, thanks to the quirky history of Unix terminals and keyboards!

Read on website | #unix | #shell | #technology

From XON/XOFF to Forward Incremental Search

2022-08-13T00:00:00Z

XON/XOFF

In the olden days of computing, software flow control with control codes XON and XOFF was a necessary feature that dumb terminals needed to support. When a terminal received more data than it could display, there needed to be a way for the terminal to tell the remote host to pause sending more data. The control code 19 was chosen for this. The control code 17 was chosen to tell the remote host to resume transmission of data.

The control code 19 is called Device Control 3 (DC3) in the ASCII chart. It is also known as "transmit off" (XOFF). The control code 17 is called Device Control 1 (DC1) as well as "transmit on" (XON). Now how does a user of the terminal really send these control codes? Well, how do they send any control code? Using the ctrl key of course.

Let us take a step back and see how a user can send familiar control codes on modern terminal emulators, like say, the terminal software we find on a Unix or Linux desktop environment. While any sane computer user would just type the tab key to insert a tab character, one could also type ctrl+i to insert a tab character. The character I has code 73 (binary 1001001) and holding the ctrl key while typing it results in a control code made by taking 73 (binary 1001001), keeping its five least significant bits and discarding the rest to get the control code 9 (binary 1001) which is the code of the tab character. In other words, we get the control code by performing a bitwise AND operation on the code of the character being modified with binary code of 31 (binary 0011111).

In case you are wondering, if we get the same result if we choose the binary code of the lowercase i to perform the aforementioned operation, the answer is, yes. While the code of uppercase I is 73 (binary 1001001), that of lowercase i is 105 (binary 1101001). The right-most five bits are same for both. Thus when we preserve the five least significant bits and discard the rest, we get the control code 9 (binary 1001) in both cases. This is a neat result due to the fact that the five least significant bits of the code of a lowercase character is exactly the same as that of the corresponding uppercase character. They differ only in their sixth least significant bit. That bit is on for the lowercase character but off for the corresponding uppercase character. This is just an interesting observation as far as this post is concerned. The very early keyboards did not have lowercase letters. They only had uppercase letters and when the ctrl key is pressed together with a letter on such keyboards, the 7th bit of the character code was flipped to get the control code.

The bitwise operation mentioned earlier explains why typing ctrl+h sends a backspace, typing ctrl+j sends a newline and typing ctrl+g plays a bell. In modern terminal emulators these days, the bell often manifests in the form of an audible beep or a visual flash. Here is a table that summarises these control codes and a few more:

Key	Modified Character			Control Character
Key	Binary	Decimal	Character	Binary	Decimal	Character
`ctrl`+`@`	1000000	64	@	00000	0	Null
`ctrl`+`g`	1000111	71	G	00111	7	Bell
`ctrl`+`h`	1001000	72	H	01000	8	Backspace
`ctrl`+`i`	1001001	73	I	01001	9	Horizontal Tab
`ctrl`+`j`	1001010	74	I	01010	10	Line Feed
`ctrl`+`m`	1001101	77	M	01101	13	Carriage Return
`ctrl`+`[`	1011011	91	[	11011	27	Escape

The last row in the table above explains why we can also type ctrl+[ in Vim to escape from insert mode to normal mode. This is, in fact, one of the convenient ways for touch-typists to return to normal mode in Vim instead of clumsily stretching the left hand fingers out to reach the esc key which is usually poorly located at the corner of most keyboards.

There is a bit of oversimplication in the description above. Throughout the history of computing, different systems have used slightly different methods to compute the resulting control code when the ctrl modifier key is held. Toggling the 7th least significant bit was an early method. Turning off both the 6th and 7th least significant bits is another method. Subtracting 64 from the character code is yet another method. These are implementation details and these various implementation methods lead to the same results for the examples in the table above. Then there are some special rules too. For example, many terminals implement a special rule to make ctrl+space behave the same as ctrl+@ thus producing the null character. Further ctrl+? produces the delete character in some terminals. These special rules are summarised in the table below.

Key	Modified Character	Resulting Character
`ctrl`+`space`	0100000	32	Space	0	0	Null
`ctrl`+`?`	0111111	63	?	1111111	127	Delete

Key

Modified Character

Resulting Character

Binary

Decimal

Character

Binary

Decimal

Character

ctrl+space

0100000

Space

Null

ctrl+?

0111111

1111111

127

Delete

For the purpose of this blog post, we don't need to worry about these special rules. Let us get back to the control codes 19 and 17 that represent XOFF and XON. Here is how the table looks for them:

Key	Modified Character	Resulting Character
`ctrl`+`q`	1010001	81	Q	10001	17	DC1 (XON)
`ctrl`+`s`	1010011	83	S	10011	19	DC3 (XOFF)

Key

Modified Character

Resulting Character

Binary

Decimal

Character

Binary

Decimal

Character

ctrl+q

1010001

10001

DC1 (XON)

ctrl+s

1010011

10011

DC3 (XOFF)

We see that a user can type ctrl+s to send the control code XOFF and then ctrl+q to send the control code XON. Although we do not use dumb terminals anymore, these conventions have survived in various forms in our modern terminal emulators. To see a glimpse of it, launch the terminal that comes with a modern operating system, then run ping localhost and while this command is printing its output, type ctrl+s. The terminal should pause printing the output of the command. Then type ctrl+q and the terminal should resume printing the output.

Incremental Search in Bash/Zsh

Let us now take a look at the shells that run within the terminals. Both Bash and Zsh are very popular these days. Both these shells have excellent support for performing incremental searches through the input history. To see a quick demonstration of this, open a new terminal that runs either Bash or Zsh and run these commands:

echo foo
echo bar
cal
uname
echo baz

Now type ctrl+r followed by echo. This should invoke the reverse search feature of the shell and automatically complete the partial command echo to echo baz. Type ctrl+r again to move back in the input history and autocomplete the command to echo bar. We can type enter anytime we use the reverse search feature to execute the automatically completed command. Typing ctrl+r yet another time should bring the echo foo command at the shell prompt.

What if we went too far back in the input history and now want to go forward? There is a good news and there is some bad news. The good news is that both Bash and Zsh support forward search using the ctrl+s key sequence. The bad news is that this key sequence may not reach the shell. Most terminal emulators consume this key sequence and interpret it as the control code XOFF.

The Conflict

In the previous two sections, we have seen that the ctrl+s key sequence is used to send the control code XOFF. But the same key sequence is also used for forward incremental search in Bash and Zsh. Since the terminal consumes this key sequence and interprets it as the control code XOFF, the shell never sees the key sequence. As a result, the forward incremental search functionality does not work when we type this key sequence in the shell.

Bash offers the incremental search facility via a wonderful piece of library known as the the GNU Readline Library. ZSH offers this facility with its own Zsh Line Editor (ZLE). So other tools that rely on these libraries to offer line editing and history capability are also affected by this conflict. For example, many builds of Python offer line editing and history capability using GNU Readline, so while ctrl+r works fine to perform reverse search in the Python interpreter, it is very likely that ctrl+s does not work to perform forward search.

Reclaim Forward Incremental Search

We can forfeit the usage of control codes XON/XOFF to reclaim forward incremental search. Here is the command to disable XON/XOFF output control in the terminal:

stty -ixon

After running the command, ctrl+s is no longer consumed by the terminal to pause the output. To confirm that forward incremental search works now, first run the above command and then run the following commands:

echo foo
echo bar
cal
uname
echo baz

Now type ctrl+r followed by echo and the input line should be automatically completed to echo baz. Type ctrl+r again and echo bar should appear at the shell prompt. Type ctrl+r one more time to bring the command echo foo at the shell prompt.

Now type ctrl+s once and the search facility should switch itself to forward incremental search. The search prompt should change to show this. For example, in Bash the search prompt changes from reverse-i-search: to i-search: to indicate this. In Zsh, the search prompt changes from bck-i-search: to fwd-i-search:.

Now type ctrl+s again and the input line should be automatically completed to echo bar. Type ctrl+s again to bring back echo baz at the shell prompt. This is the forward incremental search in action.

Conclusion

I believe the forward incremental search facility offered in shells and other tools with line editing and history capabilities is a very useful feature that can make navigating the input history very convenient. However due to the default setting of most terminals, this rather splendid feature remains unusable.

I believe heavy terminal users should add the command stty -ixon to their ~/.bash_profile or ~/.zshrc, so that the ctrl+s key sequence can be used for forward incremental search. Forfeit XON/XOFF to reclaim forward incremental search!

Read on website | #unix | #shell | #technology

Wordle With Grep

2022-01-22T00:00:00Z

Let us solve a couple of Wordle games with the Unix grep command and the Unix words file. The Wordle games #217, #218 and #219 for 22 Jan 2022, 23 Jan 2022 and 24 Jan 22 respectively are used as examples in this post. The output examples shown below are obtained using the words file /usr/share/dict/words, GNU grep 3.6 and GNU bash 5.1.4 on Debian GNU/Linux 11.2 (bullseye).

Note that the original Wordle game uses a different word list. Further, there are several Wordle clones which may have their own word lists. For the purpose of this post, we will use the word list that comes with Debian. We will solve each Wordle in a quick and dirty manner in this post. The focus is going to be on making constant progress and reaching the solution quickly with simple shell commands.

Preliminary Work
Wordle #217
Wordle #218
Wordle #219

Preliminary Work

Before we start solving Wordle games, we will do some preliminary work. We will create a convenient shell alias that automatically selects all five-letter words from the words files. We will also find a good word to enter as the first guess into the Wordle game. The following steps elaborate this preliminary work:

Make a shell alias named words that selects all 5 letter words from the words file.

$ alias words='grep "^[a-z]\{5\}$" /usr/share/dict/words'
$ words | head -n 3
abaci
aback
abaft
$ words | tail -n 3
zoned
zones
zooms
$ words | wc -l
4594

For each letter in the English alphabet, count the number of five-letter words that contain the letter. Rank each letter by this count.
```
$ for c in {a..z}; do echo $(words | grep $c | wc -l) $c; done | sort -rn | head -n 15
2245 s
2149 e
1736 a
1404 r
1301 o
1231 i
1177 l
1171 t
975 n
924 d
810 u
757 c
708 p
633 h
623 y
```
The output shows that the letter 's' occurs in 2245 five-letter words, followed by 'e' which occurs in 2149 five-letter words and so on.
Find a word that contains the top five letters found in the previous step.
```
$ words | grep s | grep e | grep a | grep r | grep o
arose
```
We will enter this word as the first guess in every Wordle game.
In case, the word "arose" does not lead to any positive result, we will need another word to enter as our second guess. Find a word that contains the next five top letters in the list found above.
```
$ words | grep i | grep l | grep t | grep n | grep d
$ words | grep i | grep l | grep t | grep n | grep u
until
```
We found that there is no such word that contains 'i', 'l', 't', 'n' and 'd'. So we got rid of 'd' in our search and included 'u' (the next highest ranking letter after 'd') instead to find the word "until". We will enter this word as the second guess if and only if the first guess (i.e. "arose") does not lead to any positive result.

Wordle #217

Let us now solve Wordle #217 for Sat, 22 Jan 2022 with the following steps:

Use the word "arose" as the first guess. The following result appears:

A R O S E
The previous result shows that the letter 'e' occurs at the fifth place. Further, the letters 'a', 'r', 'o' and 's' do not occur anywhere in the word. Look for words satisfying these constraints.
```
$ words | grep '....e' | grep -v '[aros]' | head -n 5
beige
belie
belle
bible
bilge
```
Pick the word "beige" for the second guess and enter it into the Wordle game. Note that since we are following a quick and dirty approach here, we do not spend any time figuring out which of the various five-letter words ending with the letter 'e' is the most optimal choice for the next guess. We simply pick the first word from the output above and enter it as the second guess. The following result appears now:

B E I G E
The letter 'i' occurs somewhere in the word but not at the third place. Further the letters 'b' and 'g' do not occur anywhere in the word. Also, the letter 'e' does not occur anywhere apart from the fifth place. The letter 'e' in the gray tile in the second place confirms that the letter 'e' does not repeat in the answer word. Refine the previous command to add these constraints.
```
$ words | grep '[^e][^e][^ie][^e]e' | grep i | grep -v '[arosbg]' | head -n 5
fiche
indue
lithe
mince
niche
```
Enter "fiche" as the third guess. The following result appears:

F I C H E
The previous result shows that the letter 'i' occurs at the second place. Further, the letter 'c' occurs somewhere in the word but not at the third place. Also, the letters 'f' and 'h' do not occur anywhere in the word. Refine the previous command further to add these constraints:
```
$ words | grep '[^e]i[^iec][^e]e' | grep c | grep -v '[arosbgfh]' | head -n 5
mince
wince
```
Enter the word "mince" for the fourth guess. It leads to the following result:

M I N C E
We are almost there! We now have all the letters except the first one. The previous result shows that the letter 'm' does not occur in the word. Thus the answer word must be "wince". For the sake of completeness, here is a refined search that selects the answer word based on the constraints known so far:
```
$ words | grep '[^e]ince' | grep -v '[arosbgfhm]' | head -n 5
wince
```
It looks like we have found the answer word. Enter "wince" as the fifth guess to get the following result:

W I N C E

Done!

Wordle #218

Now that the wordle for Sat, 22 Jan 2022 is solved, let us try the same method on Wordle #219 for Sun, 23 Jan 2022 and see how well this method works. Here are the steps:

Like before, the first guess is "arose". Entering this word leads to the following result:

A R O S E
Now search for words based on the previous result.
```
$ words | grep '.r...' | grep -v '[aose]' | head -n 5
brick
bring
brink
briny
bruin
```
Enter the word "brick" as the second guess. This leads to the following result:

B R I C K
Use the previous result to refine the search further.
```
$ words | grep '.ri[^c].' | grep c | grep -v '[aosebk]' | head -n 5
crimp
```
Enter "crimp" as the third guess. This leads to the following result:

C R I M P

Done!

Wordle #219

Finally, let us solve Wordle #219 for Mon, 24 Jan 2022.

Enter "arose" as the first guess to get this result:

A R O S E
The previous result shows that the third letter is 'o' and the letters 'a', 'r', 's' and 'e' do not occur anywhere in the word. Search for words that match these constraints.
```
$ words | grep '..o..' | grep -v '[arse]' | head -n 5
block
blond
blood
bloom
blown
```
Enter "block" as the second guess. This leads to the following result:

B L O C K
The previous result shows that the letter 'l' occurs somewhere in the word but not at the second place. Similarly, the letter 'k' occurs somewhere in the word but not at the fifth place. Further, the letters 'b' and 'c' do not occur anywhere in the word. Search for words that match these constraints.
```
$ words | grep '.[^l]o.[^k]' | grep l | grep k | grep -v '[arsebc]' | head -n 5
knoll
```
Enter "knoll" as the third guess. It leads to the following result:

K N O L L

Done!

Read on website | #unix | #shell | #technology | #puzzle

Shell Eval

2022-01-06T00:00:00Z

In this post, we will perform a few experiments to see the usefulness of the eval command for a particular scenario in a POSIX-compliant shell. At first, we prepare a test file that contains a space in its name and define a variable as follows:

$ echo lorem ipsum > "foo bar"
$ cmd='cat "foo bar"'

We will use this file and the variable in the experiments below. All output examples below are obtained using Dash 0.5.11 on a Debian GNU/Linux 11.2 (bullseye) system. Dash stands for Debian Almquist Shell which is a POSIX-compliant shell available in Debian. Any POSIX conforming shell should produce similar output. On Zsh, use the command emulate sh before running these examples to get similar output.

Experiment 1

Now simply enter $cmd as a command into the shell. The following error occurs:

$ $cmd
cat: '"foo': No such file or directory
cat: 'bar"': No such file or directory

The error occurs because the above command expands to the command cat followed by two arguments: "foo and bar". Such an expansion occurs due to a concept known as field splitting. Quoting from section 2.6.5 of POSIX.1-2017:

After parameter expansion, command substitution, and arithmetic expansion, the shell shall scan the results of expansions and substitutions that did not occur in double-quotes for field splitting and multiple fields can result.

The shell shall treat each character of the IFS as a delimiter and use the delimiters as field terminators to split the results of parameter expansion, command substitution, and arithmetic expansion into fields.

By default, the space character belongs to IFS. Here is an example command to verify this:

$ printf "$IFS" | od -tcx1
0000000      \t  \n
         20  09  0a
0000003

The hexadecimal code 20 in the output confirms that the space character is present in the value of IFS. Therefore, according to the POSIX specification, $cmd first expands to cat "foo bar", then it is split into three fields cat, "foo and bar" and then the command cat is executed with two arguments "foo and bar". Since no files with those names exist, an error occurs.

Experiment 2

Next we try to double-quote the previous command to prevent field splitting and see what happens:

$ "$cmd"
dash: 8: cat "foo bar": not found

The excerpt from the POSIX.1-2017 specification quoted in the previous section shows that field splitting does not occur for variable expansions within double quotes. So the entire expansion cat "foo bar" remains intact as a single field and is then executed as a command. Since there is no such weirdly named command, we get the above error.

Experiment 3

Field splitting leads to an error as seen in the first experiment. Preventing field splitting by double-quoting the variable expansion also leads to an error as seen in the second experiment. How do we execute the command in the cmd variable?

We need a way to somehow get the shell to parse cat "foo bar" like a shell normally does, i.e. treat each unquoted token as a separate field and each quoted token as a single one. How do we get the shell to do that? Well, we can just invoke the shell itself to parse our command:

$ sh -c "$cmd"
lorem ipsum

But the above command invokes a new shell process. Can we avoid that? Yes, using the eval command:

$ eval "$cmd"
lorem ipsum

Read on website | #unix | #shell | #technology

Of Course "changeme" Is Valid Base64

2020-10-24T00:00:00Z

Today, I came across this blog post regarding how the author of the post used the string "changeme" as test data while testing a Base64 decoding functionality in their application. However, the author incorrectly believed that this test data is not a valid Base64-encoded string and therefore would fail to decode successfully when decoded as Base64. To their surprise, they found that this string "changeme" does in fact decode successfully.

The post did not go any further into understanding why indeed "changeme" is a valid Base64-encoded string and why it can successfully be decoded into binary data. It appears that the author was using Base64 encoding scheme as a black box.

I think it is worth noting and illustrating that any alphanumeric string with a length that is a multiple of 4 is a valid Base64-encoded string. Here are some examples that illustrate this:

$ printf AAAA | base64 --decode | od -tx1
0000000    00  00  00
0000003
$ printf AAAAAAAA | base64 --decode | od -tx1
0000000    00  00  00  00  00  00
0000006
$ printf AQEB | base64 --decode | od -tx1
0000000    01  01  01
0000003
$ printf AQID | base64 --decode | od -tx1
0000000    01  02  03
0000003
$ printf main | base64 --decode | od -tx1
0000000    99  a8  a7
0000003
$ printf scrabble | base64 --decode | od -tx1
0000000    b1  ca  da  6d  b9  5e
0000006
$ printf 12345678 | base64 --decode | od -tx1
0000000    d7  6d  f8  e7  ae  fc
0000006

Further, since + and / are also used as symbols in Base64 encoding (for binary 111110 and 111111 respectively), we also have a few more interesting examples:

$ printf 1+2+3+4+5/11 | base64 --decode | od -tx1
0000000    d7  ed  be  df  ee  3e  e7  fd  75
0000011
$ printf "\xd7\xed\xbe\xdf\xee\x3e\xe7\xfd\x75" | base64
1+2+3+4+5/11

I think it is good to understand why any string with a length that is a multiple of 4 turns out to be a valid Base64-encoded string. The Base64 encoding scheme encodes each group of 6 bits in the binary input with a chosen ASCII character. For every possible 6-bit binary value, we have assigned an ASCII character that appears in the Base64-encoded string. Each output ASCII character can be one of the 64 carefully chosen ASCII characters: lowercase and uppercase letters from the English alphabet, the ten digits from the Arabic numerals, the plus sign (+) and the forward slash (/). For example, the bits 000000 is encoded as A, the bits 000001 is encoded as B and so on. The equals sign (=) is used for padding but that is not something we will discuss in detail in this post.

The smallest positive multiple of 6 that is also a multiple of 8 is 24. Thus every group of 3 bytes (24 bits) of binary data is translated to 4 ASCII characters in its Base64-encoded string. Thus the entire input data is divided into groups of 3 bytes each and then each group of 3 bytes is encoded into 4 ASCII characters. What if the last group is less than 3 bytes long? There are certain padding rules for such cases but I will not discuss them right now in this post. For more details on the padding rules, see RFC 4648.

Now as a natural result of the encoding scheme, it turns out that any 4 alphanumeric characters is a valid Base64 encoding of some binary data. That's because for every alphanumeric character, we can find some 6-bit binary data that would be translated to it during Base64 encoding. This is the reason why any alphanumeric string with a length that is a multiple of 4 is a valid Base64-encoded string and can be successfully decoded to some binary data.

Read on website | #unix | #shell | #technology

Unix Timestamp 1600000000

2020-09-12T00:00:00Z

At 2020-09-13 12:26:40 UTC, the Unix timestamp is going to turn 1600000000.

Unix Timestamp Conversion

The following subsections show a few examples of converting the Unix timestamp to a human-readable date.

Python

$ python3 -q
>>> from datetime import datetime
>>> datetime.utcfromtimestamp(1_600_000_000)
datetime.datetime(2020, 9, 13, 12, 26, 40)

GNU date (Linux)

$ date -ud @1600000000
Sun Sep 13 12:26:40 UTC 2020

BSD date (macOS, FreeBSD, OpenBSD, etc.)

$ date -ur 1600000000
Sun Sep 13 12:26:40 UTC 2020

Other Such Dates

All such dates (in UTC) until the end of the current century:

$ python3 -q
>>> from datetime import datetime
>>> for t in range(0, 4_200_000_000, 100_000_000):
...     print(f'{t:13_d} - {datetime.utcfromtimestamp(t)}')
...
            0 - 1970-01-01 00:00:00
  100_000_000 - 1973-03-03 09:46:40
  200_000_000 - 1976-05-03 19:33:20
  300_000_000 - 1979-07-05 05:20:00
  400_000_000 - 1982-09-04 15:06:40
  500_000_000 - 1985-11-05 00:53:20
  600_000_000 - 1989-01-05 10:40:00
  700_000_000 - 1992-03-07 20:26:40
  800_000_000 - 1995-05-09 06:13:20
  900_000_000 - 1998-07-09 16:00:00
1_000_000_000 - 2001-09-09 01:46:40
1_100_000_000 - 2004-11-09 11:33:20
1_200_000_000 - 2008-01-10 21:20:00
1_300_000_000 - 2011-03-13 07:06:40
1_400_000_000 - 2014-05-13 16:53:20
1_500_000_000 - 2017-07-14 02:40:00
1_600_000_000 - 2020-09-13 12:26:40
1_700_000_000 - 2023-11-14 22:13:20
1_800_000_000 - 2027-01-15 08:00:00
1_900_000_000 - 2030-03-17 17:46:40
2_000_000_000 - 2033-05-18 03:33:20
2_100_000_000 - 2036-07-18 13:20:00
2_200_000_000 - 2039-09-18 23:06:40
2_300_000_000 - 2042-11-19 08:53:20
2_400_000_000 - 2046-01-19 18:40:00
2_500_000_000 - 2049-03-22 04:26:40
2_600_000_000 - 2052-05-22 14:13:20
2_700_000_000 - 2055-07-24 00:00:00
2_800_000_000 - 2058-09-23 09:46:40
2_900_000_000 - 2061-11-23 19:33:20
3_000_000_000 - 2065-01-24 05:20:00
3_100_000_000 - 2068-03-26 15:06:40
3_200_000_000 - 2071-05-28 00:53:20
3_300_000_000 - 2074-07-28 10:40:00
3_400_000_000 - 2077-09-27 20:26:40
3_500_000_000 - 2080-11-28 06:13:20
3_600_000_000 - 2084-01-29 16:00:00
3_700_000_000 - 2087-04-01 01:46:40
3_800_000_000 - 2090-06-01 11:33:20
3_900_000_000 - 2093-08-01 21:20:00
4_000_000_000 - 2096-10-02 07:06:40
4_100_000_000 - 2099-12-03 16:53:20

Update

Here is a screenshot I took at Unix timestamp 1600000000: twitter.com/susam/status/130512093609862758.

Reproduced as text below:

$ date -u; date; date +%s
Sun Sep 13 12:26:39 UTC 2020
Sun Sep 13 17:56:39 IST 2020
1599999999
$ date -u; date; date +%s
Sun Sep 13 12:26:40 UTC 2020
Sun Sep 13 17:56:40 IST 2020
1600000000

An important point worth noting from the POSIX.1-2008 specification:

Coordinated Universal Time (UTC) includes leap seconds. However, in POSIX time (seconds since the Epoch), leap seconds are ignored (not applied) to provide an easy and compatible method of computing time differences. Broken-down POSIX time is therefore not necessarily UTC, despite its appearance.

See § A.4.16 of the POSIX.1-2008 specification for more details.

Unix Line Discard

2017-07-09T00:00:00Z

Type C-u (i.e. ctrl+u) in Bash or Zsh to discard the current line of input. To read more about it, enter man bash and then type /unix-line-discard to locate the relevant section of the manual. Here is an excerpt:

unix-line-discard (C-u)
       Kill backward from point to the beginning of the line.
       The killed text is saved on the kill-ring.

Similarly, for Zsh, type man zshzle and then type /kill-whole-line. We find this:

kill-whole-line (^U) (unbound) (unbound)
       Kill the current line.

By the way, Emacs-style key sequence like C-a C-k works too.

Furthermore, it is quite likely that C-u is mapped to delete the current line of input in the terminal itself. To confirm this, type the command stty -a and check the output. If the output contains the text kill = ^U, then typing C-u anytime in the terminal would delete the current line of input. This would happen regardless of what program is running in the terminal. For example, programs like cat, sbcl, etc. do not support key sequences like C-a, C-k, C-u, etc. the way Bash or Zsh does. Despite this limitation, typing C-u in cat or sbcl would delete the current line of input if the output of stty -a indicates that the terminal has mapped this key sequence to the operation of deleting the current line.

Read on website | #unix | #shell | #technology

Random String in Shell

2011-08-11T00:00:00Z

Here is a quick way to generate a random alphanumeric string in the shell:

LC_CTYPE=C tr -dc '[:alnum:]' < /dev/urandom | head -c 20; echo

Here is an example output:

GrWPmvF1oOmbeUzyJwC3

The command works both on macOS as well as Linux. The LC_CTYPE=C environment variable is set specifically to make the command work successfully on macOS. Without it tr may report an "Illegal byte sequence" error.

Read on website | #unix | #shell | #technology

Fork Bunny

2006-06-11T00:00:00Z

Have a close look at this line of shell command that can be executed on Bash, Zsh and most POSIX or POSIX-like shells:

: () { : | : & } ; :

Beware! Don't execute it on your system without understanding the consequences completely. If the command above looks puzzling, that is because it is deliberately obfuscated. Let us simplify it.

The : is a function name. It could very well have been f. Let us replace : with f and see what the code now looks like.

f () { f | f & } ; f

Now it looks familiar. We have two commands separated by a semicolon. Written in a more legible manner, the code would look like this:

f()
{
    f | f &
}

f

It creates a function f and then executes it. This function calls itself twice recursively. The control operator & executes the recursive calls to f asynchronously, i.e. in the background. The number of instances of the function executing keeps growing exponentially thereby depleting CPU cycles and memory. The system is rendered unusable soon.

This type of denial-of-service attack by self-replication is also known as a fork bunny which is a specific type of wabbit. See the following entry in the Jargon File for more information on this: wabbit.

Read on website | #unix | #shell | #technology

Vim Sudo Write Trick

2005-08-25T00:00:00Z

The Trick

You open a file, edit it and save it only to get the E45 error message that says:

E45: 'readonly' option is set (add ! to override)

You now realise that only root can edit the file. What do you? Start over? No, instead try this:

:w !sudo tee "%"

I learnt this trick recently from the comment section of Tip #975 on the Vim Tips website.

Explanation

How does the :w !sudo tee "%" trick work? Let us look at the command part-by-part:

:w !{cmd}

Execute {cmd} with all lines in buffer as standard input.
"%"

The % is replaced with the current filename. The quotes around it keeps the filename as a single argument even if it contains whitespace.
tee {file}

The tee command is a Unix command (not a Vim command). It copies standard input to standard output and {file}.

More Information

For more information on this command, enter the following commands in Vim:

:help :w_c
:help current-file
:help :_%

Also, enter the following command in shell:

man tee

I hope this was fun!

Read on website | #vim | #unix | #technology | #how-to

Cal 9 1752

2004-09-14T00:00:00Z

A very interesting thing I came across recently while learning Unix is an apparent gap in the calendar for Sep 1752. Here is how the cal output for this month appears on a Unix or Linux system:

$ cal 9 1752
   September 1752
Su Mo Tu We Th Fr Sa
       1  2 14 15 16
17 18 19 20 21 22 23
24 25 26 27 28 29 30

In the SunOS 5.9 manual page for cal(1), this gap is explained as follows:

An unusual calendar is printed for September 1752. That is the month 11 days were skipped to make up for lack of leap year adjustments. To see this calendar, type: cal 9 1752

Similarly, FreeBSD 5.0 has the following note in its manual page for cal(1):

-s country_code

Assume the switch from Julian to Gregorian Calendar at the date associated with the country_code. If not specified, ncal tries to guess the switch date from the local environment or falls back to September 2, 1752. This was when Great Britain and her colonies switched to the Gregorian Calendar.

On investigating this switch from Julian to Gregorian calendar, I learnt that the Julian calendar was introduced by Julius Caesar in 46 BC and it remained in effect across most of the Western world from 45 BC to 1582. It assumed an average year to be 365.25 days long. However, the actual solar year has been known to be around 365.2422 days since the 17th century. Although the difference appears to be too small, it leads to an error of adding 1 extra day every 128 years. To reduce this error, the Gregorian calendar was introduced in October 1582 by Pope Gregory XIII. It assumed an average year to be 365.2425 days long.

The new calendar was adopted in some European countries where 4 Oct 1582 was followed by 15 Oct 1582 thereby skipping 10 days in between. However, it took as long as September 1752 for the new calendar to be adopted by Britain. In Great Britain and the British Empire, 2 Sep 1752 was followed by 14 Sep 1752 and that is the gap of 11 days we see in the cal 9 1752 output.

Read on website | #unix | #technology

Susam's Unix Pages

Hacker News Hug of Deaf

Pretty-Printing JSON Response with HTTP Headers

Control, Escape and Meta Tricks

Terminal Tricks

Control Codes

Meta Key Sequences

Awkward Vim Tricks

From XON/XOFF to Forward Incremental Search

XON/XOFF

Incremental Search in Bash/Zsh

The Conflict

Reclaim Forward Incremental Search

Conclusion

Wordle With Grep

Contents

Preliminary Work

Wordle #217

Wordle #218

Wordle #219

Shell Eval

Experiment 1

Experiment 2

Experiment 3

Of Course "changeme" Is Valid Base64

Unix Timestamp 1600000000

Unix Timestamp Conversion

Python

GNU date (Linux)

BSD date (macOS, FreeBSD, OpenBSD, etc.)

Other Such Dates

Update

Unix Line Discard

Random String in Shell

Fork Bunny

Vim Sudo Write Trick

The Trick

Explanation

More Information

Cal 9 1752