From XON/XOFF to Forward Incremental Search

XON/XOFF

In the olden days of computing, software flow control with control codes XON and XOFF was a necessary feature that dumb terminals needed to support. When a terminal received more data than it could display, there needed to be a way for the terminal to tell the remote host to pause sending more data. The control code 19 was chosen for this. The control code 17 was chosen to tell the remote host to resume transmission of data.

The control code 19 is called Device Control 3 (DC3) in the ASCII chart. It is also known as "transmit off" (XOFF). The control code 17 is called Device Control 1 (DC1) as well as "transmit on" (XON). Now how does a user of the terminal really send these control codes? Well, how do they send any control code? Using the ctrl key of course.

Let us take a step back and see how a user can send familiar control codes on modern terminal emulators, like say, the terminal software we find on a Unix or Linux desktop environment. While any sane computer user would just type the tab key to insert a tab character, one could also type ctrl+i to insert a tab character. The character I has code 73 (binary 1001001) and holding the ctrl key while typing it results in a control code made by taking 73 (binary 1001001), keeping its five least significant bits, and discarding the rest to get the control code 9 (binary 1001) which is the code of the tab character. In other words, we get the control code by performing a bitwise AND operation on the code of the character being modified with binary code of 31 (binary 0011111).

In case you are wondering, if we get the same result if we choose the binary code of the lowercase i to perform the aforementioned operation, the answer is, yes. While the code of uppercase I is 73 (binary 1001001), that of lowercase i is 105 (binary 1101001). The right-most five bits are same for both. Thus when we preserve the five least significant bits and discard the rest, we get the control code 9 (binary 1001) in both cases. This is a neat result due to the fact that the five least significant bits of the code of a lowercase character is exactly the same as that of the corresponding uppercase character. They differ only in their sixth least significant bit. That bit is on for the lowercase character but off for the corresponding uppercase character. This is just an interesting observation as far as this post is concerned. The very early keyboards did not have lowercase letters. They only had uppercase letters and when the ctrl key is pressed together with a letter on such keyboards, the 7th bit of the character code was flipped to get the control code.

The bitwise operation mentioned earlier explains why typing ctrl+h sends a backspace, typing ctrl+j sends a newline, and typing ctrl+g plays a bell. In modern terminal emulators these days, the bell often manifests in the form of an audible beep or a visual flash. Here is a table that summarises these control codes and a few more:

The last row in the table above explains why we can also type ctrl+[ in Vim to escape from insert mode to normal mode. This is, in fact, one of the convenient ways for touch-typists to return to normal mode in Vim instead of clumsily stretching the left hand fingers out to reach the esc key which is usually poorly located at the corner of most keyboards.

There is a bit of oversimplication in the description above. Throughout the history of computing, different systems have used slightly different methods to compute the resulting control code when the ctrl modifier key is held. Toggling the 7th least significant bit was an early method. Turning off both the 6th and 7th least significant bits is another method. Subtracting 64 from the character code is yet another method. These are implementation details and these various implementation methods lead to the same results for the examples in the table above. Then there are some special rules too. For example, many terminals implement a special rule to make ctrl+space behave the same as ctrl+@ thus producing the null character. Further ctrl+? produces the delete character in some terminals. These special rules are summarised in the table below.

For the purpose of this blog post, we don't need to worry about these special rules. Let us get back to the control codes 19 and 17 that represent XOFF and XON. Here is how the table looks for them:

We see that a user can type ctrl+s to send the control code XOFF and then ctrl+q to send the control code XON. Although we do not use dumb terminals anymore, these conventions have survived in various forms in our modern terminal emulators. To see a glimpse of it, launch the terminal that comes with a modern operating system, then run ping localhost and while this command is printing its output, type ctrl+s. The terminal should pause printing the output of the command. Then type ctrl+q and the terminal should resume printing the output.

Incremental Search in Bash/Zsh

Let us now take a look at the shells that run within the terminals. Both Bash and Zsh are very popular these days. Both these shells have excellent support for performing incremental searches through the input history. To see a quick demonstration of this, open a new terminal that runs either Bash or Zsh and run these commands:

echo foo
echo bar
cal
uname
echo baz

Now type ctrl+r followed by echo. This should invoke the reverse search feature of the shell and automatically complete the partial command echo to echo baz. Type ctrl+r again to move back in the input history and autocomplete the command to echo bar. We can type enter anytime we use the reverse search feature to execute the automatically completed command. Typing ctrl+r yet another time should bring the echo foo command at the shell prompt.

What if we went too far back in the input history and now want to go forward? There is a good news and there is some bad news. The good news is that both Bash and Zsh support forward search using the ctrl+s key sequence. The bad news is that this key sequence may not reach the shell. Most terminal emulators consume this key sequence and interpret it as the control code XOFF.

The Conflict

In the previous two sections, we have seen that the ctrl+s key sequence is used to send the control code XOFF. But the same key sequence is also used for forward incremental search in Bash and Zsh. Since the terminal consumes this key sequence and interprets it as the control code XOFF, the shell never sees the key sequence. As a result, the forward incremental search functionality does not work when we type this key sequence in the shell.

Bash offers the incremental search facility via a wonderful piece of library known as the the GNU Readline Library. ZSH offers this facility with its own Zsh Line Editor (ZLE). So other tools that rely on these libraries to offer line editing and history capability are also affected by this conflict. For example, many builds of Python offer line editing and history capability using GNU Readline, so while ctrl+r works fine to perform reverse search in the Python interpreter, it is very likely that ctrl+s does not work to perform forward search.

Reclaim Forward Incremental Search

We can forfeit the usage of control codes XON/XOFF to reclaim forward incremental search. Here is the command to disable XON/XOFF output control in the terminal:

stty -ixon

After running the command, ctrl+s is no longer consumed by the terminal to pause the output. To confirm that forward incremental search works now, first run the above command, and then run the following commands:

echo foo
echo bar
cal
uname
echo baz

Now type ctrl+r followed by echo and the input line should be automatically completed to echo baz. Type ctrl+r again and echo bar should appear at the shell prompt. Type ctrl+r one more time to bring the command echo foo at the shell prompt.

Now type ctrl+s once and the search facility should switch itself to forward incremental search. The search prompt should change to show this. For example, in Bash the search prompt changes from reverse-i-search: to i-search: to indicate this. In Zsh, the search prompt changes from bck-i-search: to fwd-i-search:.

Now type ctrl+s again and the input line should be automatically completed to echo bar. Type ctrl+s again to bring back echo baz at the shell prompt. This is the forward incremental search in action.

Conclusion

I believe the forward incremental search facility offered in shells and other tools with line editing and history capabilities is a very useful feature that can make navigating the input history very convenient. However due to the default setting of most terminals, this rather splendid feature remains unusable.

I believe heavy terminal users should add the command stty -ixon to their ~/.bash_profile or ~/.zshrc, so that the ctrl+s key sequence can be used for forward incremental search. Forfeit XON/XOFF to reclaim forward incremental search!