From XON/XOFF to Forward Incremental Search

By Susam Pal on 13 Aug 2022

XON/XOFF

In the olden days of computing, software flow control with control codes XON and XOFF was a necessary feature that dumb terminals needed to support. When a terminal received more data than it could display, there needed to be a way for the terminal to tell the remote host to pause sending more data. The control code 19 was chosen for this. The control code 17 was chosen to tell the remote host to resume transmission of data.

The control code 19 is called Device Control 3 (DC3) in the ASCII chart. It is also known as "transmit off" (XOFF). The control code 17 is called Device Control 1 (DC1) as well as "transmit on" (XON). Now how does a user of the terminal really send these control codes? Well, how do they send any control code? Using the ctrl key of course.

Let us take a step back and see how a user can send familiar control codes on modern terminal emulators, like say, the terminal software we find on a Unix or Linux desktop environment. While any sane computer user would just type the tab key to insert a tab character, one could also type ctrl+i to insert a tab character. The character I has code 73 (binary 1001001) and holding the ctrl key while typing it results in a control code made by taking 73 (binary 1001001), keeping its five least significant bits, and discarding the rest to get the control code 9 (binary 1001) which is the code of the tab character. In other words, we get the control code by performing a bitwise AND operation on the code of the character being modified with binary code of 31 (binary 0011111).

In case you are wondering, if we get the same result if we choose the binary code of the lowercase i to perform the aforementioned operation, the answer is, yes. While the code of uppercase I is 73 (binary 1001001), that of lowercase i is 105 (binary 1101001). The right-most five bits are same for both. Thus when we preserve the five least significant bits and discard the rest, we get the control code 9 (binary 1001) in both cases. This is a neat result due to the fact that the five least significant bits of the code of a lowercase character is exactly the same as that of the corresponding uppercase character. They differ only in their sixth least significant bit. That bit is on for the lowercase character but off for the corresponding uppercase character. This is just an interesting observation as far as this post is concerned. We are not going to use the code of lowercase characters in the rest of this post.

The bitwise operation mentioned earlier explains why typing ctrl+h sends a backspace, typing ctrl+j sends a newline, and typing ctrl+g plays a bell. In modern terminal emulators these days, the bell often manifests in the form of an audible beep. Here is a table that summarizes these control codes and a few more:

Key Modified Character Control Character
Binary Decimal Character Binary Decimal Character
ctrl+@ 1000000 64 @ 00000 0 Null
ctrl+g 1000111 71 G 00111 7 Bell
ctrl+h 1001000 72 H 01000 8 Backspace
ctrl+i 1001001 73 I 01001 9 Horizontal Tab
ctrl+j 1001010 74 I 01010 10 Line Feed
ctrl+m 1001101 77 M 01101 13 Carriage Return
ctrl+[ 1011011 91 [ 11011 27 Escape

The last row in the table above explains why we can also type ctrl+[ in Vim to escape from insert mode to normal mode. This is infact one of the convenient ways for touch-typists to return to normal mode in Vim instead of clumsily stretching the left hand fingers out to reach the esc key which is usually poorly located at the corner of most keyboards.

There is a bit of oversimplication in the description above. Throughout the history of computing, different systems have used slightly different methods to compute the resulting control code when the ctrl modifier key is held. Turning off both the 6th and 7th least significant bits is one method. Subtracting 64 from the character code is yet another method. These are implementation details and these various implementation methods lead to the same results for the examples in the table above. Then there are some special rules too. For example, many terminals implement a special rule to make ctrl+space behave the same as ctrl+@ thus producing the null character. Further ctrl+? produces the delete character. These special rules are summarized in the table below.

Key Modified Character Resulting Character
Binary Decimal Character Binary Decimal Character
ctrl+space 0100000 32 Space 0 0 Null
ctrl+? 0111111 63 ? 1111111 127 Delete

For the purpose of this blog post, we don't need to worry about these special rules. Let us get back to the control codes 19 and 17 that represent XOFF and XON. Here is how the table looks for them:

Key Modified Character Resulting Character
Binary Decimal Character Binary Decimal Character
ctrl+q 1010001 81 Q 10001 17 DC1 (XON)
ctrl+s 1010011 83 S 10011 19 DC3 (XOFF)

We see that a user can type ctrl+s to send the control code XOFF and then ctrl+q to send the control code XON. Although we do not use dumb terminals anymore, these conventions have survived in various forms in our modern terminal emulators. To see a glimpse of it, launch the terminal that comes with a modern operating system, then run ping localhost and while this command is printing its output, type ctrl+s. The terminal should pause printing the output of the command. Then type ctrl+q and the terminal should resume printing the output.

Incremental Search in Bash/Zsh

Let us now take a look at the shells that run within the terminals. Both Bash and Zsh are very popular these days. Both these shells have excellent support for performing incremental searches through the input history. To see a quick demonstration of this, open a new terminal that runs eitehr Bash or Zsh and run these commands:

echo foo
echo bar
cal
uname
echo baz

Now type ctrl+r followed by echo. This should invoke the reverse search feature of the shell and automatically complete the partial command echo to echo baz. Type ctrl+r again to move back in the input history and autocomplete the command to echo bar. We can type enter anytime we use the reverse search feature to execute the automatically completed command. Typing ctrl+r yet another time should bring the echo foo command at the shell prompt.

What if we went too far back in the input history and now want to go forward? There is a good news and there is some bad news. The good news is that both Bash and Zsh support forward search using the ctrl+s key sequence. The bad news is that this key sequence may not reach the shell. Most terminal emulators consume this key sequence and interpret it as the control code XOFF.

The Conflict

In the previous two sections, we have seen that the ctrl+s key sequence is used to send the control code XOFF. But the same key sequence is also used for forward incremental search in Bash and Zsh. Since the terminal consumes this key sequence and interprets it as the control code XOFF, the shell never sees the key sequence. As a result, the forward incremental search functionality does not work when we type this key sequence in the shell.

Bash offers the incremental search facility via a wonderful piece of library known as the the GNU Readline Library. ZSH offers this facility with its own Zsh Line Editor (ZLE). So other tools that rely on these libraries to offer line editing and history capability are also affected by this conflict. For example, many builds of Python offer line editing and history capability using GNU Readline, so while ctrl+r works fine to perform reverse search in the Python interpreter, it is very likely that ctrl+s does not work to perform forward search.

We can forfeit the usage of control codes XON/XOFF to reclaim forward incremental search. Here is the command to disable XON/XOFF output control in the terminal:

stty -ixon

After running the command, ctrl+s is no longer consumed by the terminal to pause the output. To confirm that forward incremental search works now, first run the above command, and then run the following commands:

echo foo
echo bar
cal
uname
echo baz

Now type ctrl+r followed by echo and the input line should be automatically completed to echo baz. Type ctrl+r again and echo bar should appear at the shell prompt. Type ctrl+r one more time to bring the command echo foo at the shell prompt.

Now type ctrl+s once and the search facility should switch itself to forward incremental search. The search prompt should change to show this. For example, in Bash the search prompt changes from reverse-i-search: to i-search: to indicate this. In Zsh, the search prompt changes from bck-i-search: to fwd-i-search:.

Now type ctrl+s again and the input line should be automatically completed to echo bar. Type ctrl+s again to bring back echo baz at the shell prompt. This is the forward incremental search in action.

Conclusion

I believe the forward incremental search facility offered in shells and other tools with line editing and history capabilities is a very useful feature that can make navigating the input history very convenient. However due to the default setting of most terminals, this rather splendid feature remains unusable.

I believe heavy terminal users should add the command stty -ixon to their ~/.bash_profile or ~/.zshrc, so that the ctrl+s key sequence can be used for forward incremental search. Forfeit XON/XOFF to reclaim forward incremental search!

Comments