From Fill Prefix to TRAMP

Our tiny book club that used to meet during the weekends and holidays and discuss the book Mastering Emacs, 2022 edition concluded today. In our final meeting today, we first discussed how to work across multiple directories in the same Dired buffer. Then we did several demos of the various shells and terminal modes available in Emacs out of the box. That completed our discussion on Chapter 6. Then we moved on to Chapter 7 (the final chapter) that first reiterates the importance of using the describe-system to ask Emacs questions about itself and then offers some recommendations about third-party packages and online Emacs communities. Completing this chapter brought our book club discussions to an end.

A big thanks to Mickey Petersen for writing the book and also very generously granting me the permission to share his book on screen while discussing it.

This book club began on 16 Dec 2022 when we had our first meeting over Jitsi. About 3½ months after beginning these meetings, I posted an update about this book club in another blog post titled From Lunar Phases to Yank-Pop. If you have not read that post yet, I suggest you read it before returning to this post. Especially if you have recently begun learning Emacs, I think you will find that post useful.

Back then, when I posted that last update, we had spent about 26 hours together across 36 meetings and we were reading Chapter 5 of the book. It took another 36 meetings to complete that chapter and the remaining two chapters. After a total of 72 meetings, we completed discussing Chapter 7 of the book today which concluded this series of book club meetings. In total, we have spent a little over 52 hours together to discuss this book, trying out every concept and command introduced in the book, and sharing our insights about the material with each other.

In this post, I will share some highlights from our meetings since the last update. These highlights share some concepts and commands we learnt that most members of our book club were not familiar with earlier but were found to be very useful after having learnt them.

Contents

• Fill Prefix
• Elisp Expressions in Replacement Strings
• Keep Lines and Flush Lines
• Keyboard Macros
• DAbbrev
• TAB vs M-i
• Project Management
• Eshell with TRAMP
• Thanks

Fill Prefix

Most of us in the book discussion group knew about filling paragraphs with M-q. Consider the following badly formatted paragraphs with very long and very short lines:

Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor
incididunt ut labore et dolore
magna aliqua.  Arcu dui vivamus arcu felis bibendum ut tristique et egestas.
Bibendum arcu vitae
elementum curabitur vitae.

Now put the point (cursor) anywhere on the paragraph and type M-q. The paragraph gets neatly formatted to something like this:

Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do
eiusmod tempor incididunt ut labore et dolore magna aliqua.  Arcu dui
vivamus arcu felis bibendum ut tristique et egestas.  Bibendum arcu
vitae elementum curabitur vitae.

The key sequence M-q invokes the fill-paragraph command that reformats the paragraph such that each line is as long as possible without exceeding the fill width (70 columns by default). Most of us already used this command very often while writing and editing text. However what some of us did not know was that there is such a thing as fill prefix which is taken into account while filling paragraphs. To illustrate this concept, we will first consider this badly formatted paragraph:

:::: Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor
:::: incididunt ut labore et dolore
:::: magna aliqua.  Arcu dui vivamus arcu felis bibendum ut tristique et egestas.
:::: Bibendum arcu vitae
:::: elementum curabitur vitae.

Each line has a prefix consisting of four colons and a space. After we reformat this paragraph with M-q, we get something like this:

:::: Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do
eiusmod tempor :::: incididunt ut labore et dolore :::: magna aliqua.
Arcu dui vivamus arcu felis bibendum ut tristique et egestas.  ::::
Bibendum arcu vitae :::: elementum curabitur vitae.

This is not what we want. We want the paragraph to be formatted such that each line does not exceed 70 characters in length (which we have, in fact, accomplished above) and each line contains the four colons and a space as the prefix (this is broken above). Can we do this? Yes, by setting the fill prefix. Type C-/ or C-x u to undo the bad formatting we did just now and let us try again. This time move the point over to the L of Lorem and type C-x . to set the fill prefix to the current line up to the point. A confirmation is printed in the echo area that ":::: " has been set as the fill prefix. Then type M-q and the paragraph is now neatly formatted to look like this:

:::: Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do
:::: eiusmod tempor incididunt ut labore et dolore magna aliqua.  Arcu
:::: dui vivamus arcu felis bibendum ut tristique et egestas.
:::: Bibendum arcu vitae elementum curabitur vitae.

Note how every line is as long as possible without exceeding 70 characters in length and each line has the fill prefix. Emacs took care to remove the fill prefix from each line, subtract the length of the fill prefix from the maximum character budget it has for each line, reformat the lines, and then reinsert the fill prefix on each line of the result.

To turn off the fill prefix, simply set it to an empty prefix by typing C-x . at the beginning of the line. Thus C-a C-x . becomes an idiom for turning off the fill prefix.

Elisp Expressions in Replacement Strings

It was no surprise to anyone in the book discussion group that the key sequence C-M-% f.. RET bar RET starts a search-and-replace operation for strings that match the regular expression pattern f.. to be replaced with the text bar.

The concept of backreferences was also known to most. For example, C-M-% \(f..\)-\(b..\) RET \2-\1 RET searches for strings matching the given regular expression pattern and replaces it with a new string that swaps the positions of the first capturing group and the second capturing group. The backreference \1 refers to the string matched by the first capturing group \(f..\) and similarly \2 refers to the string matched by the second capturing group \(b..\). In this example, a string like foo-bar is replaced with bar-foo, or playful-banter with playban-fulter.

However what came as a surprise to some of us was that we could also use Elisp expressions in the replacement strings. The syntax to do so is to write \, followed by the Elisp expression in the replacement string. For example, consider the key sequence C-M-% f.. RET \,(upcase \&) RET. Note how we are using the backreference \& that refers to the whole match as the argument to the Elisp function upcase that converts its argument to upper-case. This example searches for strings that match the pattern f.. and replaces them with the upper-case form of the match. A string like foo-bar is replaced with FOO-bar.

Here is another slightly more sophisticated example: C-M-% port-\([0-9]+\) RET port-\,(+ 1000 \#1). The backreference \#1 refers to the string matched by the first capturing group \([0-9]+\) as number. The Elisp expression in the replacement pattern simply adds 1000 to that number and replaces the matched string with the result. A string like port-80 becomes port-1080.

Keep Lines and Flush Lines

A nifty set of commands that our group members enjoyed learning were the commands for keeping and flushing lines. These commands can be incredibly useful while filtering large log files. Here is a brief illustration of a couple of these commands:

• M-x keep-lines RET f.. RET: Keep lines in region that match the regular expression f.. and delete the rest. If no region is active, then keep matching lines between the point and end of buffer, and delete the rest. The deleted lines are not copied to kill ring.

• M-x flush-lines RET f.. RET: Delete lines in the region that match the regular expression f... If no region is active, then delete matching lines between the point and end of buffer. The deleted lines are not copied to kill ring.

Note how each point above mentions that the deleted lines are not copied to the kill ring. This can be an inconvenience if we want to quickly yank the deleted lines to another buffer. Emacs 28.1 introduces a couple of more commands that remedy this situation to an extent. Here they are:

• M-x copy-matching-lines RET f.. RET: Copy lines in the region that match the regular expression f... If no region is active, then copy matching lines between the point and end of buffer.

• M-x kill-matching-lines RET f.. RET: Kill lines in region that match the regular expression f.. to the kill ring. If no region is active, then kill matching lines between the point and end of buffer.

Keyboard Macros

Most experienced Emacs users in our group were aware of keyboard macros. However, some people did learn this wonderful automation feature for the first time in our meetings, so I thought this deserves its own section in this post.

Keyboard macros is a large topic on its own which is best learnt from section Keyboard Macros of the manual. On Emacs, type M-: (info "(emacs) Keyboard Macros") RET to open this section using the Info documentation browser. In this blog post though, we will very briefly talk about keyboard macros that should be enough to get someone very new to it started with them quickly.

Say, we have a buffer that looks like this:

foo:bar:baz
bar:baz:qux
quux:corge:grault
garply:waldo:fred

Now suppose we want to swap the first two fields separated by colon in each line. Of course, we could do it using regular expressions, for example, with the key sequence C-M-% ^\(.+?:\)\(.+?:\) RET \2\1 RET. But we can also solve this problem in a "dumb" way by simply performing the edits necessary to do the swap on one line and then asking Emacs to repeat what we did on the other lines. Here are the steps:

1. First move the point to somewhere on the first line.

2. Then type C-x ( to start recording a keyboard macro.

3. Then type C-a M-d C-d M-f : C-y C-n to swap the first and second fields on the first line and move the point to the next line. This is just one way to achieve the swap. You may use any editing commands you are comfortable with to make the swap happen and move the point to the next line.

4. Now type C-x ) to stop macro recording.

5. Now type C-x e to replay the macro in the second line. As soon as we type this key sequence, the swap occurs in the second line and the cursor moves to the third line. Keep repeating this key sequence to keep repeating the swap operation on subsequent lines.

To summarise, C-x ( starts recording a new keyboard macro, C-x ) stops recording the keyboard macro, and C-x e replays the last keyboard macro. Alternatively, we could also use the function keys F3 and F4. To start recording a keyboard macro, type F3. Type F4 to stop recording a keyboard macro. Then type F4 again to replay the last macro.

Pay close attention to step 3 above. We start the key sequence with C-a to move the point to the first column. This may feel redundant when the cursor is already at the first column. However in our meetings, I used to emphasise often about the importance of doing this. Typing C-a at the beginning ensures that we do not carry over any assumption about where the cursor is on the line into the rest of the keyboard macro definition we are going to record. By typing C-a, we ensure that no matter where the cursor is on the line, when we replay the macro, the cursor would first move to the beginning of the line. This guarantee allows us to confidently define the rest of the editing operations necessary to perform the swap.

Similarly, at the end we type C-n to move the point to the next line. I used to emphasise the importance of doing this too in our meetings. Moving the cursor to the next line ensures that the cursor is in a good place to allow repeating the keyboard macro again immediately. This is why we could type C-x e (or alternatively, F4) over and over again to replay the macro on subsequent lines. In fact, if we feel confident about the keyboard macro, we can repeat it several times automatically using the digit argument. For example, type C-3 C-x e (or alternatively C-3 F4) to repeat the keyboard macro 3 times. We could also type C-0 C-x e (or alternatively C-0 F4) to repeat the keyboard macro until there is an error (e.g., reaching the end of the buffer).

DAbbrev

DAbbrev stands for dynamic abbrevation. This is a pretty useful package that many of us learnt only from our book club meetings. We discussed two simple key sequences supported by this package:

• M-/: Expand the word before the point to the nearest preceding word for which the current word is a prefix. If no suitable preceding word is found, then expand the current word to the nearest succeeding word for which the current word is a prefix.

• C-M-/: Find all words in the buffer that has the current word before the cursor as the prefix and expand the current word to the longest common prefix of all these matching words. However, if the longest common prefix of the matching words is same as the word before the cursor, then present them as suggestions for completion. If there is exactly one matching word, expand the word before the cursor to that word.

Let us look at some examples. Suppose there is a buffer with the following one line of text:

abacus apple appliance application

Now if we type ap on the next line and type M-/, DAbbrev automatically expands the partially written word to application because that is the nearest word that has ap as the prefix.

But if we type ap and type C-M-/, the word expands to appl since that is the longest common prefix among all the matching words. If we type C-M-/ again, then apple, appliance, and application are presented as possible completions in a temporary buffer named *Completions*. If we type ic, so that the word before the cursor becomes applic, and type C-M-/, it is expanded to application because that is the only possible completion now.

These two commands are simpler than it sounds from the verbose descriptions of these commands presented in the above paragraphs. When we actually begin to use them, they become intuitive in no time. Roughly speaking, while M-/ expands the word before the point to the nearest preceding word, C-M-/ considers all matching words in the file for expansion and presents completion options to the user when it finds multiple of them.

TAB vs M-i

The behaviour of Emacs when we type TAB can be surprising to beginners. In most other mainstream editors, this key either inserts a tab character or it inserts enough number of spaces so that the cursor moves to the next tab stop. But in Emacs, TAB most often indents the current line according to the syntax rules implemented by the major mode enabled in the buffer.

What is a simple key on other editors happens to be a complex feature in Emacs. The exact behaviour of TAB is controlled by variables like tab-always-indent, indent-line-function, etc. Some major modes may refer to other such special variables to decide what TAB should do. However, as a user of Emacs this is not something we normally have to worry about. Most major modes set up all these variables appropriately, so that TAB almost always does what an experienced Emacs user expects, i.e., indent the current line of code correctly.

But what if we really do want to just insert a tab or enough number of spaces to move the point to the next tab stop column? That is done with M-i. If the variable indent-tabs-mode is set to t, then M-i inserts a literal tab character. If it is set to nil, then M-i inserts enough number of spaces to move the point to the next tab stop column.

To summarise, the behaviour of M-i is similar to the TAB behaviour we observe in other editors. In practice though, the key sequence M-i is rarely required. Most people just type TAB to automatically indent code. In fact, we can also select a region of code and type TAB to reindent the whole region.

Project Management

The project management commands that come out of the box (from the package named project.el) came as a surprise to some. In fact, some members of our group who never used the project management commands earlier happen to use them regularly now after learning about them in our meetings.

When we use a project command like C-x p f to visit a file in the current project, the command automatically detects the top-level directory of the project by checking parent directories for version control system artefacts (e.g., .git directory) and presents files within that top-level directory as autocomplete options.

There is a lot that can be written about the project management features that come out of the box in Emacs. The following list introduces only the very simple ones to get someone started with them:

• C-x p f logger RET: Find file with name that matches logger in the current project. This searches all subdirectories recursively. If there is only one matching file (say, src/logger.cc), that file is opened. If there are multiple matching files, they are presented as completion options. Running this command or, in fact, running any project command leads to discovering the current project and adding an entry for the discovered project to ~/.emacs.d/projects. This is useful for a command like the one presented in the next point.

• C-x p f foo TAB RET logger TAB RET: When we type C-x p f while visiting a file that does not belong to any project, then its prompts for a project path first. In this example, we type foo TAB RET to automatically expand it to a known project path such as ~/git/foo/ and enter it. Then we type logger TAB RET to automatically expand it to a file name such as src/logger.cc and visit it.

• C-x p p bar TAB RET f logger TAB RET: Say we are in project ~/git/foo/ but we want to switch to another previously discovered project ~/git/bar/ and find a file there. To do so, we first type C-x p p to switch project. At the project selection prompt, we type bar TAB to automatically complete the directory path of the known project ~/git/bar/. Then another prompt is presented to choose an action from a number of actions. In this case, we type f to find a file in the project we have switched to. Finally, we type logger TAB RET to automatically expand the partially entered name to a path like src/logger.cc and visit it. The key sequence C-x p p is very useful when the current file belongs to one project but we want to run a project command on another project.

• C-x p p ... RET ~/git/baz/ RET f logger TAB RET: This awkward key sequence discovers a new project directory at ~/git/baz/ and then finds a file in it. The key sequence C-x p p ... RET is rarely required though. See the notes after the end of this list to read why.

• C-x p g ^key\> RET: Find all matches for the regular expression ^key\> in the current project. The results are displayed in *xref* buffer.

• C-x p s: Start a shell in the current project's root directory.

• C-x p d: Start Dired in the current project's root directory.

• C-x p k yes RET: Kill all buffers belonging to the current project.

There are several more project commands but we will end the above list here for the sake of brevity. Do pay attention to the second point that mentions that if the current file does not belong to any project, we are first prompted to enter the project name. This is a common theme for all project commands. Anytime we invoke a project command, it works on the current project. However if there is no current project, then it automatically prompts us to enter a project name before executing the command.

The key sequence C-x p p ... RET is very rarely required during day-to-day editing activities. Once a project has been discovered (say, due to having run a project command on that project earlier) and added to the list of known projects at ~/.emacs.d/projects, we never have to discover it again. We can use the other key sequences to switch to or work on a known project. Most day-to-day project activities involve working on known projects.

Further, even when we do want to discover a new project and add it to the list of known projects, a much more natural way to do it is to run a project command while we are visiting a file in the project directory. In most cases, we already have a file from some project open in the current buffer. Therefore it makes more sense to just go ahead with a project command, say, with C-x p f, C-x p g, etc. directly instead of explicitly discovering the project with C-x p p ... RET. Merely running a project command while we have a file from a project open ends up discovering the current project automatically. Explicitly discovering projects with C-x p p ... RET is almost never necessary.

Eshell with TRAMP

Many members of our group knew about Eshell and TRAMP separately. For example, M-x eshell RET starts Eshell. Eshell is implemented purely in Elisp and we can use it much like a regular shell. Here is an example session:

~ $ cd /tmp/
/tmp $ echo hello > hello.txt
/tmp $ cat hello.txt
hello
/tmp $ python3 --version
Python 3.11.5
/tmp $ which cd echo cat python3 which
eshell/cd is a byte-compiled Lisp function in ‘em-dirs.el’.
eshell/echo is a byte-compiled Lisp function in ‘em-basic.el’.
eshell/cat is a byte-compiled Lisp function in ‘em-unix.el’.
/usr/bin/python3
eshell/which is a byte-compiled Lisp function in ‘esh-cmd.el’.

We also knew about TRAMP. For example, when we type the key sequence C-x C-f /ssh:alice@box:/tmp/foo.txt RET, TRAMP notices that we intend to connect to a remote system named box as the user alice via SSH and edit a file named /tmp/foo.txt on the remote system. TRAMP then transparently establishes the SSH connection for us. If public key authentication is already set up, then the connection is successfully established immediately. Otherwise it prompts for a password. In the end, we get a buffer to edit the remote file /tmp/foo.txt. Once we have this buffer, we never have to do anything special to work on the remote file. All Emacs commands work seamlessly on this buffer for the remote file. For example, when we type C-x C-s TRAMP would go ahead and save the file to the remote system using the established SSH connection. If we type C-x d, TRAMP would create a Dired buffer for the remote directory /tmp/. All the Emacs commands for working with files and directories we know just work fine with the remote file or directory.

So we knew about Eshell and we knew about TRAMP. However what many of us found pleasantly surprising was how remarkably well Eshell and TRAMP work together. Here is an example Eshell session that illustrates this point:

~ $ cd /ssh:alice@box:/tmp/
/ssh:alice@box:/tmp $ echo foo > foo.txt
/ssh:alice@box:/tmp $ ls
foo.txt
/ssh:alice@box:/tmp $ cd /tmp/
/tmp $ echo bar > bar.txt
/tmp $ ls
bar.txt
/tmp $ cp /ssh:alice@box:/tmp/foo.txt .
/tmp $ ls
bar.txt  foo.txt
/tmp $

Look at how the command cd /ssh:alice@box:/tmp/ above has seamlessly and transparently set the current directory of the shell to the remote directory. When we create a file after that, it gets created on the remote directory! We can work across directories opened with multiple TRAMP methods too. For example first consider this session where the current local user does not have the permissions to write to the local /etc/ directory:

~ $ cp /ssh:alice@box:/etc/wgetrc /etc/
Opening output file Permission denied /etc/wgetrc

But if the current user has sudo privilege, we can do something like this:

~ $ cp /ssh:alice@box:/etc/wgetrc /sudo::/etc/
~ $ ls /etc/wgetrc
/etc/wgetrc

We copied a file from a remote system and wrote it to a protected directory on the local system by using the sudo privilege. We used the ssh method to read a remote file and the sudo method to write the file to a protected local directory. TRAMP really does live up to its name: Transparent Remote Access, Multiple Protocol!

Thanks

It has been a pleasure hosting these Emacs book club meetings throughout this year. I have really enjoyed discussing the book in great detail, examining each new concept introduced in the book carefully, and performing demos to illustrate the concepts. A big thank you to the Emacs communities on Libera and Matrix networks who showed interest in these meetings, joined these meetings, participated in the discussions, and helped make these meetings successful!