Intro

Welcome back. As a reminder, in the previous part, you learned about the actual navigation logic, its execution logic, and how the "z" command is exposed as an executable. In this part, we'll deep-dive into shell integration and explore how hooks and auto-completions work on both zsh and bash. This last chapter will tie together all the flows we covered up until now.

C-10 Integrate shells

# z.sh:229 [ "$_Z_NO_RESOLVE_SYMLINKS" ] || _Z_RESOLVE_SYMLINKS="-P" # z.sh:231 if type compctl >/dev/null 2>&1; then # zsh [ "$_Z_NO_PROMPT_COMMAND" ] || { # $_Z_NO_RESOLVE_SYMLINKS # _z --add "..." # _z --complete "..." } elif type complete >/dev/null 2>&1; then # bash [ "$_Z_NO_PROMPT_COMMAND" ] || { # $_Z_RESOLVE_SYMLINKS # _z --add "..." # _z --complete "..." } fi

• type [-afptP] [name …] - For each name, indicate how it would be interpreted if used as a command name. If the name is not found, then nothing is printed, and an exit status of false is returned. Read more.
• compctl [ -CDT ] options [ command ... ] - Control the editor’s completion behavior according to the supplied set of options. Various editing commands, notably expand-or-complete-word, usually bound to tab, will attempt to complete a word typed by the user, while others, notably delete-char-or-list, usually bound to ^D in EMACS editing mode, list the possibilities; compctl controls what those possibilities are. They may for example be filenames (the most common case, and hence the default), shell variables, or words from a user-specified list. Read more.
  • This command is specific to ZSH and does not exists in other shells.
• complete (TAB) - Attempt to perform completion on the text before point. The actual completion performed is application-specific. Bash attempts completion treating the text as a variable (if the text begins with ‘$’), username (if the text begins with ‘~’), hostname (if the text begins with ‘@’), or command (including aliases and functions) in turn. If none of these produces a match, filename completion is attempted. Read more.
  • This command is specific to BASH, for a full explanation check the bash manual under “8.6 Programmable Completion” or click here.

Goal: determine which shell we operate under and integrate the z script according to the shell environment. Since the z script allows the user to define PROMPT_COMMAND, we should disable integration if a corresponding environment variable is set. Also, we need to determine how we should deal with symbolic links by a corresponding environment variable.

Walkthrough: test if the _Z_NO_RESOLVE_SYMLINKS variable is defined; if not, define the _Z_RESOLVE_SYMLINKS variable to a -P value.

Perform a type [command-name] >/dev/null 2>&1 test to confirm under which shell the script invoked. Since we don't care about the type command output, redirect stdout and stderr to the null device. This will return only the exit status, satisfying the if clause.

We perform this test twice, once against the compctl command name; if it exists, we are in a ZSH shell environment; the second time, we test against the complete command name; if it exists, we are in a BASH shell environment.

For each shell environment, we'll have different autocompletion logic that achieves the same thing and has the same goals, which are:

1. Check if the user sets the _Z_NO_PROMPT_COMMAND environment variable. If he did, it means he's handling PROMPT_COMMAND himself, effectively disabling the ability to add entries on each prompt command.

2. Check if the user sets the _Z_NO_RESOLVE_SYMLINKS environment variable. If he did, then prevent symlink resolution. In the case of the BASH shell, we use the _Z_RESOLVE_SYMLINKS variable, which is set to the string -P if _Z_NO_RESOLVE_SYMLINKS is set.

3. Utilize the _z --add control flow to add new entries and update the rank of existing ones in our data file before each shell command.

   As you might recall from the start of this breakdown (Part 1 - Add data) when we pass the ‘--add’ option to the _z function, it adds a new entry or updates an existing one in our data file (datafile).

   It also recalculates all entries with an updated rank and timestamp. This new list is saved to a temporary file (tempfile), which replaces the old data file (datafile) by renaming itself to it.

4. Utilize the _z --complete control flow to display a list of path completions when doing a shell <TAB> completion.

   As you might recall from the start of this breakdown (Part 1 - Generate completions) when we pass the ‘--complete’ option to the _z function, it prints paths of possible completions derived from our data file (datafile) and the passed query.

   It does this by iterating over all entries in our data file (datafile). If an entry matches the path part of the passed query, it prints it.

Next, we'll dive into the ZSH compctl and BASH complete functions and explore how to achieve our goals with those two autocompletion systems.

Note that the ZSH compctl command is complicated to reason about and work with; the documentation around it is anything but straightforward, and if you genuinely want to understand how it works, you'll have to get your hands dirty, don't be frustrated if you get confused by it, just try to roughly understand what's going on.

If you do get confused, then it's expected; we'll follow by covering the BASH completion system, which will make everything much more straightforward, I'll do my best to explain how ZSH compctl works for our case, but that's only the tip of the iceberg.

Also, note that the ZSH docs recommend using the newer compsys command instead of compctl for modern scripts, so it might not be worth your time to dive any deeper into this subject. Read more.

ZSH hook

# if type compctl >/dev/null 2>&1; then # zsh [ "$_Z_NO_PROMPT_COMMAND" ] || { # populate directory list, avoid clobbering any other precmds. if [ "$_Z_NO_RESOLVE_SYMLINKS" ]; then _z_precmd() { (_z --add "${PWD:a}" &) : $RANDOM } else _z_precmd() { (_z --add "${PWD:A}" &) : $RANDOM } fi [[ -n "${precmd_functions[(r)_z_precmd]}" ]] || { precmd_functions[$(($#precmd_functions+1))]=_z_precmd } }

• ${PWD:a} - Turn a file name into an absolute path. Read more under 14.1.4 Modifiers.
• ${PWD:A} - same as "${PWD:a}" but also resolve use of symbolic links where possible. Read more under 14.1.4 Modifiers.
• '&' - control operator, If a command is terminated by '&', the shell executes the command asynchronously in a subshell. This is known as executing the command in the background, and these are referred to as asynchronous commands. The shell does not wait for the command to finish, and the return status is 0 (true). Read more under 16.1 Simple Commands & Pipelines.
• : [ arg ... ] - This command does nothing, although normal argument expansions is performed which may have effects on shell parameters. A zero exit status is returned. Read more.
• $RANDOM - used outside the subshell to refresh it for the next subshell invocation. Otherwise, subsequent runs of the function get the same value and, if run simultaneously, they may clobber each others' temp .z files. This is due to how zsh distributes RANDOM values when running inside a subshell: subshells that reference RANDOM will result in identical pseudo-random values unless the value of RANDOM is referenced or seeded in the parent shell in between subshell invocations. Read more.
• [[-n string]] - true if length of string is non-zero. Read more.
• precmd[_functions] - 'precmd' is executed before each prompt, it is also possible to define an array by appending '_functions' to 'precmd'. Any element in such an array is taken as the name of a function to execute; it is executed in the same context and with the same arguments and same initial value of '$?' as the basic 'precmd' function, A function found by this mechanism is referred to elsewhere as a hook function. Read more.
• ${array[(r)pattern]} - If the opening bracket, or the comma in a range, in any subscript expression is directly followed by an opening parenthesis, the string up to the matching closing one is considered to be a list of flags, as in ‘name[(flags)exp]’. Reverse subscripting: if the flag (r) is given, the exp is taken as a pattern and the result is the first matching array element, substring or word. Read more.

Goal: invoke a pre-command hook each time a user interacts with the shell environment, this pre-command will add the current path and recalculate all ranks in the data file (datafile) for each item it currently stores.

Walkthrough: Check if the user has set _Z_NO_PROMPT_COMMAND. If not, evaluate the block after the || (OR operator). Otherwise, skip forward and ignore the block.

Check if the user has set _Z_NO_RESOLVE_SYMLINKS. If not, enter the else block; Otherwise, enter the if block.

In both the if and the else blocks, we define a _z_precmd() function that invokes z --add with the current path parameter (expansion over PWD). Depending on whether we're in the if or the else block, this parameter either has symlink resolution modifier :A or without :a. We define it to execute in a subshell asynchronously using the control operator & by terminating the command with it and seed a new $RANDOM value from outside the subshell.

Then, we test that the length of the string result of accessing the precmd_functions array at the location corresponding to the name _z_precmd is non-zero (-n). If non-zero, the hook is already defined, and we can skip the right side of the || (OR operator). If zero, we need to define it by evaluating the right side of the || (OR operator).

The definition of the _z_precmd on the precmd_functions array needs to have an unused array index. To get an unused index, we take the last index by accessing the array's length ($#) and adding + 1, then assign this index to the _z_precmd function.

Now, each time we execute a command, the _z_precmd function will execute and add entries with the current path to the data file.

ZSH completions

_z_zsh_tab_completion() { # tab completion local compl read -l compl reply=(${(f)"$(_z --complete "$compl")"}) } compctl -U -K _z_zsh_tab_completion _z

• local [...] [ name[=value] ... ] - For each argument, a local variable named name is created, and assigned value. The option can be any of the options accepted by declare. local can only be used within a function; it makes the variable name have a visible scope restricted to that function and its children. Read more.
• read [-l...] [ name ... ] - Read one line and break it into fields using the characters in $IFS as separators, except as noted below. The first field is assigned to the first name, the second field to the second name, etc., with leftover fields assigned to the last name. If name is omitted then REPLY is used for scalars and reply for arrays. Read more.
  • -l - This flag is allowed only if called inside a function used for completion (specified with the -K flag to compctl). If the -l flag is given, the whole line is assigned as a scalar.
• ${(f)…} - (f) splits the expanded result at newlines (\n). Read more under Simple word splitting.
• compctl -U -K fn1 fn2 - Control the editor's completion behavior according to the supplied set of options; in our case, fn1 will be invoked and return an array of possible completions after pressing the completion key <TAB> on fn2. Read more.

  • -U - use the list of possible completions to determine whether they match the word on the command line. The word typed so far will be deleted. This is most useful with a function (given by the -K option) that can examine the word components passed to it (or via the read built-in's -c and -l flags) and use its own criteria to decide what matches. If there is no completion, the original word is retained.

  • -K function - call the given function to get the completions. Unless the name starts with an underscore, the function is passed two arguments: the prefix and the suffix of the word on which completion is to be attempted; in other words, those characters before the cursor position and those from the cursor position onwards. The function should set the variable reply to an array containing the completions (one completion per element);

    Reply should not be made local to the function.

    The command line can be accessed with the -c and -l flags to the read builtin from the function.

    For example:

    function whoson { reply=(`users`); } compctl -K whoson talk

    Completes only logged-on users after 'talk.' Note that 'whoson' must return an array, so reply='users' would be incorrect.

Goal: create tab completions for each time we press <TAB> on the z command depending on the string followed by the z command, i.e., when we type the command z foo followed by pressing the <TAB> key, it will make the shell display all possible paths containing foo at their path from the data file (datafile) as suggested completions.

Walkthrough: define a _z_zsh_tab_completion function, which will read the supplied command using the read -l command and assign the result to a local compl variable.

Assign an array of completions to the special variable reply. Whatever array is assigned to replay will be displayed as a suggestion to the user.

Construct the completions array by executing reply=(${(f)"$(_z --complete "$compl")"}); if we break down this assignment, then it can be thought of in the following steps:

1. temp1="$(_z --complete "$compl")"
   • Perform command substitution over the result of the _z --complete command with the compl variable as a string positional argument.
   • The command returns a string containing paths ending with \n.
   • Because the substitution appears within double quotes, word splitting, and filename expansion are not performed.
2. temp2=${(f)$temp1}
   • Split into words by the default value of IFS global variable.
   • The default values of IFS include the newline (\n) character.
3. temp3=($temp2)
   • Store the resulting words from the split in an array.
   • Each item of this array is a word representing a completion path.
4. reply=temp3
   • Assign the array of suggested completion paths to the special variable reply.

Invoke the compctl function with the -U -K _z_zsh_tab_completion options over the _z function to derive completions for it.

Now, each time the user presses the <TAB> key on the alias of _z, which is just z by default, we will run the _z_zsh_tab_completion function.

We extract the positional parameter followed by the z command using the read -l command in the given function.

We get shell completions by assigning an array to the reply special variable in the given function.

BASH completions

# elif type complete >/dev/null 2>&1; then # bash # tab completion complete -o filenames -C '_z --complete "$COMP_LINE"' ${_Z_CMD:-z}

• complete ... [name …] - unique function to bash, it specifies how arguments to each name should be completed. After defining a completion for a specific command, pressing <TAB> in bash on that command will trigger an attempt to perform a completion on the text before point. The actual completion performed is application-specific. Bash attempts completion treating the text as a variable (if the text begins with ‘$’), username (if the text begins with ‘~’), hostname (if the text begins with ‘@’), or command (including aliases and functions) in turn. If none of these produces a match, filename completion is attempted Read more.

  • -o filenames - The comp-option controls several aspects of the compspec’s behavior beyond the simple generation of completions, 'filenames' Tell Readline that the compspec generates filenames, so it can perform any filename-specific processing (like adding a slash to directory names, quoting special characters, or suppressing trailing spaces).
    • (*)compspec - When word completion is attempted for an argument to a command for which a completion specification (a compspec) has been defined using the complete builtin the programmable completion facilities are invoked. Read more.
    • (*)Readline - Command line editing is provided by the Readline library, which is used by several different programs, including Bash. Command line editing is enabled by default when using an interactive shell, unless the --noediting option is supplied at shell invocation. Read more.

  First, the command name is identified. If a compspec has been defined for that command, the compspec is used to generate the list of possible completions for the word

  • -C {command} - command is executed in a subshell environment, and its output is used as the possible completions. When it is executed, $1 is the name of the command whose arguments are being completed, $2 is the word being completed, and $3 is the word preceding the word being completed. When it finishes, the possible completions are retrieved from the value of the COMPREPLY array variable.
    • COMPREPLY - An array variable from which Bash reads the possible completions generated by a shell function invoked by the programmable completion facility (see Programmable Completion). Each array element contains one possible completion. Read more.

• COMP_LINE - The current command line. This variable is available only in shell functions and external commands invoked by the programmable completion facilities. Read more.

Goal: create tab completions for each time we press <TAB> on the z command depending on the string followed by the z command, i.e., when we type the command z foo followed by pressing the <TAB> key, it will make the shell display all possible paths containing foo at their path from the data file (datafile) as suggested completions.

Walkthrough: We initialize bash completions using the complete function. The -o filenames option tells Readline that the possible completions should be treated as filenames and quoted appropriately. This option will also cause Readline to append a slash to filenames it can determine are directories.

Once installed using complete, _z --complete "$COMP_LINE" will be called every time we attempt word completion for the z command (or $_Z_CMD if defined).

In the _z function runtime, $1 will be '--complete,' and $2 will be the expansion of $COMP_LINE, which is the passed string following the z command that will serve as the search query for completions.

BASH hook

# elif type complete >/dev/null 2>&1; then # bash [ "$_Z_NO_PROMPT_COMMAND" ] || { # populate directory list. avoid clobbering other PROMPT_COMMANDs. grep "_z --add" <<< "$PROMPT_COMMAND" >/dev/null || { PROMPT_COMMAND="$PROMPT_COMMAND"$'\n''(_z --add "$(command pwd '$_Z_RESOLVE_SYMLINKS' 2>/dev/null)" 2>/dev/null &);' } }

• grep - The grep utility searches any given input files, selecting lines that match one or more patterns. By default, a pattern matches an input line if the regular expression (RE) in the pattern matches the input line without its trailing newline. An empty expression matches every line. Each input line that matches at least one of the patterns is written to the standard output. Read more.
• <<<word - The word is expanded and supplied to the command on its standard input. Read more.
• PROMPT_COMMAND - If set, the value is executed as a command prior to issuing each primary prompt. Read more.
• command - Runs command with arguments ignoring any shell function named command. Only shell builtin commands or commands found by searching the PATH are executed. If there is a shell function named ls, running ‘command ls’ within the function will execute the external command ls instead of calling the function recursively. Read more.
• pwd - current working directory. Read more.
  • -P - If the -P option is supplied, the pathname printed will not contain symbolic links.

Goal: invoke a pre-command hook each time a user interacts with the shell environment, this pre-command will add the current path and recalculate all ranks in the data file (datafile) for each item it currently stores.

Walkthrough: First, we test whether variable _Z_NO_PROMPT_COMMAND is defined; if not, we evaluate the right side of OR (||).

To invoke _z on every bash prompt command and, as a result, populate and recalculate ranks on the datafile, we need to add _z --add to our prompt command variable PROMPT_COMMAND.

Before we add _z to the prompt command, we first test if “_z --add” string already exists in the current PROMPT_COMMAND variable by using the grep utility with the string "_z --add" as a query over the PROMPT_COMMAND global variable and redirect its resulting stdout to the /dev/null device.

Then, we check if the exit status of grep is zero (we successfully found a match); if it is non-zero, it means the grep utility didn't match our query, and we should continue to evaluate the right side of the OR (||) operator to add _z to the prompt command.

To add _z to the prompt command, we reassign the PROMPT_COMMAND variable to equal the current PROMPT_COMMAND variable to preserve any previously defined commands.

Because each prompt command is separated by a newline, we first concat it with a newline (/n), and add the (_z --add "$(command pwd '$_Z_RESOLVE_SYMLINKS' 2>/dev/null)" 2>/dev/null &); string to it as the command we want to execute.

The command we defined will use the command builtin to run the pwd command to get the current path and pass it as the $2 argument to the _z function.

The pwd command accepts the -P option, instructing it to resolve symlinks; we defined this option over the _Z_RESOLVE_SYMLINKS variable earlier.

If an error occurs during the execution of either the _z --add command or the inner command pwd[-P] command, we discard the resulting stdout to the /dev/null device.

By adding & at the end of the command, we instruct bash to execute the prompt command in a subshell.

At this point, we have achieved the defined goal, which means that each time we execute any shell command, the datafile is recalculated, and new ranks are evaluated per path.

---

Conclusion

At this point we have completed going over all the parts that make the z program functional, this was a long journey and I’m proud of you for completing it, high five yourself, you earned it.

In this deep dive into the z program we covered numerous subjects and utilities crucial for bash programming, we explored the Awk programming language, Bash and Zsh completions systems, Arguments management, Worked with files and explored builtin bash commands. I’m quite confident that by now you have enough tools under your belt to build your own programs and automations.

P.S if you got inspired and wrote a program please share the repo link in the comments section, this will make all the effort put into this article to worth it.

I hope you enjoyed this journey into the depths of bash programing and had as much fun as I did exploring the amazing job the creator and maintainers of z brought to us. If you found any mistakes or have any kind of feedback or just want to say hi and hang around then please leave a comment bellow.

---

Next steps suggestions

• Building your own programs.
• Contribute to the rupa/z repository.
• Deep diving into other bash projects source codes.
• Following community influencers, my personal favorite is Dave eddy from YSAP(You Suck at Programming) Youtube channel.
• Following linux/bash community news sites, just to name a few resources:
  • Dev.to/bash
  • Redit r/bash
  • Linux handbook
  • Itsfoss
  • GeeksForGeeks
• Completing advanced tutorials, Dave from YSAP have great resources on his site though I’m sure you can find many more online resources for specific subjects you are curious about.
• Exploring the zoxide program, it’s inspired by z and adds many integrations and capabilities on top, though it’s written in Rust so it’s not quite related to bash programming.

Final words

I appreciate the effort put into Z by the creators and maintainers. In my personal opinion, those guys are brilliant. I hope both you (the reader) and I can one day craft such a well-thought-out script. I hope that after reading this article, we are at least one step closer.

“Live long, and prosper.” - Spock.