Using Bash Local History Mode

Taking a cue from the famous Bash Stride Mode post, I want to introduce a powerful tool that I have shared with multiple engineers, all of whom loved it.

  • Have you ever forgotten what your exact command line flags were for a long-running command?

  • Did you ever want to turn an interactive bash session into a script after the fact but could not remember every single command you ran, what order you ran them in, and what directories you ran them in?

I think machine learning engineers and scientists can particularly relate to the first bullet point: when tuning hyperparameters, it is an enormous headache to forget what hyperparameter values gave you specific losses in multiple-hours-long training runs. In fact, there are entire projects and even companies built to try to make tracking experimental runs and reproducing results easier! Check out Databricks’s MLFlow announcement to learn more about this state of affairs. As two personal data points, when I was at LinkedIn, there was an entire team of 10-20 engineers (depends on how you count) under the title “Productive Machine Learning”, and my friends at Deepgram also rolled their own thing a few years ago (<10 headcount startup) called Kur.

Build and devops engineers can hopefully relate to the second point. As a C++-Machine Learning hybrid, I spend a lot of time doing multi-language builds and projects.

While I understand that companies try to build a lot of infrastructure, when I’m working with clients, working across heterogenous clusters without sudo, or just trying to get something done, I prefer an 80/20 hack. I do not know where this script originated, but I would also like to thank the amazing Chris Dyer for introducing me to it.

Introducing bash local history

This is it:

# output '0' if no matches (not using -q because PIPEFAIL might not work)
useHistory() {
    egrep -v '^top$|^pwd$|^ls$|^ll$|^l$|^lt$|^dir$|^cd |^h$|^gh$|^h |^bg$|^fg$|^qsm$|^quser$|^cStat|^note |^mutt|^std ' | wc -l | tr -d ' \n'
owner() {
    ls -ld "${1:-$PWD}" | awk '{print $3}'
lasthistoryline() {
    history | tail -1 | sed 's:^ *[0-9]* *::g'
    if [[ `owner` = "$USER" ]] ; then
        local useful=`lasthistoryline | useHistory | tr -d '\n'`
        if [[ $useful != 0 ]] ; then
            # date hostname cmd >> $PWD/.history
            ((date +%F.%H-%M-%S.; echo "$HOST ") | tr -d '\n' ; lasthistoryline) >>.history 2>/dev/null
addPromptCommand() {
    if [[ $PROMPT_COMMAND != *$1* ]] ; then
        if [[ $PROMPT_COMMAND ]] ; then
            PROMPT_COMMAND+=" ; $1"
        export PROMPT_COMMAND
# convenience cmd for searching: h blah => anything matching blah in current local history
alias h='cat .history | grep --binary-file=text'

### to enable,
# Append to your ~/.bashrc or ~/.bash_profile
# source ~/
# addPromptCommand localHistory

To enable this, paste the code into a file like ~/, and then add the following two lines to your bashrc:

source ~/
addPromptCommand localHistory

What it does

This file registers a “prompt command”, which is a callback function that runs every time you hit “enter” at the bash command line. In this case, localHistory() will be called every time, which appends your just entered command and a timestamp to a file called .history in your current working directory.

There is only one case I think this could cause issues: Latency appending to the the end of the file if you are using a file system exposed via a slow network. I have not found this to be a problem despite having worked in multiple clusters. If it is for you, contact me and maybe I can help you automatically disable appending on slow disks.

In my case, in one of my experimental directories for my local checkout of kaldi, the history script looks like this:

dgalvez@host:~/dgalvez/kaldi-git/egs/mini_librispeech/s5$ tail .history
2018-12-29.03-29-23. PATH="/usr/bin/:$PATH" ./local/chain/tuning/ --stage 3 2>&1 | tee run_cnn_tdnn_1a_past_stage_3.log
2018-12-29.03-29-23. PATH="/usr/bin/:$PATH" ./local/chain/tuning/ --stage 3 2>&1 | tee run_cnn_tdnn_1a_past_stage_3.log
2019-01-01.13-01-10. local/chain/ --stage 14 --train-stage 0
2019-01-01.13-03-21. local/chain/ --stage 14 --train-stage 0
2019-01-01.13-04-40. du -h nvprof_first.log
2019-01-01.13-05-02. gzip nvprof_first.log
2019-01-01.13-05-09. du -h nvprof_first.log.gz
2019-01-01.13-25-07. qlogin  -now no -l 'hostname=b1[12345678]*|c*,gpu=1'

You can do some archaeological work to see what scripts I ran exactly at a given time, and what analysis I did afterward (in this case, I compressed a log from NVIDIA’s nvprof tool, to download to a local machine for viewing, after running a training script for time delay neural networks on the mini_librispeech dataset), while focused on that current directory. I no longer fret about a colleague asking me how to reproduce something, because I can always figure it out ad-hoc on-demand.