tl;dr; til
is a wrapper around a few other tools to simplify the management of a TIL repo, such as
https://github.com/pjambet/til. I recently published it as a ruby gem.
The code can be found on GitHub.
Intro
At first glance this small CLI might not seem like it does much, and it really doesn’t, but while building it I ended learning quite a few things, three to be exact:
- How to use an external command line tool, such as
fzf
, and feed it input, similar to using a unix pipe, likeecho "a\nb\nc" | fzf
, as well as reading the output of the command, but programmatically, in Ruby. - How to implement a flow similar to what happens when you type
git commit
without the-m/--message
option and it prompts you with an editor,vim
by default. - How to use the Gitub API to create a new commit, without using the
git
cli.
Before jumping in, here’s a summary of what til
actually does:
- It first loads the list of all existing categories in your TIL repo, and then uses
fzf
to prompt you to pick the category for your new TIL. You can also choose to add a new category. - Once you picked a category, it uses your default editor, as configured through the
$VISUAL
or$EDITOR
environment variables, orvi
if none of those are defined. You can then type what you actually learned today. - After saving and closing the text editor,
til
will grab the content of the file, and commit it to the configured GitHub repo - It also takes care of maintaining the
README.md
file so that it contains a nicely organized index of all your TILs. It keeps a list of all the categories at the top, and includes a link to each TIL below, grouped by category
Using fzf
I don’t actually use it that much, but I love how fzf
improves CLI interactions. I thought it would be a great
addition for the workflow I wanted with til
. Most of the time I will be reusing existing categories, and as someone
who makes a lot of typos, I always look for ways to avoid having to type anything.
The main way that fzf
gets its input in from STDIN. You can test that from the command line: echo "a\nb\nc" | fzf
will load fzf
with three items a
, b
& c
.
I was able to get the list of folders using the GitHub API ruby library, Octokit. Each folder is a category,
and I needed to feed that to fzf
, through STDIN, in Ruby.
It’s worth mentioning that I could have used the backtick approach, but that was a tiny bit too much magic for me
and also, there are few blog posts out there that recommend against using it, and favor the system
method. In this
case we’re not really dealing with user input, so it doesn’t matter that much from a security standpoint.
The Process.spawn
method accepts a bunch of options, there are a lot, but these are the ones that are
interesting to us here:
:in : the file descriptor 0 which is the standard input
:out : the file descriptor 1 which is the standard output
So, we can create a pipe
with IO.pipe
and pass the reader as the :in
argument to spawn
, so that we
can write with the writer from the main process and the process on the other end, the one created by spawn
will
receive it.
Note: it is important to close the writer
in the initial process, if you don’t, fzf
still thinks that there
might be more to read, and it shows it by displaying a spinner in the bottom left corner of the terminal.
We can use the same approach to get the content that fzf
will output. Once a selection is made, fzf
writes it to
STDOUT. So we create another pipe, and give the writer as the :out
option, the process started by spawn
will
be able to write to it and we can then use the other end of the pipe to read from it and get the selection from the
user in the main process.
This is what til
does, as you can see on GitHub.
A quick look at the C code defining the backtick function shows that it uses the pipe
syscall, so we’re
essentially reimplementing something fairly similar to what ruby does for us with `echo 'a' | fzf`
.
Note (another one): As I was writing this post, I realized that Ruby has another method related to spawning new
processes and dealing with STDIN and STDOUT: IO.popen
. I haven’t looked too much into it yet, but it looks like
it could simplify my code a little bit. That being said, the overall approach described above is still valid.
Using an external editor
For years I used git
from the cli and relied on its commit workflow without really wondering how it actually made that
happen. In case you’re not familiar with it, from the CLI, when you commit with git commit
you essentially have two
options, you either provide the commit message inline, through the -m/--message
option, or you leave this option blank
and git
opens an editor for you, by default vim
.
What is actually really cool with this is that you don’t have to use vim
, you could use pretty much any other editors,
that being said, you probably want to pick one that is quick to start so you don’t have to wait just to type a commit
message. It’s a little bit trickier with editors using a dedicated window such as vscode or macvim. GitHub’s
documentation has a page explaing how you can use the most common visual editors as git editors.
So, as a git cli enthusiast, I wanted to replicate the same worflow: you picked a category, now let’s write the content, in the editor you like using, so you can format things the way you want.
It turns out that it’s apparently “the right approach” to first look at the VISUAL
environment variable and then at
EDITOR
as explained here and there.
Reimplementing the git commit workflow turned out to be very little work, you first create a file, Ruby makes that easy
with the Tempfile
class, we then use either system
or spawn
with the value in $EDITOR
or $VISUAL
(we
default to vi
, just in case, so we have something).
The main difference between spawn
and system
here is how they return, system
does not return until the process is
over, whereas spawn
returns a pid. Since we basically want to wait for the user to close the editor, system
is
easier, with spawn
we would have had to use waitpid
to wait.
Once the child process is done, we can read the content of the file, and VOILA! We have the content, formatted by the
user, in their favorite editor (unless they ended up in vi
and couldn’t figure out how to exit).
You can see that til
does exactly what I just described
Creating a commit using the GitHub API
And now, the final piece of the puzzle, we have a category and a string representing a new TIL, it’s time to create a new commit on GitHub. The GitHub API must have an easy way to do this right?
Well?
You can do it! But is it easy? I’ll let you answer on your own.
The new commit needs to contain two changes, the new file we want to create, but also, and this is really one of the reasons why I wanted to create this tool in the first place, the updates to the README file, to keep the table of content and the links up to the date with the new TIL.
This blog post was really helpful but the fact that it didn’t include any code examples means that I spent a few hours (ðŸ˜) trying to get things workings, here’s a summary of what I’m doing to create a new commit, I hope you’re ready:
- Get the
ref
of the latest commit on master - Get the commit object, with the sha obtained in the previous step
- Get the tree object, based on the latest commit obtained in the previous step
- Get the readme content
- Create a blob for the new file - Interesting to note that a blob does not have a path, we only specify the path when adding blobs to a tree
- Get all blobs that are in the current tree, except the README - This is necessary because we don’t want the new tree to remove any files, so our new tree needs to contain all the old files, plus the new file, plus the updated readme, and we definitely don’t want the old README
- Figure out what the content of the README should be - We add a new category at the top if necessary and then we add a link to the new TIL at the bottom, keeping the categories sorted alphabetically, and the entries sorted, oldest at the top, newest at the bottom. The code is U (pause) GLY, but it works!
- Create a new blob, for the updated readme
- Add the new blob (at the correct path) and the blob for the updated readme to the blobs list
- Create a new tree with all blobs, all the old one, unchanged plus the new file and the updated readme
- …
- CREATE A NEW COMMIT, FINALLY
- You thought we were done? Nope, we have to update master to point to the new commit, and now, we’re done
You can find documentation about the API endpoints I’m using on the following pages:
See it in action
Conclusion
The current version (0.0.4) is very very basic, but it gets the job done, and I’ve been using it for a few days already. I have a few thoughts about what I would like to do next:
- A “real” cli, probably written in go, so that it’s easier to distribute, with
brew
for instance. There doesn’t seem to be any formulae/casks namedtil
, so I should hurry up! - A chrome/firefox extension, so you can do the same without leaving your browser
- Improve the code (if you’ve looked at it, it’s … far from great, really far)
- Show a terminal spinner at the end when creating the commit, since it can take up to a few seconds. I recently learned how to use terminal escape sequences to do this! Read more on my TIL repo (see what I did there?!). But there are also at least two gems that do that for you, here and there.
Questions? Comments? Hit me up on Twitter!
Existing gems
There are a few similar gems, both in names and features available on ruby gems, I checked all of them before publishing mine:
til_cli, there were two issues with this one:
- It does not seem to work with latest ruby 2.7.1 because of json 1.8.1 not compiling
- It runs git commands locally, it basically wraps
git
calls, which is great, but means that the tool is not really a standalone tool, I wanted something I could run from anywhere, and that would work on its own.
todayilearned: This is an interesting gem, but it works locally with sqlite, and while this is great, this is not what I wanted
til: This seems to the same code as the one above, they both point to this GH repo: https://github.com/ip2k/todayilearned
til_: This looks like an empty gem, there’s no “real” code in there as of today.