Testing a command-line tool with plain Ruby

Recently I published Qc, a tool for interacting with QuantConnect using the command-line. It lets you develop your algos in your local box, without having to open QuantConnect web editor to run them.

I wanted to write some system tests that invoked the command line tool and checked the results. As long as I can keep them fast, I love system tests that test the real thing from top to bottom.

It uses the QuantConnect API under the hood. It is HTTP-based, so I could use VCR and not having to mock the API connection for testing purposes. I love VCR, so this was an easy choice.

For testing commands execution, I first tried to use aruba. I had learned about it in this book and I knew RSpec used it for its tests. Unfortunately, I found a blocking issue: for VCR (or any mocking approach) to work, I needed to use the in_process approach, but this mode wouldn’t work with running commands interactively. I needed to exercise how the user interacted with the tool for a couple of cases, so this invalidated aruba for me.

Looking for an alternative, I ended up just using plain Ruby and writing a few simple helpers for my needs:

• For testing the output, MiniTest’s capture_io lets you capture $stdout and $stderr into strings.
• For testing the input, you can just stub the methods you need in $stdin. See this article.

The resulting helper was very simple and allowed me to write pretty expressive tests like this:

  def test_init_asks_for_project_and_store_it_in_settings_with_default_extension
    sign_in

    type_when_prompted '2', '' do
      run_command 'qc init'
    end

    assert_match /My first C# project/, last_command.output

    assert_stored_project_settings project_id: '799895', file_extensions: 'cs,py'
    assert_equal 0, last_command.exit_status
  end

For dealing with files, I liked aruba’s approach of creating a fixtures directory with files you can use to prepare your tests. Ruby support for scripting with files is wonderful, so it was easy to prepare another helper that allowed me to write tests like this:

class PushTest < SystemTest
  def setup
    prepare_local_files 'file_1.cs', 'file_2.cs'
    ...
  end

  def test_initial_push_should_push_all_the_files
    run_command 'qc push'
    assert_match /Uploading file_1.cs.../, last_command.output
    assert_match /Uploading file_2.cs.../, last_command.output
    assert_files_were_uploaded 'file_1.cs', 'file_2.cs'
  end
  ...
end  

I remember that in my early days with Ruby and Rails I was willing to use whatever gem that more or less matched my needs. These days I fight a lot before adding new libraries. This was a good reminder to check how vanilla works for you before adding third-party solutions.