Using the System.CommandLine Package to Create Great CLI Programs

Problem Statement

A recent blog post of mine showed a significant performance difference between an async API method and a synchronous one. The performance test was generating 100 new requests each second for 30 seconds, making a total of 3,000 requests.

The async code behaved much better under this load, but at what level of load did this really become true?

I hard-coded both the test to be run as well as the injection rate for the requests.

In this post I want to walk through creating a more flexible CLI experience to run the tests such that we can more easily find the "tipping point" between async and synchronous for my example, and we'll use the new System.CommandLine package to do it.

I'll do this over a few "iterations" so that (hopefully) it's clear how this all fits together.

Iteration 1: Get Started with System.CommandLine

Here's the code for this iteration: https://github.com/dahlsailrunner/async-vs-sync/tree/cli-iteration-1

Here's what we're after in this iteration:

Couple of things to notice in the above:

• The help content is auto-generated. This useful content eliminates the need to go scouring a code base to determine the usage arguments for a console app.
• Note the alternative options for each argument (both --url and -u work for the first option as an example)
• Note the first option is showing as REQUIRED
• Note the second option has a default value of 30

Iteration 1 Code Blocks and Explanations

The code for System.CommandLine usage is basically a set of predictable steps:

• Define options
• Define Command(s) (and Handlers)
• Invoke the RootCommand

Define Options

The CLI we're after right now has two options that we need to define - a base URI option for the URL that our tests will run against, and the injection rate for the requests.

Here's the code that does that:

 1var (baseUriOption, injectionRateOption) = DefineGlobalOptions();
 2//...
 3(Option<Uri> BaseUrlOption, Option<int> InjectionRateOption) DefineGlobalOptions()
 4{
 5    var baseUrlOption = new Option<Uri>(
 6        "--url", "The base URL to test, e.g. https://localhost:7213")
 7    {
 8        IsRequired = true
 9    };
10    baseUrlOption.AddAlias("-u");
11
12    var injectionRate = new Option<int>("--rate", 
13        "Injection rate. Number of new requests to generate each second.");
14    injectionRate.SetDefaultValue(30);
15    injectionRate.AddAlias("-r");
16
17    return (baseUrlOption, injectionRate);
18}

The first line is a top-level statement that calls a a method to define the options (this could have been done in-line but I like to separate things into individual methods for certain things).

The method DefineGlobalOptions returns a tuple containing two options. The code for each option is pretty self-explanatory and some content is likely recognizable based on the screenshot above.

Define Command(s) and Handlers

The CLI app will have a single command (for now) that will be invoked when the program runs and the options are validated. That is defined as a RootCommand which has a description that can be specified, as well as the options that are part of it.

1var rootCommand = new RootCommand("CarvedRock Performance Test CLI")
2{
3    baseUriOption,
4    injectionRateOption
5};

Setting up the handler for the command is done as follows:

1rootCommand.SetHandler(DoSomething, baseUriOption, injectionRateOption);

The above line bears a little explanation. The DoSomething parameter is a method that takes arguments of type Uri and int based on the Option<Uri> and Option<int> parameters that follow it (the Option parameters are of type IValueDescriptor<T>) -- note the generic types of the Option values.

Then the DoSomething method is pretty straight-forward:

1void DoSomething(Uri uriArgument, int injectionRate)
2{
3    Console.WriteLine($"The base URL is {uriArgument}");
4    Console.WriteLine($"The injection rate is {injectionRate} requests per second");
5}

Invoke the RootCommand

The last thing to do for iteration 1 is to actually invoke the RootCommand that we've defined:

1await rootCommand.InvokeAsync(args);

At this point we should have a working app that can be run.

Running the App

To run the app directly from your IDE (Visual Studio, Rider, VS Code) you can specify command line arguments in the profile:

1"profiles": {
2    "CarvedRock.PerformanceTest.Cli": {
3        "commandName": "Project",
4        "commandLineArgs": "-h"
5    }
6}

Replace the commandLineArgs with whatever you want to pass.

Alternatively, you can run it more directly from the command line with either dotnet run or after doing a dotnet publish/build.

If you do a dotnet run you need to be aware that you may have defined some options that may conflict with options in dotnet run.

For example:

1dotnet run -u https://localhost:7213 -r 40   # won't work: -r conflicts with runtime arg
2dotnet run -u https://localhost:7213 --rate 40 # works fine

If you've done a dotnet publish or a dotnet build you can run the command without those conflict worries:

1./CarvedRock.PerformanceTest.Cli.exe -u https://localhost:7213 -r 40   # works
2./CarvedRock.PerformanceTest.Cli.exe -u https://localhost:7213 --rate 40 # also works

Iteration 2: Incorporate the NBomber Async Test

Here's the code for this iteration: https://github.com/dahlsailrunner/async-vs-sync/tree/cli-iteration-2

One of the "real" questions that I wanted to answer with this CLI program was:

At what point does the async code start to fail and/or go above a 1 second response time?

So I wanted to try that as the next scenario.

In this iteration, I changed the name of the DoSomething method to a more meaningful name, and created a helper class for the NBomber code for the load test.

 1rootCommand.SetHandler(RunPerformanceTest, baseUriOption, injectionRateOption);
 2//...
 3void RunPerformanceTest(Uri baseUri, int injectionRate)
 4{
 5    var httpClient = new HttpClient();
 6    var urlFormat = $"{baseUri}Product?category={{0}}"; // category to be provided dynamically/randomly
 7
 8    var scenario = NBomberHelper.GetScenario("ASYNC requests", 
 9        urlFormat, injectionRate, httpClient);
10
11    NBomberRunner.RegisterScenarios(scenario)
12        .Run();
13}

My usage of NBomber has been explained in that recent blog post, but the source code for this slightly-changed logic is in the NBomberHelper.cs file.

I'm running a single scenario with this, and the scenario name and the path to be hit will always be on the async version of my API (we'll come back to that in the next iteration).

But with the iteration two changes in place, we can run a variety of performance tests against the async API to see where it starts to break:

 1# works fine
 2./CarvedRock.PerformanceTest.Cli.exe -u https://localhost:7213 --rate 400 
 3
 4# still works fine
 5./CarvedRock.PerformanceTest.Cli.exe -u https://localhost:7213 --rate 800 
 6
 7# starts slowing down - some responses almost 3 seconds
 8./CarvedRock.PerformanceTest.Cli.exe -u https://localhost:7213 --rate 1600 
 9
10# slower still, and some "connection refused" errors
11./CarvedRock.PerformanceTest.Cli.exe -u https://localhost:7213 --rate 2000 

It wasn't until I hit a full 2,000 requests per second that I started getting "connection refused" errors. Super impressive, and the CLI helped me quickly determine that without me having to update code!

Iteration 3: Add a Sub-Command for the Sync Tests

Here's the code for this iteration: https://github.com/dahlsailrunner/async-vs-sync/tree/cli-iteration-3

To test the syncrhonous version of the API without losing support for the async version, I opted to use the "sub-command" functionality of System.CommandLine.

There are certainly more than one way to do this, but it gave me the opportunity to explore sub-commands and also show how they work.

For this iteration, I've moved the async performance test into a sub-command and created a second sub-command for the synchronous test.

I want the URL and iteration rate options to be options available to both of the commands. So here's the new code for the RootCommand and its two sub-commands:

 1var rootCommand = new RootCommand("CarvedRock Performance Test CLI");
 2rootCommand.AddGlobalOption(baseUriOption);
 3rootCommand.AddGlobalOption(injectionRateOption);
 4
 5var asyncCommand = new Command("async", "Run load test against the async API endpoint");
 6asyncCommand.SetHandler((baseUri, injRate) =>
 7    RunPerformanceTest(baseUri, injRate, "ASYNC scenario", "Product?category={0}"),
 8    baseUriOption, injectionRateOption);
 9rootCommand.AddCommand(asyncCommand);
10
11var syncCommand = new Command("sync", "Run load test against the synchronous API endpoint");
12syncCommand.SetHandler((baseUri, injRate) =>
13        RunPerformanceTest(baseUri, injRate, "SYNCHRONOUS scenario", "SyncProduct?category={0}"),
14    baseUriOption, injectionRateOption);
15rootCommand.AddCommand(syncCommand);

I've added the two options as "Global Options" on the rootCommand.

Then I create both the asyncCommand and the syncCommand, set the handler to a method invocation with some parameters for each of them, and add the new sub-commands to the rootCommand with calls to rootCommand.AddCommand().

The method invocation is a little different now - since I want to pass some additional parameters to the method, I can use a lambda expression when calling SetHandler. The initial arguments (baseUri, injRate) will correspond to the two Option values (IValueDescriptor<T> as described above).

The updated RunPerformanceTest method is as follows:

 1void RunPerformanceTest(Uri baseUri, int injectionRate, string scenarioName, string apiRoute)
 2{
 3    var httpClient = new HttpClient();
 4    var urlFormat = $"{baseUri}{apiRoute}"; // category provided dynamically/randomly
 5
 6    var scenario = NBomberHelper.GetScenario(scenarioName,
 7        urlFormat, injectionRate, httpClient);
 8
 9    NBomberRunner.RegisterScenarios(scenario)
10        .Run();
11}

Instead of hard-coded relative paths and scenario names now, the method accepts them as parameters which get specified with different values from the SetHandler calls.

And with those sub-commands in place, the help text now looks like this:

Now to run the tests you can run commands like these examples (the first one runs the synchronous test and the second runs the async one):

1./CarvedRock.PerformanceTest.Cli.exe sync -u https://localhost:7213 --rate 60
2./CarvedRock.PerformanceTest.Cli.exe async -u https://localhost:7213 --rate 200

Expansion Thoughts

CLI programs can come in very handy. Here are some ideas for further experimentation if you're interested.

• Package the CLI app as a dotnet tool, as described here
• Model some of the other NBomber load simulation techniques as options (not just injection rate)

Happy coding!