We are excited to (pre-) announce clap 4.0! This release focuses on removing deprecated APIs and finishing what we couldn't do without breaking changes. For more details see the CHANGELOG (including the migration guide) and the documentation.

This release builds on work done in the 3.x releases and can be worth catching up on them:

To put all of this work into numbers:

Baseline2.34.03.0.03.2.214.0.0-rc.14.0.0
Builder API Surface174245282165 [1] [2]166
Lines of Code613,46217,30824,04420,653 [1] [3]20,839
Code size218.2 KiB487.0 KiB609.3 KiB605.5 KiB544.3 KiB [1] [4]542.6 KiB
Runtime7.529 us14.544 us14.657 us8.2478 us [5]7.6455 us

(see Methodology for more details)

Aside: Yes, those clap v2 -> v3 numbers are not good.

For Builder API Surface, it is understandable when you consider we mostly didn't remove functionality in v3 but deprecated it, removing it in v4.

Lines of Code is mostly accounted for with the merge of structopt into clap as clap_derive. We continued to have significant growth after that as we continued to develop replacement features for functionality in clap. These more general solutions take up more lines though not more code size.

For code size and runtime, one factor is that things fell through the cracks during clap v3's development. clap's development went dark for an extended period of time and went through several maintainers. This isn't to say one of the maintainers is at fault but that things get lost in hand offs. Towards the end, we double-downed on just getting out what we had and hadn't looked to see how we compared to v2.

For code size, it looks like it was a lot of small changes that added up, like changing a VecMap to a BTreeMap.

For runtime, it seems to mostly be a single feature that caused it which was removed in v4 [5].

Our plan is to give about a week window between the release-candidate and the official release to allow for collecting and processing feedback.

What Changed

As a caution, this will be a mix of high-level and low-level details as we try to show how each change impacts not just functionality but several different performance metrics to help understand the benefits of some features and the costs of others.

Most people will care about:

Some more specific changes:

You can also skip ahead to

Since 4.0.0-rc.1, the biggest change is that we made the new actions behave like the original clap 3 actions when multiple occurrences are used. See Issue #4261 to further details and to discuss this. See GitHub Releases for more details.

Support Policy

Before getting into the details, I want to assure you that we understand the effort required to deal with breaking changes. We both maintain CLIs using clap and have to deal with updating 500+ tests for clap. We are trying to balance the needs of clap users for whom clap is already good enough with those who want to keep using clap but are discouraged by the build times, binary size, or some missing features. I'm hoping that the improvements we made will help justify the changes.

With clap 3.0, we adopted a deprecation policy to help developers migrate to new versions. After feedback from developers using clap, we adjusted that policy soon after clap 3.2 so that deprecations are opt-in (via the deprecated feature flag) so you can choose the timetable of when to respond to them without extra noise from the deprecations. We've also improved clap_derive to avoid deprecations being reported for code it generates on your behalf.

The next change is that we are officially supporting old major versions:

VersionStatusSupport
v4activeFeatures and bug fixes target master by default
v3maintenanceAccepting trivial cherry-picks from master (i.e. minimal conflict resolution) by contributors
v2deprecatedOnly accepting fixes for ecosystem-wide critical bugs
v1unsupported-

We have not decided on when we will end-of-life each version; this is an experiment that we are iterating on as we go. This policy is trying to balance the need for developers to upgrade at their own pace with the burden for maintainers to support old versions.

See also our CONTRIBUTING.md

Polishing Help Output

BeforeAfter
screenshotscreenshot
screenshotscreenshot

During clap 3.0.0 development, we were looking at removing AppSettings::ColoredHelp, making it always on. This helped identify several problems with our existing colored help and started a broader re-examination of our --help output. For more background on each decision, I recommend you check out the parent issue

Problems with the colored help

In the short term, 6079a871, 8315ba3d, e02648e6 starts us off with a more neutral color palette by focusing on bold/underline, rather than colors. Longer term, we need to allow users to customize the colors (#3234) and provide their own colored output (#3108). We plan to focus on this during clap 4.x.

We then extended the styling to the usage line as well as the general structure of the help output. This required dropping textwrap and included clean up that helped offset the extra code size from the extra formatting logic. This happened in 2e2b63fa..a6cb2e65

A frequent point of confusion for users is that -h shows an abbreviated help and --help shows a detailed help. rg and bat include messages in the output explaining the distinction. I realized we could bake this directly into the description for -h and --help. This happened in c1c269b4

When getting user feedback on --help, I noticed that their usage statement was meaningless, something like prog [OPTIONS] [ARGS]. While showing everything can be too much (see git -h), showing too little negates the reason for the usage in the first place. We've changed the usage to never collapse positional arguments into [ARGS] as that seems to strike a nice balance. Longer term, we should provide ways for users to force certain args to not be collapsed or to allow specialized usages per case in an ArgGroup. This happened in f3c4bfd9..a1256a6e

We also

More Specific Derive Attributes

We've added support for new derive attributes, deprecating the old ones.

Before:

/// Simple program to greet a person
#[derive(Parser, Debug)]
#[clap(author, version, about, long_about = None)]
struct Args {
    /// Name of the person to greet
    #[clap(short, long, value_parser)]
    name: String,

    /// Number of times to greet
    #[clap(short, long, value_parser, default_value_t = 1)]
    count: u8,
}

After:

/// Simple program to greet a person
#[derive(Parser, Debug)]
#[command(author, version, about, long_about = None)]
struct Args {
    /// Name of the person to greet
    #[arg(short, long)]
    name: String,

    /// Number of times to greet
    #[arg(short, long, default_value_t = 1)]
    count: u8,
}

As we look forward to some planned clap_derive features (e.g. #1807, #1553 ), the use of a single #[clap(...)] attribute is limiting. In addition, we have seen users frequently confused by how the derive and builder APIs relate ( #4090 ). We are hoping that by migrating users to #[command(...)], #[arg(...)], and #[value(...)] attributes, code will be clearer, the derive will be easier to use, and we can expand on the capabilities of the derive API.

This happened in 2f4e42f2..ba5eec31 and 20ba828f..f5138629 and as they are focusing on a code generating, they should have negligible impact on code size either way.

This also provided an opportunity for us to do some much needed cleanup of clap_derive, including

num_args

The new num_args defines how many command-line arguments should be consumed as values, per occurrence:

et cmd = Command::new("prog")
   .arg(Arg::new("file")
       .action(ArgAction::Set)
       .num_args(2)  // single value
       .short('F'))
    .arg(Arg::new("mode")
        .long("mode")
        .num_args(0..=1))  // range of values
        .default_value("plaid")
        .default_missing_value("slow");

Previously, clap has had several ways for controlling how many values will be captured without always being clear on how they interacted, including

These have now all been collapsed into Arg::num_args. Most changes happened between ee06707c..179faa6b

See Issue 2688 for more background.

Removing Deprecated Features

A lot of deprecations were merely the removal of functions. Others involved changing defaults, percolating the effect throughout, and simplifying the code as a result.

The most fundamental changes were the introduction of

Removing deprecated APIs were done in 16bd7599..017b87ab (#3963) and 017b87ab..ee06707c

This is in addition to the Builder API Surface shrinking dramatically, making it easier to discover and use the features that clap provides.

Reducing Code Size

Along the way some more specific actions were taken to reduce code size.

The first was to move off of IndexMap, IndexSet, HashMap, and HashSet to a custom built FlatSet and FlatMap. clap does not deal in big numbers and can get away with just using a Vec and manually enforcing uniqueness by iterating over it. FlatSet codifies that approach. FlatMap takes the same approach but uses separate Vecs for the keys and values.

Benefit

This happened in b344f1cf..084a6dff

I tried doing the parallel-Vec approach for our formatted output (Vec<(Style, String)>) but it wasn't an immediate win, so I dropped that effort as it didn't look like there was enough to optimize about my proof-of-concept to get a noticeable win.

Some of our increases in binary size this release came from being more detailed in our help formatting. Code that previously wrote to a String now worked with our formatted output (Vec<(Style, String)>). My hope is we can re-work this data structure in a future 4.x release. To see how much this will help reduce binary size, I switched to a straight String when the color feature flag is disabled.

This happened in e75da2f8

Next, we split the formatting of errors out and wrapped it in a trait, allowing more naive implementations or just not rendering any details at all. This is an example of using traits for reducing size rather than throwing in more feature flags which can be harder to maintain and harder for a developer to discover and use well.

This happened in 956789c3..d219c69c. An application that switches from the "rich" to the "null" formatter dropped code size by 6 KiB. This won't show up in our usual metrics as this is opt-in and the example we are using for measuring does not opt-in.

When adopting StyledStr, we also needed some behavior that textwrap didn't seem to provide, so we re-implemented a subset that fit our needs (incremental wrapping).

This happened in 37f2efb0

In cleaning up clap_derive, we found some redundant generated code that we cleaned up in 0b5f95e3. Our usual benchmark doesn't cover this case. I ran another one and didn't see any noticeable change, so congrats to the compiler optimizing it?

Looking for another way to break down where all of the code size for clap comes from, I split out auto-generated help, auto-generated usage, and error-context features.

This happened in 94c5802a..c26e7fd6

Breaking clap down by feature:

Removing Lifetimes

In the most common case, a developer never sees that clap borrows data or deal with the lifetimes on the data types. You just use the builder or derive and it is all taken care of for you and is fairly fast with low code size.

However, when you do notice them they can be a pain to deal with. Consider these use cases

When doing this, we wanted to use newtypes to allow

To keep things convenient for the common case, we updated our APIs to accept impl Into<Str> and impl Into<OsStr>. This gave us the opportunity to re-evaluate APIs that accepted a &'static str instead of a &'static OsStr or had separate str and OsStr versions to clean this all up.

The use of impl Into<Str> did run into problems with the few places we had impl Into<Option<T>> for resetting of fields because None was ambiguous on what type it was coming from, so we created the IntoResettable trait to resolve these ambiguities.

a5494573..5950c4b9

e81b1aac..c6b8a7ba

d4ec9ca5..956789c3

We originally accepted String in the API and had a perf feature flag to control whether you prefer code size or runtime performance. In looking back at these numbers, the cost of accepting Strings in clap's API seemed too high for the how often this feature would be used. We instead shifted it so clap accepts only &'static str by default and the string feature flag changes the API to accept Strings as well, restoring most of the code size and runtime performance from before we made any of these changes.

1365b088..6dc8c994

As I mentioned, we have to support some fields being resettable. We've made this more consistent by allowing nearly all fields to be reset in ee387c66..7ee121a2

Removing Implicit Version/Help Behavior

clap 3.0.0 tried to make --version and --help flags as easy to modify as possible

All of those automatically show help output when present which you could disable, making it a regular flag.

Magic, while making some cases easy, also makes it difficult to divine the rules and force it to do what you want. This had come up several times in supporting developers but Issue #3405 was the tipping point for re-thinking how we do this.

Now, things are simplified down to

Most changes happened between 0129d99f..6bc8dfd3

Specifically f70ebe89 is where almost all of the parse speed gains happened between clap v3 and clap v4, returning clap back to clap v2's performance.

Storing Ids for ArgGroup

Something that can easily be missed in clap is that when we are parsing and store a value in ArgMatches, we also store that value in all ArgGroups that the arg is a part of.

So we've changed things up and now for each occurrence (not value) of an argument, we are storing its arg Id in the ArgGroup, reducing the number of allocations involved (since an Id might not even be on the heap) and making it easier to make programmatic decisions about the argument.

This happened in 41be1bed

Introspecting on ArgMatches

A fairly regular request is for iterating over the arguments in ArgMatches, whether for re-organizing the data by order of arguments for commands like find or for layered configuration. This was blocked on how we were storing argument Ids. During the development of clap 3.0, Ids were switched from a string to a hash of the string. This removed the need for a lifetime on ArgMatches, reduced memory overhead, and opened the door for allowing alternative Id types in the future. The problem is that it is a one way process (can't get a string back from the hash) and the way the traits were laid out it made it error prone (easy to accidentally look up a hash of a hash).

After re-examining the alternative Id types feature request (#1104), we decided to not move forward with that route. This meant we could go back to strings, especially with our previous lifetime changes, and make it fairly ergonomic to allow people to enumerate the arguments that were parsed.

This happened in 7486a0b4 though it was dependent on a batch of the lifetime work mentioned earlier. See the lifetime-removal work for metrics.

Non-bool Flags

Sometimes you want a flag to convey something more specific than true/false, like a --https flag tracking the relevant port:

let cmd = Command::new("mycmd")
    .arg(
        Arg::new("port")
            .long("https")
            .action(clap::ArgAction::SetTrue)
            .value_parser(
                value_parser!(bool)
                    .map(|b| -> usize {
                        if b { 443 } else { 80 }
                    })
            )
    );

let matches = cmd.get_matches_from(["mycmd", "--https"]);
let port = matches.get_one::<usize>("port").copied();
assert_eq!(port, Some(443));

Originally, this was possible in clap_derive but not with the builder API. While the value_parser / ArgAction APIs brought a lot of features from clap_derive to the builder API, this is one area where it regressed.

I had noticed we could re-work SetTrue's semantics to reuse existing machinery within clap to accomplish this:

This happened between 5950c4b9..e81b1aac

Fixing Parsing for Hyphenated Values

clap supports parsing values that look like flags, whether they are numbers or flags to forward to another command. This initially applied to all arguments in a Command with AppSettings::AllowLeadingHyphen. A more Command-wide specialization was added with AppSettings::AllowNegativeNumbers. Then we got an argument-specific ArgSettings::AllowHyphenValues. Unfortunately, the latter missed some cases in AppSettings::AllowLeadingHyphen. Developers were finding the Arg-specific one first and running into road blocks. We updated the now Arg::allow_hyphen_values to operate like the Command version.

In working on this, it gave us an opportunity to re-evaluate this API. Normally, a single Arg needs to accept hyphen values, especially numbers, and it can lead to unexpected behavior to enable it on all Args via a Command setting. We've now mirrored Command::allow_negative_numbers to Arg and deprecated all of the Command variants of these settings. We then did similar for a closely related setting, Command::trailing_var_args which brings it inline with its related Arg::last.

This happened in f731ce70..f97670ac

Looking Forward

More immediately, I plan to focus on an experiment with how we style the output. clap has had several challenges in supporting more customization

For the first, I've started work on anstyle (ANSI Styling) for providing a common definition of ANSI styles with a minimal API so it is safe to put in public APIs (once it hits 1.0).

The second is because of a fundamental design restriction in existing terminal styling crates; they couple together styling with terminal capabilities. I'm going to experiment with an alternative route where the user styles their output with their preferred ANSI styling crate and then the output stream adapts it as needed, whether that means stripping the escape codes when piping to a file or turning them into wincon API calls for older Windows versions. The biggest risk is the performance hit when needing to adapt the output. My hope is that the cost will be acceptable for most applications.

In addition to this being a low maintenance way of accepting styled output, my hope is that this will simplify clap's rendering code to get the same size benefits as our optimization for StyledStr without the color feature which saved 15 KiB in code size.

Speaking of code size, while we've made reductions during clap 4.0, our work is still not done ( #2037, #1365 ). Seeing the success of clap v4, we are optimistic that a more open API design will continue to reduce code size while making clap more flexible.

How Can I Help

First, we welcome any feedback on what is going into clap 4.0.0 which is why we are pre-announcing so we have opportunities to re-evaluate based on feedback. It can be difficult to weigh feedback on Issues as the audience tends to be small and self-selecting to those who agree, so we recognize the need for finding ways to remove that bias and collect feedback from a wider audience.

Second, we would love for you to share what is great or what doesn't work well with other CLI parsers so we can learn from all of the best and avoid NIH syndrome. For example, I've started digging into click's documentation to see what works well and how it might apply to clap. This has already led to value_parser!'s design being made extensible so we could support features like FileReader and FileWriter with value_parser! and clap_derive.

Lastly, of our help-wanted Issues, the one I'm most excited for is #3166 as it will make completions more featureful, less buggy, and more consistent across shells.

Methodology