I figured a great way to close out the year 2021 is to wrap up the long awaited clap 3.0 release!

Some major milestones along the way:

Thanks to:

For users who helped us through testing, see our release-candidate changelog and beta changelog

Port Status

Ported

Remaining:

Highlights

Everyone will have their own favorite aspect of the release. For me, they include:

Reducing Gotchas

With StructOpt, I used a arg: Vec<T> and thought I got what I wanted: collecting each flag's value into a Vec (--arg alice --arg bob). What I didn't expect is it also supported an unbounded number of arguments per flag (--arg alice bob). StructOpt's Vec<T> mapped to Arg::multiple. In clap 3, these have been split into separate concepts: multiple_occurrences (what I wanted) and multiple_values (what surprised me). Now, arg: Vec<T> maps to multiple_occurrences though you can enable multiple_values if you want.

There are other examples I've seen as I've supported clap users but I don't remember enough to be able to enumerate them.

StructOpt Integration

StructOpt provides a serde-like declarative approach to defining your parser.

As a user, I didn't mind using structopt as a separate crate. As I've started maintaining clap, I found the integration provided a missing feedback loop to ensure new features weren't just capable of being exposed as a derive macro but fit natural with how people used a StructOpt. Custom help headings were one particular area where there was a lot of iteration.

Custom Help Headings

Most of the CLIs I've made have been for work and have been in Python with argparse. One of the aspects I always missed was being able to categorize the help so I can highlight the important arguments and shove into a corner the unimportant. clap finally has this feature!

#[derive(Debug, Clone, Parser)]
struct Cli {
    #[clap(long, help_heading = "CONFIG")]
    isolated: bool,

    #[clap(long, parse(from_os_str), help_heading = "CONFIG")]
    config: std::path::PathBuf,

    #[clap(short, long, help_heading = "DEBUG")]
    verbose: bool,
}

which produces

$ cargo run -- --help
test-clap 

USAGE:
    test-clap [OPTIONS] --config <CONFIG>

OPTIONS:
    -h, --help    Print help information

CONFIG:
        --config <CONFIG>
        --isolated

DEBUG:
    -v, --verbose

Lessons Learned

I feel like a 4 year release cycle can't be passed up without talking about what could be improved. I can't speak for all of the other maintainer's along the way, so this will be my personal experience and interpretation. I'm sorry if I misattribute anything.

Maintainer Availability

XKCD: Dependency

For most, open source is done on the side and personal obligations take priority. I'm grateful that Foundation sponsors are hiring Rust developers to keep things progressing and that the Foundation and DevX are exploring ways of more sustainable open source.

I'm also grateful the WG-CLI stepped in from time to time to help keep things moving forward. My personal life took over for me, so it originally prevented me from helping as part of these efforts. In working for Futurewei, its opened things up so I could step in and help out while still maintaining my obligations in my personal life. Unfortunately, WG-CLI has also mostly gone into maintenance mode.

Some ideas I want to play with for improving things further:

Avoiding Breaking Compatibility

Earlier in the life of Rust's community, it felt like there was an aura around maintaining compatibility. When the next breaking release is an indefinite time away, it puts pressure on the current release to be "perfect", to slip in every breaking fix possible. There is always something to improve though.

Thankfully we've come to better recognize when we need to avoid breaking compatibility and when it is more acceptable (e.g. if types are so called "vocabulary terms", being used for interop between crates). Its ok to release 3.0 now and push off some of those improvements to 4.0 because its not going to be that far away.

Holding Onto Pets

Even when being willing to break compatibility, it can be too easy to have a pet improvement you want to make but that isn't critical to the release.

We need to be willing to say "not yet". This can be hard. One pet I gave into before I hardened myself against them was getting Arg::help_heading to work well with derive. Once I recognized what I was doing and started watching for it, I found many dear pets that I had to say "not yet" to or "only the minimal amount to unblock this fix". It hurt. I hated seeing an area to improve and passing it up at the risk of forgetting but I also knew that if I gave in, I risked pushing back the 3.0 release further.

Another method for dealing with pets is to timebox them. Its too easy for one improvement to uncover another problem or to introduce a regression. Setting a time for how long you are willing to let this continue and then deciding how much to leave in is an important tool. A recent example of this is that clap 3.0 originally removed a hard-break token in help ({n}) because users of the Builder API could just use \n. The problem is the derive API has a poor-man's markdown parser for detecting hard breaks and it doesn't work well, so people have had to rely on {n}. We looked into our options for a period of time before decided to add {n} back in to give ourselves more time for resolving it.

And finally, there is using feature flags to isolate your pets from impacting a release. We took several features that we felt weren't ready and put them behind unstable- prefixed feature flags.

Long Release Cycles

Several of the others help lead to a long release cycle but having a long release cycle is a problem in of itself.

People keep having ideas and want to keep contributing. Halting development for years would just kill any investment people have in contributing. This can introduce regressions and there isn't the forcing function of a release to make sure these features are polished enough.

To help, we introduced our stablization process with unstable- feature flags with stablization issues.

We also had to take the hard stance of treating master is if we would release it any day, rather than one day. This raised the bar for what we'd accept for contributions.

In the end, as we got to our release-candidate phase, we did introduce a feature freeze but we knew it had a limited time frame (about a month).

Plans for the Future

This will be different to each person though applying Bevy's "focus" goal should lead to some alignment over time.

For me, I feel like clap has let a 1,000 flowers bloom and is ready to rip 999 of them out by the roots. clap has grown organically and has a lot of built-in features. Each new feature makes it harder to discover every other feature (I never knew half of what clap could do until I started contributing). All of these features are also built-in, controlled by runtime flags. This makes it harder for the compiler to identify dead code, requiring compiling and including nearly everything into the final binary. This slows down compile times and bloats binary size.

This doesn't necessarily mean we'll be removing features. There might be some shifting of features where we keep the common case easy but still make less common cases possible. The main focus for this will instead be on making clap more modular. In part, this will be done by splitting out building blocks like a lexer or help generation. These won't do much on their own but enable changes down the road for customizing clap without a flag for every detail like changing out the parser to support different CLI conventions or allowing build-time help generation.

These changes are a step towards making the API open ended; more of a library of tools rather than a framework. Moving more of the validation logic out of clap will also be a step in that direction.