Define a Porter Proposal Process ✅

[carolynvs]

Background

I have been thinking about how I can improve how proposals are handled going forward. Right now, people make proposals either in an issue or on our forum or slack messages, etc, we hash it out there, sometimes making a new issue to track the final set of work, and then start on it.

I’m finding that it’s really hard to engage with these discussions and give people a proper turnabout time. It takes a while to read through them, understand the use cases, potential impact of the change, understand if there are gaps or unintended consequences, etc and that usually ends up looking like no response from me for too long. I think it would help if we created a single process for proposals, so that I can quickly weigh in on “Is this worth pursuing” without implicitly giving people the impression that I have done all the mentioned due diligence and a false impression that it’s approved or ready to start on.

I’m hoping that if we define a process and identity what due diligence needs to happen before it’s finally considered and approved, it would help us make changes more safely, and collaborate on a design.

Below is a suggestion for a PEP process. It would be great if contributors can look it over and let us know if it seems okay or not. Once I get feedback on this, I'll turn this into a document in Porter's repo.

Porter Enhancement Proposal (PEP)

I would like to do a lightweight version of the Python Enhancement Proposal.

• Before submitting a PEP, discuss the idea first with the maintainers and make sure they agree that this is something worth pursuing. This is to make sure someone doesn't waste time starting a PEP when a solution already exists, doesn't have a duplicate and that the general idea is going in the right direction. The best place to submit an idea is our forums (here).
• Have a template for PEPs that outlines what a PEP should look like and the sections it should contain, etc.
• After getting a 👍 from maintainers, the proposer drafts a PEP and submits it to our proposals repository (TBD).
• The PEP is reviewed and iterated upon before being marked "implementable" and then is implemented in Porter by the author(s).

PEP Roles

• Author(s) - Contributors writing the PEP and the implementation. It can be a group effort. 💪
• Owner - Author who is responsible for moving the PEP forward. They submit pull requests to keep the PEP up-to-date, for example updating the status or recording answers or changes to the design, and is the main point of contact for the PEP.
• Reviewer(s) - Maintainer(s) who help get the PEP to match PEP requirements, ask questions and ensure that the PEP has been fully considered before final approval. The particular maintainers depends on the affected areas, scope of change, the lunar cycle, etc. At least one person is assigned to make sure the process moves forward to a resolution.
• Commenter(s): Community members are welcome to join in on the discussion. Ask clarifying questions, provide your own use cases, identify edge cases, etc.

When is a PEP required?

A PEP is helpful when we need to have an agreed upon design, analysis of risk/scope, or evaluation of user experience up-front. Based on that a PEP is required when altering:

• CLI syntax
• Configuration syntax (such as the porter.yaml or config.toml)
• Communication between Porter and mixins/plugins
• Essential behavior of Porter (such as defaults or anything that significantly impacts what a mixin developer, bundle author or end-user needs to know)

In general it isn't necessary for bug fixes, documentation updates, website tweaks, build changes or meta project concerns. A maintainer may ask for a PEP when they feel that a change would benefit from
the PEP process.

PEP Contents

Every PEP should have the following sections:

Each PEP should have the following parts/sections (modified from Python's PEP):

1. Preamble -- Meta-data about the PEP, including the PEP number, a short descriptive title (limited to a maximum of 44 characters), the names of the author(s) and status.

2. Abstract -- a short (~200 word) description of the technical issue being addressed.

3. Motivation -- It should clearly explain why Porter's existing functionality is inadequate to address the problem that the PEP solves and the impacted audience(s) (mixin developers, bundle authors, end-users). PEP submissions without sufficient motivation may be rejected outright. This is the most important part at the beginning and is required before moving forward.

4. Rationale -- The rationale fleshes out the specification by describing why particular design decisions were made. It should
   describe alternate designs that were considered and related work.

   The rationale should provide evidence of consensus within the community and discuss important objections or concerns raised during discussion.

5. Specification -- The technical specification should describe the command and/or configuration syntax and semantics of any new feature.

• If this is a command, we are looking for what the porter COMMAND --help would look like: description of command, arguments, flags, default behavior and error handling.
• If this is a syntax change to a configuration file, define the allowed syntax, at least one example per use case, covering defaults and error handling.
• All PEPs will be reviewed for user experience. So make sure to think about the common use case, how people can accomplish more advanced scenarios, precedence from existing Porter features or other tools in the ecosystem, and how the change fits into Porter workflows and tasks.

6. Backwards Compatibility -- All PEPs that introduce backwards incompatibilities must include a section describing these
   incompatibilities and their severity. The PEP must explain how to deal with these incompatibilities, possibly with defaulting or migrations. PEP submissions without a sufficient backwards compatibility treatise may be rejected outright.

7. Security Implications -- If there are security concerns in relation to the PEP, those concerns should be explicitly written out to make sure reviewers of the PEP are aware of them.

8. Rejected Ideas -- Throughout the discussion of a PEP, various ideas will be proposed which are not accepted. Those rejected ideas should be recorded along with the reasoning as to why they were rejected. This both helps record the thought process behind the final version of the PEP as well as preventing people from bringing up the same rejected idea again in subsequent discussions.

9. Open Issues -- While a PEP is in draft, ideas can come up which warrant further discussion. Those ideas should be recorded so people know that they are being thought about but do not have a concrete resolution. This helps make sure all issues required for the PEP to be ready for consideration are complete complete and reduces people duplicating prior discussion.

PEP Lifecycle

We are using the process from Kubernetes Enhancement Proposal for our status's because they work better for our implementation than Python (Porter isn't a spec like python, so Kubernetes process fits us better).

This is the happy path, where a PEP is accepted.

1. A pull request for the PEP is submitted. The status is PROPOSED.
2. Use the pull request number to assign a number to the PEP. This number helps people quickly find and refer to PEPs.
3. Reviewer(s) are assigned to the PEP.
4. Reviewer(s) work with the author(s) to get the PEP to address all sections. Do not do the implementation at this time but otherwise should have everything else.
5. PEP is merged. The status is PROVISIONAL.
6. The owner notifies the mailing list about the PEP. This is the community's chance to discuss the PEP, ask questions, provide their own use cases, and participate in the design.
7. The owner continues to edit the PEP as new open issues come up, decisions are made and changes to the design are made.
8. After the design has settled, the owner submits a PR to change the status to implementable.
9. Reviewer(s) evaluate the PEP and ensure that due diligence has been followed and all pertinent concerns addressed. If it is acceptable it is merged. The status is now IMPLEMENTABLE.
10. Once the PEP is implementable, author(s) submit a pull request to Porter's repositories that implements the design.
11. Reviewer(s) evaluate the implementation. Beyond determining if the implementation is acceptable, they should use this to identify gaps in the PEP, for example the scope of change was much larger than originally thought, the proposed implementation wasn't actually possible as specified, etc. The pull request may be closed so the author(s) can go back and iterate on the PEP. We should be careful about follow-up pull requests at this stage so that a PEP isn't incompletely implemented, critical changes should be made together. Limit follow-ups to unrelated changes that not relevant to the PEP.
12. When the implementation is merged, the reviewer or owner submits a pull request to change the status to implemented.
13. Reviewer(s) merge the pull request and the final state is IMPLEMENTED.

Resting States

• The PEP may be rejected at certain points by the reviewer. Reviewer should explain the rejection in the active pull request and ultimately close the pull request unmerged. The reviewer submits a pull request updating the status to rejected and reiterate the rejection, including enough context that someone can understand it without going back to old pull requests. The status is REJECTED.
• The owner can withdraw the PEP by submitting a pull request changing the status to WITHDRAWN.
• When progress on the PEP ceases but we aren't ready to reject it, for example people are busy for months and but they would like to come back to it later or we want to do it but now isn't the right time because other things need to be done first, the owner or reviewer can submit a pull request changing the status to DEFERRED.
• Once a PEP has been merged, and a new PEP is started to replace it, or supersede it, a pull request should be opened changing the status to replaced. If the original PEP was implemented, wait to update it to replaced until the new PEP is implemented. For in-progress PEPs, it can be merged immediately. The status is now REPLACED.

[vdice]

This looks great to me.

One question: Where is the PEP status (Proposed, Provisional, etc.) tracked? In the PEP content itself or elsewhere? (In the GitHub PR?)

[carolynvs]

The status would be in the PEP content itself, in the preamble/header. Fixing now.

[iennae]

Two questions

(1) Where does Owner come up in the lifecycle?
(2) When are PEPs the appropriate mechanism? i.e. significant change to user experience, architecture change, development/testing processes, etc.

I think the preamble metadata should include the status for the PEP and refer to lifecycle stages (unless status is tracked elsewhere ?)

In the motivation section, can we add something about the who is impacted by the PEP? This might not be the right section for this, but just thinking "with a community hat on" .. if I'm reading a PEP and it tells me upfront who the impact is for, consumer of bundles, bundle author, etc that may be helpful.

[carolynvs]

Where does Owner come up in the lifecycle?

Oops, thanks yes let me go back and revise to use Owner instead of submitter/author(s) in some places. I'm thinking that even if multiple people work on a PEP, it sure is helpful to have one person responsible for doing tasks like updating the PEP status and essentially representing it, and making sure it is still worked on.

When are PEPs the appropriate mechanism?

I think this is something we need to clearly define for people. Right now Vaughn and I do a lot of what is in a PEP just informally in github issues/comments. Every time we change user-facing areas of porter such as the CLI command syntax or configuration syntax or essential behavior (not just for example error handling for some condition), we answer a lot of the questions that you see in the suggested PEP. Those are times when having an agreed upon design, and analysis of risk/scope up-front is useful. We don't bother doing that for bug fixes, update documentation, website design tweaks, etc. I'll add a section to answer that.

preamble metadata should include the status for the PEP

Agreed! I'll update that.

In the motivation section, can we add something about the who is impacted by the PEP?

Good idea! Will add that too.

[carolynvs]

I have created a repository for proposals. Our first proposal will be the PEP process. 😀 I am working on that today and should have it ready soon.

Here is my PEP to add a PEP process. 😀