Project Lifecycle: Annual Review Process

[Naomi-Wash]

Comment moved from Project Lifecycle Document Section 4. Annual Review Process

The TAC shall develop an annual review process to determine whether projects are in the stage that accurately reflects their needs and goals.

Discussion

Not directly a comment on the doc, but validation thereof.
PQCA currently has two projects.

• OQS ~ Production/Growth? (moving towards Impact?)
• PQCP ~ Production/Incubation ?
  We need to understand the lifecycle of a project overall. However within each project there may be more subtlety around parts of the project.
  For example oqs has some implementations of more experimental algorithms (Mayo) but others that are more stable and being adopted (Kyber/ML-KEM). Some code is aimed at production, but doesn't meet many criteria (oqs-provider).
  So it would seem there is a responsibility for each subproject to further clarify the lifecycle of individual components/features along similar lines

Absolutely agree on the need for correct sub project classification. Input solicited in https://github.com/open-quantum-safe/tsc/issues/2.
I absolutely do not agree on your implied classification of oqs-provider (but again, this may be due to a difference in understanding of the quality criteria for "production"): What about making them explicit here and/or in the above TSC issue?

This classification would be the responsibility of the OQS TSC, not the overall TAC.

Should we add it as text to make clear the responsibility distribution between TAC and TSCs?

I think that would be at a higher level in the document, as the split between TAC and TSC is a recurring theme?
TODO: Find place in text & update this comment

Explaining (and ideally, agreeing) that "split of responsibility" would make an awful lot of sense in my eyes: I only got involved in this discussion on the TAC as it seems to be the place that IBM wants to use to wield the control over the project I so consistently object against since their take-over proposal a year ago wrt OQS (basically, defending the "O" in the name): No objection against influence via technical contribution; but corporate control over a FOSS project via committees and definitions (e.g., of what is "healthy") in my eyes is "not Good" (harsher word deleted :).

[baentsch]

OQS ~ Production/Growth? (moving towards Impact?)

This classification proposal is too optimistic and would raise wrong expectations by users if published as such.

This is to urge PQCA to act responsibly and carefully wrt this classification: Particularly OQS and PQCP are security software. Under no circumstances should PQCA entities overly "optimistic" or controlled by marketing-oriented people following their corporate agenda(s) be allowed to set externally visible "lifecycle" expectations contradicting the positions of responsible (also sub project) maintainers of such software, if they see things more conservatively.

Also here, please tag me explicitly as/if progress is made: I have not subscribed to this project and otherwise probably would not get notified as my preference is to only work on technical GH issues. In this case, though (the resolution of) this issue is one that can have impact on my personal reputation (e.g., if I don't withdraw visibly and quickly enough from projects that PQCA markets too aggressively).

[planetf1]

The lifecycle document talks about providing two examples of growth/production, and uses terms like 'production usefulness'

I don't think the definition is very tight, and that the way forward is for projects - particularly the more mature ones (ie liboqs) to make a statement they are comfortable with about what this means -- and indeed there's been some discussion in the community.

Transparency over the characteristics of the code/docs/testing are a large part of this so that any consumer can review and make their own decision as to whether that means they can use in production.

It also implies some responsibility within the liboqs community to critically assess where they are, and figure out sensible ways forward to improve (as the community has discussed, and is very aware of some of the things they'd like to do). Ideally any high line items would feature in a roadmap, again providing insight to consumers.

I think that's in the spirit of the lifecycle categorization - words like 'production' and 'support' need to be treated carefully and we need to be clear it's the consumer's responsibility to assess.

It might be interesting to review what other subprojects state - particularly smaller communities with great ideas but limited resources!

[maximilien]

I believe this is solved in other issues. Let me suggest that we proactively close this issue by the next TAC meeting (Oct.9th) unless I hear otherwise..

[baentsch]

we need to be clear it's the consumer's responsibility to assess.

Can you please point to the section in the document making this clear? Doesn't this statement void the whole point and purpose of the lifecycle document giving users guidance as to the quality of the software?

we proactively close this issue

Can you please explain what this practically means? Do you expect (further?) input or will you just close it?

[maximilien]

As per discussion in the TAC call I am closing this issue since it was agreed that we will do an annual review next year as we approach one year. The actual date for the review shall be decided in March as we will celebrate one year anniversary.

[baentsch]

The procedure showcased here troubles me: IBM's @maximilien asks for input by the OSS community (in June 2024), gets it, pushes resulting questions to GH without answering for months and then just closes them (November 24) pointing further "down the road" (February 25)... Again tagging @SWilson4 for background information/pointer to public information about the (rationale stated in the) discussion mentioned above.

And just for completeness regarding the input by @planetf1: You make correct references to ongoing discussions but your summary statement leads the whole PQCA life-cycle document in my opinion pretty much ad absurdum (hence I thought feedback is not warranted -- but now that the issue got closed, it's required, I think):

If I understand it right, PQCA wants to position this project lifecycle document exactly as an external "quality indicator" to people who cannot assess the quality properties of the projects/their software. So this statement sounds pretty wrong:

we need to be clear it's the consumer's responsibility to assess

If PQCA pushes the responsibility for assessment of project/software quality to the user, what's the purpose of a project/software quality lifecycle documentation?

Or phrased differenty, if PQCA is willing to put this statement

it's the consumer's responsibility to assess [the project/software quality]

at the very top of the PQCA lifecycle documentation, I'm fine with closing all issues around this document as any classification based on this document then clearly cannot be misunderstood as a "quality recommendation" by PQCA: Tagging @brian-jarvis-aws (as the probable editor of the document): Would that be OK for you?

[hartm]

@baentsch I think the core of your argument is a good one, and it's something we should probably clarify. The lifecycle document is focused on the community of a project and the procedures the community follows, and assessing those. It is not about assessing the code quality. Now, often times these are correlated, but of course in some cases they are not. It would be totally fine to add something to the top of this document saying the focus of the document is on community structures, diversity, and processes, and not on the quality of code itself, and that users should do their own due diligence before using code in production applications.

In other words, the lifecycle document tries to indicate whether or not the right people and processes are in place for a project to succeed, but is clearly not a substitute for a security audit or any other kind of code analysis.

Would a statement like this near the top of the document satisfy you? If so, do you want to draft one?

Open source software would never work if the maintainers and contributors were held directly responsible for their code being used in production deployments; in fact, this point came up in the early versions of the EU's cyber resiliency act, and such provisions that might make maintainers liable were eliminated.

[baentsch]

@hartm Thanks for your willingness to clarify the purpose of the document. Indeed, a significant part of my concern is undue user expectations forced upon all OQS contributors and maintainers by PQCA statements like

[...] the PQCA project lifecycle policy was explicitly designed to support research implementations as well as production code, and we have both research tracks and production tracks for codebases. This dual-track approach will let OQS and its contributors to be able to continue to be leaders in research as well as move towards ironclad production code

--> This puts "ironclad production code" in direct relation to the lifecycle document, and probably not just for me implying the latter ensures the former (which also warrants a definition, imo).

Therefore, indeed a clarification of this document's purpose would be goodness to avoid misleading users.

A proposal for a possible disclaimer on page 1:

This document is meant to provide project-internal guidance and tracking only as to whether the PQCA sub projects have a chance to succeed in their internal classification areas. In no case should any classification according to this document be understood to infer actual utility, let alone quality properties of the software: Any such quality considerations are explicitly disclaimed by the projects, their contributors and maintainers and are the responsibility of each user of the software to validate before use.

But again, I'm not a native speaker nor an LF nor TAC member, so I'm leaving this to @SWilson4 who kindly agreed to contribute time to the TAC.

In general, I'd be grateful if all PQCA publications would try to be considerate of the obligations (user expectations) they put on this software (and its creators) that it has not been created for and cannot fulfill -- certainly not without significant improvements in code and community support.

Finally, when re-reading this blog post, I admit to having also overlooked so far the following:

we have both research tracks and production tracks for codebases

Could you please clarify what you mean by this, @hartm ? I am not aware of this being the case for the OQS codebase: There is no technical distinction between pure research code (e.g., oqs-demos, oqs-provider, SHBS, CROSS in liboqs) and code aiming to be more robust (e.g., ML-KEM in liboqs). There's just an ongoing discussion how to possibly/eventually achieve this. But maybe such codebase separation exists for PQCP?

Or to phrase it differently, if the wording were

we have both research tracks and production tracks for PQCA projects

the statement would be more in line with reality/purpose of the lifecycle document as it becomes clearer now: Would you be OK changing the blog post along that line, @hartm ?

@SWilson4 : Would you be willing to lift all relevant comments in this discussion to one or several new (open/openly discussed and) suitably focused issue(s), e.g., "Clarify lifecycle document purpose", "Revise blog post to be in line with lifecycle document purpose", "Define concrete steps to deliver on PQCA promise for ironclad production code" (or also change that wording), and carry these forward and to conclusion?

[hartm]

@baentsch Happy to clarify!

This puts "ironclad production code" in direct relation to the lifecycle document, and probably not just for me implying the latter ensures the former (which also warrants a definition, imo).

I'm happy to take out ironclad if you think that term is problematic. Is that the main problem you have with that phrase?

A proposal for a possible disclaimer on page 1:

This document is meant to provide project-internal guidance and tracking only as to whether the PQCA sub projects have a chance to succeed in their internal classification areas. In no case should any classification according to this document be understood to infer actual utility, let alone quality properties of the software: Any such quality considerations are explicitly disclaimed by the projects, their contributors and maintainers and are the responsibility of each user of the software to validate before use.

How about the following:

"This document is meant to provide project-internal guidance and tracking only as to whether the PQCA sub projects have the necessary community and infrastructure to succeed in their internal classification areas. In no case should any classification according to this document be understood to imply properties of the software: any such quality considerations are explicitly disclaimed by the projects, their contributors, and maintainers and are the responsibility of each user of the software to validate before use."

The main point is to emphasize that the document focuses on community, not software properties. What we want to impress upon folks is that (a) not all good communities write secure software but (b) you probably shouldn't trust software that doesn't have a good community behind it at all.

Or to phrase it differently, if the wording were

we have both research tracks and production tracks for PQCA projects

the statement would be more in line with reality/purpose of the lifecycle document as it becomes clearer now: Would you be OK changing the blog post along that line, @hartm ?

I use "projects" and "codebases" interchangeably in almost all of my writing, and treat them as almost exact synonyms in many cases. This is probably slightly imprecise, but it makes the flow of writing much smoother when you don't have to use the word "project" repetitively. If it really bothers you, I'm happy to change it though.

[baentsch]

I'm happy to take out ironclad if you think that term is problematic.

I think this would be goodness: In my eyes this term sounds like an unfulfillable promise: Can you give examples of software that is ironclad or of an actionable definition of that term for software, @hartm ? I don't think you meant to write this text only for marketing people that will assume/understand this is a term without real meaning, right?

Is that the main problem you have with that phrase?

It's the most significant. The former part ("continue to be leaders in research") I'm also not sure as being accurate any more, see IETF-Hackathon/pqc-certificates#165 (comment) or the start of the discussion to integrate openssl PQ code into liboqs rather than the other way around but that's surely a claim without practical consequences if found wrong (unlike "ironclad" security software that isn't).

Your suggested changes to my proposal seems to carve out the project-internal classification towards utility from the disclaimer, right? My goal with the original formulation was to make clear that also the distinction into research and product use ("utility") is a purely PQCA internal one and nothing (in terms of utility) should be inferred from that either. IMO the goal should be to avoid anything that could mislead a reader into thinking this is "production quality code": All this documents is that someone outside chose to use it in that way and PQCA therefore changed a classification label for internal processes. If you don't agree with this line of thinking, please let me know. If the problem with my wording is just one of use of English, maybe @SWilson4 has another suggestion.

I use "projects" and "codebases" interchangeably [...] If it really bothers you, I'm happy to change it though.

That'd be good I think, exactly because it is imprecise, again: A code base (probably not just to me) is software, a project is the processes around the software. This life cycle document only covers the processes but not the code. Hence, mixing the two for "smoothness" is in my opinion not good: lacking precision leads to corners being cut -- which I deem not good for security software. Again, I am at heart an engineer who wants things precise; a marketing person of course may prefer "smoothness" such as to win over more people. But if such imprecision establishes wrong assumptions, it's going to bite back eventually.

[hartm]

I think this would be goodness: In my eyes this term sounds like an unfulfillable promise: Can you give examples of software that is ironclad or of an actionable definition of that term for software, @hartm ? I don't think you meant to write this text only for marketing people that will assume/understand this is a term without real meaning, right?

We'll get it taken out.

Fundamentally, that blog post is targeted towards business people, though. We would always welcome more technical content, though, if people want to write that sort of article for the blog.

Your suggested changes to my proposal seems to carve out the project-internal classification towards utility from the disclaimer, right? My goal with the original formulation was to make clear that also the distinction into research and product use ("utility") is a purely PQCA internal one and nothing (in terms of utility) should be inferred from that either.

In this context, I read utility to mean "what [something] does". The research and product projects would do similar things at a high level, but they just would have different levels of maturity. That's why I reworded and took out the word "utility". If you want to reword this again, please feel free to do so, but I wouldn't use the word "utility" to mean "(high) quality"; at least Americans will probably not understand this clearly.

That'd be good I think, exactly because it is imprecise, again: A code base (probably not just to me) is software, a project is the processes around the software. This life cycle document only covers the processes but not the code. Hence, mixing the two for "smoothness" is in my opinion not good: lacking precision leads to corners being cut -- which I deem not good for security software. Again, I am at heart an engineer who wants things precise; a marketing person of course may prefer "smoothness" such as to win over more people. But if such imprecision establishes wrong assumptions, it's going to bite back eventually.

Again, this is business-focused text, but this is a completely reasonable change that won't affect the overall flow. We'll make it.

[baentsch]

Again, this is business-focused text, but this is a completely reasonable change that won't affect the overall flow. We'll make it.

Thanks for these improvements, @hartm. I never understood (good) business to necessitate imprecision in the way it words things -- to the contrary, clarity in my experience very much supported the pursuit of common business goals.

at least Americans will probably not understand this clearly.

Thanks for this advice. In that case, allow me to ask @SWilson4 for advice as to how to phrase things to be clear and understandable also for more people.