metrics.ltx

\hypertarget{metrics-archetypes}{}
\unnumberedsection{Signs of Health: Metrics and Archetypes}\label{metrics-archetypes}

The purpose of metrics, in the context of the archetypes, is to tell
you whether your investments are having the desired effect.

Metrics are only meaningful when you start from your
goals\footnote{See section \fullref{goals}, p. \pageref{goals}}.  What
are you trying to maximize, what is your investment so far, and are
you seeing the kinds of results you want?  It's rarely a question of
``Are we doing a good job or a bad job?''.  Rather, it's ``Are we
doing the job we meant to do, and how can we do it better?''

For example, your goal is to create an ecosystem that spans an
industry, with a Multi-Vendor Infrastructure project, then an
important metric to watch is what companies are contributing to the
project --- not just who, but the frequency and depth of their
involvement, how often third-party organizations work with each other
as peers in the project without the founding organization's
involvement, etc.

Compare that with a Wide Open project, where long-term stability is a
key goal.  There, signals of health would be diversity of
contributors, reduction in onboarding time, rate of non-employee
contribution, etc.  If one of your goals in the project is to use open
source involvement to develop a hiring pipeline for new talent, that
could affect how you approach onboarding: a broad but shallow pool may be
good enough.  Broad and deep sounds even better of course, but it also
takes more investment and may not be worth the cost.

The point of metrics is not to \emph{identify} the archetype --- an
archetype is more a target one aims for than a reality one discovers
--- but rather to tell if the project is succeeding at its goals.  A
secondary purpose is to get advance notice of forces that may cause
the project to evolve from one archetype to another.  If you're
expecting certain metrics but seeing different ones, it could indicate
that the project is transitioning to, or possibly from, some other
archetype.\footnote{See \fullref{transitions},
  p. \pageref{transitions}.}  Whether this is desirable or not depends
on your goals.

Because archetypes overlap, and because a given metric usually applies
to multiple archetypes, we treat metrics as their own unified topic
here, rather than organizing them by archetype.  The combinatorics of
looking at every archetype with every metric would be prohibitive, so
instead we just mention the archetypes for which a given metric is
especially relevant.

These metrics below are in no particular order, and are intended to be
used creatively --- it might be that, for your particular project,
variations of some of these would be most useful:

\begin{itemize}

  \item \textbf{Ratio of one-off inquirers to repeat inquirers.}

    For \textit{Mass Market}, expect very high; for \textit{Wide
      Open}, expect high.

  \item \textbf{Duration of organizational participation, and ``weight''
  (of contribution and influence) per organization.}  Duration might
    need to be measured relative to market cycles.

    \textit{B2B}: look for these metrics to be higher for the
    organizational partners you care most about.

  \item \textbf{Ratio of large change requests to small / one-off change
  requests.}

    \textit{B2B}: look for this ratio to be higher than in other types
    of projects.

  \item \textbf{Ratio of organizationally-motivated changes to
  individually-motivated ones.}

    This ratio should be higher for \textit{B2B} and
    \textit{Multi-Vendor Infrastructure} projects than for other
    archetypes.

  \item \textbf{Existence of customized forks / vendor branches.}

    Long-lived customized forks are more common, and less worrisome,
    in \textit{B2B} projects than in other types of projects.

  \item \textbf{Rate of conversion of bug reporters to code
  contributors.}

    Higher in \textit{Specialty Library}, \textit{Controlled
      Ecosystem}, and \textit{Upstream Dependency} than in other types
    of projects.

  \item \textbf{Frequency of appearance as topic of conference
  presentations, hackathons, etc.}

    Higher in \textit{B2B} and \textit{Multi-Vendor Infrastructure}
    projects than elsewhere.

  \item \textbf{Ratio of lead-organization contributors to external
  contributors.}

    In \textit{B2B} and \textit{Multi-Vendor Infrastructure}, a high
    proportion of contributors should be coming from the organizations
    that are most involved (probably even more so in the former than
    the latter).

  \item \textbf{Ubiquity of use within a defined industry area.}

    If a \textit{B2B} project is not widely-used in the business area
    in which it is applicable, it is unlikely to be meeting its goals
    (unless it has a very unusual goal set).

  \item \textbf{Packaging by OS distributions; frequency of being
  offered as a pre-defined add-on by cloud vendors.}

    Look for these in \textit{Multi-Vendor Infrastructure}, and
    perhaps in \textit{B2B} depending on the nature of the project.

  \item \textbf{Migration of functionality from core to plugins.}  

    This is a special case of increasing modularization; it should be
    especially common in \textit{Controlled Ecosystem} projects.

  \item \textbf{Frequency with which a representative from the lead
    organization (``Benevolent Dictator Organization'' / ``BDO'')
    participates in discussion, especially with respect to discussions
    that lead toward a decision.}  

    Look at both ratio across all discussions and ratio within each
    discussion thread.  By ``discussions'' here, we include issue
    tracker threads.

    For \textit{B2B}, \textit{Rocket Ship To Mars}, \textit{Upstream
      Dependency}, and \textit{Trusted Vendor}, this frequency should
    be high.

    For \textit{Controlled Ecosystem}, same, but only for core development, not
    for plugins.

    For \textit{Multi-Vendor Infrastructure}, if it's very high for
    one organization, that might be a sign that the project isn't as
    MVI as you thought.

    For \textit{Wide Open} and \textit{Mass Market}, this metric is
    not particularly relevant.  It could be high or it could be low,
    but it doesn't necessarily tell you much either way.

  \item \textbf{Organizational diversity of responders.}

    In \textit{Multi-Vendor Infrastructure}, again watch out for one
    organization doing most of the responding.  One use of these
    metrics is to be able to tell a project --- perhaps a project that
    doesn't want to hear it --- that although they are aiming for MVI,
    they're actually B2B or something else.

    In \textit{Trusted Vendor}, \textit{B2B}, \textit{Rocket Ship To
      Mars}, \textit{Specialty Library}, expect low.

    In \textit{Wide Open}, \textit{Mass Market}, and
    \textit{Controlled Ecosystem}, expect high.

    In \textit{Upstream Dependency}, this metric probably doesn't tell
    one very much.

  \item \textbf{Diversity of participants and contributors along several axes:}

    \begin{itemize}
      \item Nationality (also, location --- not necessarily the same)
      \item Native language
      \item Profession
      \item Gender
      \item Ethnicity
      \item Paid contributor vs voluntary
    \end{itemize}

    These are of special interest to \textit{Wide Open} and
    \textit{Mass Market} projects.  An argument --- or perhaps several
    distinct arguments, depending on the archetype --- can be made for
    their importance to other archetypes too.

  \item \textbf{Number of discrete interactions from a newcomer's
  arrival to them making a contribution (and thence to becoming a
  committer, perhaps).  Relatedly, how many arrivals drop somewhere
  along the way, and exactly where.}

    In a sense, this measures speed of onboarding, but with
    interactions instead of calendar time as the clock tick.  We
    define an interaction as participation in a distinct thread
    (including an issue tracker thread).

    The reason to measure by interaction instead of by calendar time
    is that calendar time can vary for reasons that are entirely about
    the individual and not about the project.  For example, someone
    who is busy and not focused on this project might only interact a
    few times per year while still eventually getting to the point of
    being a regular contributor, while someone who has more time or
    more motivation might do the same much more quickly.

    This \emph{may} be independent of archetype, and it may be the
    first of a whole class of metrics that are useable as signs of
    health but that don't necessarily correlate strongly with any
    particular archetype.  We welcome feedback on this question and on
    the need for deeper discusison of cross-archetype metrics.

\end{itemize}