Request for Comments: 3774 Nortel Networks
Category: Informational May 2004
IETF Problem Statement
Status of this Memo
This memo provides information for the Internet community. It does
not specify an Internet standard of any kind. Distribution of this
memo is unlimited.
Copyright Notice
Copyright (C) The Internet Society (2004). All Rights Reserved.
Abstract
This memo summarizes perceived problems in the structure, function,
and processes of the Internet Engineering Task Force (IETF). We are
attempting to identify these problems, so that they can be addressed
and corrected by the IETF community.
The problems have been digested and categorized from an extensive
discussion which took place on the ’problem-statement’ mailing list
from November 2002 to September 2003. The problem list has been
further analyzed in an attempt to determine the root causes at the
heart of the perceived problems: The result will be used to guide the
next stage of the process in the Problem Statement working group
which is to recommend the structures and processes that will carry
out the corrections.
Table of Contents
1. Introduction: Issues/Problems in the IETF Process . . . . . . 2
1.1. Consequences of Past Growth . . . . . . . . . . . . . . 3
1.2. The Aim is Improvement, not Finger-pointing . . . . . . 4
1.3. Perceived Problems - Consensus on Solutions . . . . . . 4
2. Root Cause Problems . . . . . . . . . . . . . . . . . . . . . 5
2.1. Participants in the IETF do not have a Common
Understanding of its Mission . . . . . . . . . . . . . . 5
2.2. The IETF does not Consistently use Effective
Engineering Practices . . . . . . . . . . . . . . . . . 7
2.3. The IETF has Difficulty Handling Large and/or Complex
Problems . . . . . . . . . . . . . . . . . . . . . . . . 9
2.4. Three Stage Standards Hierarchy not properly Utilized . 11
2.5. The IETF’s Workload Exceeds the Number of Fully
Engaged Participants . . . . . . . . . . . . . . . . . . 12
2.5.1. Lack of Formal Recognition . . . . . . . . . . . 13
2.6. The IETF Management Structure is not Matched to the
Current Size and Complexity of the IETF . . . . . . . . 13
2.6.1. Span of Authority . . . . . . . . . . . . . . . 13
2.6.2. Workload of the IESG . . . . . . . . . . . . . . 13
2.6.3. Procedural Blockages . . . . . . . . . . . . . . 15
2.6.4. Consequences of Low Throughput in IESG . . . . . 15
2.6.5. Avoidance of Procedural Ossification . . . . . . 15
2.6.6. Concentration of Influence in Too Few Hands . . 16
2.6.7. Excessive Reliance on Personal Relationships . . 17
2.6.8. Difficulty making Technical and Process Appeals. 18
2.7. Working Group Dynamics can make Issue Closure Difficult. 18
2.8. IETF Participants and Leaders are Inadequately Prepared
for their Roles . . . . . . . . . . . . . . . . . . . . 19
3. Security Considerations . . . . . . . . . . . . . . . . . . . 20
4. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . 20
5. References . . . . . . . . . . . . . . . . . . . . . . . . . . 21
5.1. Normative References . . . . . . . . . . . . . . . . . . 21
5.2. Informative References . . . . . . . . . . . . . . . . . 21
6. Editor’s Address . . . . . . . . . . . . . . . . . . . . . . . 21
7. Full Copyright Statement . . . . . . . . . . . . . . . . . . . 22
1. Introduction: Issues/Problems in the IETF Process
Discussion started in the second half of 2002 has shown that a
significant number of problems are believed to exist in the way the
Internet Engineering Taskforce (IETF) operates. Before attempting to
change the IETF procedures and rules to deal with these problems, the
IETF should have a clear, agreed-upon description of what problems we
are trying to solve.
The Problem Statement working group was chartered to create this
document, which contains a description of the problems, and to use
this analysis to suggest processes to address the identified
problems.
Taken in isolation, this document may appear to be exceedingly
negative. The IETF needs to refresh its management and processes to
address today’s challenges, but it should not be forgotten that the
IETF has produced a large body of high quality work which has lead to
an extremely successful and pervasive network infrastructure.
Against this background, we should see the current document as a
necessary piece of self-criticism leading to renewal and continued
success. The discussion of the positive aspects has been
deliberately confined to the IETF Problem Resolution Processes
document [5] which considers the core values that the IETF needs to
maintain whilst correcting the problems that participants perceive as
affecting the IETF at present.
The raw material for this document was derived by summarizing the
extensive discussions which initially took place on the ’wgchairs’
mailing list and subsequently on the ’problem-statement’ mailing list
from November 2002 through to September 2003, incorporating
additional input from relevant drafts published during this period
(see [2], [3] and [4]), and the minutes of recent plenary
discussions. This produced a list of perceived problems which were
classified into a number of related groups using a classification
suggested by the processes which go on in the IETF.
This document has digested these perceived problems into a small set
of root cause issues, and a short list of subsidiary issues which
appear to be the most pressing items engendered by the root cause.
This list is set out in Section 2.
Section 1.1 gives a short explanation of the thinking that has taken
place in coming to the current view of the root causes.
The original summary of perceived problems has been posted to the
Problem Statement Working Group mailing list so that it can be
referred to in future. Note that it remains classified according the
original scheme so that the raw data is available if alternative root
cause analysis is needed.
1.1. Consequences of Past Growth
As the problems of the IETF were examined, it became clear that they
are neither new nor are they symptoms of a problem which is novel in
the science of organizations.
The IETF started off as a small, focused organization with a clearly
defined mission and participants who had been working in this area
for a significant period of time. Over the period 1989-1999, the
IETF grew by a factor of ten or more in terms of number of
participants, and volume of work in progress. The effects of this
growth have been compounded by the extension of the scope of the IETF
which makes the work much more varied. Also during this period, the
Internet has become more complex and the requirements placed on it by
a far larger user community have changed as the network has come to
have a pivotal role in many areas of life.
Many of the problems and symptoms appear to be fundamentally caused
by the organization failing to fully adapt its management structure
and processes to its new larger size and the increased complexity of
the work. The IETF has also failed to clearly define its future
mission now that the initial mission has been completed or outgrown.
These failures are just those that afflict many small organizations
trying to make the transition from a small organization, which can be
run informally and where essentially all participants fully share the
aims, values, and motivations of the leadership, to a medium sized
organization, where there are too many participants for informal
leadership and later arrivals either do not fully understand or have
a different perception of the ethos of the organization.
Some IETF participants have been aware of these issues for a long
time. Records dating back to at least 1992 drew similar conclusions.
1.2. The Aim is Improvement, not Finger-pointing
Many of the problems identified in this memo have been remarkably
persistent over a 15-year period, surviving a number of changes in
personnel. We see them as structural problems, not personnel
problems. Blame for any of the perceived problems should not be
directed to any individual. The sole aim of this review process is
to identify how the IETF can improve itself so that it knows what it
is about and becomes fit for that purpose in the shortest possible
time frame.
1.3. Perceived Problems - Consensus on Solutions
The working group participants emphasize that both the long list of
problems and the root cause issues that were derived from them are
problems that are believed to exist by a significant constituency,
either on the mailing list and/or in private discussions. We also
note that many of these problems appear to be of long standing, as a
very similar list has survived from the discussions in the first
POISED working group that took place prior to the IETF organizational
changes approved in 1992.
We, in line with many contributors to the mailing list, believe that
it is important to try and identify what appear to be the root causes
of the perceived problems, but trying to prioritize or assign a
relative importance to the problems would not be useful: rough
consensus on an unordered list of real and important root causes will
be sufficient. The root causes identified will provide a guide in
setting up the processes needed to resolve the problems: the
perceived problems can be viewed as multiple symptoms of the root
causes which should provide input to those trying to resolve the
problems in achieving consensus on solutions.
2. Root Cause Problems
This section forms the heart of this analysis, and lists the issues
which we believe lie at the core of the problems. Apart from the
first issue which is fundamental, the problems are not necessarily in
priority order, but they will be seen to be interlinked in various
ways.
2.1. Participants in the IETF do not have a Common Understanding of
its Mission
The IETF lacks a clearly defined and commonly understood Mission: as
a result, the goals and priorities for the IETF as a whole and any
Working Groups (WGs) that are chartered are also unclear.
The IETF needs to understand its mission in the context of the
greatly increased scope and complexity of the Internet, and the
changing requirements of the much larger user community that the
success of its previous work has engendered.
The lack of a common mission has many consequences, of which the
principal ones appear to be:
o The IETF is unsure what it is trying to achieve and hence cannot
know what its optimum internal organization should be to achieve
its aims.
o The IETF cannot determine what its ’scope’ should be, and hence
cannot decide whether a piece of proposed work is either in-scope
or out-of-scope.
o The IETF is unsure who its stakeholders are. Consequently,
certain groups of stakeholder, who could otherwise provide
important input to the process, have been more or less sidelined
because it has seemed to these stakeholders that the organization
does not give due weight to their input.
o Working Groups can potentially be hijacked by sectional interests
to the detriment of the IETF’s mission.
o The misty vision has inhibited the development of roadmaps that
would inform the IETF’s stakeholders of our longer term
intentions, as well as restricting the associated architectural
views to an outline top level view which does not fully reflect
the developing nature of the Internet. It would be desirable to
have roadmaps and architectural views for portions of work which
extend beyond a single working group: it may also be the case
that it is no longer possible to fit the whole Internet within a
single architecture.
o The IETF is unable to determine explicitly what effect it desires
to have in the marketplace, and is therefore unable to determine
what requirements of timeliness are appropriate when planning work
and setting expectations for stakeholders which will further the
IETF’s mission.
o The lack of precision regarding our goals leads to WG charters and
requirements that are poorly thought out and/or not aligned with
the overall architecture. The resulting poorly defined charters
are a major factor in poor quality and/or late deliveries from
some WGs and the total failure of other WGs.
o The IETF needs to avoid focusing on a too-narrow scope of
technology because this would be likely to blinker the IETF’s view
of ’the good of the Internet’, and will harm the long-term goal of
making the Internet useful to the greatest number stakeholders;
this argues for allowing a relatively wide range of topics to be
worked on in the IETF - cross-fertilization has always been one of
the IETF’s strengths.
An additional barrier to achieving a common understanding is that the
IETF does not have a recognized forum in which all stakeholders
participate and in which organization wide consensus might be
reached. Plenary meetings during regular IETF meetings allow a large
cross-section of the community to offer views, but there is not
generally sufficient time to achieve consensus and there is no single
mailing list which all stakeholders can be guaranteed to monitor.
The IETF creates standards and is therefore necessarily a Standards
Development Organization (SDO), but many participants would like to
differentiate the IETF and its way of working from the ’conventional’
SDOs which emphasize corporate involvement and mandated delegates.
Externally, the IETF is often classified with these conventional
SDOs, especially by detractors, because the differentiation in the
IETF’s mission and processes and the rationale for those differences
are not clear. This can lead to the IETF being misunderstood by
other SDOs which can make communications between SDOs less effective,
harming the IETF’s ability to achieve its mission.
2.2. The IETF does not Consistently use Effective Engineering Practices
For an organization with ’engineering’ in its title and participants
who are likely to trot out the statement "Trust me, I’m an engineer!"
when confronted with the need to find a solution to a particularly
knotty problem, the IETF has, at least in some cases, extremely
ineffective engineering practices. Effective engineering practices,
as used here, covers both the techniques used to derive and verify
the technical solutions needed, and the management and organizational
strategies that are commonly accepted to help with the engineering
process.
A major symptom of this lack is that WGs do not consistently produce
timely, high-quality, and predictable output. As discussed in
Section 2.1, this problem is exacerbated because the IETF currently
finds it difficult to determine what is timely, and hence what are
appropriate deadlines for the delivery of WG output. Some of the
contributing problems which interfere with effective engineering in
WGs include:
o Failure to ensure that there is a uniform view in the WG of the
scope of the WG activity, especially the intended purpose of the
solution.
o Failure to identify the issues that need to be resolved at an
early stage (before the design is frozen), and/or then to ensure
that there is a uniform view in the WG of the issues that need to
be resolved to bring the work to a satisfactory conclusion.
o Failure to identify and articulate engineering trade-offs that may
be needed to meet the deadlines that the WG has set without
inappropriately reducing the ’fitness for purpose’ for the
intended customers.
o Continued refinement of the solution beyond the point at which it
is adequate to meet the requirements placed on it by the intended
purpose.
The IETF standards engineering process is not set up to deliver
iterative process improvement. Particular areas that need
improvement include:
o The charter may not be sufficiently detailed to document the
process and timeline to be followed by the WG. Additional
documents may be needed, such as a roadmap or detailed plans.
o Poorly defined success criteria for WGs and individual documents.
o Lack of written guidelines or templates for the content of
documents (as opposed to the overall layout) and matching lists of
review criteria designed to achieve appropriate quality in output.
o Lack of auditing against explicit criteria throughout the
standards development process.
o Lack of review, especially early review, by reviewers who are not
directly interested members of the WG, and by subject matter
experts for topics related to, but not necessarily the immediate
focus of the document.
o Lack of documentation about likely problem areas that might arise
due to interactions with other popular IETF protocols.
o Lack of metrics to measure the achievement of the desired quality
and the performance of both WGs and the whole IETF.
o Lack of metrics and ’post mortem’ procedures to drive the
improvement of the standards development and other IETF processes.
o Lack of criteria for determining when a piece of work is
overrunning and/or is unlikely to be concluded successfully,
either at all or within an acceptable time frame. Lack of process
for extending the time frame, adjusting the scope, or terminating
the work item or the whole Working Group.
o Automated tools to support the engineering process are minimal.
o Despite its commitment to ’running code’, the IETF is not
proactive in providing ways for developers to verify their
implementations of IETF standards.
In addition, IETF processes, and Working Group processes in
particular, suffer because commonly accepted Project Management
techniques are not regularly applied to the progress of work in the
organization.
o Project entry, goal setting, dependency identification,
coordination, and tracking processes are all either missing or
implemented less effectively than the norm for commercial
organizations in related activities. Dependencies and
coordination should cover both other WGs within the IETF and any
outside SDO with which the IETF is collaborating.
o Charters regularly fail to set enough milestones with sufficiently
small granularity at which progress of WGs, individuals, and
documents can be evaluated. Also, WGs often do not make more
detailed work plans to refine the charter plans.
o The acceptable deadlines for finishing a piece of work, and the
criteria used to determine them, are rarely, if ever, documented.
Also, the estimated time required to complete the work often
differs widely from the time actually taken. The combination of
these factors makes determining the feasibility of delivering
within the required time frame, and then adjusting the scope of
the work to fit the time frame requirements, extremely difficult.
One problem which the IETF does not appear to suffer from is
excessive bureaucracy, in the sense that transfer of information is
generally kept to the minimum necessary to accomplish the task. It
is important that any changes introduced do not significantly
increase the bureaucratic load whilst still recording sufficient
information to allow process improvement.
Finally, even where the IETF does have Engineering Practices defined,
there are frequently cases where they are ignored or distorted. One
area of particular concern is the tendency for protocols to be
assessed and issues resolved primarily through static analysis of the
written specification rather than by practical experiment with
’running code’.
2.3. The IETF has Difficulty Handling Large and/or Complex Problems
The IETF has historically been most successful when dealing with
tightly focused problems that have few interactions with other parts
of the total problem solution. Given that the Internet has become
more complex, such tightly focused problems are becoming the
exception. The IETF does not always seem to be aware of the
interactions between protocols that are bound to be thrown up by
deployment in more complex situations and so fails to minimize the
chances of unwelcome consequences arising unforeseen when a new
protocol is deployed. This may be exacerbated by inadequate review
from outside the WG as suggested in Section 2.2.
IETF standardization procedures are optimized for tightly constrained
working groups and are generally less effective if ’engineering in
the large’ is needed to reach a satisfactory solution. Engineering
in the large can encompass many aspects of system design including:
Architecture
Frameworks
Security
Internationalization
The IETF has historically standardized protocol components rather
than complete systems, but as we have learned more about the ways in
which systems on the Internet interact, design of components needs to
take into account more and more external constraints, and the
understanding of these constraints tends to require more engineering
in the large.
Part of the cause of this difficulty may be that the formal reporting
structure of the IETF emphasizes communication between the Internet
Engineering Steering Group (IESG) through the ADs and the WGs, and
does not place much reliance on inter-WG communications:
o The IETF is not consistently effective at resolving issues that
cross WG or area boundaries.
o The IETF does not possess effective formal mechanisms for inter-WG
cooperation, coordination, or communication, including the
handling of dependencies between deliverables and processes
specified in WG charters.
o The IETF does not have an effective means for defining
architectures and frameworks that will shape the work of multiple
WGs.
The IETF also has to work with other SDOs, and the liaison mechanisms
for coordination and cooperation do not always work efficiently.
This needs to be remedied because some of the interactions which IETF
work has to take into account will involve protocols and systems
standardized by these other SDOs.
A possible consequence of the need for more engineering in the large
is that protocol specifications have become larger: as a result they
now take longer to develop. Some people perceive that this is
because the IESG has tended to require protocol specifications to