The CUP'A Framework for Evaluating Sources
Caliber, Utility, Potency, and Appeal
Every day, you decide what to read, trust, save, or ignore. Most of those decisions are made quickly—and often poorly.
Assessing knowledge resources is a constitutive skill of knowledge work that requires ongoing attention, and should not be allowed to fossilize upon graduation.
Here I present a framework for making those decisions deliberately and well, called the CUP’A framework. When you encounter a resource, ask:
Caliber: “Does this stand up to high standards of quality, validity, rigour and logic?”
Utility: “Is this worth my time, given my current motives (projects, etc.)?”
Potency: “Can this change how I think or act?”
Appeal: “Do I like this information for the wrong reasons? How does it relate to my preferences”
I refer to the caliber, utility, and potency of a resource as its helpfulness. The first three of these criteria are actually based on Ortony, Clore & Collins (1988)’s schema of value. (You’ll notice a trend in my writing: drawing on integrative cognitive science).
In contrast to frameworks one finds in study skills texts (e.g., the CRAAP test, which focuses on currency, relevance, authority, accuracy, purpose, and point of view), philosophy of science texts and elsewhere, this framework:
is developed not only for students but also for professional knowledge workers;
is inscribed in integrative design-oriented psychology;
includes suggestions for using information-processing software in powerful ways; and
functionally specifies information-processing software (the ‘future of text’).
Table 1: CUP’A criteria
1. Caliber
Caliber concerns how well a resource is constructed, judged against expert standards, irrespective of your particular goals, knowledge or preferences. General standards of caliber include:
the clarity of its thesis and its overall clarity,
the suitability of research methods used,
the rigor of its arguments, backing and statistics,
the originality of its concepts, claims, findings, etc.
its actual or potential significance for the literature,
its grounding in previous literature (e.g., missing or misused references),
its conceptual richness and coherence (relevant to its potency), and other criteria.
These criteria vary by domain. A resource can measure well against some standards and poorly against others.
Often a factual resource conveys, or at least should reference, an explanatory theory or model. Hence criteria for assessing theories are relevant, which include
assessing its generality, parsimony, extensibility, mechanistic plausibility and practical usefulness; and
determining whether it (a) can account for fine structure and (b) is part of a progressing or degenerating research programme.
There are other general criteria of caliber, as well as criteria that are specific to particular domains.
2. Utility
The utility of a resource is a measure and description of how instrumental it would be to one’s projects, goals, plans and areas of responsibilities — more generally, to one’s motives. This is its value relative to time, effort, and opportunity cost. A resource may be of high caliber but irrelevant to one’s intentions, considering its processing cost (time, etc.), risks and constraints. Moreover, a resource may be deeply flawed but potentially useful. We must try to prevent our utility judgments from affecting caliber judgments.
Assessing utility requires explicit knowledge of one’s projects. Personal task/project management software can help one track and pursue one’s projects, goals, plans and actions. Ideally software would enable one to:
link knowledge resources to specific projects or motives, e.g. using our Hookmark app; and
quantify the utility of the resource.
Explicitly making such judgments may help one judiciously select and use information.
3. Potency
A resource’s potency is the extent to which it can affect you as a person: your beliefs, understanding, attitudes, goals, standards, etc. Potency is inherently subjective to you but objective as a matter of psychology. A paper may be high-caliber but low potency if it merely confirms what you already know. Conversely, a difficult, conceptually rich work may be highly potent because it forces you to reorganize your understanding — for instance encountering the cognitive shuffle for the first time.
A potent resource is typically difficult to assimilate: it calls for accommodation (Piaget’s term): elaboration, restructuring, productive practice etc. Productive practice is particularly important for actualizing the mind-bending potential of a resource, i.e., its potency.
4. Appeal
Appeal operates through motivators that can bias attention, belief, and persistence—sometimes independently of caliber or utility. Appeal can adversely bias one’s judgments of caliber, utility and potency. For instance, dubious information (clickbait, idea pathogens, etc) may appeal to one’s preferences (whether on specific topics, ‘my side’ bias etc). Not all truths are beautiful, some are ugly! For example, The Goodness Paradox book by Richard Wrangham exposed anthropologists who unfortunately rejected high-caliber papers because the papers clashed with their political [pacifist] attitudes, which unnecessarily delayed scientific progress. (Ironically, such clashing is itself often based on misunderstanding.)
A promising direction is to cultivate dispositions where discovering one’s errors is intrinsically rewarding. For example, experiencing mirth and debugging one’s software involve discovering one’s errors, and yet are both pleasant. It might be possible to generalize, transfer and nourish such dispositions (e.g., enjoying having one’s flawed ideas corrected).
CUP’A is not just for evaluating information, but for selecting what to engage with in productive practice.
Information technology and strategies
To fully realize CUP’A in practice requires the following innovations.
Assign global assessments to resources. Not merely “likes” or “upvotes”, but systematic categorical ratings of caliber, utility and potency.
Not merely highlight text but tag one’s annotations. For instance, one should be able to tag text as I disagree or as containing a particular fallacy. Common categories of assessment should be built-into the information processing software and new tags addable.
Filter annotations by tag, for instance to list everything in a PDF, book or web page with which one disagrees.
Robustly link entire sources to multiple other resources, such as one’s evaluative notes about them (“meta-docs”), one’s projects, and related documents (such as others’ reviews). These other resources may be developed in arbitrary software (outliners, mind mappers, etc.) and stored locally or remotely. Ubiquitous linking software enables navigating between a source and metadocs without searching. See “A manifesto for user and automation interfaces for hyperlinking” and Hook productivity software. See Figure 1 below.
Find previously encountered resources designated as pertinent to a (sub)project.
Share entire sources , meta-docs, and annotations; and links to said information.
Figure 1. Documents link to a source that explicitly or implicitly evaluate it
Relevance of psychology
Assessing knowledge is inherently difficult. Compounding the psychological and technical challenges summarized above are problematic social trends that reject the humanist ideals of an open society (enlightenment liberalism). Postmodernism rejects the possibility of separating truth and value (they lack the CUP’A framework). Fear of the ‘tyranny of the cousins’ (per Wrangham’s theory), of being ‘canceled’, can adversely bias CUP’A judgments. (See Cynical Theories by Pluckrose and Lindsay for an exposition of epistemological trends).
Research in psychology is required to help us design software and strategies to assess information. It would help us deal with the fact that some truths may look ugly. It might help us understand and counter the memetic fitness of parasitic information. It would help us evaluate information analytically, systematically and rigorously.
Beyond non-fiction: assessing art
I first developed the CUP’A framework for assessing practical and expository information (non-fiction). I then generalized it to apply to fiction and other forms of art. Art has particular potency.
To realize the potency of art, one may need to practice productively with it, answering transformative questions about the story at spaced intervals. (E.g., Question: “What Shakespearian play can help me stop worrying?” Answer: “King Lear, because worrying is a transdiagnostic road to mental instability. I do not want to become like King Lear.” Not that Lear worried — he did not. It’s just that worrying (a form of mental perturbance) can lead to psychopathology. [More generally: cognitive defusion is not enough, one needs cognitive-affective defusion and fiction + productive practice can help with that.]) Learning from stories is discussed in detail in Discontinuities: Love, Art, Mind.
Call to action
The next time you encounter an article, paper, or book:
Write a one-line CUP’A assessment.
Note where your judgment might be biased by appeal.
If it is sufficiently helpful, develop a productive practice challenge for it.
This turns the framework into a habit and a discipline.
Learn more
The CUP’A framework is described in my three books:
Chapter 11, “Assess” of Cognitive Productivity: Using Knowledge to Become Profoundly Effective
Principle 3, “Assess Analytically” of Cognitive Productivity with macOS: 7 Principles for Getting Smarter with Knowledge.
My Discontinuities: Love, Art, Mind book has a chapter on evaluating fiction with the CUP’A framework.
CUP’A is also discussed on the CogZest.com blog
NB
An earlier version of this article was published as a chapter in the 2022 volume of The Future of Text