More on strong open-mindedness

For the last couple of days I have been exploring what I like to call strongly open-minded accuracy scoring rules. It’s well known that every proper scoring rule is open-minded in the sense that it never requires you to reject free information: the expected epistemic utility of updating on the free information is always at least as good as your current expected epistemic utility. It’s strictly open-minded provided that in non-trivial cases (i.e., when the information has a non-zero probability of having statistical relevance to the credences you are scoring) you are required to accept the free information.

Now there are two reasons why one might accept free information about some proposition q. First, you might be wrong about q: your credence may be high while q is false or your credence might be low while q is true. Second, even if you are right about q, the free information may boost your credence in the right direction. I say that a scoring rule is strongly open-minded provided that it licenses you to accept and update on the free information even if you disregard the first consideration. We can then tack on “strictly” if it requires you to do so in non-trivial cases. In the case of a strongly open-minded scoring rule, your acceptance of free information is not a sign of doubt in your propositions—it is not a way of hedging your bets—and thus is arguably compatible with faith in the propositions being evaluated.

A strongly open-minded scoring rule can also be characterized in the following way. There is a more ordinary kind of epistemic paternalism where I might have reason to block another from receiving free information on the grounds that this information could mislead due to the fact that the other has different likelihoods from the ones I think are right. For instance, if too many people have an unjustified mistrust of Dr. Smith such that they are likely to believe the opposite of what Dr. Smith’s experiments reveal, there is reason to give a grant to someone else, because Dr. Smith’s experiments are likely to lead people away from the truth, for no fault of Dr. Smith’s. Call this likelihood-based paternalism. But there is another kind of motivation of the refusal of free information for another, which we might call pure-risk-based paternalism. Even if someone else has the same likelihoods as you do—trusts Dr. Smith just as you do—perhaps the risk that Dr. Smith’s experiments will, by pure chance, provide evidence away from the truth is enough to justify not funding these experiments.

I’ve been collecting results about these issues. Here’s what I seem t have so far, though I have to emphasize that sometimes the proofs are just in my head and I might be wrong. I will specialize on scoring rules for a single proposition, given as a pair of functions T and F, where T(x) is the value of having credence x when the proposition is true and F(x) is the value of having credence x when the proposition is false.

1. A scoring rule sometimes calls for pure-risk-based paternalism if and only if it is not strongly open-minded.

2. A scoring rule that’s strongly open-minded is open-minded.

3. A scoring rule (T,F) is (strictly) strongly open-minded if and only if xT(x) and (1−x)F(1−x) are both (strictly) convex.

4. The logarithmic scoring rule is strictly strongly open-minded. The Brier and spherical rules are not strongly open-minded.

5. If a proper scoring rule is generated by the Schervisch-style integral representation T(x) = T(1/2) + ∫_x^1/2(1−t)b(t)dt and F(x) = F(1/2) + ∫_1/2^xtb(t)dt and b is sufficiently differentiable, then the scoring rule is strongly open-minded if and only if the derivative of log b(x) lies between (3x−2)/[x(1−x)] and (3x−1)/[x(1−x)].

6. A strongly open-minded scoring rule whose logarithm is sufficiently differentiable is unbounded.

7. [Item deleted as I discovered it to be false.]

8. For any credences p and r such that 1/2 < r and p < r, there is a strictly proper scoring rule and a situation where the scoring rule calls for the individual with credence r to have purely-risk-based epistemic paternalism for that hypothesis.

A feature of the logarithmic scoring rule

Accuracy scoring rules measure the epistemic utility of having some credence assignment. For simplicity, let’s assume that all credence assignments are probabilistically coherent. A strictly proper scoring rule has the property that always by one’s own lights, the expected value of one’s actual credence assignment is better than that of any other credence assignment.

A well-known fact is that a strictly proper scoring rules always makes it rational to update on non-trivial evidence. I.e., by one’s present lights, the expected epistemic utility after examining and updating on non-trivial evidence will be higher than the expected epistemic utility of ignoring that evidence. We might put this by saying that a strictly proper scoring rule is strictly open-minded.

The logarithmic scoring rule makes the score of assigning credence r be log r when the hypothesis is true and log (1−r) when the hypothesis is false. It is strictly proper and hence strictly open-minded.

The logarithmic scoring rule, however, satisfies a condition even stronger than strict open-mindedness. This condition is easiest to describe in a binary case where one is simply evaluating the score of one’s credence in a single hypothesis H. Assuming some non-triviality assumptions, it turns out that not only is the expected epistemic utility increased by examining evidence, but the expected epistemic utility conditional on H is increased by examining evidence. (This is a pretty easy calculation.)

So what?

Well, there are several reasons this matters. First, on my recent account of what it is to have a no-hedge commitment to a hypothesis H, if your epistemic utilities are measured by some scoring rules (e.g., Brier) and you have a no-hedge commitment to H but you do not have credence 1 in H, then you will sometimes have reason to refuse to look at evidence. But the above fact about the logarithmic scoring rule shows that this is not so for the logarithmic scoring rule. With the logarithmic scoring rule, it makes sense to look at the evidence even if you have a no-hedge commitment to H—i.e., even if all your betting behavior is “as if H”.

Second, let’s imagine that I run a funding agency and you come to me with an interest in doing some experiment relevant to a hypothesis H. Let’s suppose that the relevant epistemic community agrees on the relevant likelihoods with respect to the evidence obtainable from the experiment, and is perfectly rational, but differs with regard to the priors of H. I might then have this paternalistic worry about funding the experiment. Even though updating on the results of the experiment by my lights is expected to benefit me epistemically, if a strictly proper scoring rule is the appropriate measure of benefit, it may not be true that by my lights other members of the community will benefit epistemically from updating on the results of the experiment. I may, for instance, be close to certain of H, and think that some members of the community have credences that are sufficiently high that the benefit to them of getting a boost in credence in H from the experiment is outweighed by the risk of misleading evidence. If it is my job to watch out for the epistemic good of the community, this could give me reason to refuse funding.

But not so if I think the logarithmic rule is the right way to evaluate epistemic utility. If everyone shares likelihoods, and we differ only in priors for H, and everyone is rational, then when we measure epistemic utility with the logarithmic rule, I have a positive expectation of the epistemic utility effect of examining the experiment’s results on each member of the community. This is easily shown to follow from my above observation about the logarithmic scoring rule. (By my lights the expectation of a fellow community member’s epistemic utility after updating on the experimental results is a weighted sum of an expectation given H and an expectation given not-H. Each improves given the experiment.)

What is it like not to be hedging?

Plausibly, a Christian commitment prohibits hedging. Thus in some sense even if one’s own credence in Christianity is less than 100%, one should act “as if it is 100%”, without hedging one’s bets. One shouldn’t have a backup plan if Christianity is false.

Understanding what this exactly means is difficult. Suppose Alice has Christian commitment, but her credence in Christianity is 97%. If someone asks Alice her credence in Christianity, she should not lie and say “100%”, even though that is literally acting “as if it is 100%”.

Here is a more controversial issue. Suppose Alice has a 97% credence in Christianity, but has the opportunity to examine a piece of evidence which will settle the question one way or the other—it will make her 100% certain Christianity is true or 100% certain it’s not. (Maybe she has an opportunity for a conversation with God.) If she were literally acting as if her credence were 100%, there would be no point to looking at any more evidence. But that seems the wrong answer. It seems to be a way of being scared that the evidence will refute Christianity, but that kind of a fear is opposed to the no-hedge attitude.

Here is a suggestion about how no-hedge decision-making should work. When I think about my credences, say in the context of decision-making, I can:

1. think about the credences as psychological facts about me, or

2. regulate my epistemic and practical behavior by the credences (use them to compute expected values, etc.).

The distinction between these two approaches to my credences is really clear from a third-person perspective. Bob, who is Alice’s therapist, thinks about Alice’s credences as psychological facts about her, but does not regulate his own behavior by these credences: Alice’s credences have a psychologically descriptive role for Bob but not a regulative role for Bob in his actions. In fact, they probably don’t even have a regulative role for Bob when he thinks about what actions are good for Alice. If Alice has a high credence in the danger of housecats, and Bob does not, Bob will not encourage Alice to avoid housecats—on the contrary, he may well try to change Alice’s credence, in order to get Alice to act more normally around them.

So, here is my suggestion about no-hedging commitments. When you have a no-hedging commitment to a set of claims, you regulate your behavior by them as if the claims had credence 100%, but when you take the credences into account as psychological facts about you, you give them the credence they actually have.

(I am neglecting here a subtle issue. Should we regulate our behavior by our credences or by our opinion about our credences? I suspect that it is by our credences—else a regress results. If that’s right, then there might be a very nice way to clarify the distinction between taking credences into account as psychological facts and taking them into account as regulative facts. When we take them into account as psychological facts, our behavior is regulated by our credences about the credences. When we take them into account regulatively, our behavior is directly regulated by the credences. If I am right about this, the whole story becomes neater.)

Thus, when Alice is asked what her credence in Christianity is, her decision of how to answer depends on the credence qua psychological fact. Hence, she answers “97%”. But when Alice decides whether or not to engage in Christian worship in a time of persecution, her decision on how to answer would normally depend on the credence qua regulative, and so she does not take into account the 3% probability of being wrong about Christianity—she just acts as if Christianity were certain.

Similarly, when Alice considers whether to look at a piece of evidence that might raise or lower her credence in Christianity, she does need to consider what her credence is as a psychological fact, because her interest is in what might happen to her actual psychological credence.

Let’s think about this in terms of epistemic utilities (or accuracy scoring rules). If Alice were proceeding “normally”, without any no-hedge commitment, when she evaluates the expected epistemic value of examining some piece of evidence—after all, it may be practically costly to examine it (it may involve digging in an archaeological site, or studying a new language)—she needs to take her credences into account in two different ways: psychologically when calculating the potential for epistemic gain from her credence getting closer to the truth and potential for epistemic loss from her credence getting further from the truth, and regulatively when calculating the expectations as well as when thinking about what is or is not true.

Now on to some fun technical stuff. Let ϕ(r,t) be the epistemic utility of having credence r in some fixed hypothesis of interest H when the truth value is t (which can be 0 or 1). Let’s suppose there is no as-if stuff going on, and I am evaluating the expected epistemic value of examining whether some piece of evidence E obtains. Then if P indicates my credences, the expected epistemic utility of examining the evidence is:

3. V_E = P(H)(P(E|H)ϕ(P(H|E),1)+P(∼E|H)ϕ(P(H|E),1)) + P(∼H)(P(E|∼H)ϕ(P(H|E),0)+P(∼E|∼H)ϕ(P(H|∼E),0)).

Basically, I am partitioning logical space based on whether H and E obtain.

Now, in the as-if case, basically the agent has two sets of credences: psychological credences and regulative credences, and they come apart. Let Ψ and R be the two. Then the formula above becomes:

3. V_E = R(H)(R(E|H)ϕ(Ψ(H|E),1)+R(∼E|H)ϕ(Ψ(H|∼E),1)) + R(∼H)(R(E|∼H)ϕ(Ψ(H|E),0)+R(∼E|∼H)ϕ(Ψ(H|∼E),0)).

The no-hedging case that interests us makes R(H) = 1: we regulatively ignore the possibility that the hypothesis is false. Our expected value of examining whether E obtains is then:

4. V_E = R(E|H)ϕ(Ψ(H|E),1) + R(∼E|H)ϕ(Ψ(H|∼E),1).

Let’s make a simplifying assumption that the doctrines that we are as-if committed to do not affect the likelihoods P(E|H) and P(E|∣H) (granted the latter may be a bit fishy if P(H) = 1, but let’s suppose we have Popper functions or something like that to take care of that), so that R(E|H) = Ψ(E|H) and R(E|∣H) = Ψ(E|∣H).

We then have:

5. Ψ(H|E) = Ψ(H)R(E|H)/(R(E|H)Ψ(H)+R(E|∼H)Ψ(∼H)).

6. Ψ(H|∼E) = Ψ(H)R(∼E|H)/(R(∼E|H)Ψ(H)+R(∼E|∼H)Ψ(∼H)).

Assuming Alice has a preferred scoring rule, we now have a formula that can guide Alice what evidence to look at: she can just check whether V_E is bigger than ϕ(Ψ(H),1), which is her current score regulatively evaluated, i.e., evaluated in the as-if H is true way. If V_E is bigger, it’s worth checking whether E is true.

One might hope for something really nice, like that if the scoring rule ϕ is strictly proper, then it’s always worth looking at the evidence. Not so, alas.

It’s easy to see that V_E beats the current epistemic utility when E is perfectly correlated with H, assuming ϕ(x,1) is strictly monotonic increasing in x.

Surprisingly and sadly, numerical calculations with the Brier score ϕ(x,t) =  − (x−t)² show that if Alice’s credence is 0.97, then unless the Bayes’ factor is very far from 1, current epistemic utility beats V_E, and so no-hedging Alice should not look at the evidence, except in rare cases where the evidence is extreme. Interestingly, though, if Alice’s current credence were 0.5, then Alice should always look at the evidence. I suppose the reason is that if Alice is at 0.97, there is not much room for her Brier score to go up assuming the hypothesis is correct, but there is a lot of room for her score to go down. If we took seriously the possibility that the hypothesis could be false, it would be worth examining the evidence just in case the hypothesis is false. But that would be a form of hedging.