Home > Philosophy, Society > Maybe the scientific method just isn’t that powerful

Maybe the scientific method just isn’t that powerful

But now all sorts of well-established, multiply confirmed findings have started to look increasingly uncertain. It’s as if our facts were losing their truth: claims that have been enshrined in textbooks are suddenly unprovable. This phenomenon doesn’t yet have an official name, but it’s occurring across a wide range of fields, from psychology to ecology. In the field of medicine, the phenomenon seems extremely widespread, affecting not only antipsychotics but also therapies ranging from cardiac stents to Vitamin E and antidepressants: Davis has a forthcoming analysis demonstrating that the efficacy of antidepressants has gone down as much as threefold in recent decades.–Jonah Lehrer

Lehrer’s recent article on the decline effect, where initially strong and robust results shrink over time, has caused quite a stir (see hereherehere, and here).  It’s similar to Freedman’s Atlantic profile of Ioannidis in that it identifies some allegedly deep problems in the structure of science.

As both Lehrer and Freedman note, the fight for funding, the bias against null results, the drive for more publications, a desire for personal glory, and conflicts of interest all make the practice of science quite a bit different from how it’s usually portrayed.  Both, however, also note that there may be something fundamentally amiss here.  These relatively straightforward explanations only go so far.  It appears we can’t quite determine what’s really going on, and Lehrer’s subtitle asks: “Is there something wrong with the scientific method?”

I think it’s important to remember that the scientific method (TSM) was first used to solve what are, conceptually at least, fairly simple questions.  Planetary motion, as beautiful, intricate and awe-inspiring it may be, is solved by high-school students every day before lunch recess.  After all, it’s just giant balls circling each other.  Even quantum mechanics and relativity can be described almost perfectly with existing mathematics, and it’s usually first taught to college sophomores.  Penicillin was discovered in a petri dish, where we can reliably control most variables.  And so on.

None of this is meant to diminish Newton, Einstein and Fleming, or to trivialize their accomplishments.  They were geniuses, and we are forever indebted to them.  But it appears to me that what we consider the canonical scientific accomplishments were often fairly discrete questions.  If you can effectively use mathematics (as you can in much of physics), or if you can control all the relevant variables (physics, molecular biology), the problem is tractable.  But determining the effect of Vitamin E on public health is a different beast altogether.  After all, what we call health is really a combination of traditional medical science, economic status, level of stress, culture, pollution, and who knows what else.  TSM may never be able to disentangle all these effects regardless of how many perfect, unbiased studies we run.  From Freedman’s article:

But even if a study managed to highlight a genuine health connection to some nutrient, you’re unlikely to benefit much from taking more of it, because we consume thousands of nutrients that act together as a sort of network, and changing intake of just one of them is bound to cause ripples throughout the network that are far too complex for these studies to detect, and that may be as likely to harm you as help you.

And so asking what’s wrong with TSM kind of misses the point.  There’s nothing really wrong with it.  It’s that TSM, this crowning achievement of human intellect, the best tool ever developed to investigate the natural world, may just not be as powerful as we want it to be. Even if we disclose conflicts of interest, release all collected data (not just the pretty ones), publish more null results, and quadruple funding, some problems will be beyond our reach. There are ultimately some things in heaven and earth not dreamt of in our philosophies.

Advertisements
Categories: Philosophy, Society
  1. Michael S. Pearl
    January 12, 2011 at 11:26 am

    And so asking what’s wrong with TSM kind of misses the point. There’s nothing really wrong with it. It’s that TSM, this crowning achievement of human intellect, the best tool ever developed to investigate the natural world, may just not be as powerful as we want it to be.

    I agree that “asking what’s wrong with TSM” is very likely the wrong question. As to whether it is the “crowning achievement”, well, I see it as an extension of logic, and therein may rest the problem. Many scientific studies – certainly biomedical studies and especially clinical investigations – seem to have transformed the method of science into a veritable rote form which, once achieved, (seemingly) reduces the burden of having to ask proper questions, questions which first and maybe even foremost pertain to the identification and/or discovery of relevant variables.

    When the focus is on variables (and discrepancies), good thinking necessarily involves logic, thinking in terms of possibilities and seeking out the limits of applicability. It is this very sort of thinking that seems to be missing from a lot of what passes for science today, certainly that science which most readily passes into the domain of the general public. I discuss similar points here.

    • January 13, 2011 at 9:08 am

      Michael,

      I’m not sure I would agree that TSM is “extension of logic”. Rather, I see it as a matter of ingenuity, first in the devising of new ‘crazy’ questions no-one thought to ask before, usually by introducing new metaphors into discussion, which introduction need have nothing to do with logic and indeed may well be inhibited by strict adherence to logic. Second, by devising observational circumstances (lab experiments or field situations) the outcome of which provide answers to those questions with the minimum of ambiguity.

      Peter

      • Michael S. Pearl
        January 13, 2011 at 10:54 am

        I’m not sure I would agree that TSM is “extension of logic”. Rather, I see it as a matter of ingenuity … need have nothing to do with logic and indeed may well be inhibited by strict adherence to logic. … devising of new ‘crazy’ questions … usually by introducing new metaphors

        Peter, I am very well aware that logic is nearly always portrayed and treated as if it were strictly and nothing more than a matter of form, a form that is in no way affected by content. “[S]trict adherence” to such a logic would definitely inhibit creativity; it would inhibit the imagination necessary for moving much beyond mere observation. Indeed, the notion of creative or imaginative logic almost seems oxymoronic. But, it is not.

        The logic I had in mind is the logic of possibilities – which is what modal logic is supposed to be. Granted, some (actually too many) philosophers (and it is actually the preposterous among the analytic philosophers who) unfortunately have misused the logic of possibilities until it seems to pertain to nothing that would interest a sane person; they have even expended significant amounts of effort to put it in symbolic forms as if that were to make modal logic somehow more broadly useful.

        It is not that sort of modal logic that I have in mind. Indeed, formalized modal logic has been abstracted from the (let us just call it a) genuine modal logic employed in the narrative form of explication – which definitely relies heavily upon metaphors and other such “literary” forms.

        The modal logic that I have in mind is simply the logic used to seek out explanations by first imagining what might possibly explain the apparent connections between occurrences and then by setting about to identify from amongst those possibilities the factors or conditions which will be sufficient for assuring or determining that one occurrence rather than some other follows from some previous condition or occurrence given the factors that have been identified.

        It should be apparent, that deviations from expectations would strongly indicate the importance of pursuing identification of factors that have been filtered out or which have not even been taken into account.

        I also think it is clearly apparent that this process of thinking in terms of possibilities requires imagination or “ingenuity”, and I also think it rather apparent that this process fosters further imaginative thinking – especially when failures in explanation or deviations are encountered.

        Now, I am not saying that inspiration never occurs, but I am saying that thinking trained in terms of possibilities will open up the imagination much more than would occur with thinking trained in terms of such alleged necessities as “laws”.

        To put it another way, ingenious scientific thinking (would that it were redundant to say “ingenious scientific”) is reducible to the logic of possibilities, even when the moment of ingenuity (which might even be a “revelation”) does not itself appear to be the product of thinking consciously in terms of possibilities.

        It is in this sense that scientific methods are extensions of logic, the logic of possibilities – a mode of thought which necessitates and fosters imagination.

  2. RickA
    January 12, 2011 at 1:43 pm

    Very nice Praj.

    I think cost benefit plays into this question as well – which is kind of an economic analysis.

    For example, we all know we need vitamin D and we put it into juice and cereal because it is good. You can also get vitamin D by catching some sun – but too much sun can also harm.

    Some questions are not really scientific – like what is the best climate for Earth.

    This is the type of question which many people can have different opinions on, for all sorts of good objective reasons.

    Anyway – nice post.

    • January 13, 2011 at 9:21 pm

      Thanks for the nice comment Rick! Good to hear from you. Happy new year, and hope all is well.

  3. January 13, 2011 at 11:49 am

    Good reply. Thanks.
    In view of your earlier comments on “what passes for science today”, do you suppose that science today suffers from a deficit of modal logic?
    Could science perhaps be improved by training scientists in modal logic, or is its contemporary degradation more likely a consequence of the demand for scientists outstripping the supply of individuals equipped to apply modal logic?

    • Michael S. Pearl
      January 13, 2011 at 1:35 pm

      In view of your earlier comments on “what passes for science today”, do you suppose that science today suffers from a deficit of modal logic?

      Yes, I think it does suffer from just such a deficit. I turn to clinical biomedical studies (if only because that is the area to which I am most often exposed) for good examples of where there is a predominance of correlative or correlational studies which seem cast with hardly an interest in the possibilities that are essential to the intentional expansion of the scope of knowledge.

      Correlations can most definitely be part of the best science, but correlations are most properly to be regarded more as initial rather than used as if they are conclusive or treated as if they are in some important way definitive. It might be that this particular predominance in the biomedical field is a consequence of both commercial and political factors. Nonetheless, correlative research seems to have become the normative science that “justifies” as “scientific” those demands put forth by such regulatory bodies as the FDA in the US. Whether those demands shape what is expected of submissions for publication or the convention for communication, or whether the convention shapes the demands, I do not know. But that is not especially important in that either way what we are left with is a convention that seems to have minimal appreciation for the importance of thinking in terms of possibilities.

      I expect that similar skewing occurs in other scientific fields, even if the sources of that skewing are different, but the key factor seems to be that convention tends to be disinterested in or to move away from a greater immersion in the logic of possibilities.

      Could science perhaps be improved by training scientists in modal logic, or is its contemporary degradation more likely a consequence of the demand for scientists outstripping the supply of individuals equipped to apply modal logic?

      It would be nice if scientists could become better aware of the usefulness of modal logic. But, how would they be trained in modal logic? By philosophers?!?!?! Perish the thought!!! Just joking. Sort of. I think it can be done, but I think that the best approach would be in terms of philosophy in science — which, apparently, is not quite the same thing as the philosophy of science, at least inasmuch as some philosophy of science seems more like the sociology or the anthropology of science.

      Maybe it is just me, but I think that a lot of the misunderstandings people have about the likes of Feyerabend and (particularly recently) Laudan is the fact that these philosophers (effectively if not explicitly) warn of the importance of making one’s self cognizant of the limited scope for the validity of findings and claims. This, of course, pertains directly to thinking in terms of possibilities, and it also emphasizes the tentativeness which is supposed to be a virtue of science.

      I think that there is a sort of “outstripping” at work. For one thing, the public has no ear for tentativeness.

  4. Mark
    January 13, 2011 at 5:37 pm

    As an Earth scientist who has never before been exposed to the ideal of modal logic, I went to Wikipedia, as you do. Hmmm, that epistemic modal logic stuff looks interesting.

    But I couldn’t agree more with this (Michael S. Pearl): “I turn to clinical biomedical studies (if only because that is the area to which I am most often exposed) for good examples of where there is a predominance of correlative or correlational studies which seem cast with hardly an interest in the possibilities that are essential to the intentional expansion of the scope of knowledge.”

    In Earth sciences, correlative and correlational studies (is there a difference?) predominate in areas where we most want to answer applied questions and have the least knowledge about mechanisms, e.g. fisheries science. There is a famous paper in the field (I forget the reference) that looked at the robustness of statistically-established relationships between fish numbers and environmental variables. With some exceptions, the majority of such relationships evaporate when more data become available.

  5. January 13, 2011 at 9:27 pm

    Hi All. Great discussion here. I enjoy following it. Peter, Michael: I like your suggestions of training scientists in fields like logic and philosophy. It would even be nice to have actual philosophers involved! I hope my efforts here show that I think philosophy, sociology, etc. does have something to offer science. I especially like the distinction between philosophy in science vs. philosophy of science.

    I think biomedical sciences focuses on correlational studies because that’s often the best they can do. In this I sympathize with Mark. I was a space physicist at one point, and so I appreciate how difficult data are to come by. And Mark, like you I also will have to Wikipedia some of the terms Michael and Peter were discussing!

    Happy new year all.

  6. November 20, 2013 at 5:23 am

    We were on the same page here, Sriram:

    http://utopiaordystopia.com/2013/03/31/how-science-and-technology-slammed-into-a-wall-and-what-we-should-do-about-it/

    Would be interested in your thoughts..

  1. January 17, 2011 at 6:49 pm
  2. February 11, 2011 at 7:08 pm

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: