In 1999 at the Annual convention of the American Academy of Child and Adolescent Psychiatry, Henggeler and colleagues presented a study on the efficacy of treatment of adolescent patients in situations of psychiatric urgency.  In particular, two approaches to the problem were compared,  the first carried out with hospitalization, while the second envisaged an integrated treatment through clinical interviews, home visits, etc. Outcome measurements connected to symptom assessment had revealed that integrated therapy produced a better performance than hospitalization. At the same time, however, measurements connected to the assessment of self-esteem had shown that hospitalization worked better.
One might therefore think that if we wanted to work on the reduction of symptomatology with this kind of patient it would be preferable to apply an integrated treatment, while if we wanted to work on self-esteem we would choose hospitalization. This would however be a partial conclusion since broadening the research and taking other outcome indicators into account would probably be enough to lead to a rethinking of the most suitable treatment.
What is highlighted by Henggeler is not so much the greater or lesser efficacy of one type of intervention over another, but rather the methodological difficulties involved in assessing clinical treatments. In empirical psychotherapy research when we talk about outcome research we mean a form of research that aims at defining the changes obtained through therapy, by studying what happens at the end of the treatment compared to the starting point. The biggest problem lies in what we consider starting point,  and in fact, as we have just seen, it is enough to change the outcome indicator to get diametrically opposed results. One of the indicators used in the past to evaluate treatment was the reduction of evident symptomatology. But we know very well that the emphasis on the symptom and on its importance in assessing the good outcome of a treatment has become progressively weaker due in part to the heated debate over the nature and significance of the symptom in psychopathology. Symptoms can indicate the presence of a pathological process but they do not say much about the nature of the problem. But if the attitude to the weight of symptoms tells us little about the nature of the problem, at the same time it tells us a lot about the representation of mental health and the concept of the cure, found both in the clinical and the research context.  The first outcome studies were based on the criterion of cure in the sense of the reduction of symptomatology, taking for granted that an absence of symptoms corresponded to an absence of mental suffering.

Much has certainly been done since the first provocative research by Eysenck (1952) on the inefficacy of psychotherapy and since the studies by Bergin (1963) confuting his conclusions by showing the non casual effects of psychotherapy. Present-day research tends to favour an outcome measurement based on several levels of functioning, taking into account not only the individual level but also that of the environment. However it must always be remembered that the evaluation of the efficacy of a treatment is not in itself an absolute result, but depends substantially on the  outcome measurements identified in each situation. It becomes essential to uphold the general methodological principle that calls for congruency in the definition of the clinical problem, the therapeutic process and the clinical outcome. In other words, it is necessary for the definition of the initial pathological state (clinical problem) to be somehow isomorphic and coherent with the definition of the type of intervention applied (therapy) and with the definition of what modification is expected at the end of the treatment, compared to the initial problem (clinical outcome) (Dahl, H., Kächele H., & Thomä H. 1988).
This approach is widely represented in adolescent research in that the results related to children’s mental health cannot be considered in absolute terms since, more than with an adult population, the problems to be faced are tied to the functioning of the social and family environment.
Currently there is the tendency to favour outcome measurement based on several levels of functioning, taking into account not only the individual level but also that of the environment. In fact several levels or areas of functioning have been identified, which should be studied simultaneously so as to have reliable outcome indicators available. Fonagy and colleagues (2002), in a recent overview of outcome studies in child psychotherapy, propose a re-elaboration of the two main classifications put forward respectively by Hoagwood and colleagues (1996) and by Kazdin and Kendall (1998) and identify five areas of functioning which correspond to five different levels of outcome:
1. Symptomatic or Diagnostic level
2. Adaptation level
3. Mechanism level
4. Transactional level
5. Service use and satisfaction level

Symptomatic or Diagnostic level

In the field of development psychology, the disappearance of a symptom can no longer be considered the only and most important criterion with which to assess therapeutic efficacy. Nevertheless, as Kazdin (2000) points out, in most published research the reduction of a symptom remains the favoured criterion for defining a successful therapy outcome. Over the years, increasingly sophisticated tools have been developed for the measurement of symptoms; these tools may differ in length, ease of administration or standardization quality. However, in contrast to what one might expect, the recording of symptoms in adolescence is particularly complex. A low level of agreement has often been found between those who provide information about the symptomatology or about the child’s adaptation. Descriptions of a child or an adolescent’s symptomatology obtained from teachers or from individual parents show a low level of agreement, as can be seen in the research by Achenbach (1995). Moreover, positive changes in the symptomatology during treatment are more often recorded by parents than by teachers, with a significant increase in the perception gap every time parents and not teachers participate actively in the therapy. Few studies have been done on the descriptions of their conditions given by children or adolescents, since until a few years ago they were not considered a reliable source, but from the few research projects carried out, there emerges a further reduction in agreement.
In view of all these problem factors associated to the issue of recording symptoms, research is moving more and more towards a multidimensional measurement of symptomatology in adolescents. The tools designed for the measurement of symptoms, also during treatment, mainly envisage an assessment coming from more than one source of information. Currently the most reliable tool offering both attention to this issue and an accurate statistical validation, also in the Italian version, is the Child Behavior Checklist (CBCL) which includes forms both for parents and teachers and for self-assessment.

Adaptation level

Closely connected to the issue of symptomatological assessment is that of the assessment of the child or adolescent’s social adaptation. It is quite natural to imagine that mental health problems affect the child’s functioning in various contexts. Although symptomatology and adaptation are strictly interdependent, for the purposes of an outcome assessment of the treatment, it is more correct and practical to make a distinction between the two levels.
In a meta-analysis, Weisz and colleagues (1995) showed that assessing a treatment by focusing exclusively on the effect on the main symptoms entails overestimating the real effect of such treatment. We know in fact that many treatments, while bringing a change at the symptomatic level, do not act on other areas of functioning.
While the measurement of symptoms is complex and requires a multidimensional approach for a correct assessment, there is such a variety of events and situations involved in evaluating adaptation, i.e. measuring the quality of the child’s relational life,  that it is also necessary in this domain to use a number of indicators. In fact, if assessing adaptation coincides with evaluating the child’s quality of life, from an empirical viewpoint the task presents no small number of difficulties. Due to the constraints of the research, only the most important contexts for adaptation are taken into consideration. It is well known that one of the criteria used for the evaluation of pathologies is to what extent the symptomatology impedes the conducting of a normal life, and one of the primary objectives of the treatment is to remove these impediments.
It is also possible to identify an ideal level of social functioning for childhood and adolescence. From studies on childhood development it emerges that the areas of social functioning can be limited to three main contexts: the fulfilment of the requirements imposed by the family,  of those imposed by the school context and good social interaction with the peer group. As has been mentioned already, for adaptation assessment, too, it is preferable to use multidimensional measures  that take these main functioning areas into account. The best known global assessment tool of child functioning is the Children’s Global Assesment Scale (CGAS; Shaffer, Gould, & Brasic, 1983), which consists of an adapted version for childhood of the Health Sickness Rating Scale developed by Luborsky and Bachrach in 1974 for an adult population.
Over the years difficulties have emerged connected to the low level of agreement between different assessors in the use of this tool, reducing the statistical validity of the measurement. At the same time there are currently many questionnaires available to evaluate the single areas of adaptive functioning. It should however be considered that while on the one hand the assessment of single aspects of adaptation simplifies the global assessment, on the other it poses serious methodological problems especially as far as the comparability of the measurements obtained is concerned. For these reasons from the methodological point of view it is preferable to obtain an adaptation profile of the child in the different aspects of social functioning by using a single measurement.  In the literature, various tools can be found with these characteristics: among those most widely used at present are the Child and Adolescent Functioning Assesment Scale (CAFAS) (Hodges, Bickman, Ring-Kurtz, & Rieter, 1991), the Social Adjustment Inventory for Children and Adolescents (SAICA) (John, Gammon, Prusoff, & Warner, 1987) and the Hampstead Child Adaptation Measure (HCAM)(Target & Fonagy, 1992). They are all multidimensional tools based on the exploration of the different areas of social functioning through semi-structured interviews with the parents. These however are tools that call for a specific training, otherwise the reliability of the assessment will diminish. At the moment there are no studies demonstrating the superiority of one or the other of these measurements and this is not the right place for a detailed analysis of the individual tools which naturally offer each of the specific features. We only wish to recall that perhaps one of the advantages offered by the Hampstead Child Adaptation Measure is that in investigating the single areas of social adaptation, it considers the variability between the aspects as one of the adaptation indicators, therefore paying greater attention to the interdependence  of the different areas of social functioning.

Mechanism level

Following the ideal division of the various outcome levels described above, the third area of functioning taken into consideration is the one defined by Fonagy and Target as the mechanism level, using ‘mechanisms’ in the sense of the level of functioning of the cognitive and emotional capacities which probably underlie both the child’s symptomatology and his/her adaptation. The mechanism level certainly represents the most complex area of study since it refers to the child’s global mental functioning. It involves knowledge connected to normal and pathological development, and therefore depends partly on the researcher and also on the different theoretical orientations, as well as on the different theories of therapeutic action. It is well known that different approaches tend to favour one aspect or another of mental functioning, considering it less or more important in the development of the pathology.  Both clinicians and researchers, who have to deal more with these issues, underline the importance of identifying and specifying the processes and the mechanisms through which treatments achieve their results, stressing the need to establish with great precision a theory of normal or pathological child development strictly linked to a theory of therapeutic action.  

Transactional level

Current theorizations in the field of adolescent psychopathology place importance, for the development of both mental health and psychopathology,  on the transactional interactions between the child’s mental state, his/her behavioural predispositions and the reactions of the environment to the child in the course of time. The transactional level therefore refers to the set of environmental situations of which the child is part, and the corresponding outcome measurement deals with the assessment of the quality of these transactions.   Naturally when one talks about transactional levels what is meant is all those contexts that have a direct effect on the child’s life, like the relationship with parents, the quality of relationships between family members or the features of the social and school community. What one tends to assess are essentially the elements of the child’s environmental functioning which were traditionally covered by the definition of psychopathological risk factors.  

Service use and satisfaction level

The last measurement level proposed by Fonagy and Target concerns the experience of using the services and the degree of satisfaction with them. Theoretically the hypothesis is of a reduction in the use of the services after an effective treatment although, as regards adolescence, such a linear relation has not been demonstrated. On the contrary, the provision of a service seen as effective often produces a short term increase in the request for similar services. For instance Olds and Kitzman, in a research from about 1993, underlined that one of the main effects of a treatment aimed at parental support, carried out through home visits on the babies’ births, produced, due to the satisfaction shown by the customers, an increase in the use of a range of paediatric and medical services.  
In the United Kingdom a tool has been constructed ad hoc for economic assessment in the mental health sector, with a variation for the field of child and adolescent mental health called the Client Service Receipt Inventory (Beecham, 1995). There are therefore a number of tools for the collection of data on the use of the service but one should consider the fact that most of these tools were developed with the National Health Service in mind. Even though they are very well validated tools, they cannot be used in health settings other than their original context.
From this brief exposition, it seems obvious that the most complex level in terms of assessment is that of mechanisms. Too often the literature has favoured the other measurement areas, producing results that are not always reliable.  As has been pointed out a number of times, also by Fonagy in 2002, inevitably the demonstration of the efficacy of a treatment cannot be considered something absolute, inevitably its meaning is in fact determined by the cultural context that requests it and gives it meaning. Employing mainly the outcome indicators belonging to the symptom reduction area or to that of adaptation entails the risk of superimposing the definition of mental health with the normative aspects of the cultural context in which the research is carried out.
Furthermore as clinicians and researchers in the field of adolescence, we must be extremely careful with the use of tools created for adult research. Although we are well aware when the measurements we take with tests or more sophisticated tools do not do justice to the complex cognitive, affective and physiological processes involved in these measurements, we now have at our disposal enough valid tools for the assessment of the therapeutic process. It should however be remembered that most of these tools were developed in the context of adult psychotherapy research. For instance, sophisticated scales like that of Perry for the assessment of the functioning of defence mechanisms can create problems when applied to adolescence. I am not referring only to methodological problems linked to the standardization of the child language scale but more to the mental health model implicit in the tool itself. While with an adult patient it may  make sense to use the maturation level of defence mechanisms as an indicator of outcome, the same cannot be said for treatment involving children. We are well aware that the maturation process of defence mechanisms, or rather, of the defence organization, is not linear during development.
For the specific evaluation of the efficacy of psychotherapy in childhood and adolescence, we therefore need to anchor the criteria for assessing normal and pathological functioning in adolescent subjects to the data from empirical research into development. The results produced in the field of infant research in fact seem to provide clinicians and researchers with new hypotheses for an operative definition of the mental processes that protect against stressful events (see the studies by Fonagy and Target on mentalization) and above all indicate the flexibility and reversibility of adolescent development disorders, with interesting consequences for clinical work with child and adolescent patients.

References

Achenbach, T.H. (1995). Diagnosis, assessment and comorbidity in psychosocial treatment research. Journal of Abnormal Child Psychology, 23, 45-64..

Beecham, J.K. (1995). Collecting and estimating costs. In Knapp, M.R. (ed). The economic evaluation of mental health care, pp. 61-82, Aldershot, UK.

Bergin, A.E. (1963). The empirical emphasis in psychotherapy. A symposium. The effects of psychotherapy: negative results revisited. Journal of Consulting Psychology, 10, 224-250.

Bergin, A.E, & Garfield, S.L. (1971). Handbook of psychotherapy and behaviour change. New York: Wiley.

Dahl H., Kächele H., & Thomä H. (1988). Psychoanalytic Process Research Strategies. Berlin-Heidelberg-New York: Springer Verlag.

Eysenck, H.J. (1952). The effects of psychotherapy: an evaluation. Journal of Consulting Psychology, 116, 319-324.

Fonagy, P., Target, M., Cottrell, D., Phillips, J., & Kurtz, Z. (2002). What Works for Whom? A Critical Review of Treatments for Children and Adolescents. The Guifort Press.

Henggeler, S.W., Schoenwald, S.K., Rowland, M.., & Ward, D. (1999). MST versus psychiatric hospitalization: Placement outcome post-treatment. 46th Annual Meeting of the American Academy of Child and adolescent Psychiatry, Chicago.

Hoagwood, K., Jensen, P.S., Pretti, T., & Burns, B.J, (1996). Outcomes of mental health care for children and adolescents: I. A Comprehensive conceptual model. Journal of American Academy of Child and Adolescent Psychiatry, 35, 1055-1063.

Hodges, K, Bickman, L., Ring-Kurtz, S., & Rieter, M. (1991). Multi-dimensional measure of level of functioning for children and adolescent. In Algarian, A., & Friedman, R. (eds). A system of care for children’s mental health: Expanding the research base, pp. 149-154, Tampa.

John, K., Gammon, G.D., Prusoff,, B.A., & Warner, V. (1987). The Social Adjustment Inventory for Children and Adolescent (SAICA): Testing of a new semi-structural interview. Journal of the American Academy of Child and adolescent Psychiatry, 26, 898-911.

Kazdin, A.E. (2000). Psychotherapy for children and adolescents: Directions for research and practice. Oxford: Oxford University Press.

Kazdin, A.E., & Kendall, P.C., (1998). Current progress and future plans for developing effective treatments: Comments and perspective. Journal of Clinical Child Psychology, 27, 217-226.

Luborsky, L., & Bachrach, H. (1974). Health Sickness Rating Scale, Archives of General Psychiatry, 31, 292-299.

Olds, D.,&  Kitzman, H. (1993). Review of research on home visiting for pregnant women and parents of young children, Future of Children. 3, 53-92.

Shaffer, D., Gould,M., & Brasic, J.A. (1983). Children’s Global Assessment Scale (CGAS). Archives of General psychiatry, 40, 1228-1231.

Target, M., & Fonagy, P. (1992). Raters’ manual for the Hampstead Child Adaptation Measure (HACAM), London, UCL.

Weisz, J., Weiss, B., Han, S., Granger, & D., Morton, T. (1995). Effects of psychotherapy with children and adolescent revisited: A meta-analysis outcome study. Psychological Bullettin, 117, 450-468.

Notes

* Researcher, Faculty of Psychology, “La Sapienza” University of Rome, Italy. Top