Using concept mapping as a planning tool: Child welfare citizen review panels

Citizen Review Panels (CRPs) are groups of citizen volunteers authorized by U.S. federal law to examine state child welfare agencies. These groups inspect policies and practices related to child protection responsibilities and are tasked with making recommendations for systemic improvement. Despite the federal mandate for each state to develop a CRP and the potential of these groups to positively impact child welfare practices, there is a dearth in the literature related to CRPs. Consequently, planning and evaluation processes of these groups vary widely. This study reports on the use of concept mapping (CM) to outline a framework for planning and subsequently evaluating the CRP in one southeastern state. CM is a mixed-method research approach that uses multi-dimensional scaling and hierarchical cluster analyses to explore an area of study. Through these analyses, the method creates visual depictions of conceptual relationships between ideas. Data yielded a seven cluster concept map that CRP members (N = 36) utilized for planning processes, and subsequently for developing an internal evaluation tool. Results from this study offer a unifying framework by which CRPs, and similar groups in other areas can utilize for planning and evaluation purposes. After a review of pertinent literature on CRPs, this article explicates CM processes utilized in this study, describes results, discusses lessons learned, and outlines apposite areas for future CRP research.     

Staff and youth views on autonomy and emancipation from residential care: A participatory research study

The use of participatory approaches in designing services is still relatively uncommon. In this study, we helped design a service to support the transition of youth from residential care to independent living by exploring the perspectives of staff and of youth regarding: (a) the concept and development of autonomy; and (b) key factors in developing this type of service. We gathered the data through 10 interviews with staff (n = 10) and 4 focus groups with youth (n = 21), and subjected the data to a thematic content analysis. Staff defined autonomy as self-regulation and self-care, and identified three paths to foster autonomy – a sense of normality, meaningful relationships, and planning for emancipation. The staff and youth identified the following important aspects in designing the service: achieving normality (e.g. limited number of residents), promoting youth capacity (e.g. skill-building activities), providing social support (e.g. trust and respect between residents), and assuring guidance and boundaries (e.g. supervision of youth).     

Asking for Change: Feasibility,acceptability, and preliminary outcomes of a manualized photovoice intervention with youth experiencing homelessness

Photovoice is a participatory action research method that empowers participants to photograph their everyday lives as a means of documenting and advocating for their needs; it has rarely been utilized with young people experiencing homelessness. The current study examined the feasibility, accessibility, and preliminary outcomes associated with participation in Asking for Change, a manualized Photovoice intervention, among youth (ages 18–21) staying in a homeless shelter (N = 22). Multiple sources of data, including field observation, standardized pre-post measures, and qualitative exit interviews were collected across two cohorts of Asking for Change. Results suggest the intervention was feasible and highly acceptable to many young people, created new opportunities to connect with young people, and, among those surveyed pre and post intervention (n = 9), was associated with improvements in communication skills, social connectedness, resiliency, and well-being. This article discusses the challenges and benefits inherent in doing this work and outlines a robust research agenda to move this knowledge base forward.     

Barriers and facilitators to evaluation of health policies and programs: Policymaker and researcher perspectives

Our research sought to identify the barriers and facilitators experienced by policymakers and evaluation researchers in the critical early stages of establishing an evaluation of a policy or program. We sought to determine the immediate barriers experienced at the point of initiating or commissioning evaluations and how these relate to broader system factors previously identified in the literature.We undertook 17 semi-structured interviews with a purposive sample of senior policymakers (n = 9) and senior evaluation researchers (n = 8) in Australia.Six themes were consistently raised by participants: political influence, funding, timeframes, a ‘culture of evaluation’, caution over anticipated results, and skills of policy agency staff. Participants also reflected on the dynamics of policy-researcher relationships including different motivations, physical and conceptual separation of the policy and researcher worlds, intellectual property concerns, and trust.We found that political and system factors act as macro level barriers to good evaluation practice that are manifested as time and funding constraints and contribute to organisational cultures that can come to fear evaluation. These factors then fed into meso and micro level factors. The dynamics of policy-researcher relationship provide a further challenge to evaluating government policies and programs.     

The effects of organizational context on autonomy-enhancing policies in assisted living

This study examines the effects of facility size, ownership, chain membership, and residents' characteristics on autonomy-enhancing policies in assisted living. The theoretical framework for the study is based on the open systems perspective, which views organizations as being influenced by environmental context (e.g., ownership and chain membership). Data were collected from interviews with administrators of 60 facilities in Maryland. Autonomy-enhancing policies were assessed with the MEAP Policy Choice and Resident Control scales. Facility size and residents' disability were strong predictors of autonomy-enhancing policies (R² = .39, p < .001). Higher levels of policies that foster resident autonomy were associated with larger facility size (β = .54, p < .001) and lower residents' disability (β = .23, p < .05). Chain membership had an indirect influence on autonomy-enhancing via facility size and residents' disability. Chain-related facilities were larger, and their residents were less disabled. The study contributes to a better understanding of the mechanism through which organizational factors influence resident-oriented policies.     

Measuring and explaining tax evasion: Improving self-reports using the crosswise model

Due to its sensitive nature, tax compliance is difficult to study empirically, and valid information on tax evasion is rare. More specifically, when directly asked on surveys, respondents are likely to underreport their evasion behavior. Such invalid responses not only bias prevalence estimates but may also obscure associations with individual predictors. To generate more valid estimates of tax evasion, we used a new method of data collection for sensitive questions, the crosswise model (CM). The CM is conceptually based on the randomized response technique (RRT), but due to its advanced design, it is better suited for large surveys than classical RRTs. In an experimental online survey, we compared the CM (N = 862) to standard direct questioning (DQ; N = 305). First, our results showed that the CM was able to elicit a higher proportion of self-stigmatizing reports of tax evasion by increasing privacy in the data collection process. Second, on average, we found stronger effects of our predictor variables on tax evasion in the CM condition compared with the DQ condition such that an egoistic personality and the opportunity for tax evasion predicted actual tax evasion only in the CM condition.     

Finish what you start: A study of Design Team change initiatives' impact on agency climate

This study employs a multi-site longitudinal design to examine the effect of a Design Team intervention on organizational climate. Thirteen private, not-for-profit child welfare agencies from one state participated in a Design Team intervention to address workforce needs. A total of 407 workers from those agencies responded pre and post intervention to a survey that measures worker perceptions of the psychological climate of their organization using the Parker Psychological Climate Survey. Workers in organizations that completed the Design Team intervention had statistically significant increases in three of the four dimensions of the Parker scale. On the role dimension, significant change was noted on all three subscales on the interaction between Time 1 and Time 2 (ambiguity: p = 0.012; conflict: p = 0.04; overload: p = 0.05). On the organization dimension, the justice and support subscales had significant differences in the desired direction (justice: p = 0.05; support: p = 0.03). On the supervisor dimension, significant change was observed in the desired direction for both the goal emphasis and work facilitation subscales (goal emphasis: p = 0.02; work facilitation: p = 0.00). Statistically significant improvements in the organizational climates of child welfare agencies suggest the benefit of future research to test the effectiveness of Design Team interventions in other service areas. These findings build on intervention research with organizations by linking the ability of an organization to fully implement a change initiative to their capacity to improve the workplace climate for employees.     

Using the port of entry report as a benchmark: Survey results of on-the-job training among public relations internship site managers

The more up-to-date educators are regarding what the public relations industry expects from students, the more lucrative are students’ opportunities for potential employment upon graduation. This study looks through the lenses of internship supervisors within the framework of the “Port of Entry” study. Its purpose is to determine the skills and professional characteristics site supervisors deem most necessary for interns employed in their organizations. Results (n = 109) indicate that in general, writing, oral, and organizational skills are most necessary for interns; professional characteristics include motivation and personality/attitude.     

Ready or not: The influence of readiness on healthy marriage and relationship education training outcomes

This exploratory study utilizes a sample of human service providers and administrators to examine personal, agency, and collaborative readiness as predictors of training outcomes (utility reaction, gains in knowledge, and transfer of learning). The study is informed by the child welfare training evaluation model to address the effect of participants' (n = 58) readiness indicators and learning impact in relation to the transfer of learning that takes place following a one-day training. Participants' personal and agency readiness predicted their utility reaction, while personal readiness predicted gains in knowledge. The association between agency readiness and transfer of learning was partially mediated by utility reaction. Implications are drawn for both human service providers and researchers.     

Social Return On Investment (SROI): Problems,solutions … and is SROI a good investment?

The conclusion of this special issue on Social Return On Investment (SROI) begins with a summary of both advantages and problems of SROI, many of which were identified in preceding articles. We also offer potential solutions for some of these problems that can be derived from standard evaluation practices and that are becoming expected in SROIs that follow guidances from international SROI networks. A remaining concern about SROI is that we do not yet know if SROI itself adds sufficient benefit to programs to justify its cost. Two frameworks for this proposed metaevaluation of SROI are suggested, the first comparing benefits to costs summatively (the resource → outcome model). The second framework evaluates costs and benefits according to how much they contribute to or are caused by the different activities of SROI. This resource → activity → outcome model could enable outcomes of SROI to be maximized within resource constraints (such as budget and time limits) on SROI. Alternatively, information from this model could help minimize the costs of achieving a specific level of return on investment from conducting SROI. Possible problems with this metaevaluation of SROI are discussed.     

From assessing needs to designing and evaluating programs: Case study of a family support program in Portugal

In the last years, researchers have been emphasizing the importance of promoting needs-led, context-specific, user-centered services in the context of child protection. However, policy-makers and service planners around the world largely depend on US-based research evidence of what is effective in the domain of family support. This work presents, in two studies, the process of design and evaluation of a targeted family support intervention that was developed and implemented in Portugal.Following the Common Language Approach to needs assessment (Dartington Social Research Unit, 2001), in study 1 we screened 100 children and their families attending a generalist child care service for risk and protective factors. Four different clusters of needs were identified. One cluster was selected as target-group for the design of a service to match their needs. Results indicated that families in this cluster had socioeconomical disadvantages, a challenging family environment and inadequate parenting practices. Children were showing signs of problematic social behaviors.In study 2 we describe the theoretical process model and the logic model for the intervention, and experimentally evaluate the program's efficacy. Results from pre–post assessments indicate improvements in the intervention group (N = 20) in several areas of parental empowerment and family relations, comparing with the control group (N = 20).This approach to need-service matching seems to be a viable pathway to design needs-led, context-specific, and user-centered services, and to assess their efficacy, thereby informing policy makers and service planners.     

Internalizing and externalizing behavioral problems of American Indian children in the child welfare system

Children involved in the child welfare system display elevated or clinically significant behavioral problems. However, there is a dearth of literature on the behavioral problems of American Indian children following child welfare involvement. Grounded in Patterson's Family Adjustment and Adaptation Response theory, this study fills that gap. Baseline, 18-month, and 36-month follow-up data from the National Survey of Child and Adolescent Well-being were utilized. The sample (n = 3498) consisted of American Indian, African American, and Caucasian children ages 2–16 at baseline (M = 8.13 years old, SD = 3.85) and 51.7% were female. Nearest neighbor propensity score matching analyses were used to estimate the effect of race on clinically significant internalizing and externalizing behavioral problems. Findings suggest that although externalizing behavioral problems do not differ based on race after controlling for other important factors, internalizing behavioral problems do differ. American Indian children are more likely to display clinically significant internalizing behavioral problems.     

The plans,goals, and concerns of pre-emancipated youth in foster care

This study focuses on the plans, goals, and concerns of foster care youth prior to leaving care. Participants were 179 pre-emancipated youth between the ages of 17 and 20 years old (M = 17.82, SD = 0.79) from a large metropolitan area in Southern California. Self-articulated immediate plans were grouped into 4 major categories and self-articulated life goals were grouped into 10 categories while also examining the prioritization of, estimated time frame for, and youth's sense of control over their life goals. Survey and interview data reveal the importance of educational and occupational life goals and their prioritization. Youth reported a high level of certainty in their immediate plans, but the youth were less explicit in describing their immediate plans for the year after foster care. Foster youth may have difficulty identifying concrete steps to make plans a reality despite their ideas for the future. Youths' worries and concerns about their post-emancipation plans and life goals typically centered around academics and finances. This study contributes to the limited literature on the life goals and plans for foster youth; these results reinforce the need for greater support in planning and goal setting prior to emancipation.     

A comparison of outcomes for children and youth in foster and residential group care across agencies

Working collaboratively with two state associations and their member (nonprofit) agencies providing out-of-home care to children and youth, University researchers conducted a multi-site project to examine whether there were any differences in individual child-level outcomes between children placed in residential group care and those placed in foster. The study employed a quasi-experimental repeated measures design, with data collected at a minimum of two intervals (at intake and 3-month follow-up) and at subsequent intervals of 6 and 12 months for children remaining in care. Samples for analyses were drawn from 1082 youth in either residential group care (n = 903) or foster care (n = 179), in one of 37 agency sites across two southeastern states, who participated in a broader evaluation project. The average ages of participating youth in residential and foster care were 13.97 (SD = 2.43) and 13.65 (SD = 2.73), respectively. Based on his or her score on the Children's Global Assessment Scale (CGAS) at intake, each participant was also assigned to the low functioning group (n = 526; 53.1%), the borderline group (n = 232; 23.4%), or the high functioning group (n = 232; 23.4%). Analyses confirmed that youth in foster care tended to have higher levels of general functioning at baseline than did youth placed in group care. However, the degree to which youth progressed in care on measures of general functioning and mental and behavioral health problems did not differ based on placement setting; youth in residential group care settings progressed at the same rate as youth in community-based settings, regardless of their level of functioning at intake. The only exception to this pattern was in regard to anxiety; there was an observable, but non-significant trend of youth in foster care reporting decreases in anxiety levels, while those in group care reported increased anxiety.     

Exploring the self-care practices of child welfare workers: A research brief

Child welfare workers experience higher rates of vicarious trauma, workplace stress, and compassion fatigue, when compared to other social service workers. Increasingly, social service agencies, in general, and child welfare agencies, specifically, recognize the importance of self-care in assuaging these problematic employee outcomes. However, research that explicitly examines the self-care practices of child welfare workers in nominal. This study brief explores the self-care practices of child welfare workers (N = 222) in one southeastern state. Results reveal that child welfare workers only engage in self-care at moderate levels. Additionally, data suggests that variables such as health status, current financial status, and relationship status significantly impact personal and professional self-care practices, respectively. After a terse review of relevant literature, this brief will explicate findings associated with this study, and identify salient discussion points and implications for child welfare training, practice, and research.     

Using a community-based participatory research approach to evaluate resident predictors of involuntary exit from permanent supportive housing

ObjectiveThe aim of this study was to determine an association between resident characteristics at time of entry to permanent supportive housing and exit status.MethodsA community-based participatory research (CBPR) approach was the guiding framework for the design, implementation and evaluation of this project. This retrospective observational study used an administrative data source from a local permanent supportive housing provider to compare resident characteristics among those who left housing voluntarily or involuntarily.ResultsThe population based sample (n = 407) was comprised of 51% males and 47% African-Americans with a mean age of 40 years (SD = 11.8). Involuntary exits (IEs) occurred in 40% of the sample (n = 166). IE was less likely with receipt of mainstream benefits, compared to employment income (O.R. = .546, p = .032). IE was more likely for residents self-identifying as African-American (O.R. = 1.56, p = .037) and when children resided in the household (O.R. = 2.03, p = .013).ConclusionsDespite limitations of community-derived data, findings suggest that supportive housing providers consider income source and family status when designing interventions to decrease IEs. A CBPR approach is a promising framework to guide evaluation efforts for supportive housing programs.     

Is child protective services effective?

A number of studies have concluded that there is little observable connection between CPS involvement and improved outcomes for children and families. Evidence of CPS effectiveness is complicated by the presence of selection bias and difficulty controlling for confounding. To understand outcomes by group and intervention effects, comparable groups are necessary and difficult to ascertain using CPS administrative case record data. This study examines the causal effect of CPS involvement on the likelihood of future maltreatment using administrative case management records from July 1, 2010 and June 30, 2011. The current study accounts for differences in pre-existing condition between groups to establish sound estimates of CPS involvement effects. Logistic regression models were used to examine the difference in subsequent substantiated investigation between families with comparable risk and differing service recommendation (p = 0.83), recurrence among families with comparable risk, the same service recommendation that did or did not receive services (p = 0.83). Hazard models were used to explore risk of substantiated investigation among families with comparable risk and differing service recommendation (p = 0.77). Results indicate receipt of CPS services had no observable effect on recurrence of maltreatment overall and among families with similar levels of risk of recurrence. Further inquiry into worker attributes, decision-making, types of and quality of services offered to families could help explain the effective, or ineffectiveness, or services.     

Gang involvement moderates the effectiveness of evidence-based intervention for justice-involved youth

There are no validated, evidence-based intervention approaches to helping youth who are involved in gang activity. The goal of this study was to evaluate the impact of gang involvement on the effectiveness of evidence-based intervention services for problem behavior delivered to youth referred by the justice system. We analyzed data drawn from 421 youth (69% male; M age = 15.08 years, SD = 1.32; 38% Black/African-American, 18% Latino/a, 34% White, 10% other) referred consecutively over a 13-month period for Multisystemic Therapy (MST; Henggeler, Schoenwald, Borduin, Rowland, & Cunningham, 2009) across clinical service sites in 7 different Eastern states of the US. Outcomes were indicated by successful or unsuccessful case closures, and gang involvement was indicated by a validated multi-factored classification scheme. We conducted analyses of outcomes related to gang involvement for the whole sample as well as a propensity score-matched (PSM) reduced sample. Analyses of treatment success rates indicate that gang involvement significantly and substantially reduces the effectiveness of MST in this population.