Imagine: You’ve got a vacancy. It’s for a supervisory position that deals directly with the heart of your mission. The person who fills it needs to effectively motivate teams, build morale and ensure the productivity of each and every individual. You’ve got to make a choice among candidates. Some have previous experience; others come highly recommended. But you’ve never seen any of them leading this particular group to do this particular task. You essentially have to take a leap. How can you predict the future? “Everybody is looking for a crystal ball to predict what a person will actually be like on the job” (White, 2006, B3).

That’s a dilemma, whether it is selecting a manager for a call center or a community center. How do you choose the best leader for the job? Or at the very least, how do you avoid some of the most common pitfalls? Two traps are, according to the CEO of Development Dimensions International (Byham, 2002): “believing in the accuracy of an internal nomination system” and “being fooled by someone's excellent experience record or lack thereof” (p. 59).

What’s needed is a practical predictor that taps interpersonal behaviors found effective in motivating employees. This study addresses all three issues. First, it uses an in-basket simulation (Schippmann, Prien, & Katz, 1990), in which respondents take only an hour or so to respond in their own words to emails, reports, and phone messages in their in-basket. Because all respondents are faced with identical situations, you can make meaningful comparisons among them. Second, instead of cognitive tests commonly given to candidates (e.g., Goldstein, Zedeck, & Goldstein, 2002), the focus is on interpersonal behaviors. Third, the behaviors – monitoring or gathering performance information and providing consequences – are based on operant conditioning theory and deal with back-and-forth exchanges. Furthermore, empirical evidence supports highlighting these behaviors (Komaki, 1998). The time leaders spent on one or both of these behaviors made a difference in every one of seven field studies conducted on three continents. Exemplary insurance managers, for example, asked employees about their performance, observed employees in action, and examined the products of their work (Komaki, 1986). Top-rated sergeants in Australia communicated their evaluations of performance, sometimes positive, sometimes negative, often neutral (Brewer, Wilson, & Beck, 1994). Effectiveness in motivating others ranged from ratings of superiors or subordinates to actual outcomes (Komaki, Desselles, & Bowman, 1989).

Hypothesis 1 states that motivational effectiveness will be significantly related to monitoring and providing consequences, as measured using an in-basket, the Motivational Effectiveness Exercise (MEE). Although empirical evidence exists, data about the two behaviors were gathered by observing managers on the job -- as many as 10 hours (e.g., Komaki, Zlotnick, & Jensen, 1986). This laborious method of data collection would be impossible when screening candidates, even internal ones. But the MEE, which would be well suited in selecting or promoting candidates, has been the subject of only one study (Komaki, Newlin, & Desselles, 1990). A multi-trait, multi-method assessment, it compared behaviors measured via different methods and showed that the information obtained by observation was comparable to that via an in-basket. No study, however, has looked at the MEE and effectiveness. If it could be demonstrated that managers who are effective appropriately monitor and provide consequences as measured on an in-basket, this would provide evidence of a practical and valid measure with which to select managers.

Hypothesis 2 introduces a category of behavior identified in operant conditioning as secondary for motivational purposes, that of antecedents. These rules, reminders, and exhortations are portrayed with chagrin by Aubrey Daniels (1994): “In most organizations we attempt to manage performance by telling people what to do. … We send memos, have meetings, write policies, hold classes, and make informational and inspirational speeches. Interestingly, when these methods don’t get the desired response …, we … send new memos around (with bolder type, capital letters, and even exclamation marks)” (pp. 17-18). Given the widespread reliance on antecedents in the field, persons choosing managers are likely to weigh favorably the touting of information and inspiration. Hence, antecedents might be helpful to include in the prediction equation.

Alas, no direct evidence exists for the possibly deleterious impact of relying on antecedents in managing performance. In the seven field studies showing that monitoring and/or providing consequences were positively related, correlation coefficients for antecedents frequently hovered around zero. Only one study, in which data were painstakingly collected every five seconds during sailboat races, showed giving many antecedents before monitoring was associated with being less effective as a leader (Reynard & Komaki, 1995). Another study, with newspaper managers, found antecedents to be a classic suppressor variable (Jensen & Komaki, 1993). That is, they were not related to effectiveness but, in a multiple regression analysis with monitoring and providing consequences, had a negative beta weight. The latter indicates that unless managers raise their delivery of monitors and consequences, they should refrain from increasing antecedents. Pertinent here was evidence showing that adding antecedents actually enhanced prediction. That is, knowing the score on managers delivering monitors, consequences, and antecedents yielded a better prediction than relying on monitors and consequences. Without antecedents, the coefficient was .37. Including antecedents, however, raised the multiple correlation to .50 (p < .05). Hence, Hypothesis 2 states that prediction will be enhanced by including monitors, consequences, and antecedents. If it could be demonstrated that the three behaviors together were better predictors than providing monitors and consequences alone, then one would have a basis for including antecedents, which in turn would assist in making a better-informed decision when selecting among notoriously difficult-to-predict candidates.

Method

Subjects were 35 investment bankers in a multinational investment bank who volunteered to participate in the study. All had created, led, and participated in “deal teams,” moving deals to closure (e.g., arranging a multi-million dollar transaction for a company in Los Angeles who was financing ships built in Germany and “flagged” in Liberia, to be owned by United Kingdom subsidiaries).

The Managing Director of the group introduced the study as a personal growth and development opportunity including individual coaching. After the MEE was administered and scored, one-on-one feedback sessions were conducted with 94% of the participants. A list of the behaviors and scores in chart form were provided. For U.S. participants, sessions were in person, for those in Europe and Asia-Pacific, via videoconference.

An in-basket exercise was revised. Respondents took on the role of publisher of a food magazine and responded to 22 items (memos, reports, phone messages, and e-mail messages) from direct reports, e.g., directors of production, human resources, advertising, marketing/circulation. For sample items, refer below to Tables 2, 3, and 4.

Categories. Three categories of behavior (monitoring, providing consequences, and giving antecedents), as well as numerous subcategories (e.g., method of monitoring, evaluation of consequences) or aspects (e.g., timeliness of monitoring, diplomacy of consequences) were scored (Table 1).

Table 1
Major Categories and Subcategories of Behavior Measured in the In-basket

^a The portion of the verbatim example that is in italics is an illustration of the category or subcategory.

Scoring. Point values were assigned depending upon category and subcategory or aspect. Higher points were awarded for higher quality responses, e.g., monitoring via work sampling counted for more than via self-report. It was possible to receive points for more than one category of behavior on an item. Overall category scores were calculated by summing subcategory scores across items (e.g., consequences overall consisted of the sum of positive/neutral, and negative).

Responses from all participants on all items were scored independently by both authors. Inter-rater agreement was calculated. Average agreement scores were 91.4% for monitors, 90.5% for consequences, and 82.5% for antecedents overall.

Participants were asked to rate only those they had directly observed in action. Ratings came from several sources, including managers, peers, direct reports, and others who may have worked on a team with a person. Ratings from A to F were made, with A being the best. Averages of all ratings received for each participant were calculated for effectiveness at motivating others and technical expertise.

Results

Hypothesis 1 was supported. Both monitoring (r = .30, p < .05) and providing consequences (r = .29, p < .05) were significantly correlated with effectiveness. Furthermore, the effective bankers monitored via work sampling (r = .28, p < .05). What this meant was that the top rated motivators were more likely to monitor by requesting to watch employees work or sample/inspect the work. In Table 2, for example, the banker actually wanted to see the aforementioned forms. Top rated motivators also took the time to provide consequences, especially positive ones (r = .33, p < .05). In addition, they delivered a newly designated subset of positive consequences, referred to as thanking the bearer of bad news (r = .40, p < .01). Bankers who acknowledged people for drawing attention to unwanted information, as shown in Table 3, were more likely to be effective. In contrast, the responses of the bottom-rated motivators are particularly illustrative. Although not necessarily representative, the bottom-rated banker in Table 2 recommended, again and again, using antecedents to solve the problem.

Table 2
Monitoring via Work Sampling in Responses to MEE Item from Top and Bottom Rated Motivators

Note. Monitoring via work sampling is shown in italics.

Table 3
Providing Positive Consequences by Acknowledging the Bearer of Bad News in Responses to MEE Item from Top and Bottom Rated Motivators

Note. Acknowledging the bearer of bad news is shown in italics.

Although no support was found for Hypothesis 2, an interesting but unanticipated finding was found regarding antecedents and technical expertise. Providing antecedents, and in particular the tacked-on traditional variety, was related to technical skill (r = -.44, p < .01). Rarely did bankers rated high in technical skill add on directives, instructions, or reminders when they were not needed or requested. In contrast, the low-rated bankers more often repeated what needed to be done, even if they ended with an inspirational nugget, as one banker did in Table 4: “[M]ake sure the information is correct. … [L]et’s keep up the improvement … and aim to reduce that to zero.”

Table 4
Tacked on Traditional Antecedents in Responses to MEE Item from Top and Bottom Rated Technical Experts

Note. The tacked on traditional antecedents are shown in italics.

Discussion
As hypothesized, the effective bankers who completed an in-basket exercise monitored via work sampling and provided consequences that were positive. These results not only affirm the importance of delivering monitors and consequences but they also have practical implications for selecting and promoting managers. To select a manager whose job necessitates motivating others, we recommend asking interested candidates to take the MEE, computing overall scores for monitoring and providing consequences, and using the MEE scores to select among candidates for the job.

Based on these findings, several avenues are wide open for future research:

1. Consider replicating the results, for example, with school principals and health care administrators.

2. Build upon the list of different types of positive consequences that we have generated:

a) giving simple, short acknowledgements (“Thanks for taking my messages.”);
b) broadcasting positive consequences to multiple parties (To all direct reports: “Great work by the Ad Team.”);
c) letting a memo sender know what action is being taken (“I’ll talk to Marketing.”); and
d) acknowledging the bearer of bad news (as shown in Table 3). To our knowledge, the latter two, while more subtle, have not been explicitly identified as positive consequences. In the case of acknowledging the bearer of bad news, we found that it was significantly related to effectiveness.

3. Revamp or expand the concept of antecedents. In this study, we differentiated between two types, relying on antecedents to motivate others (Table 2) and the tacked-on variety (Table 4). While the former fit the more traditional definition of antecedents as conveying expectations for performance, the data indicated that tacking on antecedents when they were not explicitly asked for were the ones distinguishing between high-and-low rated technical experts. No doubt, there are other ways of categorizing antecedents. Ideas about other categorizations, which might lead to a fuller understanding of the complex role of antecedents in motivation, would be a major contribution to the leadership literature.

4. Lastly, use the in-basket, as we have done recently with emergency medical service managers (Komaki, Minnich, Grotto, & Kern, 2006), as the dependent variable to assess leadership development training.

In sum, if you’re concerned about choosing the best person for a managerial job, consider using an in-basket simulation.  As Fred Fishbach of Applied Psychometrics Inc. explains: “People can fake an interview. .  . . But they can’t bluff their way through (a simulation)” (Lopez, 1993, p. B1). Furthermore, managers who consistently monitored via work sampling and provided positive consequences on the MEE were those judged to be effective in motivating others.  Hence, evidence exists that the Motivational Effectiveness Exercise can be both a practical and valid predictor for those incredibly difficult to predict softer skills.

References

Brewer, N., Wilson, C., & Beck, K. (1994). Supervisory behaviour and team performance
amongst police patrol sergeants. Journal of Occupational and Organizational Psychology, 67, 69-78.

Byham, W.C. (2002). 14 Leadership traps: Why many CEOs don't have the leadership bench strength they need. Business and Management Practices, 56(3), 56-64.

Daniels, A.C. (1994). Bringing out the best in people:  How to apply the astonishing power of positive reinforcement. New York, NY:  McGraw-Hill, Inc.

Goldstein, H.W., Zedeck, S., & Goldstein, I.L. (2002). g: Is this your final answer?. Human Performance, 15, 123-142.

Jensen, M. & Komaki, J.L. (1993, May). Beware of too many directives: Spotting a suppressor variable in the operant model of effective supervision. In J. Cannon-Bowers (Chair), Optimizing team performance through team leader behavior. Symposium conducted at the conference of the Society of Industrial and Organizational Psychology, San Francisco, CA.

Komaki, J.L. (1986). Toward effective supervision:  An operant analysis and comparison of managers at work. Journal of Applied Psychology, 71, 270-278.

Komaki, J.L. (1998). Leadership from an operant perspective. London: Routledge.

Komaki, J.L., Desselles, M.L., & Bowman, E. (1989). Definitely not a breeze:  Extending an operant model of effective supervision to teams. Journal of Applied Psychology, 74, 522-529.

Komaki, J.L., Minnich, M.L.R., Grotto, A., & Kern, M. (2006, May). Developing leaders using a model and criterion reflecting everyday complications. Poster presented at the conference of the Society for Industrial and Organizational Psychology, Dallas.

Komaki, J.L., Newlin, M.H., & Desselles, M.L. (1990, April). Walking on the wild side: Criterion related validation of an in-basket exercise of supervisory behaviors. In M. H. Newlin (Chair), Simulated performance assessment: Fact or fantasy? Symposium conducted at the conference of the Society of Industrial and Organizational Psychology, Miami, FL.

Komaki, J.L., Zlotnick, S., & Jensen, M. (1986). Development of an operant-based taxonomy and observational index of supervisory behavior. Journal of Applied Psychology, 71, 260-269.

Lopez, J. A. (1993, October 6). Management: Firms force job seekers to jump through hoops. The Wall Street Journal, p. B1.

Reynard, M. & Komaki, J.L. (1995, May). The music of management: Rhythms of effective leaders. Poster session presented at the conference of the Society of Industrial and Organizational Psychology, Orlando, FL.

Schippmann, J.S., Prien, E.P., & Katz, J.A. (1990). Reliability and validity of in-basket performance measures. Personnel Psychology, 43, 837-859.