EVALUATION OF MERI LEGISLATOR RATINGS METHODOLOGY

by Douglas I. Hodgkin, Professor Emeritus of Political Science

Bates College

            The methodology that MERI uses to rate Maine legislators is a sophisticated process that comes much closer to the goals that such ratings attempt to achieve than do most such efforts.  Most ratings are based solely upon a few roll calls selected by the staff of the organization.  This is a quick and cheap method because roll calls are so readily accessible.  However, the roll call is only one dimension of the legislative process and of the record of a legislator.  It also is problematic to rely solely upon the judgment of staff, for there is evidence that staff may be out of touch with their membership and may have ideological agendas of their own.  They may select roll calls to achieve a particular result.  They may include roll calls that have little to do with the central purpose of their organization.

            MERI is to be commended for a methodology that relies upon the judgment of those whom they represent.  Moreover, the ratings are based in significant part upon a “qualitative” assessment of the attitudes and activities of legislators.  While these are difficult to measure and to convert into a rating, these elements of legislators’ records are just as important, if not more important than the roll calls.

            It is impossible to develop a methodology that is immune to criticism.  At every step in the process, judgment is required.  If different judgments are possible, critics who dislike the outcomes will challenge the objectivity of the ratings.

            At least three central questions guide the critique of a methodology:

            1.  Does a method measure what the user says is being measured?

            2.  Does a method measure accurately?  Is there a better measure of the same thing?

            3.  Once a measure is selected, is there a chance for bias in the collection of data?

I.  Senior Management Survey.  This is a survey of business leaders to determine the issue areas relevant to the business community.  There seem to be two overlapping, but possibly distinct criteria for the selection:  (1) issues “essential to their success” and (2) issues “which are fundamental to a strong economy.”  (The criterion of “their ability to create quality jobs” is possibly a distinct goal, but may be a component of the latter.)  Are both criteria used equally by the business leaders?  Are the criteria always consistent with one another?  Which are you really measuring: issues essential to business success (how defined?) or issues “fundamental to a strong economy”?

            The use of business leaders to judge what is essential to their businesses’ success is probably “as close as you can get to knowing what” business leaders consider best public policy of benefit to their business.  The selection of a particular issue to include in the rating undoubtedly depends upon business consensus on this issue.  Undoubtedly there are some issues of disagreement, as when small businesses compete with large businesses or when businesses who are customers of other businesses have different interests.

            Even if businesses are in consensus on an issue as helping business, we have the stickier question of what is “fundamental to a strong economy.”  Economists disagree on this.  Are large tax cuts good or bad for the economy?  Is NAFTA good or bad for Maine’s economy?  As long as you have these disagreements, you can be challenged on whether the issues selected are good or bad for the economy.  The challengers essentially will say that you are not measuring what you say you are measuring.  The bottom line is that you are measuring what the business community says is “fundamental to a strong economy.”

            The overlap between these criteria comes with the assumption that if businesses are healthy, the economy is healthy.  Policies that promote healthy businesses thereby promote a healthy economy.

            For purposes of influencing general public opinion, it could be difficult to argue that this is without bias.  However, I understand that a major purpose of the rating is to address the business community, and only secondarily the general public.  Most business people do not have the time to investigate the record of relevant legislators.  The MERI rating is an excellent service to business people.  It tells them how legislators perform on issues that their own colleagues judge are the relevant pieces of legislation.

            A breakdown of the respondents by number of employees is provided.  Since there is probably a degree of self-selection in the decision to participate, it would be helpful to know whether the distribution is similar to that of the business sector as a whole.  It would also be helpful, but probably impossible to determine, to know whether the survey participants had the same party distribution as the business sector as a whole.  These are questions concerning the representativeness of the participants.

II.  Key Legislation.  The staff reviews all bills introduced in the Legislature to identify those germane to the issues identified in the management survey.  This is an appropriate and essential function of staff.  Although staff can have influence, MERI has built in some checks with the participation of an Advisory Committee and final approval of the legislation selected by MERI’s board.  The composition of the two bodies might be examined for bias.

III.  Legislator Ratings.  As noted, the inclusion of both roll calls and a measure of other forms of legislative participation is a major strength of  this methodology.

MERI Voting Record Evaluation:  Governmental affairs representatives recommend roll calls for inclusion in the voting record evaluation.  They are undoubtedly the best judges of what they consider to be crucial roll calls for the business(es) they represent.  MERI’s Board apparently determines who is eligible to participate.  Is there any bias in the selection of these participants?  Are they representative of business representatives?

            If the business representatives are representative of the pool from which they are chosen, strict staff deference to the governmental affairs representatives regarding selection of roll calls and determination of the correct position ensures that the composition of this portion of the rating adheres to the intent that the rating reflect the judgment of business people on these issues.

            Most ratings weight each roll call or bill equally.  The MERI weighting of each bill according to frequency of submission is a further sophistication of this system.  Those roll calls of broader importance to business and the economy should carry greater weight.  MERI’s Board rather than staff make the final check on whether a roll call is included.

MERI Qualitative Evaluation.  By definition, a qualitative evaluation is more subjective.  However, it is essential to get at behavior that is not easily quantifiable from documents or other records.  It is indicated that the governmental affairs representatives evaluate “behavior in committees, issue advocacy, and general support for important economic and business issues.”  Apparently, the evaluator provides a general rating as a total impression of the legislator.  It is hoped that the evaluator is given a brief checklist of behaviors to consider so that the one doing the ratings is encouraged to think of a variety of behaviors and contexts.

            The ten-point scale which ranges from 10 to 100 has a bias toward raising an anti-business legislator’s pro-business score.  There is no 0 (zero) permitted even though a legislator might be rabidly anti-business in philosophy and behavior.

            It is an excellent feature that the evaluators are to by-pass rating legislators not known well enough.  This enhances the accuracy of the assessment.  What happens when a legislator has such a low profile that there are no ratings?  It could also be problematic if there are only one or two ratings;  the rating could be skewed by the judgment of one or two people.

MERI Legislator Rating Calculation.  The combination of the two scores is always an arbitrary process.  Critics can always question their weighting in the formula for the calculation of the summary score.  However, a rationale for the formula needs to be thought through.

            As indicated above, the roll calls may be less important than other dimensions of the legislative process.  It is more accurate to weight the “qualitative” dimension more than the roll call score.  MERI’s system does this, but it does so in a cautious manner.  If two-thirds or more of the roll calls are decided by less than two-thirds of the vote, the qualitative and voting record components are weighted equally.  The apparent rationale is that closely contested roll calls reflect highly contentious issues that test most strongly the legislator’s position.  One-sided roll calls reflect consensus and votes on which a legislator’s philosophy is less challenged.  If most of the roll calls are highly contentious, then it is less necessary to weight the qualitative dimension more highly; the more objective roll calls do an adequate job of sorting out the legislators.  However, when more of the roll calls are less contentious, it is more important to get at the behind-the-scenes activity to distinguish among legislators.  Hence the qualitative rating then receives greater weight, namely, a two-to-one rating when roughly half the roll calls are closely divided or a three-to-one rating when less than a third of the roll calls are contentious.

            The prime criticism for such a system is that it enhances the likelihood of a partisan division in the ratings.  When the roll calls are closely divided, there has been a failure to achieve consensus.  A prime factor that undercuts legislative consensus is the party divide.  Nevertheless, this is simply a reflection of a basic underlying fact that there are real differences between the parties on many economic issues.

            Another criticism is that a roll call that reflects some consensus could be on a bill that is highly significant for the Maine economy and businesses.  In choosing their roll call nominations, governmental affairs representatives may choose those on which they had to fight the hardest to win.  Other, “more important” bills are passed over because they were not such a tough fight.

            It is not specified how the two ratings are combined.  Presumably when the one-to-one ratio is used, it is a simple average.

Conclusion.  This rating system is one of the most sophisticated, most realistic and most accurate I have seen in decades of studying interest-group ratings of legislators.  It measures what business leaders consider the important legislative issues.  It includes objective and easily quantifiable roll call data, but with consideration of the meaningfulness of these votes.  It makes a good attempt to include other dimensions of the legislative process than roll calls.

            I have critiqued each component of the rating system with the intent to alert a reader to the points at which a critic can challenge the rating.  There are elements of the system that have a potential for inaccuracy and bias, but the MERI system does well at eliminating or checking against many problems found in other interest-group ratings.  Fundamentally, the attempt to calculate a summary rating does oversimplify the complexities of political philosophy.  A challenge to the rating need only point at some complexities that might not be captured.

            Nevertheless, for the purposes of providing information to a business audience that does not have the time or other resources to research the records of their legislators, the MERI system does a first-rate job.