There is no right or wrong answer when doing pairwise, just stay consistent. If voters are unsure of priorities, have them do an original ranking on paper to collect their thought and priorities prior to voting in the tool. Help people understand voting tendencies and different rationales to voting.
Here are a few rationales:
• People are wearing the hat of what is good for the overall organization at the same time as the hat of the department they are there representing. These two can be conflicting at times.
• Voting in favor of something that is the most important regardless of any other data points.
• Voting in favor of a criterion that may be less important than another criterion but performance is much further behind in the less important area than in the most important area. Extra focus and catch up may be needed.
• Scaling back dominance in an area because there is not enough data to evaluate or data integrity is questionable (i.e., Future Cost Savings is something you can estimate but it’s not absolute). Some people don’t put as much weight into the unknown. The other side of this is scaling back dominance because you know that there is not that much differentiation across alternatives for that criterion.