What is this statistic really measuring?

Mark from At Home on the Court once posed a question on Facebook. It went like this:

What does number of transition attacks divided by number of serves measure? If anything.

After a bit of discussion, Mark made the following clarification:

So now I have removed service errors and direct block points. I am left with transition attack attempts divided by number of times the attack gets past the block. What am I measuring now?

So we have this formula:

Transition Attacks / (Opponent Attacks – Blocks – Hitting Errors)

My response at the time was to say that broadly speaking you are measuring dig %. After all, you must dig the ball to get a transition swing. If you consider Transition Attacks to be something other than simply playing the 3rd ball over the net, then you are effectively deriving a “good” dig %. The first requirement of attacking back at the opposition is a settable dig (or, in theory, hittable on 2).

I’d be curious to hear what you would say in response to Mark. Beyond that, though, I wanted to point out the importance with regards to statistics of actually understanding what you’re measuring. It might not be the same as what you think you’re measuring.

Considering a set rating system

In my Volleyball player efficiency ratings post I referred to a short paper proposing a way to evaluate setting. That paper is A Reconceptualisation of Traditional Volleyball Statistics to Provide a Coaching Tool for Setting by Alexis Lebedew. The paper’s basic idea is to combine a pass/dig rating (like the kind mentioned here) with a 0-3 rating for the outcome of a spike to get a set rating (0 for error, 3 for kill). The pass rating is effectively inverted. Good results off poor passes are rated higher than ones off good passes. This accounts for set decision and/or execution.

Conceptually, I like the idea proposed by the paper. There are a couple of issues with this particular ranking process, though.

No real new information

There was an evaluation of set rating system to see if it related to the probability of winning a set. It does, but that’s no real surprise since outcomes heavily influence the ratings. Specifically, the highest ratings coming on kills. We already know that getting lots of kills generally results in more winning. We also know that both hitting efficiency and pass quality relate to scoring points and winning sets. As a result, we don’t really have any new information from this rating in that context.

Problematic comparisons

Hitter skill is a major factor in attack outcomes. Good hitters can make up for poor sets. Poor hitters, though, struggle even with great sets. Further, the quality of the block and defense the hitters face is a meaningful factor in attack outcomes. As such, it is problematic to compare setting across teams. A great setter on a poor team simply won’t rate as well as a setter on a good team.

So what to do

If the idea of rating sets is to be able to compare setters directly then I fully support factoring in pass quality. It is the primary control variable determining what can be done with any given setting opportunity. Including it allows us to look at setting on an even playing field.

In order to really assess the quality of sets, though, we need to also be controlling for the outcomes in a similar way we do for passing. If a passer puts up a perfect pass, we don’t punish them because the setter mishandled the ball, nor do we bump up their pass rating because the setter was able to put up a good set off a poor pass. I think we end up circling back to the idea of rating sets similarly to how we rate passes. The problem, though, is we really should be factoring in play calling and decision making in the process, which is no easy thing.

All that said, though, I do think the type of set rating proposed in the paper could be useful in comparing setters on the same team or in a situation where teams and players are quite similar in level of play. In that case, hitting quality and the block/defense of the opposition would presumably be the same for each, allowing for a level assessment.

Volleyball player efficiency ratings

A while back I came upon a 3-part series of blog posts. It conceptualizes a unified way of looking at volleyball statistics across all categories. It’s dubbed the Volleyball Player Efficiency Rating by its creator. The three installations are here, here, and here respectively. The second refers to a paper on setter-specific ratings I address separately in another post. A number of folks came up with different variations on the player efficiency theme in the recent years.They are interesting conceptual exercises. At least the stat geek side of me thinks so!

When I coached at Brown I even developed one myself. I dubbed it the Point Contribution Ratio. Basically, I took the basic match stats for kills, blocks, digs, and assists. I then added in the 0-3 ranking we did for serve reception and the 0-5 score we used for serving into the mix. Each stat was weighted based on how directly it contributed to points scored or conceded. The calculation looked something like this:

PCR = Kills + Blocks + Aces + 0.5 x Assists + 0.5 x 3-passes + 0.25 x 2-passes + 0.5 x 4-serves + 0.25 x 3-serves – Hitting Errors – Service Errors – Ball-Handling Errors – Block Errors

That’s not exactly it, but I think you probably get the idea. Comparison was made on a positional basis because of the way different positions scored. Setters, for example, had the highest PCR because of their assists.

I never did actually test the PCR out statistically, though. That means I can’t give you an idea of how useful it might have been given the right weightings. Therein lies the problem with many of these volleyball statistical measures. We don’t know if they are meaningful when it comes to winning and losing points and matches. Jim Coleman actually did the statistical work on passing. He showed that how a team passed on the 0-3 scale related to their probability of scoring points (see his chapter in The Volleyball Coaching Bible). Those who propose new statting methods must do the same. Those who use statistical techniques to evaluate teams and players need to know that they actually have a measurable relationship to what we’re using them for. They can’t just sound good. Otherwise, we’re just spinning our wheels to no real purpose.

The good and the bad of volleyball statistics

Coach Rey has an interesting post discussing the use of stats in modern volleyball. It includes an idea for his own sort of team scoring metric. I found his comments about how complex and advanced stats are these days to be quite interesting. I’d started having that feeling as well at times. I can imagine how overwhelming the mass of numbers are for some. In particular, Rey brings up how confusing things can be for players. I think that is probably something which varies from player to player. Some don’t care about stats while others get quite into them.Personally, from a player perspective, I like to use stats to help track development and performance over time. I also like to provide points of comparison where appropriate.

I can imagine coaches getting too caught up in the numbers. This is a little bit of the PhD in me talking. The fact of the matter is that as coaches we tend to get only small sample sizes. You need a fairly large number of observations to draw proper conclusions. This is fine in a situation where you can track lots of reps (like serve receive over several training sessions). It’s more problematic when you have to make quick judgements in the middle of a match. For example, when a hitter has 10 swings, 1 kill or 1 hitting error either way has a major impact on their hitting %.

And let’s face it. There’s a lot of stuff that goes on in a volleyball match which doesn’t show up in the stats. Having numbers at hand makes it seem like we can make nice clean assessments. The bottom line, though, is that we’re still dealing with people. People aren’t machines. Inevitably there are things they do (or don’t do) which aren’t so easily converted into neat objective measures. Some coaches seem to get so obsessed with the numbers that they forget this fact.

Don’t get me wrong! Stats are quite handy so long as one doesn’t get carried away.

While coaching in England I was jealous of the stats my coaching peers back in the States got. There is very little in the way of statistics there, and most of what does exist comes from coaches collecting their own. I did a bit of stat-tracking in training to give players progress reports and to do some comparisons. At one point I had an assistant who tracked some things during matches when he was available. That’s about as far as it went, however.

I struggle personally as a head coach to keep stats during matches because I find it distracts me from observing the big picture of what’s going on with my team (different when I was an assistant). I wouldn’t have minded having consistent box score type stats provided to me, though. 🙂

Coaching Log – Nov 25, 2013

This is an entry in my volleyball coaching log.

With matches for everyone coming up, playing 6 v 6 in segregated teams was the main theme of training. The B side focus was on teamwork, calling the ball, staying switched on, and not looking for someone else to play the ball. I told the B side focus was on continuing to be aggressive in attack, but underlying that was also an evaluation of the setter position. This harkens back to my observations from the Student Cup performance.

One of my primary OHs thus far is also a capable setter (sets for her German club team). I haven’t used her there thus far because I saw her value as an attacker as being in the area of greater need. Even had to use her at MB on a couple of occasions when we were thin on bodies. In the last match of the Cup I had her do some setting for the first time and it was an eye-opener just from the warm-ups. With other hitters coming along, I no longer feel so concerned about losing out on her as an OH.

At the same time, our starting setter up to now is only here for this term. As a result, I need to evaluate our setting options moving forward in any case. One of the teams on the schedule for the upcoming matches is currently top of the table, having beat us a couple weeks ago. We need to be at our best for the rematch, so I took a hard look at the setting in training to start evaluating who would give us our best chance at victory.

Following what was basically a pre-match warm-up, we went straight into 6 v 6. It started with a game featuring one side serving 3 times in a row (not counting missed serves). I put the starting setter with the B team (their setter was missing) and the other one with the A team. I used that to gauge a rough baseline point differential I could use to spot the B side in the straight-up games to follow. I came up with 8.

We then played two regular games which started with the B side serving, up 8-0. In the first the starting setter was with the A side. I then flipped setters for the second game. The A team lost the first set 25-20, and the second 25-23.

Stats were kept for all three games to evaluate offensive effectiveness as a way to compare the setters. More of this needs to be done in the next training, but the early results show quite a stark difference. The kill % for the team when the starting setter ran things on the A side was was 21% vs 36% for the former OH, while the hitting efficiency numbers were .063 vs .190. Need to see if that gap holds up.

Scoring Serving and Passing Effectiveness

For the sake of making solid objective assessments of your team and players, and to see how they progress over time, it is worth compiling as many volleyball statistics on them as you can during both training and competition. One of the easier stats to keep is that for serving and passing effectiveness.

The common practice among volleyball coaches is to score passing on a 0 to 3 scale. This is primarily for serve reception, but one could also rate free ball passing and even digging in the same way. The scale looks like this:

3 – Perfect or near perfect pass giving the setter all setting options

2 – Good pass, but the setter has primarily just two options (forward or back)

1 – Poor pass allowing the setter only one option, or forcing a non-setter to set.

0 – Ace or over-pass

Generally speaking, teams want to aim for an average score of 2.0 or better. Squads who are able to do that will usually run an effective offense.

On an individual basis, the best passers will come in around the 2.3-2.4 level on average. Obviously, you probably won’t see that kind of average for lower level players.

I have seen some coaches use modifications on this system. For example, 1 could be an over-pass, shifting the rest of the scale up such that a perfect pass is a 4 rather than a 3. This might be suitable for lower level teams where an over-pass doesn’t translate into points for the opposition as frequently as it does at upper levels. In any case, feel free to adapt the system to suit the needs of your team.

Scoring Serves

As for serving, we use a 0-5 scale which is largely an inverted version of the passing scale.

5 – Ace

4 – Over-pass

3 – Opposing team passes a 1

2 – Opposing team passes a 2

1 – Opposing team passes a 3

0 – Error

As with passing, the objective here it to average 2 or better. Doing so means the other team cannot run its offense consistently, making your defense and transition game more effective. Again, you can make adjustments to suit your needs.

Stat Both Training and Matches

I strongly recommend you score serving and passing in training drills and games as well as in matches. If you only score during matches then your bench players won’t ever get scored. Part of the reason for keeping volleyball statistics like this is to give your players very specific feedback on where they are currently and where they need to get.

Scoring serving and passing also gives you a clear an unambiguous way of ranking players for lineup decisions. You’re less likely to have ruffled feathers when you decide to have Joe hidden in serve receive or Jane serving last in the rotation if the player knows they are not one of the better performers in those skills.

Make Sure It’s Consistent

Since the serve and pass are two sides of the same coin, keeping these serving and passing stats is quite easy. The one requirement, though, is that a consistent metric is used to make scoring judgements. If you don’t have consistent ratings then the averages derived won’t be reliable. It may sound easy to define a 3-pass, but it’s going to vary based on the athleticism of your setter and/or the ability of your middle hitter(s) to stay available for a front quick set. You can be more liberal with your scoring if your setter is quick and your middles mobile, but if you have a more slow-footed setter and/or lumbering middles the range of passes which could reasonably be called a 3 will be narrow.

• Website Question or Comment?

Go to the Contact page to submit a query or share your thoughts.