Information is (accessible) everywhere. It’s changed the way we think, problem solve, and interact. In an age where developers can cut/paste code into demo products, marketers can re-purpose infographics into rapid-fire click-bait, and anyone with 10 minutes on the right Wikipedia page can sound like an expert – it’s worth asking: what skills can we foster that remain valued? There are plenty of these skills – deep cognition for example, or the ability to see connections no one else sees, create new generalizations, and make an inference to collect more information.
I’d argue that empirical thinking is one of the skills that remains high value. Folks who I work with who are particularly good at applying the scientific method to their everyday work tend to rise to the top. Along those lines – I’d promised a second look at the now “infamous” product comparison between AirMedia 2 and other wireless content collaboration systems in a brochure distributed by Crestron. In a previous post, I pointed out how misleading the Crestron marketing team was with their “data”, and made the case for the damage that misinformation can cause with customers. Here, I want to focus on the importance of careful empirical experimentation when measuring data, solving problems, and (yes) creating a marketing brochure.
The brochure in question failed to report one of the most important aspects of any product built on wireless video – frame rate. As the video cable makes it’s slow exit from our lives, some pretty great things can happen – displays become shared, more easily accessed, and new opportunities to work with our colleagues emerge. However, to do these things, wireless video needs to accurately replace what was there before. This translates to the ability to share a 1080p desktop to the display and view it at 30 frames per second.
Some wireless collaboration vendors will tell you their products are really for PowerPoint viewing – but I disagree. The customers I know are looking to wireless as a part of a larger movement away from presentation oriented meetings. So, how do we accurately measure the video frame rate of existing products – we could simply ask each vendor to supply it – but I have serious doubts about the veracity of those claims given what I covered in my last post. Instead, I devised an experiment to measure the frame rates of Crestron AirMedia, Barco Clickshare, and the Solstice Pod. I’ll describe the experiment in enough detail below that you should be able to recreate it yourself (a common requirement in academia).
The products I tested were a standalone Crestron AirMedia 2 with the latest firmware, a Barco CSE-200, and the Solstice Pod. The Solstice Pod will report instantaneous frame rate directly in the product (share a desktop, select “Info” on that Post in the user-interface and it’ll show how many frames per second are being rendered). But to be absolutely fair, I computed frame rates for all three products using an experimental setup. First, I aimed a high-speed camera at a flat panel display. The camera was configured to capture 240 frames every second. The need for a high speed camera is related to what’s known as the Nyquist Rate. The Nyquist Rate is the principle that to measure a periodic signal accurately, you need to sample that signal at greater than twice the frequency of the signal. For example, imagine that if you’re looking out a car window, blinking in the wind, and you want to count fence posts that are flying by at 1 post per second – you’d need to open your eyes at least 2 times per second to make sure you don’t miss a post.
In our case the signal is a complete rendered frame being displayed on the flat panel (somewhere around 30 frames per second hopefully), so we need a camera that can sample those frames at least 60 fps. For each test, I used a Microsoft Surface Pro 4, displaying a video in full screen on the device while sharing that video wirelessly to each of the products attached to the display. We designed a video that rendered at 60 frames per second that we would then display on the desktop – let’s call this the “raw” video. Each frame in this raw video contained a unique, easy to read number in sequence. We then made sure that approximately 2,000 frames of the raw video could be displayed to the laptop without losing a single frame, just to ensure that Windows did not act as a bottleneck to displaying a 60Hz video. We did this by recording the desktop while it played the raw video and then compared the captured video to the raw video – looking for dropped frames – all looked good.
To measure each product’s ability to render this video to the display – we used a high-speed video camera to record the output of each product, while displaying the raw video on the Windows laptop and displaying it using the wireless product. We called this the “captured” video. Once the video was captured, we loaded the captured video into a video editing tool and looked at each captured frame (thanks to a couple great interns for your help!). We counted the number of unique sequence numbers in the captured video. If any of these products was capable of 60fps rendering so we’d see all of the sequence numbers somewhere in the captured video. If the product being tested was hitting 30fps perfectly, then we’d see exactly half of the frames displayed than were included in the video. Sure – they’d be repeated because we’re sampling them quickly, but we shouldn’t have seen any missing/half numbers. To compute a frame rate, we simply counted the unique sequence numbers in the captured video (displayed on the screen) versus the number of sequence numbers in the raw footage:
60 * (number of unique sequence numbers in captured video / number of frames in the raw video).
For each experiment, we displayed 150 frames of raw video.
Good empirical design means repeatable experiments and controlling variables. To that end, we ran the same test on all products 3 different times and computed the average frame rate for each. Another empirical value is avoiding experimenter bias. To avoid bias, for each of the three trials a different individual looked at each frame to determine how many unique sequence numbers were captured. Finally, the individual that was counting unique frames in the captured video did not know which product the video corresponded to, so the objective data was collected blind.
I also thought it would be valuable to compute not only frame rate, but how consistent the frame rate seemed to be. This is usually expressed as variance – the expected value of the squared deviation from the mean frame rate.
On to the results drumroll please:
(41 frames observed / 150 displayed)
|Barco Clickshare CSE-200||27.6
(69 frames observed / 150 displayed)
(73 frames observed / 150 displayed)
I won’t try to explain the results because that would include speculation, but I think they speak for themselves. I suspect that the wireless network conditions contributed to some of the variance we observed in frame rate based on congestion and throughput variability on the network. It’s interesting that the AirMedia has a low variance, but low variance is not necessarily a good thing if that variance is centered on a poor frame rate. The Clickshare unit had good throughput, and for all it’s limitations in user-experience, it’s proven itself to have good frame rate when only one post is shared to the screen in this study. The high variance around Clickshare’s product manifested itself as occasional video jitter – for a ½ second or so the frame rate would drop into the low 20’s. To be fair, this only occurred a few times and at other times the throughput jumped to over 30 frames per second. The Solstice Pod, by design, will limit itself to a 30 frames-per-second clock when sharing only business applications, so this result makes sense. On average, based on network conditions it was bound to lose one or two frame boundaries, giving the Pod a mean throughput over 3 trials of 29.2 frames per second.
Why did we go to so much trouble to design and execute these tests? It’s not only driven by my need to set the record straight when I see misleading marketing but to reinforce the value of empirical thinking and how it can create real value in how you design products, how you frame and then solve problems, and how you communicate with your customers – like I’m doing here!
We ran these tests in full view of both our marketing and engineering teams and continue to encourage them to adopt this type of experimentation in their own processes. It will no doubt lead to careful thinking, objectivity, and ultimately creating better products for our customers.