How We Rate And Review

How We Rate And Review

We’re obsessed with the gadgets, car tech, science and geek culture that change the way we live. Here’s some background on how our Australian team tests the gear that comes through the Gizmodo offices in Sydney and Melbourne.

Gizmodo Australia’s 100-Point Score System

Our scoring system is an incremental scale from 0 to 100. Naturally, you’ll see more devices within the upper middle of the range — from 50 to 70 — and fewer outliers. Exceptionally good pieces of tech — those that pioneer new ideas and technologies and do so successfully, or those that manage their tried-and-tested everyday tasks excellently — will rate more highly, and those that are notably poor or that have some significant flaw will rate lower.

Gizmodo’s 100-point scoring system is based around a personalised, per-category scoring sheet filled out for each individual device by Australian editors. On each of these sheets, a device will be rated against its peers in a predetermined series of tests — objectively rating its build quality, the quality of its bundled accessories, its out-of-the-box usability, and so on. Beyond that, there’s a bit of wiggle room for that X factor in a device’s overall user experience that can’t be quantified.

How We Test: Smartphones

To test smartphones, we make sure to test them in real-world settings for at least a fortnight to see how they work in the real world, and how they perform in everyday use. With smartphones of all operating systems and designs, we try to use them to their limits to get an understanding of the best possible customer for that particular device.

We evaluate a smartphone based on the version of the operating system that it runs, what pre-installed and bundled apps come with it, as well as its hardware specifications and the quality of its touchscreen display (both in specifications and in real-world usage).

With smartphones, design and price are the two most important factors in deciding how we rate any device, since these are the two factors at which most consumers make a buying decision. Battery life, and other features like a phone’s integrated earpiece and speakers, are less important but still factor into our scoring system.

How We Test: Laptops And Tablets

To test a laptop or tablet, we use it in a real-world setting for at least a fortnight, installing all the apps and using all the features that we expect you to during long-term ownership. Within the laptop category, we run a suite of standardised performance-measuring tests — PCMark 8, CrystalDiskMark, and 3DMark Fire Strike — to determine some synthetic scores that can be compared across different devices and brands.

For Windows laptops, we test them with every Windows update installed and the latest graphics drivers downloaded and installed. Our two-part battery test involves:

Gizmodo Torture Test Extreme: High Performance power mode in Windows, Wi-Fi + Bluetooth enabled, full screen brightness, fullscreen 1080p video of TimeScapes looping in VLC

Gizmodo Torture Test: Balanced power mode in Windows, Wi-Fi + Bluetooth enabled, half screen brightness (biased to lower half), fullscreen 720p video of TimeScapes looping in VLC

For Android tablets, we use PCMark For Android benchmarks and the 3DMark GPU benchmark to compare devices across different devices and brands and operating systems. When necessary we use Geekbench on iOS to provide a broadly similar testing experience and equivalent results on a completely different operating system.

How We Test: Cameras

When we test a digital camera we use it for at least a fortnight in a real-world setting, to get an idea of what it is like to live with as a long-term owner. We get an idea of what it’s like to pick up, control and shoot a camera with everything from the out-of-the-box experience to advanced manual shooting modes.

We test cameras by shooting a controlled-lighting studio scene, under both ambient fluorescent lighting and flash, as well as a landscape outdoor scene, to provide some comparability between different models. We also take a wide range of everyday photographs in different lighting conditions to determine how a camera performs in various scenarios.

When scoring a camera we also take into account the established lens and accessory ecosystems for that brand and camera line-up, as well as the price of these extras to determine the ongoing post of purchase.

How We Test: Televisions

To test televisions, we set them up in a normal viewing environment and use them for at least a fortnight to approximate real-world use. We try out everything to do with a TV’s picture quality, from viewing standard free-to-air digital TV channels to viewing Blu-ray and other high definition video and DVD content, to get a clear idea of what it’s like to live with any given TV.

We determine how easy a television is to set up both from a hardware perspective and from the aspect of its out-of-the-box software setup and online services. As well as investigating the layout of a TV’s user interface, we try out any bundled apps or standout features to determine their worth.

Our review score of a TV is based on its out-of-the-box picture performance as well as its potential for adjustment. We determine whether a TV will stand the test of time by checking its future expandability and inputs, and decide whether its Smart TV interface will age well. We also balance a TVs price against its running costs and power consumption and the expected lifespan of its technology.

How We Test: Personal And Home Audio

When we test home and portable audio products, we use them in real-world settings for at least a fortnight to get an understanding of how they perform for long-term use. We use speakers and headphones at everything from maximum to minimum volume and using all inputs to see whether they perform consistently.

If a product has a wireless connection method we test it to determine whether it has an effect on the quality of the speaker, and determine the range and versatilty of that connection method. Where both wired and wireless methods are available we decide which is superior for rating audio quality.

When a home or portable audio product has an optional or mandatory mobile app, we test that app to see whether it works seamlessly and increases the usability of that device. If an app is poor it can detract from the overall experience, while a good app can add value.

How We Test: Cars

To test cars, we spend at least a week with them and drive them on a daily basis to understand them and to see how they compare to the other vehicles we have tested in the past within a comparable price bracket or market segment. Since each and every car, even yearly updates of the same model, is different, we try to view them both in abstract and in the context of their peers.

When testing cars, we try to get a holistic idea of any vehicle’s day-to-day performance. We’re not about testing a car to its maximum potential, but rather about understanding how it might perform for the average driver. Because you’re not going to be driving a car on a race track for 99 per cent of its life, the day-to-day experience is far more important.

With a car, we take into account both its driving experience — engine, suspension, chassis dynamics — and the level and quality of its exterior and interior. Gizmodo is a technology website at heart, so the calibre of a car’s smart driving features, multimedia system, navigation and stereo is of special importance.

How We Test: Wearables, Other Gadgets And Peripherals

There are plenty of technology devices that don’t fit into our larger categories. For these miscellaneous pieces of tech — fitness trackers, computer peripherals like keyboards and mice, random smart home gadgets and so on — we follow the same procedure as any other technology product that we test. We try to spend at least a fortnight with any serious gadget, learning its ins and outs and trying to use it in the way that any everyday person might.

With any device, there are a few things that we’ll test out no matter what technology it’s based upon. If it’s a battery-powered device, for example — like almost everything we review — we’ll test the capacity of its battery and how long it lasts under everyday usage conditions. The speed with which it charges is equally important, too. The out-of-the-box experience is something that can be quantified, as is its construction (unless it’s a piece of software, of course). Whatever the device, there’s something we can do to quantify its quality.

With this catch-all miscellaneous category, we don’t compare devices against each other unless they’re directly competing — like two fitness trackers or two mechanical PC gaming keyboards. Because of this, you can still take each device’s 100-point score as gospel, but just not against the entire range of miscellaneous tech. And, of course, if you have any questions, you can easily leave a comment or get in touch with us to ask.