When I logged onto Roblox’s platform to take the company’s preselection assessment for entry-level candidates, I was instructed to design a factory workflow that would maximize returns in a toy factory. (For the millennials out there, it was like a high-tech Lemonade Stand.)
As I pulled virtual levers to build cars, robots, and playsets, I watched the returns trend frustratingly down and figured I probably didn’t have the chops to get hired at the video game company. But only probably—it wasn’t entirely clear what skills I was being tested on or how they related to software creation.
Though some employers rely only on coding assessments to evaluate early career candidates for computer science positions, Roblox has taken a different approach. Jack Buckley, Roblox’s VP of people science, said it’s designed to test how candidates think, not just what coding languages they know. (Don’t worry—applicants also take a coding test.) Applicants’ approach to the factory assessment, for example, gauges their aptitude for systems thinking, one of Roblox’s 15 critical skills.
The gamified test takes a lot of work to create—even for a company like Roblox that’s known for building virtual worlds. Here’s why Roblox went to all the trouble.
White out. The video game industry—and tech as a whole—is not very diverse. Though 45% of gamers are women, only 24% of employees were female as of 2022, and just 2% were Black as of 2021. Coding tests can perpetuate these trends: If underrepresented groups are not told to study STEM in school, they may not have coding skills, but that doesn’t mean they don’t have the aptitude to learn. Roblox wanted to design a test that screened for the latter group.
To do so, in 2020 they acquired Imbellus, a company that had created gamified cognitive assessments for McKinsey and Company. Imbellus was charged with making an efficient, fair, accurate test that could be taken anywhere at any time.
Since implementing the gamified tool in late 2020, Roblox has grown its percentage of “non-elite” hires (those from underrepresented schools) from 32% from 2020 to 2021 to 48% in 2022 and has hired from new schools including the University of Alabama, Oregon State, Auburn University, and Arizona State, according to a press release shared with HR Brew.
How it works. Some 13,000 applicants have taken Roblox’s assessments so far, according to the press release. The immersive test is designed in two parts, each taking applicants 25 to 30 minutes to complete. (Applicants can request accommodations, including extra time, under the ADA.)
Quick-to-read HR news & insights
From recruiting and retention to company culture and the latest in HR tech, HR Brew delivers up-to-date industry news and tips to help HR pros stay nimble in today’s fast-changing business environment.
Aside from the factory segment I road-tested, other assessments include a robot building challenge—which Buckley said tests creative problem solving—where applicants are instructed to build as many robots as possible, as quickly as possible.
While the games may sound silly, they’re the product of hundreds of hours of observing and talking to entry-level Roblox employees about what skills they need to do their jobs well, and months of cross-team collaboration to create and validate results. Before finalizing a block, the team hosts “think alouds” to watch Roblox employees work through the scenarios, and compares the test scores across various subgroups to ensure success isn’t an accidental byproduct of demographics. Whether game performance actually predicts job performance, only time will tell. Buckley said his team is still tracking the performance of new hires who took the assessment.
The challenges. Buckley spent a 30-year career building paper-pencil tests for institutions including the College Board before coming to the world of gamified testing—and he is the first to admit there are challenges. When designing for the SAT, he said he thought about questions. At Roblox, he thinks about scenarios. For applicants to build items in a factory, he has to build an interconnected world for them to operate in. The stakes are higher in every sense: It costs more money, time, and effort to create a good testing environment.
“Maybe that [question] took five minutes of testing time, but it might have taken our data scientists two weeks,” Buckley said. “Every little change, it turns out, you need to make, you need to involve a lot more…people, and it takes a lot longer.”
Still, Roblox says the gamified approach is saving HR time in other ways. By adding the pre-assessment screening to the top of the recruitment funnel, recruiters cut down on résumé screening by over 94% and saved 3,000 person-hours of review.—SV
Do you work in HR or have information about your HR department we should know? Email [email protected]. For completely confidential conversations, ask Susanna for her number on Signal.