Sweet Little Lies: Researchers Explore How to Detect Deception
If a stranger lied to you, how would you know? Would their speech pattern change? Would their heart rate accelerate? Would they break eye contact? Would they provide more details or fewer?
Detecting deception is difficult, especially if the person lying isn’t someone that you know well. Investigators are often in this position when conducting background check interviews as part of the security clearance process. They speak to numerous individuals, asking a series of questions that range from academic experiences to associations with foreign nationals to treason. They then have to assess if the individuals were being honest.
These investigators spend a great deal of time interviewing subjects and then determining which answers garner more attention or further investigation—potentially eliminating the applicant’s ability to obtain a security clearance, a process that can take approximately 81 days. It requires significant human resources to conduct the in-person interviews.
This predicament recently led a group of researchers to explore how technology might play a role in detecting if someone was telling a lie—highlighting areas for investigators to follow up on and making the security clearance process more efficient. And their initial research published this month is promising.
Based on findings from 103 participants, the most accurate approach for detecting deception is a machine learning model that counts the number of times interviewees use common words, according to Looking for Lies: An Exploratory Analysis for Automated Detection of Deception.
“What this study shows is that these tools can work in a controlled environment,” says Marek N. Posard, military sociologist and professor at the Pardee RAND Graduate School who participated in conducting the study. “When it’s trained on a larger sample, you can get more fidelity. This shows that there’s a 'there' there that should be tested more in the field.”
The Study
The Performance Accountability Council sponsored the RAND Corporation’s National Security Research Division to assess how machine learning and other interviewing methods could be used in the background check process to detect speech patterns that show an individual is being deceitful or truthful during a simulated investigation.
Researchers Posard, Christian Johnson, Julia L. Melin, Emily Ellinger, and Hillary Reininger then collaborated to create a scenario to test four different models of deception detection. They recruited 103 participants, who all read a story about Edward Snowden leaking classified information from the National Security Agency (NSA) in 2013.
A storyline using the Snowden leak was used because the researchers “wanted the story to be realistic to the clearance process and something memorable,” Posard says.
Some participants read a news report about the incident, while others read a memo with markings that the message contained sensitive information. Participants were then assigned to two groups: one which was told to lie about what had been read (58 participants; 56 percent of the total) and the other which was told to tell the truth (45 participants; 44 percent of the total).
Former FBI agents then interviewed the participants via video teleconference or text-based chat, asking interviewees to “fully recall” what happened, recount visual and auditory details, share how they felt while reading the story, respond to a “temporal recall question,” and disclose any information they might have left out.
Researchers uploaded the video interview recordings to Amazon Web Services (AWS) Transcribe, which uses automatic speech recognition—a deep learning process—to convert the recordings from speech to text. People also reviewed the transcriptions and cross-referenced the recordings to ensure they were accurate. The text chats were compiled in a Microsoft Word document and were then uploaded to AWS.
Researchers then reviewed the transcripts to train multiple machine learning models to see if they could tell the liars from the honest participants. The models included:
- Word Vectors: Counting the number of times an interviewee used common words.
- Linguistic Stance: Using stance vectors—sorting words into categories.
- Metadata: Examining meta-level features of speech like speaking cadence, average unique word length, and fraction of words with more than six letters.
- Chat and Video Word Vectors: Looking at word vectors in transcripts of interviews done via text-based chat and video.
The Model Analysis
After analyzing the transcripts and the models, the researchers found that word vectors were the most accurate way to tell if someone was lying (76 percent), followed closely by chat and video word vectors (72 percent), linguistic stance (64 percent), and metadata (63 percent).
“Across all our models, autobiographical statements (use of I) are much more common for truth-tellers, while liars appear to be far more likely to deflect attention away from themselves with the word you,” the researchers concluded. “We note that other studies have found the opposite result, with second-person pronouns used less by deceivers than truth-tellers.”
When it came to assessing the transcripts using a linguistic stance model, the researchers found that the top stances for truth-tellers were autobiographical statements, linguistic references, time and date, common authorities, and anger.
“In comparison, the top stance vectors identified for those in the deceptive condition were references to you/attention (e.g. look), uncertainty (e.g. maybe), projecting back (e.g. used to), subjective perception (e.g. it seems), and oral cues (e.g. you guys),” the researchers noted. “The uncertainty stance is a particularly good test of the hypothesis that lying places a mental strain on interviewees. [Results] show that the uncertainty stance was used more by liars than by truth-tellers.”
Deceivers and honest individuals were not different in a statistically significant sense when their transcripts were analyzed using the metadata model. One aspect the researchers found, however, is that liars are more likely to use fewer complex words. This, in turn, could allow them to speak more quickly than truth tellers.
When analyzing the chat and video transcripts, the researchers also found that you-based language is more prevalent for liars while I-based language is more prevalent for truth-tellers. Combining the video and chat based transcripts also helped improve the accuracy of the results.
“This suggest that the more text that can be gathered in an interview setting, the better this class of model will perform,” the researchers wrote. “Engaging the interviewees in a longer, back-and-forth chat will be more productive than asking just a few questions, and better still is to combine the data across different interview methods, which will help alleviate the potential effects of bias that may be present in one medium.”
Identifying Bias
One of the main takeaways from the limited study where participants were evenly represented by gender was the bias that can be baked in to the interview process, Posard says. Ninety-nine percent of participants recorded their gender affiliation, and all the interviewers were men.
“When the model was trained on both men and women, it was 76 percent accurate at detection deception,” the researchers found. “When the models were trained using participants’ genders, they were more accurate for men.”
The models also found that men and women used different words when attempting to deceive, such as men using “I” more often when telling the truth. Women were also more likely to use “man” to describe Snowden during the interview.
“This is an important distinction because it suggests that the subject’s gender was more salient for women than men,” according to the researchers. “It also raises the question of whether the subject’s or the interviewer’s gender could have influenced a subject’s responses to interview questions.”
This is especially important because the workforce that conducts background investigations for the security clearance process is predominantly men, while roughly 25 percent of applicants are women. If potential bias is not addressed—or there is no system in place for a check and balance—then the investigation and application process could accept or reject individuals unequally.
Future Recommendations
This particular study had only 103 participants, so Posard says he hopes it will be replicated with a larger participant group to give researchers better data to work with.
In the meantime, however, the researchers did release four findings and six recommendations based on the limited study results. Those included that there are linguistic signals that people use when attempting to deceive and machine learning can detect some of these signals, so the federal government should test machine learning models of interview data with word vectors to identify attempts at deception.
We set out to explore whether machine learning could detect deception during the interview portion of security clearance investigations.
— RAND Corporation (@RANDCorporation) October 20, 2022
**Spoiler alert** It could https://t.co/LwTLAsTG9p
They also suggested that, since the accuracy rates for deception were similar for video and chat interviews, the federal government explore conducting interviews via video and chat services as an alternative to the in-person interview process. The researchers further recommended utilizing machine learning tools to augment the existing investigation processes by conducting additional analysis of pilot data, as well as validate machine learning models to limit bias of interviewees based on race, gender, and age.
The key moving forward, however, is that humans be kept in the loop and involved in this process. Machine learning tools should compliment the work of human investigators, but not replace them, Posard says.
"The old system should not be scrapped," he adds. Instead, interviewees in the future might use a chat or video conference to collect a set of initial data that is analyzed using a machine learning tool. A human investigator could then follow-up on data that is flagged as a potential issue.
“The chat might be effective for basic info,” Posard adds. “And if you combine some of that chat with the interview itself, it can be potentially promising.”
