Using NLP to gain insights from employee review data
Utilize information hidden away in online employee reviews.
Written in collaboration with Adam Azzam and the Thinknum team.
For the Fall 2018 session of the Insight Fellows Program in NYC, we launched a new partnership with Thinknum, a company that provides alternative data indexed from the web to institutional investors and corporations — think online product listings, store location data, job listings, social media presence, and more — and allows its customers to access all this data in a time series format through a single platform. Thinknum provided our Fellows access to nearly 2.6 million Glassdoor reviews of publicly traded companies in the US, and we gave our Fellows the creative freedom to decide how they would use the Thinknum Glassdoor review data set.
In this blog, we’ll highlight the work of three Insight Fellows, who used Glassdoor employee reviews and a variety of Natural Language Processing (NLP) techniques to build innovative data science products that help job seekers identify roles that best match their interests and qualifications, better assess and compare companies, and facilitate socially responsible investment decisions.
The potential of online employee reviews
Sites like Glassdoor and Indeed have transformed the way job seekers identify and evaluate prospective employers. Anonymous employee reviews allow an unprecedented level of transparency regarding a company’s culture, management style, career ladder, and other facets of the workplace. In fact, Glassdoor estimates that 70% of job seekers use online reviews to guide their career choices.
But the power of online employee reviews extends far beyond the scope of finding employment. In recent years, analysts have leveraged the detailed information left by employees in a variety of use cases. Perhaps the most straightforward example is of employers using reviews to guide internal improvements, or to distinguish themselves in branding materials. In the financial sector, rising interest in alternative data has coincided with a flurry of public proof-of-concepts demonstrating that employee sentiment actually shifts in advance of layoffs or managerial departures, and can be a useful signal of company strength. These results aren’t surprising; a company’s employees can be excellent natural barometers for its health.
The power of online employee reviews extends far beyond the scope of finding employment.
Analytics using the numerical reviews of employees is relatively straightforward, but Glassdoor only asks employees to rate across five dimensions: Culture and Values, Work/Life Balance, Senior Management, Compensation and Benefits, and Career Opportunities. These categories are relatively broad (e.g. employees may rate Culture and Values poorly for a variety of reasons) and may not completely cover someone’s work experience. Additionally, numerical ratings don’t capture the importance of an issue to an employee relative to other topics.
Valuable information left by employees is hidden inside the text reviews they leave. These reviews are unstructured — Glassdoor asks employees to describe their experiences across several dimensions, but ultimately employees are free to write what they want. For example, a typical review could look something like this:
- Pros: The perks are great — catered lunch/happy hour every day in the office and the gym is amazing. Transit benefits are good, we even get concerts once a month where we’ve had some big names come in. Management seems passionate about the mission.
- Cons: Sadly, this company has no respect for work-life balance and, even though the perks are good, they basically mean that everyone stays at the office all the time, which contributes to a toxic environment where nobody feels like they can go home. Unlimited paid time off just means nobody takes time off.
The employee in this example clearly likes the benefits of the job, but feels work/life balance is poor, and spends virtually the entire review discussing these two dimensions. However, management was only mentioned in passing and career advancement was left out altogether, suggesting these aspects are less important to this employee. Additionally, poor work/life balance seems to be reflective of deeper problems with the company’s work culture. Clearly, the text reviews provide a significantly more detailed and nuanced picture of employee experience.
Products to Leverage Employee Sentiment
Insight Fellows utilized the Thinknum Glassdoor review data set, using the reviews and NLP techniques to build a trio of innovative data science products. The following describes each of these products in more detail.
xPea — Find your best job experience
Derek Jung built xPea, a web app designed to help job seekers identify roles that match their interests and qualifications. The inspiration for his product stems from the sheer difficulty associated with applying to jobs — a well-written job application can take 10–20 hours, and each job typically receives hundreds of applications, meaning a lot of work is invested in a process that often feels like random chance. xPea facilitates the job application process by identifying jobs that a user is not only interested in, but prioritizes jobs that closely match the user’s current skillset, raising the likelihood of application success.
To use xPea, a user starts by providing the role they are interested in and the company where they’d most like to work. xPea then compares this role/company pair with 13,000 unique job titles at over 4,000 companies to find similar roles and companies. It then returns a list of jobs at companies that are good matches to the original input.
To identify companies that are similar to a user’s dream company, Derek utilized the Glassdoor reviews data set to find companies with similar employee sentiment. To measure this sentiment, Derek classified each sentence in a review as belonging to one of five categories: Culture & Values, Work/Life Balance, Senior Management, Compensation & Benefits, Career Opportunities (the same five dimensions Glassdoor asks employees to rate along). He then used VADER, a sentiment analysis tool, on the classified sentences to quantify how employees felt about each category for a given company. These sentiment scores were then used to determine how similar companies were to one another.
In order to determine which jobs had similar skill requirements, Derek used the O*NET database, which provides 120 numerical skill scores for 1,000 occupations. As with the employee sentiment data, the numerical skill scores were then used to quantify the similarity between jobs. Job titles input by the user of xPea are mapped to the closest O*NET matches, and jobs with similar skills are recommended.
One of the major challenges Derek faced was accurately classifying each sentence in the review, since reviews themselves are unlabeled, unstructured text. His solution was to hand-label a small subset of the reviews, sentence by sentence — about 4,000 in all. These labeled sentences were then used to train a pair of custom word2vec models (one for Glassdoor pro reviews, and one for cons) that converted each of the remaining sentences to vector representations that encoded the meaning of the sentence. Finally, these numerical representations were used as inputs to a set of logistic regression classifiers which assigned each sentence to one of the five topics typically discussed in a Glassdoor review.
Users of xPea can select how important the skills match is between the role they provided and what xPea recommends — for some users, finding a good company is of paramount importance and they are willing to learn new skills to work there. Ultimately, xPea allows users to identify roles/companies in very different industries that are similar to their inputs. For example, one surprising result from Derek’s analysis was that Kraft and Starbucks, which operate in very different industries, have similar company culture sentiment scores. xPea allows users to discover these and other surprising matches as they search for their dream jobs.
CompanySnapshot — Visualize Glassdoor reviews from your dream companies
Yanxia Li built CompanySnapshot, a tool to help individuals visualize Glassdoor employee reviews, reducing the time needed to thoroughly review a company’s Glassdoor profile. Companies often have well over 300 reviews on Glassdoor, so users either invest significant time reading every review, or only read the top few and risk having incomplete information on a company. CompanySnapshot aggregates these reviews so they can all be considered at once, and compared with industry peers, in a matter of minutes.
Yanxia wanted users to have the ability to assess companies at a glance, but maintain the ability to dive deeper for more information. She built a product with an intuitive dashboard interface that not only provides an overall rating for a company relative to industry peers, but also a breakdown of sentiment within 14 different topic categories. Users can see which categories employees talk about most frequently in reviews and, like Glassdoor, read one or two reviews that match the overall picture of the company. Additionally, CompanySnapshot tracks the evolution of the company’s overall sentiment score over the past 10 years.
Like Derek, the first step Yanxia took was to classify each review’s sentences into topic categories, but she opted to use a machine learning-based approach to identify 14 latent topics. This approach involved finding context-aware embedding of each word or phrase in a review, then using latent Dirichlet allocation (LDA) to identify how words correspond to latent topic distributions. Yanxia then used a combination of quantitative and qualitative metrics to interpret each topic group identified by LDA. Categories cover subjects such as company product, training, compensation, career growth, office perks, and corporate management.
Once every topic was determined, sentences were classified as belonging to a topic via majority voting (e.g. if two words in a sentence referred to company product, and only one referred to training, then the sentence was classified as referring to company product). After categorizing every sentence in a review, Yanxia used VADER to assign a sentiment score for each topic, as well as overall review sentiment. To validate these sentiment scores, she compared the overall sentiment scores with Glassdoor’s overall company rating, and found them to be in agreement, suggesting that the sentiment scores match with employees’ numerical company reviews.
This tool aggregates reviews in such a way that the user is provided with more granularity than by just looking at Glassdoor ratings. For example, Yanxia was able to show that the most discussed topic in reviews are the company’s product. Employees also care a lot about personal growth — training, career growth, and team development are topics that all show up in many reviews. Using CompanySnapshot, users are able to specifically see how employees feel across these very specific categories.
InvestInMe — Using employee reviews to identify socially responsible investments
John Phillips built InvestInMe, a product that facilitates socially responsible investing. He built his product to serve the needs of individual investors who have limited access to the information required to make investment decisions in the rapidly growing environmental, social, and governance (ESG) space. ESG investing takes into account concerns regarding the ethics and sustainability of a given business, and John saw an opportunity to extract this information from Glassdoor reviews.
By entering the name of a publicly-traded company in this web app, investors get a dashboard view of the overall sentiment of employees of that company, as well as a breakdown of employee reviews by topic area (Enjoyment, Flexibility, Work/Life Balance, Pay, and Growth Opportunity). The dashboard also displays how employee sentiment has evolved over time, and uses this information to predict future changes in sentiment. Because different investors will have different personal guidelines that shape their ESG strategies, John worked to make the dashboard view as information-rich as possible to maximize the value of his tool to a wide range of investors.
To quantify employee sentiment, John used VADER to derive overall employee sentiment for a company as well as sentiment by topic area. The topic areas were identified using LDA, and limiting the number of topics to 5, for simplicity, resulted in remarkable agreement between the LDA-derived topics and Glassdoor’s suggested areas of review. In order to handle the common case of an employee referring to two topic areas in the same sentence (e.g., “The pay is good, but the work is terrible”), John elected to treat each independent clause as a separate entity when performing topic and sentiment analysis.
A challenge John faced when trying to predict how employee sentiment would evolve in the future was the large fluctuation in sentiment over time for a given company. These fluctuations and non-periodic nature of employee sentiment made traditional time series analysis methods, such as ARIMA or additive regression, perform poorly. His solution was to engineer lagging indicators, such as the 30-day moving average sentiment score, and use these as input features to a gradient-boosted regression using trees. This approach significantly improved the accuracy of the sentiment predictions.
InvestInMe stands as an innovative use of Glassdoor reviews to inform decisions in an unrelated domain. Reviews often contain “insider information” regarding how a company treats its workers, and can provide insights into a company’s philosophy around sustainability. By distilling employee sentiment data, John ensured that individual investors can be confident they have the information they need to invest in line with their own ethics.
It’s clear that there are many applications of employee sentiment data. These three products serve as powerful examples of how this data can be leveraged to find a job, summarize a company and compare it to its peers, and drive investment decisions. The creative use of unstructured text reviews in the Thinkum Glassdoor data set was key to each project’s success. Still, it’s important to emphasize that these are just a few examples. There are tremendous insights still hidden away in employee sentiment data, just waiting for someone to find.
Are you interested in working on high-impact projects and transitioning to a career in data? Learn more about the Insight Fellows programs and start your application today.
Insight Fellows
Before coming to Insight, Derek Jung earned a Ph.D. in mathematics from the University of Illinois at Urbana-Champaign, where he discovered techniques for finding optimal paths between points in constrained spaces. After his time at Insight, Derek was hired by Macy’s as a data scientist.
Yanxia Li earned her Ph.D. in Astronomy at the University of Hawaii at Manoa, where she developed algorithms to detect extremely faint sources in large astronomical survey data sets. After her time at Insight, Yanxia was hired by Wayfair as a data scientist.
John Phillips holds a Ph.D. in Physics & Astronomy from the University of California, Irvine, and recently completed a postdoc at the University of Minnesota, where he investigated how galaxies interact with dark matter. After his time at Insight, John was hired by Deep Macro as a data scientist.
