Battle for Data Pros Heats Up as Burnout Builds
“Tell me about your motherboard.” This bit of grim psychological humor accompanied a rather gruesome statistic in a recent data engineering survey, which found 78% of data engineers say they wished their jobs came with a therapist. The flip side of the burnout-inducing workload is exploding salaries amid heated competition for data professionals.
Besides the building demand for therapists, there were a few additional eye-popping statistics in the 2021 Data Engineering Survey, which DataKitchen and data.world jointly produced and released on October 19. For instance, 97% of the 600 data engineers surveyed reported feeling “burned out.” Another 70% say they are likely to quit in the next 12 months.
Several factors are driving data engineers to despair, the survey found, including the need to spend large amounts of time:
- Playing whack-a-mole with wonky data pipelines;
- Continuously finding and fixing data errors;
- Dealing with unreasonable requests and expectations from business users;
- Adhering to increasingly stringent data regulations and governance requirements;
- Working when they’re supposed to be playing;
- Being “shamed and blamed” when analytics programs go off the rails, which is more often than not.
“Our survey serves as a wake-up call to data organizations,” data.world CTO and co-founder Bryon Jacob said. “Data engineers are calling for change. That means instituting reliable, efficient and repeatable workflow processes that will improve analytics collaboration and productivity while restoring sanity to the lives of data engineers.”
While DataKitchen and data.world say their respective DataOps and data catalog offerings can help with the low morale among data engineers, automation isn’t a panacea. Automation can certainly help, but skilled individuals will continue to be critical to big data success.
Unfortunately, finding those skilled data professionals is getting harder. Data from the recruiting firm Burtch Works earlier this year shows that 81% of US companies were planning to expand their data science, analytic, and engineering teams in the third quarter and fourth quarter of the year.
“It’s no surprise to us that with the economic restart kicking into high gear and data science and analytics being a key element for digital transformation efforts (which are crucial to maintaining a competitive advantage in most industries), we’re seeing many companies keen to beef up their teams this year,” Burtch Works stated in a blog post.
More insights into the state of the data professional can be gleaned from Harnham, a UK-based data and analytics recruiter that is making a big push into the US job market with offices in San Francisco, New York and, more recently, Arizona. The company tracks the balance between the supply and demand for data professionals, and currently, the balance is out of whack.
According to Harnham’s Head of Marketing Owain Wood, recent data indicate that companies in the US are having difficulty filling open data and analytics positions. For instance, there were 33,000 folks searching for a particular data job in September 2020. In September 2021, the number had dropped to around 15,000.
“The number of people looking for jobs has halved basically,” Wood tells Datanami. “Vacancies are rising higher than the job seekers are looking for roles, which is really, really crazy to see.”
The increased competition for data professionals is good for Harnham’s business. As businesses struggle to attract talent using traditional techniques, such as job postings, they turn to headhunters like Harnham to fill open positions. Owain says Harnham has tripled the volume of data professionals that it has placed over the past year.
“In some niche roles, we’re only seeing one or two active applicants per job where we used to be seeing 10 to 15. As a result, our headhunting services for passive candidates are in high demand,” Wood says. “It’s that war for talent, the war for skilled talent.”
That war is driving up personnel costs for the companies with open jobs in data analytics, engineering, and data science. According to its recent annual salary survey, entry-level positions in advanced analytics are now fetching salaries in the $90,000 to $100,000 range, while executive-level positions are paying a quarter million dollars a year and up.
Some specific jobs pay much more. Experts in computer vision and robotics, for instance, can command $125,000 for entry-level jobs, while experienced vice presidents can command up to $350,000.
“The amount of money that’s being thrown around is insane,” Wood says. “I’ve seen some jobs increased salaries maybe 100%, 150% over the past six months.”
These types of salaries aren’t offered in every industry. For instance, demand is still weak for businesses impacted by COVID, such as travel and hospitality. However, in other areas, demand is growing quickly.
“For in-demand industries like life sciences, healthcare, if you have touch points in your track record, in your history, you’re being snapped up left, right, and center,” Wood says. “Finding people in those spaces is really, really tough.”
Data professionals in the US were happy to have a job when COVID was front and center in 2020. But the sentiment is quickly changing as the burnout and lure of higher salaries at other jobs is leading folks to quit their jobs in droves.
Data from the Bureau of Labor Statistics backs that up. The latest BLS report states that 4.3 million Americans overall quit their jobs in August, which is a record high. In the information sector, of which data professionals are part, the number of job openings doubled, from 78,000 in August 2020 to 154,000 in August 2021. But the number of hires fell, going from 111,000 in August 2020 to 97,000 in August 2021, the BLS report shows.
American workers across industries are in a state of flux. An August survey by Bankrate found that more than half of US workers plan to look for a new job in the next 12 months. The workers most likely to split for greener pastures skews towards youth and lower pay. But even among the Gen X and Baby Boom cohorts, 45% and 33%, respectively, say they will be looking for a new job.
Taken as a whole, the data would seem to indicate that the data profession–if not all American professionals in general–is at a crossroads. Data engineers and data scientists are in high demand, but increasingly are unhappy with the escalating demands of their jobs. A large number of data professionals appear poised to quit their jobs over the next year. That could boost their salaries even higher if they choose to stay in the profession, particularly given the overall scarcity of talent.
It’s a mixed bag, to be sure. For individual data professionals, the pay has never been higher, but the demands have never been higher too. For businesses, the need to become a “data-driven organization” has also never been higher, but route to get there is not simple, given the scarcity and expense of the talented individuals who are necessary to take businesses to the data promised land.
In the short run, individuals seemed poised to capitalize on this imbalance. In the long run, automation of the sort that data.world and DataKitchen in data engineering–and multitudes of other processes in data analytics and data science–seems destined to occur. Buckle up your seatbelts, because the AI revolution is upon us, and it’s going to be a bumpy ride.
Related Items:
Hiring Outlook Bright for Data and Open Source Pros
Data and AI Salaries Continue Upward March, O’Reilly Says
Hiring, Pay for Data Science and Analytics Pros Picks Up Steam