Syncsort, a global provider in Big Iron to Big Data solutions, announced the results from its fourth annual Big Data survey. Results revealed the top use cases and challenges faced by organisations as they progress their modern data architecture and data lake initiatives, and see significant benefits from Hadoop/Spark, with nearly 60% citing both increased productivity across the organisation and improved efficiency to reduce costs as their biggest gains.
Compared with last year’s survey, the most dramatic increase in reported benefits was higher revenue and accelerated growth, which 55% named as a benefit this year compared with 37% last year. And though organisations are using Big Data insights in more sophisticated ways to improve revenue and customer service, they continue to face some of the same challenges reported in past years, including keeping up with rapidly changing technologies and tools.
The survey also found that Hadoop and Spark, which had high interest but low adoption at the time Syncsort launched the survey in 2014, are now in production or test at 70% of responding organisations. Specifically, this year, more than 40% of respondents say they are in production with either Hadoop or Spark, and 30% say they are engaged in a proof of concept or pilot program.
Based on the survey results, the five key trends in Data Lake initiatives that organisations need to monitor in 2018 include:
1. The composition of the data lake shifts. Traditional sources remain most popular for filling the data lake. Relational Database Management Systems (RDBMS) were chosen as the top source at 69%, up from 61% last year, surpassing enterprise data warehouses (EDW) at 63%. But newer sources grew, with NoSQL databases identified by 46% of respondents compared to 35% last year. Cloud repositories are also gaining strength as a data source (accounting for 40%) as more organisations leverage the cloud as a deployment platform.
2. Legacy platforms continue to play a significant role. Data from legacy platforms (such as the mainframe and IBM i) also make major contributions to the data lake: over 97% of respondents with mainframe believe it’s valuable to access and integrate mainframe data into the data lake for real-time analytics – a 27% increase over last year. Over 90% of those with IBM i believe it’s valuable to access and integrate that data into the data lake, not surprising as leaving behind decades of valuable data stored on these systems would seriously hamper their companies’ analytics.
3. Data quality and regulatory compliance challenges are top of mind. While the skills shortage had been ranked the top challenge for three consecutive years, this year it fell to number two, replaced by mounting concerns over improving the quality of data in the data lake. Indeed, 40% said data quality was a significant struggle for their organisation, likely a result of expanded use of data lakes driving an emphasis on improving data quality. But the survey also showed not everyone is making the connection between data quality and ROI. 60% of Financial Services and Insurance professionals said ensuring data quality is a top […]
Read more here:: www.m2mnow.biz/feed/Posted on: January 3, 2018