2020 Data Engineering & Analytics Predictions.
Reflections and predictions by Siarhei Oshyn, Lead Software Engineer (Data Engineering Division) at Godel Technologies
1. Why have data engineering & analytics come so far in the last decade?
The evolution of technology in our lives overall has driven a transformation of business intelligence – the amount of data we have and our abilities to understand it are worlds apart from the state of BI in 2010.
Ten years ago business intelligence services were only really accessible to professionals with specific skills and education. Today, the transformation of BI services has delivered accessibility directly to end-users – many applications are functional and make data interpretation quick and simple.
Data analysis projects also used to be very expensive – lots of hardware and software resources were necessary to work with large amounts of data. There’s been an evolution of pricing models around data storage in the last decade with the rise of cloud, and this alongside the availability of open source solutions means that it is a lot cheaper to conduct BI work today.
The digital transformation of our lives is providing vast amounts of data sources that didn’t exist before. Daily experiences – shopping, travel, exercise – generate data points that businesses can analyse to build a 360-degree view of individuals that is defined and evidenced, and they can do this in house rather than conducting expensive surveys or relying on intuition.
2. What should business intelligence developers do to stay ahead of customer expectations?
Today data engineers and BI developers have a bigger toolkit than ever – the resources and data sources easily available make business intelligence a discipline full of opportunity. It is a great idea for people in other disciplines – development, testing, or even roles outside of technology like marketing – to learn how to use business intelligence software and understand data more easily. There are many courses available online to achieve this.
Data engineering tasks are often quite domain-specific, so data engineers should gain a close understanding of the domain they are working in – it could be anything from fraud prevention to loyalty programmes. Alongside this, they should endeavour to have a cross-domain skillset so that they can apply their skills to different work. Above all learning technologies that can support business intelligence – artificial intelligence, machine learning, how the internet of things works – are all helpful.
3. What are your future predictions for data engineering and analytics?
Business intelligence today is very available to businesses, and based on this trajectory I think in the next decade it will only continue to become more accessible. Artificial intelligence is getting better and eventually, it will be a helpful asset from an advisory position to business intelligence solutions.
My crazy prediction is the unification of data – one digital space. So for example, I have a smartphone app and I’ve added some information about myself to it. This data goes into a digital space so that when I walk into a coffee shop the waiter knows in advance my favourite order because each device is connected and sharing information. It could even go as far to predict what I like – a suggestion of a new drink that I’d probably like but wouldn’t necessarily order.
I would say that this is far in the future because there are a lot of tricky factors to it – like who controls the data – the state, a private company or is it a community solution? Maybe this is in 2050, but that’s not so far away!