The four-day training "Data Engineering: Skalierbare Datenpipelines und Big Data Technologien" shows you how to build and maintain modern data infrastructures. You will learn to plan scalable data pipelines, work with big data technologies, and apply best practices for data integration and data backup. Hands-on exercises and case studies help you transfer the knowledge directly to your own data problems.
In terms of content, you start with the basics: the role of the data engineer and the architecture of data platforms. You also learn the difference between data engineering and data science. Then it's about data modeling: designing relational databases and using NoSQL databases. For ETL processes, you learn about tools like Apache Airflow, Talend, and Informatica. In addition, data quality and incremental data processing are covered.
The second part is all about scalable data processing with Hadoop and Spark. This includes cluster management and real-time processing with Apache Kafka and Flink. You work with cloud services like AWS, Azure, and Google Cloud and set up data warehousing solutions like Amazon Redshift or Google BigQuery. I find the combination of cloud architecture and real-time streaming particularly practical, because that's exactly what is missing in many companies.
Other topics are data integration, API development, and monitoring data pipelines with monitoring tools, CI/CD pipelines, and fault tolerance strategies. At the end, you learn how to integrate machine learning pipelines and what data ethics looks like in practice.
The training is aimed at IT professionals and software developers as well as data analysts and data scientists. Programming skills and database knowledge will help you, but are not mandatory.
You attend the course either as an in-person seminar in Nuremberg or as a virtual classroom with a trainer. Both variants run over four days and mix theory with practical exercises and exchange with other participants.
At the end, you receive a certificate of participation. In addition, you get a digital open badge that you can share on social networks.
