Data Engineering Job Trends Analysis

A comprehensive analysis of data engineering job roles, focusing on the most in-demand skills, tools, and technologies. This project helps professionals align their learning and development with industry trends by providing actionable insights.

Features

• Job Repository: Contains data from 1,000+ data engineering job postings.
• Skill Trends: Identifies key programming languages, tools, and certifications.
• Insights Dashboard: Visualizes job market trends for effective decision-making.
• Automation: Uses Python and web scraping for up-to-date data collection.

Technologies Used

• Jupyter Notebooks, Python and BeautifulSoup : Web scraping job postings.
• AWS Glue, ETL, ATHENA: Data cleaning and transformation.
• AWS QUICKSIGHT:* Data visualization.

Insights

• Top Programming Languages: Python, Java, and Scala dominate the field.
• Popular Cloud Platforms: AWS is the most frequently mentioned, followed by Azure and GCP.
• In-demand Tools: Apache Spark, Hadoop, and Airflow lead the list of big data tools.
  dashboard

Check detail steps on kaggle

https://www.kaggle.com/code/sanchayr/dataengineering-linkedin-job-analysis-with-aws