UNDP Job Postings

This Repo contains all currently (October 2024) available UNDP Job Postings ranging from 2004 to 2024. The Repository entails the script for scraping and storing the raw job postings from UNDP's server as well as a script for preprocessing the data. Analysis and Dashboarding is planned.

Cleaning scripts are combined with amazing open sourc work from https://github.com/alinacherkas

Data after processing

import pandas as pd

df = pd.read_parquet("data/undp_jobs_processed.parquet")

df.head(20)

	job_id	content	title	country	region	year	type_of_contract	post_level	languages_required	staff_category	background	duties_responsibilities	competencies	skills_experiences	language
0	JOB20338108876596148de9fba9a926ad06	['hiv and sti clinical consultant (ic)', 'loca...	hiv and sti clinical consultant (ic)	FJ	Australia and Oceania	2022	Individual Contract	International Consultant	English	International Consultant	the united nations development programme (undp...	project description and consultancy rationale ...	strong interpersonal and communication skills;...	educational qualifications : minimum master s ...	en
1	JOB579f6f45d13396afc130172715c7bba3	['internship- pacific digital economy programm...	internship- pacific digital economy programme ...	SB	Australia and Oceania	2022	Internship	Intern	English	Intern	the united nations capital development fund (u...	under the guidance and supervision of unc df s...	uncdf/undp core competencies : communication d...	education : candidate must be enrolled in a de...	en
2	JOB2b8aa5ccf8370e1acb7ba55cdc1e27d9	['consultant international spécialisé dans le ...	consultant international spécialisé dans le co...	DJ	Sub-Saharian Africa	2022	Individual Contract	International Consultant	French	International Consultant	a vis de r ecru te ment d un consultant in div...	2. description du pro jet le pro jet d ap pui ...	4. liv rables attenduslivrables/résultatsdurée...	7. qualifications et experiences requisesi. qu...	fr
3	JOB6c058f6289a3634356df9788d72fb6a8	['consultant national pour l’élaboration du pl...	consultant national pour l’élaboration du plan...	DJ	Sub-Saharian Africa	2022	Individual Contract	National Consultant	French	National Consultant	a vis de r ecru te ment d un consultant in div...	3. object if s l object if principal de la mis...	9. qualification le ou la consultant do it pos...	10. term es de pai e men tle consultant sera p...	fr
4	JOB72bdf033a137fb5a2dd18ad4b6941cdd	['individual consultant - national project off...	individual consultant - national project officer	SA	Western Asia	2022	Individual Contract	National Consultant	English, Arabic	National Consultant	post title : national project officer starting...	scope of work : ensure effective and eff...	competencies : corporate competencies : demons...	required skills and experience : educat...	en
5	JOBaf1f60615b47a0e25d635679003042ee	['ic-023-22 national ecotourism site design ex...	ic-023-22 national ecotourism site design expe...	Home-based	Home-based	2022	Individual Contract	National Consultant	English, Arabic	National Consultant	undp iraq seeks to implement a 3-year “climate...	the national ecotourism site design expert wil...	corporate competencies : demonstrates integrit...	qualifications, skills, and professional exper...	en
6	JOB089a82f6bf0f937885ea7a1189429ff5	['development of materials and facilitation of...	development of materials and facilitation of t...	MZ	Sub-Saharian Africa	2022	Individual Contract	National Consultant	English	National Consultant	the sd g "localization" and the 2030 agenda re...	training course on sd g localization methodolo...	corporate competences : demonstrates integrity...	experience and qualifications academic qualifi...	en
7	JOB80883842e87f7df110ac4239ed0abb25	['international consultant – ppg gender specia...	international consultant – ppg gender speciali...	FJ	Australia and Oceania	2022	Individual Contract	International Consultant	Unspecified	International Consultant	following from the success of the tonga r 2 r ...	scope of work the international gender special...	strong interpersonal and communication skills;...	educational qualifications : master s degree o...	en
8	JOB2f3528147d720b140c9bbc7a8a7f2d5d	['international specialist(s) on climate chang...	international specialist(s) on climate change ...	Home-based	Home-based	2022	Individual Contract	International Consultant	English	International Consultant	undp cambodia country office works in partners...	the main role of the senior international spec...	functional competencies : strong reporting and...	education : master s degree in climate change...	en
9	JOB97bc4f050ce776d284ae23045c15c043	['operations and programme support coordinator...	operations and programme support coordinator	JO	Western Asia	2022	FTA Local	NO-A	English, Arabic	National Professional	within the framework of the regional programme...	under the direct supervision of the uno dc cou...	corporate competencies : demonstrates commitme...	education : bachelor s degree in business admi...	en
10	JOB8bdd536e33093ed8fae47bfd74485189	['consultant national pour l’élaboration de la...	consultant national pour l’élaboration de la s...	DJ	Sub-Saharian Africa	2022	Individual Contract	National Consultant	Unspecified	National Consultant	a vis de r ecru te ment d un consultant in div...	2.justificationl’un des object if s de la visi...	9. qualification le ou la consultant do it pos...	term es de pai e men tle consultant sera payé ...	fr
11	JOB3e512e141d58b9fda8c756d0b7ed362b	['programme associate - ending violence agains...	programme associate - ending violence against ...	VU	Australia and Oceania	2022	Service Contract	SB/SC/GS-3	English	General Service	un women, grounded in the vision of equality e...	provide advanced administrative and logistical...	key performance indicators : timely and accura...	education : completion of secondary education ...	en
12	JOB5ebb99b2e183f711995b8ce115fb20c4	['lead consultant on humanitarian assistance a...	lead consultant on humanitarian assistance and...	Home-based	Home-based	2022	Individual Contract	International Consultant	English	International Consultant	the revised national development strategy (202...	undp and w fp have commissioned this study to ...	core competencies : demonstrating/safeguarding...	education a phd or a master s degree in the fi...	en
14	JOB919010545cb6dcb0a514d33f9cd983b4	['call for volunteers: sustainable finance exp...	call for volunteers : sustainable finance exp...	Home-based	Home-based	2022	UNV	UNV	English	UNV	un women, grounded in the vision of equality e...	key functions : review the assessment of about...	integrity : demonstrate consistency in upholdi...	education : advanced university degree in busi...	en
15	JOBde40cafc4ca944d09912920952fa4e22	['english/dutch translation and interpretation...	english/dutch translation and interpretation	Home-based	Home-based	2022	Individual Contract	International Consultant	English	International Consultant	un women, grounded in the vision of equality e...	purpose of the consultancy as part of implemen...	core values : respect for diversity;integrity;...	education degree or certification or equivalen...	en
16	JOB1e2cea0a0a4693a591faf7990b896202	['onumujeres/ecu/ps/22-008 asistencia técnica ...	onumujeres/ecu/ps/22-008 asistencia técnica pa...	EC	Latin America and the Carribean	2022	Individual Contract	National Consultant	Spanish	National Consultant	1.contexto organ i zac ional la e ntida d de l...	4.alcance de los obj et ivo sse esp era que el...	8.indicadores de rendimientoproductos/ entre g...	11.procedimiento de s ele cci n y re quis it o...	es
17	JOB6e257188735c11befb96c09fedda58ee	['bbrso181270:review and update of national dr...	bbrso181270 : review and update of national dr...	BB	Latin America and the Carribean	2022	Individual Contract	International Consultant	English	International Consultant	to apply, interested persons should upload the...	1.1review and analysis of the draft gender pol...	sound understanding of national and local deve...	master s in development studies, socio-economi...	en
18	JOBd62c41b2f523acf8e5bf3411a28d6067	['long-term agreement - national senior govern...	long-term agreement - national senior governan...	LB	Western Asia	2022	Individual Contract	National Consultant	English, Arabic	National Consultant	undp launched the anti-corruption for trust in...	under the supervision of act project manager, ...	functional competencies : cultural, gender, re...	i. academic qualifications : a minimum of a ba...	en
19	JOB5851042b6c81564dc853bd062a4f4ecd	['national consultant-conduct capacity needs a...	national consultant-conduct capacity needs ass...	ET	Sub-Saharian Africa	2022	Individual Contract	National Consultant	English	National Consultant	ethiopia has adopted normative frameworks on g...	the national consultant will be responsible to...	core values and guiding principles demonstrate...	education advanced university degree (master's...	en
21	JOBbebd1008e8e31383d3a5d26e2085f293	['international consultant legal analysis for ...	international consultant legal analysis for in...	LA	South-Eastern Asia	2022	Individual Contract	International Consultant	Unspecified	International Consultant	unc df is the united nations capital investmen...	the legal review documentation and associated ...	owns a comprehensive knowledge of the organic ...	minimum master s degree in finance related are...	en

Project Plan: United Nations Development Programme Jobs Analysis

Project Overview

Project Name: United Nations Development Programme Jobs Analysis

Team

• Jonas Nothnagel

Background and Objective: The United Nations Development Programme (UNDP), founded in 1965, is one of the largest UN entities spanning over 177 countries. Interestingly, UNDP has been publicly posting all their vacancies since 2006. This makes up to over 120000 job postings in 177 countries. The purpose of this capstone project is to build a data engineering framework that automatically scrapes all available job postings, transforms the data into an actionable dataset, investigates the dataset and ultimately visualises the data and insights from the analysis in an interactive web application. The main objective is to build the robust data engineering backend but also to understand how the job market for international development has evolved over the last years - for example whether technical jobs have become more promiment or what skillsets are most sought after now compared to a few years ago.

Key Features and Components:

1. Data Retrieval: Automate the retrieval of UNDP job postings. This will include writing a web scraper (Python) and scrape the full dataset. Additionally, the scraper shall be automatically re-run periodically adding any new datapoints to the raw dataset.

   • Python Python for building the web scraper - mostly using request and beautifulsoup.

2. Data Storage: Store the scraped raw data in a Cloud database. I am aiming to store the data in Microsoft Azure since it is most relevant for my work. One initial idea was to write the raw data from scraping in a datalake first and then store the transformed structured data into a datawarehouse and expose the final (gold-standard) dataset from a Azure Datawarehouse to the web application.

   • Microsoft Azure (SQL) Microsoft Azure Datalake, Datawarehouse, DataFactory - to be determined what serves my purpose best. But I am planning to store the data on Azure.
   • Apache Airflow: Automate the data retrieval and storage processes using Airflow DAGs to ensure data is updated periodically.

3. Data Transformation: ETL. This step will include the ETL - mostly transformation of the data. Currently, I am thinking of cleaning and transforming the data using Python. Once the data is in a clean structured format I would like to write a few SQL scripts that exposes partitioned data for analysis. These could be set up with DBT potentially.

   • Python For initial cleaning and transformation.
   • SQL, DBT For additional ETL scripting and serving data for analysis.

4. Data Analysis: Perform advanced data analysis to uncover insights and answer a few questions I will define along the way. Data analysis will be conducted with Python. Since a lot of the data is in text form the analysis will be NLP heavy.

5. Interactive Web Application: The main insights of the transformed data as well as insights from the data analysis shall be displayed in a web application. I am planning to use streamlit for this and make it interactive for the user.

   • Streamlit For web deployment and interactive application.

Dataset

The dataset will be scraped from https://jobs.undp.org/cj_view_job.cfm?cur_job_id=115937 whereas the scraper must iterate over the job_id.

There are currently around 120000 job postings online which will result in a multi-dimensional dataset with the same amount of rows. Columns will feature dimensions such as:

job_id	job_description	title	country	region	year	type_of_contract	post_level	languages_required	staff_category	background	duties_responsibilities	competencies	skills_experiences	language

Next Steps:

• Finalize the project plan.
• Map out all components and add a high-level overview of the components and the interactive (visualisation).
• Set up Azure credits and choose most suiting products.
• Write web scraper and write data to Azure.
• Set up Airflow to run web scraping periodically.
• write data cleaning and transformation script and add to automated workflow such that is triggered by Airflow when new data is added.
• Test robustness of first environment: Scraping - storing - transforming - storing gold standard transformed data.
• Write analytic scripts.
• Serve analytic insights and data to Streamlit.