@ -16,7 +16,7 @@ Don't worry if things feel a bit challenging—that's part of the process! Just
> We can mention thing you do not know; but by this time, you know what to do! Search for it, ask your peers and use clever prompts ;)
- **You need to continue on the HTML, CSS, JS code you submitted for the exercise `first-move`, but with an empty JavaScript file and do not forget to change the name of the linked files to the name of this exercise!**
- **You need to continue on the HTML, CSS, JS code you submitted for the exercise `first-move`, but with an empty JavaScript file. Do not forget to change the name of the linked files to the name of this exercise as well as following the new instructions!**
### Resources
@ -33,12 +33,12 @@ We provide you with some content to get started smoothly, check it out!
#### Task 1:
Let's put a button on the top right corner of the page, that will toggle (close or open) the left eye when clicked.
Let's put a button on the top right corner of the page with the `id` set to "eye-btn", that will toggle (close or open) the left eye when clicked.
Add it in the HTML structure:
```js
<button>Click to close the left eye</button>
<buttonid="eye-btn">Click to close the left eye</button>
```
And add the style in the CSS file:
@ -55,18 +55,18 @@ button {
#### Task 2:
Select the button in your JavaScript file that will allow the user to control the robot’s left eye.
Select the button in your JavaScript file by its `id`. That will allow the user to control the robot’s left eye.
```js
// Select the button element using its ID so we can interact with it in our JavaScript
- "How do I use `querySelector` to select an HTML element by its ID?"
- "How do I use `getElementById` to select an HTML element by its ID?"
#### Task 3:
@ -74,44 +74,53 @@ Write a function that will be triggered when the button is clicked.
This function will make the robot "wink" by toggling the `eye-closed` class on the left eye and change the `button` text based on the current state of the eye.
- It changes the text content of the button: if the eye is open, write "Click to close the left eye", if the eye is closed, write "Click to open the left eye".
1- It changes the text content of the button: if the eye is open, write "Click to close the left eye", if the eye is closed, write "Click to open the left eye".
- It toggles the class eye-closed in the `classList` of the eye-left HTML element.
2- It toggles the class `eye-closed` in the `classList` of the `eye-left` HTML element.
It changes the background color of the eye-left: if the eye is open, to "red", if the eye is closed, to "black"
3- It changes the background color of the `eye-left`: if the eye is open, to "red", if the eye is closed, to "black"
**Code Example:**
```js
const button = document.querySelector('button')
const button = document.getElementById('eye-btn')
const handleClick = (event) => {
// Select the left eye element by its ID
const myDiv = ...
// Select left eye by its ID and assign it to the variable eyeLeft
const eyeLeft = ...
// Check if the eye is currently closed by looking at its background color
// Check if the eye is currently closed by looking at the eyeLeft background color, if it's 'black', that means it's closed.
if (...) {
// If the eye is closed, open it and update the button text
/*
- Add to the 'button' element the text: "Click to close the left eye"
- Change the 'eyeLeft' element background color to red
*/
} else {
// If the eye is open, close it and update the button text
/*
If the eye is open:
- Add to the 'button' element the text: "Click to open the left eye"
- Change the 'eyeLeft' element background color to black
*/
}
// Toggle the 'eye-closed' class on the 'eye-left' div
eyeLeft.classList.toggle("eye-closed")
};
// register the event:
/* Register the event:
here we ask the button to call our `handleClick` function
on the 'click' event, so every time it's clicked
*/
button.addEventListener('click', handleClick)
// here we ask the button to call our `handleClick` function
// on the 'click' event, so every time it's clicked
```
### Expected result
You can see an example of the expected result [here](https://youtu.be/wuYTorfBViE)
You can see an example of the expected result [here](https://youtu.be/IQ6-3X3JBss)
**`Prompt Examples:`**
- "As a beginner, explain to me what is `querySelector` in JavaScript, and how do I use it to select an HTML element by its ID or class?"
- "As a beginner, explain to me how can I change the text content of an `HTML element` using JavaScript?"
- "As a beginner, explain to me how can I change the text content and the background color of an `HTML element` using JavaScript?"
- "As a beginner, explain to me how do I use `addEventListener` to make a button respond to a click event in JavaScript?"
The position of an element in an array is called its index, starting from 0. So, our `batteryLevels` array is roughly equivalent to writing this object:
The position of an element in an array is called its `index`, starting from `0`. So, our `batteryLevels` array is roughly equivalent to writing this object:
```js
let batteryLevelsObject = {
@ -109,7 +109,7 @@ Now, the array looks like this:
#### Task 1:
You must declare a variable `components` that contains 4 strings, one for each robot component "motor", "sensor", "battery" and "camera" (respect that order).
You must declare a variable `components` that contains 4 strings, one for each robot component : `"motor"`, `"sensor"`, `"battery"` and `"camera"` (respect that order).
#### Task 2:
@ -128,7 +128,7 @@ We provide you a variable `robotParts` that contains some elements. You will hav
#### Task 3:
You must replace the third element of the provided `replaceComponents` array with the string 'enhanced'.
- You must replace the third element of the provided `replaceComponents` array with the string 'enhanced'.
Example:
@ -137,7 +137,9 @@ let replaceComponents = ["motor", "sensor", "battery"];
// expect -> ['motor', 'sensor', 'enhanced']
```
You must swap the first and second element of the provided `swapComponents` array.
- You must `swap` the first and second element of the provided `swapComponents` array using a variable `temp`.
_Hint: Use a `temp` variable to store the first element, you must modify the array swapComponents, not create a new one !_
@ -24,7 +24,7 @@ We provide you with some content to get started smoothly, check it out!
- Video [Basic set up of an HTML page](https://www.youtube.com/watch?v=QtKoO7tT-Gg&list=PLHyAJ_GrRtf979iZZ1N3qYMfsPj9PCCrF&index=1)
- Video [Different HTML tags overview](https://www.youtube.com/watch?v=Al-Jzpib8VY&list=PLHyAJ_GrRtf979iZZ1N3qYMfsPj9PCCrF&index=2)
Those videos are accompanying you step by step in each exercise, but if you want to check right away all the notions covered in the quest, you can watch the whole playlist throughout your next exercices[Web - HTML, CSS & DOM JavaScript](https://www.youtube.com/playlist?list=PLHyAJ_GrRtf979iZZ1N3qYMfsPj9PCCrF).
Those videos are accompanying you step by step in each exercise, but if you want to check right away all the notions covered in the quest, you can watch the whole playlist throughout your next exercises[Web - HTML, CSS & DOM JavaScript](https://www.youtube.com/playlist?list=PLHyAJ_GrRtf979iZZ1N3qYMfsPj9PCCrF).
> Your working environment may not be exactly the same as what you see in the videos or documentation, just try to adapt your work according to your research and discoveries.
The goal of this project is to perform a Backtest on the SP500 constituents, which represents the 500 largest companies by market capitalization in the United States.
## Data
### Role Play
You are a quantitative analyst at a prestigious hedge fund. Your manager has tasked you with developing and backtesting a stock-picking strategy using historical data from the S&P 500 index. The goal is to create a strategy that outperforms the market benchmark. You'll need to clean and preprocess messy financial data, develop a signal for stock selection, implement a backtesting framework, and present your findings to the investment committee.
### Learning Objectives
By the end of this project, you will be able to:
1. Optimize data types in large datasets to improve memory efficiency
2. Perform exploratory data analysis on financial time series data
3. Identify and handle outliers and missing values in stock price data
4. Preprocess financial data, including resampling and calculating returns
5. Develop a simple stock selection signal based on historical performance
6. Implement a backtesting framework for evaluating trading strategies
7. Compare the performance of a custom strategy against a market benchmark
8. Visualize financial performance data using appropriate charts and graphs
9. Write modular, reusable code for financial data analysis and strategy testing
10. Interpret and communicate the results of a quantitative trading strategy
### Instructions
#### Data
The input files are:
@ -24,42 +45,15 @@ _Note: The quality of this data set is not good: some prices are wrong, there ar
_Note: The corrections will not fix the data, as a result the results may be abnormal compared to results from cleaned financial data. That's not a problem for this small project !_
## Problem
#### Problem
Once preprocessed this data, it will be used to generate a signal that is, for each asset at each date a metric that indicates if the asset price will increase the next month. At each date (once a month) we will take the 20 highest metrics and invest $1 per company. This strategy is called **stock picking**. It consists in picking stock in an index and try to over perform the index. Finally, we will compare the performance of our strategy compared to the benchmark: the SP500
It is important to understand that the SP500 components change over time. The reason is simple: Facebook entered the SP500 in 2013 thus meaning that another company had to be removed from the 500 companies.
The structure of the project is:
```console
project
│ README.md
│ environment.yml
│
└───data
│ │ sp500.csv
│ | prices.csv
│
└───notebook
│ │ analysis.ipynb
|
|───scripts
| │ memory_reducer.py
| │ preprocessing.py
| │ create_signal.py
| | backtester.py
│ | main.py
│
└───results
│ plots
│ results.txt
│ outliers.txt
```
There are four parts:
## 1. Preliminary
#### 1. Preliminary
- Create a function that takes as input one CSV data file. This function should optimize the types to reduce its size and returns a memory optimized DataFrame.
- For `float` data the smaller data type used is `np.float32`
@ -71,7 +65,7 @@ There are four parts:
4. Find the min and the max value
5. Determine and apply the smallest datatype that can fit the range of values
## 2. Data wrangling and preprocessing
#### 2. Data wrangling and preprocessing
- Create a Jupyter Notebook to analyze the data sets and perform EDA (Exploratory Data Analysis). This notebook should contain at least:
@ -112,7 +106,7 @@ At this stage the DataFrame should look like this:
- Resample data on month and keep the last value
- Compute historical monthly returns on the adjusted close
## 3. Create signal
#### 3. Create signal
At this stage we have a data set with features that we will leverage to get an investment signal. As previously said, we will focus on one single variable to create the signal: **monthly_past_return**. The signal will be the average of monthly returns of the previous year
@ -121,7 +115,7 @@ The naive assumption made here is that if a stock has performed well the last ye
- Create a column `average_return_1y`
- Create a column named `signal` that contains `True` if `average_return_1y` is among the 20 highest in the month `average_return_1y`.
## 4. Backtester
#### 4. Backtester
At this stage we have an investment signal that indicates each month what are the 20 companies we should invest 1$ on (1$ each). In order to check the strategies and performance we will backtest our investment signal.
@ -137,7 +131,7 @@ A data point (x-axis: date, y-axis: cumulated_return) is: the **cumulated return
**The command `python main.py` executes the code from data imports to the backtest and save the results.**
### Project repository structure:
```console
project
│ README.md
│ requirements.txt
│
└───data
│ │ sp500.csv
│ | prices.csv
│
└───notebook
│ │ analysis.ipynb
|
|───scripts
| │ memory_reducer.py
| │ preprocessing.py
| │ create_signal.py
| | backtester.py
│ | main.py
│
└───results
│ plots
│ results.txt
│ outliers.txt
```
### Tips:
1. Data Quality Management:
- Be prepared to encounter messy data. Financial datasets often contain errors, outliers, and missing values.
- Develop a systematic approach to identify and handle data quality issues.
2. Memory Optimization:
- When working with large datasets, optimize memory usage by selecting appropriate data types for each column.
- Consider using smaller data types like np.float32 for floating-point numbers when precision allows.
3. Exploratory Data Analysis:
- Spend time understanding the data through visualization and statistical analysis before diving into strategy development.
- Pay special attention to outliers and their potential impact on your strategy.
4. Preprocessing Financial Data:
- When resampling time series data, be mindful of which value to keep (e.g., last value for month-end prices).
- Calculate both historical and future returns to avoid look-ahead bias in your strategy.
5. Handling Outliers:
- Develop a method to identify and handle outliers that is specific to each company's historical data.
- Be cautious about removing outliers during periods of high market volatility (e.g., 2008-2009 financial crisis).
6. Signal Creation:
- Start with a simple signal (like past 12-month average returns) before exploring more complex strategies.
- Ensure your signal doesn't use future information that wouldn't have been available at the time of decision.
7. Backtesting:
- Implement your backtesting logic without using loops for better performance.
- Compare your strategy's performance against a relevant benchmark (in this case, the S&P 500).
8. Visualization:
- Create clear, informative visualizations to communicate your strategy's performance.
- Include cumulative return plots to show how your strategy performs over time compared to the benchmark.
9. Code Structure:
- Organize your code into modular functions for better readability and reusability.
- Use a main script to orchestrate the entire process from data loading to results visualization.
10. Results Interpretation:
- Don't just focus on total returns. Consider other metrics like risk-adjusted returns, maximum drawdown, etc.
- Be prepared to explain any anomalies or unexpected results in your strategy's performance.
Remember, the goal is not just to create a strategy that looks good on paper, but to develop a robust process for analyzing financial data and testing investment ideas.
###### Does the readme file contain a description of the project, explain how to run the code from an empty environment, give a summary of the implementation of each python file and contain a conclusion that gives the performance of the strategy?
###### Does the environment contain all libraries used and their versions that are necessary to run the code?
###### Does the requirements contain all libraries used and their versions that are necessary to run the code?
###### Does the notebook contain a missing values analysis? **Example**: number of missing values per variables or per year
The goal of this project is to implement a scoring model based on various source of data ([check data documentation](./readme_data.md)) that returns the probability of default. In a nutshell, credit scoring represents an evaluation of how well the bank's customer can pay and is willing to pay off debt. It is also required that you provide an explanation of the score. For example, your model returns that the probability that one client doesn't pay back the loan is very high (90%). The reason behind is that variable_xxx which represents the ability to pay back the past loan is low. The output interpretability will appear in a visualization.
The ability to understand the underlying factors of credit scoring is important. Credit scoring is subject to more and more regulation, so transparency is key. And more generally, more and more companies prefer transparency to black box models.
### Role play
### Resources
Hey there, future credit scoring expert! Ready to dive into the exciting world of predicting loan defaults? You're in for a treat! This project is all about building a nifty model that can help figure out how likely someone is to pay back their loan. Cool, right?
### Learning Objective
The ability to understand the underlying factors of credit scoring is important. Credit scoring is subject to more and more regulation, so transparency is key. And more generally, more and more companies prefer transparency to black box models.
Historical timeline of machine learning techniques applied to credit scoring
- [Machine Learning or Econometrics for Credit Scoring: Let’s Get the Best of Both Worlds](https://hal.archives-ouvertes.fr/hal-02507499v3/document)
### Scoring model
### Instructions
#### Scoring model
There are 3 expected deliverables associated with the scoring model:
@ -18,21 +26,28 @@ There are 3 expected deliverables associated with the scoring model:
- The trained machine learning model with the features engineering pipeline:
- Do not forget: **Coming up with features is difficult, time-consuming, requires expert knowledge. ‘Applied machine learning’ is basically feature engineering.**
- The model is validated if the **AUC on the test set is higher than 75%**.
- The model is validated if the **AUC on the test set is at minimum 55%, ideally to 62% included (or in best cases higher than 62% if you can !)**.
- The labelled test data is not publicly available. However, a Kaggle competition uses the same data. The procedure to evaluate test set submission is the same as the one used for the project 1.
- Here are the [DataSets](https://assets.01-edu.org/ai-branch/project5/home-credit-default-risk.zip).
- A report on model training and evaluation:
- Include learning curves (training and validation scores vs. training set size or epochs) to demonstrate that the model is not overfitting.
- Explain the measures taken to prevent overfitting, such as early stopping or regularization techniques.
- Justify your choice of when to stop training based on the learning curves.
### Kaggle submission
#### Kaggle submission
The way the Kaggle platform works is explained in the challenge overview page. If you need more details, I suggest [this resource](https://towardsdatascience.com/getting-started-with-kaggle-f9138b35ae18) that gives detailed explanations.
- Create a username following that structure: username*01EDU* location_MM_YYYY. Submit the description profile and push it on the Git platform the first day of the week. Do not touch this file anymore.
- A text document that describes the methodology used to train the machine learning model:
- A text document `model_report.txt`that describes the methodology used to train the machine learning model:
- Algorithm
- Why the accuracy shouldn't be used in that case?
- Limit and possible improvements
### Model interpretability
#### Model interpretability
This part hasn't been covered during the piscine. Take the time to understand this key concept.
There are different level of transparency:
@ -55,16 +70,16 @@ Choose the 3 clients of your choice, compute the score, run the visualizations o
- 1 on which the model is correct and the other on which the model is wrong. Try to understand why the model got wrong on this client.
- Take 1 client from the test set
### Optional
#### Bonus
Implement a dashboard (using [Dash](https://dash.plotly.com/)) that takes as input the customer id and that returns the score and the required visualizations.
### Deliverables
### Project repository structure:
```
project
│ README.md
│ environment.yml
│ requirements.txt
│
└───data
│ │ ...
@ -93,17 +108,17 @@ project
│ │ preprocess.py
```
- `README.md` introduces the project and shows the username.
- `environment.yml` contains all libraries required to run the code.
- `README.md` introduces the project, how to run the code, and shows the username.
- `requirements.txt` contains all libraries required to run the code.
- `username.txt` contains the username, the last modified date of the file **has to correspond to the first day of the project**.
- `EDA.ipynb` contains the exploratory data analysis. This file should contain all steps of data analysis that contributed or not to improve the score of the model. It has to be commented so that the reviewer can understand the analysis and run it without any problem.
- `scripts` contains python file(s) that perform(s) the feature engineering, the model's training and prediction on the test set. It could also be one single Jupyter Notebook. It has to be commented to help the reviewers understand the approach and run the code without any bugs.
Remember, creating a great credit scoring model is like baking a perfect cake - it takes the right ingredients, careful preparation, and a dash of creativity. You've got this!
###### Does the readme file introduce the project, summarize how to run the code and show the username?
###### Does the environment contain all libraries used and the versions that are necessary to run the code?
###### Does the requirements contain all libraries used and the versions that are necessary to run the code?
###### Does the `EDA.ipynb` explain in details the exploratory data analysis?
@ -46,7 +46,7 @@ project
###### Is the model trained only the training set?
###### Is the AUC on the test set higher than 75%?
###### Is the AUC on the test set is between 55% (included) to 62%(included) or higher than 62%?
###### Does the model learning curves prove that the model is not overfitting?
@ -59,7 +59,7 @@ project
```prompt
python predict.py
AUC on test set: 0.76
AUC on test set: 0.62
```
@ -75,11 +75,13 @@ This [article](https://medium.com/thecyphy/home-credit-default-risk-part-2-84b58
### Descriptive variables:
###### These are important to understand for example the age of the client. If the data could be scaled or modified in the preprocessing pipeline but the data visualised here should be "raw". Are the visualisations computed for the 3 clients?
##### These are important to understand for example the age of the client. If the data could be scaled or modified in the preprocessing pipeline but the data visualized here should be "raw".
- Visualisations that show at least 10 variables describing the client and its loan(s).
- Visualisations that show the comparison between this client and other clients.
###### Are the visualisations computed for the 3 clients?
##### SHAP values on the model are displayed through a summary plot that shows the important features and their impact on the target. This is optional if you have already computed the features importance.
###### Are the 3 clients selected as expected? 2 clients from the train set (1 on which the model is correct and 1 on which the model's wrong) and 1 client from the test set.
Cameras are everywhere. Videos and images have become one of the most interesting data sets for artificial intelligence.
Image processing is a quite broad research area, not just filtering, compression, and enhancement.
Besides, we are even interested in the question, “what is in images?”, i.e., content analysis of visual inputs, which is part of the main task of computer vision.
### Role play
you're going to train a computer to be like a mind reader, but instead of reading thoughts, it's reading emotions! You'll be working with a bunch of pictures of faces, teaching your AI to tell the difference between a big grin and a grumpy frown, or a surprised gasp and a fearful wide-eyed look.
### Learning Objective
The study of computer vision could make possible such tasks as 3D reconstruction of scenes, motion capturing, and object recognition, which are crucial for even higher-level intelligence such as image and video understanding, and motion understanding.
For this project we will focus on two tasks:
@ -18,7 +26,9 @@ With the computing power exponentially increasing the computer vision field has
- The history behind this field is fascinating! [Here](https://kapernikov.com/basic-introduction-to-computer-vision/) is a short summary of its history.
### Project goal and suggested timeline
### Instructions
#### Project goal:
The goal of the project is to implement a **system that detects the emotion on a face from a webcam video stream**. To achieve this exciting task you'll have to understand how to:
@ -32,7 +42,7 @@ Then starts the emotion detection in a webcam video stream step that will last u
The two steps are detailed below.
### Preliminary:
#### Preliminary:
- Take [this course](https://www.coursera.org/learn/convolutional-neural-networks). This course is a reference for many reasons and one of them is the creator: **Andrew Ng**. He explains the basics of CNNs but also some more advanced topics as transfer learning, siamese networks etc ...
- I suggest to focus on Week 1 and 2 and to spend less time on Week 3 and 4. Don't worry the time scoping of such MOOCs are conservative. You can attend the lessons for free!
@ -41,7 +51,7 @@ The two steps are detailed below.
- Start first with a logistic regression to understand how to handle images in Python. And then train your first CNN on this data set.
### Face emotions classification
#### Face emotions classification
Emotion detection is one of the most researched topics in the modern-day machine learning arena. The ability to accurately detect and identify an emotion opens up numerous doors for Advanced Human Computer Interaction.
The aim of this project is to detect up to seven distinct facial emotions in real time.
@ -57,7 +67,7 @@ Your goal is to implement a program that takes as input a video stream that cont
This dataset was provided for this past [Kaggle challenge](https://www.kaggle.com/competitions/challenges-in-representation-learning-facial-expression-recognition-challenge/overview).
It is possible to find more information about on the challenge page. Train a CNN on the dataset `train.csv`. Here is an [example of architecture](https://www.quora.com/What-is-the-VGG-neural-network) you can implement.
**The CNN has to perform more than 60% on the test set**. You can use the `test_with_emotions.csv` file for this. You will see that the CNNs take a lot of time to train.
You don't want to overfit the neural network. I strongly suggest to use early stopping, callbacks and to monitor the training using the `TensorBoard`.
You don't want to overfit the neural network. I strongly suggest to use early stopping, callbacks and to monitor the training using the `TensorBoard` 'note: Integrating TensorBoard is not optional'.
You have to save the trained model in `final_emotion_model.keras` and to explain the chosen architecture in `final_emotion_model_arch.txt`. Use `model.summary())` to print the architecture.
It is also expected that you explain the iterations and how you end up choosing your final architecture. Save a screenshot of the `TensorBoard` while the model's training in `tensorboard.png` and save a plot with the learning curves showing the model training and stopping BEFORE the model starts overfitting in `learning_curves.png`.
@ -82,7 +92,7 @@ For that step, I suggest again to use **OpenCV** as much as possible. This link
- Optional: **(very cool)** Hack the CNN. Take a picture for which the prediction of your CNN is **Happy**. Now, hack the CNN: using the same image **SLIGHTLY** modified make the CNN predict **Sad**.
You can find an example on how to achieve this in [this article](https://medium.com/@ageitgey/machine-learning-is-fun-part-8-how-to-intentionally-trick-neural-networks-b55da32b7196)
### Deliverable
### Project repository structure:
```
project
@ -90,7 +100,7 @@ project
│ ├── test.csv
│ ├── train.csv
│ └── xxx.csv
├── environment.yml
├── requirements.txt
├── README.md
├── results
│ ├── model
@ -148,12 +158,17 @@ Preprocessing ...
```
### Useful resources:
### Tips
Balance technical prowess with psychological insight: as you fine-tune your CNN and optimize your video processing, remember that understanding the nuances of human facial expressions is key to creating a truly effective emotion detection system.
The goal of this **1 week** project is to get the highest possible score on a Data Science competition. More precisely you will have to predict who survived the Titanic crash.
@ -8,11 +8,11 @@ The goal of this **1 week** project is to get the highest possible score on a Da
[titanic]: titanic.jpg "Titanic"
### Kaggle
#### Kaggle
Kaggle is an online community of data scientists and machine learning practitioners. Kaggle allows users to find and publish data sets, explore and build models in a web-based data-science environment, work with other data scientists and machine learning engineers, and enter competitions to solve data science challenges. It’s a crowd-sourced platform to attract, nurture, train and challenge data scientists from all around the world to solve data science, machine learning and predictive analytics problems.
### Titanic - Machine Learning from Disaster
#### Titanic - Machine Learning from Disaster
One of the first Kaggle competition I did was: Titanic - Machine Learning from Disaster. This is a not-to-be-missed Kaggle competition.
@ -22,47 +22,28 @@ The sinking of the Titanic is one of the most infamous shipwrecks in history. On
While there was some element of luck involved in surviving, it seems some groups of people were more likely to survive than others.
In this challenge, you have to build a predictive model that answers the question: **“what sorts of people were more likely to survive?”** using passenger data (ie name, age, gender, socio-economic class, etc). **You will have to submit your prediction on Kaggle**.
### Preliminary
The way the Kaggle platform works is explained in the challenge overview page. If you need more details, I suggest this [resource](https://towardsdatascience.com/getting-started-with-kaggle-f9138b35ae18) that gives detailed explanations.
- Create a username following this structure: username*01EDU* location_MM_YYYY. Submit the description profile and push it on GitHub the first day of the week. Do not touch this file anymore.
- It is possible to have different personal accounts merged in a team for one single competition.
### Role play
### Deliverables
Ahoy, data explorer! Ready to set sail on the most thrilling voyage of your data science career? Welcome aboard the Kaggle Titanic challenge! You're about to embark on a journey through time, back to that fateful night in 1912.
Your mission, should you choose to accept it (and let's face it, you're already hooked), is to dive deep into the passenger manifest and uncover the secrets of survival. Who lived? Who perished? And most importantly, can you build a model that predicts it all?
```console
project
│ README.md
│ environment.yml
│ username.txt
│
└───data
│ │ train.csv
│ | test.csv
| | gender_submission.csv
│
└───notebook
│ │ main.ipynb
### Learning Objective
In this challenge, you have to build a predictive model that answers the question: **“what sorts of people were more likely to survive?”** using passenger data (ie name, age, gender, socio-economic class, etc). **You will have to submit your prediction on Kaggle**.
```
### Instructions
- `README.md` introduction of the project, shows the username, describes the features engineering and the best score on the **leaderboard**. Note the score on the test set using the exact same pipeline that led to the best score on the leaderboard.
#### Preliminary
- `environment.yml` contains all libraries required to run the code.
The way the Kaggle platform works is explained in the challenge overview page. If you need more details, I suggest this [resource](https://www.kaggle.com/code/alexisbcook/getting-started-with-kaggle) that gives detailed explanations.
- `username.txt` contains the username, the last modified date of the file **has to correspond to the first day of the project**.
- Create a username following this structure: username*01EDU* location_MM_YYYY. Submit the description profile and push it on GitHub the first day of the week. Do not touch this file anymore.
- `main.ipynb` This file (single Jupyter Notebook) should contain all steps of data analysis that contributed or not to improve the accuracy, the feature engineering, the model's training and prediction on the test set. It has to be commented to help the reviewers understand the approach and run the code without any bugs.
- **Submit your predictions on the Kaggle's competition platform**. Check your ranking and score in the leaderboard.
- It is possible to have different personal accounts merged in a team for one single competition.
### Scores
#### Scores
In order to validate the project you will have to score at least **79% accuracy on the Leaderboard**:
In order to validate the project you will have to score at least **78.9% accuracy on the Leaderboard**:
- 78.9% accuracy is the minimum score to validate the project.
@ -75,20 +56,48 @@ Scores indication:
#### Cheating
It is impossible to get 100%. Who would have predicted that Rose wouldn't let [Jack on the door](https://www.insider.com/jack-and-rose-werent-on-a-door-in-titanic-2019-7) ?
It is impossible to get 100%. Who would have predicted that Rose wouldn't let [Jack on the door](https://www.reddit.com/r/titanic/comments/14i0v5j/for_all_the_newbies_proof_its_not_a_door/?rdt=35268) ?
All people having 100% of accuracy on the Leaderboard cheated, there's no point to compare with them or to cheat. The Kaggle community estimates that having more than 85% is almost considered as cheated submissions as they are element of luck involved in the surviving.
**You can't use external data sets than the ones provided in that competition.**
### The key points
#### The key points
- **Feature engineering**:
Put yourself in the shoes of an investigator trying to understand what happened exactly in that boat during the crash. Do not hesitate to watch the movie to try to find as many insights as possible. Without a smart the feature engineering there's no way to validate the project ;-)
- The leaderboard evaluates on test data for which you don't have the labels. It means that there's no point to over fit the train set. Check the over fitting on the train set by dividing the data and by cross-validating the accuracy.
### Advice
### Project repository structure
```console
project
│ README.md
│ requirements.txt
│ username.txt
│
└───data
│ │ train.csv
│ | test.csv
| | gender_submission.csv
│
└───notebook
│ │ main.ipynb
```
- `README.md` introduction of the project, shows the username, describes the features engineering and the best score on the **leaderboard**. Note the score on the test set using the exact same pipeline that led to the best score on the leaderboard.
- 'requirements.txt` contains all libraries required to run the code.
- `username.txt` contains the username, the last modified date of the file **has to correspond to the first day of the project**.
- `main.ipynb` This file (single Jupyter Notebook) should contain all steps of data analysis that contributed or not to improve the accuracy, the feature engineering, the model's training and prediction on the test set. It has to be commented to help the reviewers understand the approach and run the code without any bugs.
- **Submit your predictions on the Kaggle's competition platform**. Check your ranking and score in the leaderboard.
### Tips
Don't try to build the perfect model the first day. Iterate a lot and test your assumptions:
###### Is the accuracy associated with the username in `username.txt` higher than 79%? The best submission score can be accessed from the user profile.
###### Is the accuracy associated with the username in `username.txt` higher than 78.9%? The best submission score can be accessed from the user profile.
The goal of this project is to build an NLP-enriched News Intelligence
platform. News analysis is a trending and important topic. The analysts get
@ -10,7 +10,25 @@ The platform connects to a news data source, detects the entities, detects the
topic of the article, analyses the sentiment and performs a scandal detection
analysis.
### Scraper
### Role Play
You're a Natural Language Processing (NLP) specialist at a tech startup developing a sentiment analysis tool for social media posts. Your task is to build the preprocessing pipeline and create a bag-of-words representation for tweet analysis.
### Learning Objectives
1. Set up an NLP-focused Python environment
2. Implement basic text preprocessing techniques (lowercase, punctuation removal)
3. Perform text tokenization at sentence and word levels
4. Remove stop words from text data
5. Apply stemming to reduce words to their root forms
6. Create a complete text preprocessing pipeline
7. Implement a bag-of-words model using CountVectorizer
8. Analyze word frequency in a corpus of tweets
9. Prepare a labeled dataset for sentiment analysis
### Instructions
#### Scraper
News data source:
@ -29,7 +47,7 @@ Use data from the last week otherwise the volume may be too high.
There should be at least 300 articles stored in your file system or SQL
database.
### NLP engine
#### NLP engine
In production architectures, the NLP engine delivers a live output based on the
news that are delivered in a live stream data by the scraper. However, it
@ -41,7 +59,7 @@ the stored data.
Here how the NLP engine should process the news:
#### **1. Entities detection:**
##### **1. Entities detection:**
The goal is to detect all the entities in the document (headline and body). The
type of entity we focus on is `ORG`. This corresponds to companies and
@ -52,7 +70,7 @@ organizations. This information should be stored.
In this project, you'll apply machine learning to finance. Your goal as a Quant/Data Scientist is to create a financial strategy that uses a signal generated by a machine learning model to outperform the [SP500](https://en.wikipedia.org/wiki/S%26P_500).
@ -6,15 +8,19 @@ The S&P 500 Index is a collection of 500 stocks that represent the overall perfo
The S&P 500 started in 1926 with only 90 stocks and has grown to include 500 stocks since 1957. Historically, the average annual return of the S&P 500 has been about 10-11% since 1926, and around 8% since 1957.
### Role play
As a Quantitative Researcher, your challenge is to develop a strategy that can consistently outperform the S&P 500, not just in one year, but over many years. This is a difficult task and is the primary goal of many hedge funds around the world.
The project is divided in parts:
### Learning Objective
- **Data processing and feature engineering**: Build a dataset: insightful features and the target
- **Machine Learning pipeline**: Train machine learning models on the dataset, select the best model and generate the machine learning signal.
- **Strategy backtesting**: Generate a strategy from the Machine Learning model output and backtest the strategy. As a reminder, the idea here is to see what would have performed the strategy if you had invested.
### Data processing and features engineering
### Instructions
#### Data processing and features engineering
The file `HistoricalData.csv` contains the open-high-low-close (OHLC) SP500 index data and the other file, `all_stocks_5yr.csv`, contains the open-high-low-close-volume (OHLCV) data on the SP500 constituents.
@ -42,7 +48,7 @@ We assume it is day `D`, and we want to take a position on the next n days. The
> Remark: The target used is the return computed on the price and not the price directly. There are statistical reasons for this choice - the price is not stationary. The consequence is that a machine learning model tends to overfit while training on not stationary data.
### Machine learning pipeline
#### Machine learning pipeline
- Cross-validation deliverables:
- Implements a cross validation with at least 10 folds. The train set has to be bigger than 2 years history.
@ -80,7 +86,7 @@ Once you'll have run the grid search on the cross validation (choose either Bloc
- (optional): [Train an RNN/LSTM](https://towardsdatascience.com/predicting-stock-price-with-lstm-13af86a74944). This is a nice way to discover and learn about recurrent neural networks. But keep in mind that there are some new neural network architectures that seem to outperform recurrent neural networks. Here is an [interesting article](https://towardsdatascience.com/the-fall-of-rnn-lstm-2d1594c74ce0) about the topic.
### Strategy backtesting
#### Strategy backtesting
- Backtesting module deliverables. The module takes as input a machine learning signal, convert it into a financial strategy. A financial strategy DataFrame gives the amount invested at time `t` on asset `i`. The module returns the following metrics on the train set and the test set.
- Profit and Loss (PnL) plot: save it as `strategy.png`
@ -107,7 +113,7 @@ Once you'll have run the grid search on the cross validation (choose either Bloc
- PnL plot
- strategy metrics on the train set and test set
### Example of strategies:
#### Example of strategies:
- Long only:
- Binary signal:
@ -172,7 +178,7 @@ Here's an example on how to convert a machine learning signal into a financial s
project
├── data
│ └── sp500.csv
├── environment.yml
├── requirements.txt
├── README.md
├── results
│ ├── cross-validation
@ -199,7 +205,10 @@ project
Note: `features_engineering.py` can be used in `gridsearch.py`
### Files for this project
### Tips
Remember, the goal of this project is not just to beat the S&P 500 in a backtest, but to learn about the process of developing and testing trading strategies using machine learning techniques.
### Resources
You can find the data required for this project in this :
- User Interface Design is a hybrid role and can bring together concepts from interaction design, visual design, information architecture and even Front End Development.
- Pay attention to the global aspect of the deliverables. They must be clear, simple and easy to read. You can get inspiration [canva](https://www.canva.com/) but don’t overload your design with too much details!
- Pay attention to the global aspect of the file. It must be clear, simple and easy to read. You can get inspiration [canva](https://www.canva.com/) but don’t overload your design with too much details!
## UI Challenge - Watches
### UI Challenge - Watches
### Instructions
@ -332,7 +332,7 @@ Don't forget to:
- You can use existing Libraries by browsing the Figma community resources.
- Pay attention to the global aspect of the file. It must be clear, simple and easy to read. You can get inspiration [canva](https://www.canva.com/) but don’t overload your design with too much details!
## UI Challenge - Flight
### UI Challenge - Flight
### Instructions
@ -362,7 +362,7 @@ Don't forget to:
- [Figma article about Symbols and Variants](https://help.figma.com/hc/en-us/articles/360056440594-Create-and-use-variants).
- Pay attention to the global aspect of the file. It must be clear, simple and easy to read. You can get inspiration [canva](https://www.canva.com/) but don’t overload your design with too much details!
## UI Challenge - Spotify
### UI Challenge - Spotify
### Instructions
@ -394,7 +394,7 @@ Don't forget to:
- [Figma article about Symbols and Variants](https://help.figma.com/hc/en-us/articles/360056440594-Create-and-use-variants).
- Pay attention to the global aspect of the file. It must be clear, simple and easy to read. You can get inspiration [canva](https://www.canva.com/) but don’t overload your design with too much details!
- [Example here with Nintendo.com](https://www.figma.com/file/ahSnWRZeKBO62oJDiXltxY/UI-III---Ex-2) (The screenshots date back to nov 2021. They may differ from the current version of [Nintendo.com](http://Nintendo.com) but the rules are the same).
## Accessibility (website)
### Accessibility (website)
### Instructions
@ -106,7 +106,7 @@ Don't forget to:
- Error states.
- etc.
## Accessibility (app)
### Accessibility (app)
### Instructions
@ -144,7 +144,7 @@ Don't forget to:
- Error states
- etc.
## Breadcrumbs
### Breadcrumbs
### Instructions
@ -167,7 +167,7 @@ Don't forget to:
- [What are breadcrumbs?](https://www.seoptimer.com/blog/breadcrumbs-website/)
- [What is a radio button?](https://www.justinmind.com/blog/radio-button-design-examples/)
## Calendars
### Calendars
### Instructions
@ -232,7 +232,7 @@ Don't forget to:
- Pay attention to the global aspect of the file. It must be clear, simple and easy to read. You can get inspiration [canva](https://www.canva.com/) but don’t overload your design with too much details!
## Time pickers
### Time pickers
### Instructions
@ -251,7 +251,7 @@ Don't forget to:
- Pay attention to the global aspect of the file. It must be clear, simple and easy to read. You can get inspiration [canva](https://www.canva.com/) but don’t overload your design with too much details!
## Micro-interactions
### Micro-interactions
### Instructions
@ -282,7 +282,7 @@ Don't forget to:
- [Why use micro-interactions in your design](https://hike.one/update/why-use-micro-animations-in-your-design)
- [Micro-interactions: why, when, and how to use them to boost the UX](https://uxdesign.cc/micro-interactions-why-when-and-how-to-use-them-to-boost-the-ux-17094b3baaa0)
“To ask open-ended questions is the best approach, but it’s easy to get into the weeds in data analysis when every answer is a paragraph or two of prose. Users quickly tire of answering many open-ended questions, which usually require a lot of typing and explanation.” Norman Nielsen Group
## Run interviews
### Run interviews
### Instructions
@ -68,7 +68,7 @@ Then, write down the script on a written document, and give elements of context
- Pay attention to the global aspect of the document. It must be clear, simple and easy to read. You can get inspiration [canva](https://www.canva.com/) but don’t overload your design with too much details!
## Affinity diagram & Empathy map
### Affinity diagram & Empathy map
### Instructions
@ -108,7 +108,7 @@ Don't forget to:
Remember, you are not the user, so your opinions are not absolute truth.
## Problem statement
### Problem statement
### Instructions
@ -137,7 +137,7 @@ You may use the "How Might We" exercise in pairs.
"Out of clutter, find simplicity. From discord, find harmony. In the middle of difficulty, lies opportunity.’ Albert Einstein
## Ideation
### Ideation
### Instructions
@ -180,7 +180,7 @@ Quote:
- "Brainstorming means using the brain to storm a creative problem. Do so in commando fashion, with each stormer attacking the same objective.’ Alex Faickney Osborn
## User flow
### User flow
### Instructions
@ -210,7 +210,7 @@ Don't forget to:
"Each use case is represented as a sequence of simple steps, beginning with a user’s goal and ending when that goal is fulfilled.’ Usability.gov
@ -41,7 +41,7 @@ Conduct a heuristic analysis on **one** of these websites:
'If you think design is expensive, you should look at the cost of bad design’ Ralf Speth | Former CEO Jaguar Land Rover
## Site map
### Site map
### Instructions
@ -81,7 +81,7 @@ Stick to the website you have chosen in the previous exercise "heuristics" and d
"The organization, search, and navigation systems that help people to complete tasks, find what they need, and understand what they’ve found’ Peter Morville | Information Architecture for the WWW.
## JTBD
### JTBD
### Instructions
@ -121,7 +121,7 @@ And remember: You are not your user!
- "Documenting the what and why of each element promotes organization and makes the handoff to the development team much smoother." UX Booth
- "When we buy a product, we essentially "hire" something to get a job done. If it does the job well, when we are confronted with the same job, we hire that same product again. And if the product does a crummy job, we "fire" it and look around for something else we might hire to solve the problem." Clayton M Christensen
## Card sorting
### Card sorting
### Instructions
@ -168,7 +168,7 @@ Don't forget to:
- [10 things to know about card sorting](http://www.measuringu.com/blog/card-sorting.php)
- [Card sorting: a definitive guide](http://boxesandarrows.com/card-sorting-a-definitive-guide/)
## Music label wireframes
### Music label wireframes
### Instructions
@ -218,7 +218,7 @@ Vocabulary:
- Figma.
## Test protocol
### Test protocol
### Instructions
@ -257,7 +257,7 @@ Don't forget to:
- You need to detect: How many errors do users make? How severe are these errors? How easily can they recover from the errors?
- Design needs to fail.Failure is even a necessary step, but ideally it should happen before a product is launched, during the prototype and test phases
- “UX Strategy lies at the intersection of UX and business. It provides a much better chance of creating successful products. It enables teams to see the “Big Picture” to achieve the business goals under uncertain conditions” Jamie Levy | UX Strategy: How to Devise Innovative Digital Products That People Want
- UX Strategy is the method by which you validate that your solution solves a problem for real customers in a dynamic marketplace because the market is constantly changing.
- UX Strategy is the method by which you validate that your solution solves a problem for real customers in a dynamic marketplace because the market i#s constantly changing.
## Empathy
@ -111,7 +111,7 @@ Don't forget to:
- With 5 to 10 criteria
- 10 organizations appear on the table
## Define
### Define
### Instructions
@ -158,7 +158,7 @@ Don't forget to:
- The "I want to" part involves a practical action.
- The "So I can" part involves a psychological or emotional purpose.
## Problem statement
### Problem statement
### Instructions
@ -199,7 +199,7 @@ Design is about solving problems. Fall in Love with the Problem, Not the Solutio
- How might make sure Pierre doesn’t get drunk the night before the race?
- Out of scope —> This is not about dehydration, nor about running.
## Ideation
### Ideation
### Instructions
@ -221,7 +221,7 @@ Don't forget to:
- Use as many tools as needed, and make sure you do all the process in 4 working days!
- Pay attention to the global aspect of the deliverables. They must be clear, simple and easy to read. You can get inspiration [canva](https://www.canva.com/) but don’t overload your design with too much details!
## Prototype
### Prototype
### Instructions
@ -265,7 +265,7 @@ Don't forget to:
- High-Fidelity: Wireframes with color, styles, graphical details, and micro-interactions - [Example here](https://miro.medium.com/max/1400/1*Xn0HSKAvhr4TZzC9lN5udw.gif)
- Pay attention to the global aspect of the deliverables. They must be clear, simple and easy to read. You can get inspiration [canva](https://www.canva.com/) but don’t overload your design with too much details!
- "77% of users return to content and information sites because of ease-of-use. Only 22% return because the site belongs to a favorite brand." Forrester
## Wireframes animation
### Wireframes animation
### Instructions
@ -106,7 +106,7 @@ Don't forget to:
- [Efficiently Manage Your Designs - How Mockplus State Page Helps To View & Manage Multiple State](https://help.mockplus.com/p/372)
## Test on features
### Test on features
### Instructions
@ -140,7 +140,7 @@ Don't forget to:
- [The art of guerrilla usability testing](http://www.uxbooth.com/articles/the-art-of-guerrilla-usability-testing/)
- [How to write a user testing report that people will actually read](https://uxdesign.cc/how-to-write-a-user-testing-report-that-people-will-actually-read-652d15d2f92e)
- 'Always design a thing in its larger context: a chair in a room, a room in a house, a house in an environment, an environment in a city plan’ Eero Sarrinen | Knoll
- Creating a great design isn’t just about understanding what the user wants. It’s also about understanding and delivering on what the business needs.
- Creating a great design isn’t just about understanding what the user wants. It’s also about understanding and delivering on what the business needs.#
## Surveys
@ -99,7 +99,7 @@ Here are the recommended steps:
- “Surveys measure and categorize attitudes or collect self-reported data that can help track or discover important issues to address.” Norman Nielsen Group
- Surveys are not accurate in providing behavioral data because USERS OMIT STEPS IN THE MIDDLE AND MEMORIES ARE FAULTY.
## Broadcast strategy
### Broadcast strategy
### Instructions
@ -141,7 +141,7 @@ Once your strategy is prepared, run it!
- [How to get more survey responses](https://rafflepress.com/how-to-get-more-survey-responses/)
## Interviews
### Interviews
### Instructions
@ -183,7 +183,7 @@ Here are the recommended steps :
- [How to design better products through user interviews](https://uxdesign.cc/how-to-design-better-products-through-user-interviews-4c5142bb1fc4)
- [Asking the right questions](https://uxdesign.cc/asking-the-right-questions-on-user-research-interviews-and-testing-427261742a67)
## Personas
### Personas
### Instructions
@ -215,7 +215,7 @@ Your persona card should include common trends amongst the people you got data f
- "If you design for everyone, you delight no one."
- "A good user persona is the one based on user research, without regard to how many attributes we can describe."
## User journey
### User journey
### Instructions
@ -254,7 +254,7 @@ Most articles you'll find will be about user journeys or customer journeys. Keep
- “More options = More problems.” Scott Belsky | VP of Product & Community Adobe
- "A customer journey map is a visualization of the process that a person goes through in order to accomplish a goal. It’s used for understanding and addressing customer needs and pain points’ Norman Nielsen Group
- [The myth of brainstorming](https://uxdesign.cc/the-myth-of-brainstorming-8517e02facc0?sk=995d601cbf988d574e86dd71364cb92f)
- [Ideation method: Worst possible idea](https://www.interaction-design.org/literature/article/learn-how-to-use-the-best-ideation-methods-worst-possible-idea)
- [Ideation method: Worst possible idea](https://www.interaction-design.org/literature/article/learn-how-to#-use-the-best-ideation-methods-worst-possible-idea)
## Ideation B
@ -102,7 +102,7 @@ Pay attention not to use the same ideation technique as in Ex 1!
- [The myth of brainstorming](https://uxdesign.cc/the-myth-of-brainstorming-8517e02facc0?sk=995d601cbf988d574e86dd71364cb92f)
- [Ideation method: Worst possible idea](https://www.interaction-design.org/literature/article/learn-how-to-use-the-best-ideation-methods-worst-possible-idea)
## Ideation C
### Ideation C
### Instructions
@ -151,7 +151,7 @@ Pay attention not to use the same ideation technique as in Ex 2!
- [The myth of brainstorming](https://uxdesign.cc/the-myth-of-brainstorming-8517e02facc0?sk=995d601cbf988d574e86dd71364cb92f)
- [Ideation method: Worst possible idea](https://www.interaction-design.org/literature/article/learn-how-to-use-the-best-ideation-methods-worst-possible-idea)
## User flow
### User flow
### Instructions
@ -178,7 +178,7 @@ Don't forget to:
- [Site flows vs User Flows](https://uxmovement.com/wireframes/site-flows-vs-user-flows-when-to-use-which/)
- [How to make a User Flow diagram](https://www.lucidchart.com/blog/how-to-make-a-user-flow-diagram)
## Prototyping
### Prototyping
### Instructions
@ -219,7 +219,7 @@ Don't forget to:
- "To prototype your solution, you’ll need a temporary change of philosophy: from perfect to just enough, from long-term quality to temporary simulation."
- "The prototype is meant to answer questions, so keep it focused. You just need a real-looking facade to which customers can react."
## Animation
### Animation
### Instructions
@ -244,7 +244,7 @@ This exercise is to be made individually.
- [Efficiently Manage Your Designs - How Mockplus State Page Helps To View & Manage Multiple State](https://help.mockplus.com/p/372)
@ -4,7 +4,7 @@ Creation of an ecommerce platform for second-hand clothes on desktop and mobile
Bruno loves fashion. He loves buying clothes! As he is cautious with the environment, he prefers buying second-hand clothing. However, the existing platforms are not very user-friendly.
"I need to know more about the environmental impact of the pieces I buy. I’d like to know how old they are, where they are coming from, to measure their life and the amount of kilometres they have traveled. When a new pair of jeans arrive at the ready-to-wear store in Europe, they consumed approximately 11,000 litres of water and travelled 65,000 kilometres. I’d like to monitor that kind of data.’
"I need to know more about the environmental impact of the pieces I buy. I’d like to know how old they are, where they are coming from, to measure their life and the amount of kilometers they have traveled. When a new pair of jeans arrive at the ready-to-wear store in Europe, they consumed approximately 11,000 litres of water and traveled 65,000 kilometers. I’d like to monitor that kind of data.’
You gather a team of 3 to think and design a website and an app for second-hand products.
Oumaima Fisaoui
1 month ago
committed by