You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
 
 
 
 
 
 
davhojt 1c1ff5e557 docs(data-wrangling): rename path to match ref 2 years ago
..
README.md docs(data-wrangling): rename path to match ref 2 years ago

README.md

Exercise 0: Environment and libraries

The exercise is validated is all questions of the exercise are validated.
Activate the virtual environment. If you used conda run conda activate your_env.
Run python --version.
Does it print Python 3.x? x >= 8
Does import jupyter, import numpy and import pandas run without any error?


Exercise 1: Concatenate

This question is validated if the outputted DataFrame is:
|    | letter   |   number |
|---:|:---------|---------:|
|  0 | a        |        1 |
|  1 | b        |        2 |
|  2 | c        |        1 |
|  3 | d        |        2 |


Exercise 2: Merge

The exercise is validated is all questions of the exercise are validated.
The question 1 is validated if the output is:
|    |   id | Feature1_x   | Feature2_x   | Feature1_y   | Feature2_y   |
|---:|-----:|:-------------|:-------------|:-------------|:-------------|
|  0 |    1 | A            | B            | K            | L            |
|  1 |    2 | C            | D            | M            | N            |
The question 2 is validated if the output is:
|    |   id | Feature1_df1   | Feature2_df1   | Feature1_df2   | Feature2_df2   |
|---:|-----:|:---------------|:---------------|:---------------|:---------------|
|  0 |    1 | A              | B              | K              | L              |
|  1 |    2 | C              | D              | M              | N              |
|  2 |    3 | E              | F              | nan            | nan            |
|  3 |    4 | G              | H              | nan            | nan            |
|  4 |    5 | I              | J              | nan            | nan            |
|  5 |    6 | nan            | nan            | O              | P              |
|  6 |    7 | nan            | nan            | Q              | R              |
|  7 |    8 | nan            | nan            | S              | T              |

Note: Check that the suffixes are set using the suffix parameters rather than manually changing the columns' name.


Exercise 3: Merge MultiIndex

The exercice is validated is all questions of the exercice are validated.
The question 1 is validated if the outputted DataFrame's shape is (1305, 5) and if merged.head() returns a table as below. One of the answers that returns the correct DataFrame is market_data.merge(alternative_data, how='left', left_index=True, right_index=True)
Open Close Close_Adjusted Twitter Reddit
(Timestamp('2021-01-01 00:00:00', freq='B'), 'AAPL') 0.0991792 -0.31603 0.634787 -0.00159041 1.06053
(Timestamp('2021-01-01 00:00:00', freq='B'), 'FB') -0.123753 1.00269 0.713264 0.0142127 -0.487028
(Timestamp('2021-01-01 00:00:00', freq='B'), 'GE') -1.37775 -1.01504 1.2858 0.109835 0.04273
(Timestamp('2021-01-01 00:00:00', freq='B'), 'AMZN') 1.06324 0.841241 -0.799481 -0.805677 0.511769
(Timestamp('2021-01-01 00:00:00', freq='B'), 'DAI') -0.603453 -2.06141 -0.969064 1.49817 0.730055
The question 2 is validated if the numbers that are missing in the DataFrame are equal to 0 and if filled_df.sum().sum() == merged_df.sum().sum() gives: True


Exercise 4: Groupby Apply

The exercise is validated is all questions of the exercise are validated and if the for loop hasn't been used. The goal is to use groupby and apply.
The question 1 is validated if the output is:
        df = pd.DataFrame(range(1,11), columns=['sequence'])
        print(winsorize(df, [0.20, 0.80]).to_markdown())
|    |   sequence |
|---:|-----------:|
|  0 |        2.8 |
|  1 |        2.8 |
|  2 |        3   |
|  3 |        4   |
|  4 |        5   |
|  5 |        6   |
|  6 |        7   |
|  7 |        8   |
|  8 |        8.2 |
|  9 |        8.2 |
The question 2 is validated if the output is a Pandas Series or DataFrame with the first 11 rows equal to the output below. The code below give a solution.
|    |   sequence |
|---:|-----------:|
|  0 |       1.45 |
|  1 |       2    |
|  2 |       3    |
|  3 |       4    |
|  4 |       5    |
|  5 |       6    |
|  6 |       7    |
|  7 |       8    |
|  8 |       9    |
|  9 |       9.55 |
| 10 |      11.45 |
    def winsorize(df_series, quantiles):
    """
        df: pd.DataFrame or pd.Series
        quantiles: list [0.05, 0.95]

    """
    min_value = np.quantile(df_series, quantiles[0])
    max_value = np.quantile(df_series, quantiles[1])

    return df_series.clip(lower = min_value, upper = max_value)


    df.groupby("group")[['sequence']].apply(winsorize, [0.05,0.95])


Exercise 5: Groupby Agg

The question is validated if the output is as below. The columns don't have to be MultiIndex. A solution could be df.groupby('product').agg({'value':['min','max','mean']})
product ('value', 'min') ('value', 'max') ('value', 'mean')
chair 22.89 32.12 27.505
mobile phone 100 111.22 105.61
table 20.45 99.99 51.22


Exercise 6: Unstack

The question 1 is validated if the output is similar (as the values are generated randomly, it's obvious the audit doesn't require to match the values below) to what unstacked_df.head()returns:
| Date                |   ('Prediction', 'AAPL') |   ('Prediction', 'AMZN') |   ('Prediction', 'DAI') |   ('Prediction', 'FB') |   ('Prediction', 'GE') |
|:--------------------|-------------------------:|-------------------------:|------------------------:|-----------------------:|-----------------------:|
| 2021-01-01 00:00:00 |                 0.382312 |                -0.072392 |               -0.551167 |             -0.0585555 |                1.05955 |
| 2021-01-04 00:00:00 |                -0.560953 |                 0.503199 |               -0.79517  |             -3.23136   |                1.50271 |
| 2021-01-05 00:00:00 |                 0.211489 |                 1.84867  |                0.287906 |             -1.81119   |                1.20321 |
The question 2 is validated if the answer is: unstacked.plot(title = 'Stocks 2021'). The title can be anything else.