Assignment 2: Distributions, Outliers and Transformations

Course: Exploratory Data Analysis · Weeks: 4–5

Dataset: Ames Housing (Dean De Cock, 2011, Journal of Statistics Education) — every residential property sold in Ames, Iowa, 2006–2010. 2 930 sales × 82 variables (numeric, ordinal and categorical). The same dataset is used for all assignments of the course.

Source: De Cock, D. (2011). Ames, Iowa: Alternative to the Boston Housing Data as an End of Semester Regression Project. Journal of Statistics Education 19(3), http://jse.amstat.org/v19n3/decock.pdf. Also on Kaggle: https://www.kaggle.com/datasets/prevek18/ames-housing-dataset (file AmesHousing.csv).

Recap of Assignment 1

Each row is one house sale. The target of interest is SalePrice (USD); the other columns describe size, lot, quality, age, location and sale conditions.

Exploratory questions formulated in A1:

1. Q1. What does a typical house in Ames cost, and how unequal are prices?
2. Q2. How strongly does living area (Gr Liv Area) drive the sale price?
3. Q3. Is lot size (Lot Area) related to price, or does it mostly reflect location (rural vs. city)?
4. Q4. How do quality, neighbourhood and age of the house change the price?

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
from scipy import stats

# Figures are sized to stay readable on a phone screen
sns.set_theme(style="whitegrid", context="notebook")
plt.rcParams.update({"figure.figsize": (7, 4), "figure.dpi": 110})
pd.set_option("display.float_format", lambda x: f"{x:,.2f}")

df = pd.read_csv("AmesHousing.csv", index_col=0)
print(df.shape)
df[["PID", "SalePrice", "Gr Liv Area", "Lot Area", "Overall Qual",
    "Neighborhood", "Year Built", "Sale Condition"]].head()

(2930, 82)

	PID	SalePrice	Gr Liv Area	Lot Area	Overall Qual	Neighborhood	Year Built	Sale Condition
0	526301100	215000	1656	31770	6	NAmes	1960	Normal
1	526350040	105000	896	11622	5	NAmes	1961	Normal
2	526351010	172000	1329	14267	6	NAmes	1958	Normal
3	526353030	244000	2110	11160	7	NAmes	1968	Normal
4	527105010	189900	1629	13830	5	Gilbert	1997	Normal

Part 1. Exploring Distributions

Three numerical variables, chosen because they answer Q1–Q3:

Variable	Meaning	Related question
SalePrice	sale price, USD	Q1, Q2, Q3
Gr Liv Area	above-ground living area, sq ft	Q2
Lot Area	lot size, sq ft	Q3

VARS = ["SalePrice", "Gr Liv Area", "Lot Area"]

def describe(s: pd.Series) -> pd.Series:
    q1, q3 = s.quantile([0.25, 0.75])
    return pd.Series({
        "mean": s.mean(), "median": s.median(),
        "min": s.min(), "max": s.max(),
        "Q1": q1, "Q3": q3,
        "std": s.std(), "IQR": q3 - q1,
        "skewness": s.skew(),
    })

desc = df[VARS].apply(describe).T
print("Missing values:", df[VARS].isna().sum().to_dict())
desc

Missing values: {'SalePrice': 0, 'Gr Liv Area': 0, 'Lot Area': 0}

	mean	median	min	max	Q1	Q3	std	IQR	skewness
SalePrice	180,796.06	160,000.00	12,789.00	755,000.00	129,500.00	213,500.00	79,886.69	84,000.00	1.74
Gr Liv Area	1,499.69	1,442.00	334.00	5,642.00	1,126.00	1,742.75	505.51	616.75	1.27
Lot Area	10,147.92	9,436.50	1,300.00	215,245.00	7,440.25	11,555.25	7,880.02	4,115.00	12.82

1.1 Histograms (3 histograms)

units = {"SalePrice": "USD", "Gr Liv Area": "sq ft", "Lot Area": "sq ft"}
for v in VARS:
    s = df[v]
    fig, ax = plt.subplots()
    sns.histplot(s, bins=50, ax=ax, color="steelblue")
    ax.axvline(s.mean(), color="red", ls="--", label=f"mean = {s.mean():,.0f}")
    ax.axvline(s.median(), color="black", ls="-", label=f"median = {s.median():,.0f}")
    ax.set(title=f"Histogram of {v}", xlabel=f"{v}, {units[v]}", ylabel="Number of houses")
    ax.legend()
    plt.show()

1.2 Boxplots (3 boxplots)

fig, axes = plt.subplots(3, 1, figsize=(7, 7))
for ax, v in zip(axes, VARS):
    sns.boxplot(x=df[v], ax=ax, color="lightsteelblue", flierprops={"marker": "o", "markersize": 3})
    ax.set(title=f"Boxplot of {v}", xlabel=f"{v}, {units[v]}")
plt.tight_layout()
plt.show()

1.3 Additional distribution plots: density (KDE) and ECDF

fig, ax = plt.subplots()
sns.kdeplot(df["SalePrice"], fill=True, ax=ax)
x = np.linspace(df.SalePrice.min(), df.SalePrice.max(), 400)
ax.plot(x, stats.norm.pdf(x, df.SalePrice.mean(), df.SalePrice.std()),
        "r--", label="normal curve with same mean/std")
ax.set(title="Density of SalePrice vs. normal distribution", xlabel="SalePrice, USD")
ax.legend()
plt.show()

fig, ax = plt.subplots()
sns.ecdfplot(df["Gr Liv Area"], ax=ax, label="Gr Liv Area")
for q in (0.25, 0.5, 0.75):
    ax.axhline(q, color="grey", lw=0.7, ls=":")
ax.set(title="ECDF of Gr Liv Area", xlabel="Gr Liv Area, sq ft", ylabel="Share of houses ≤ x")
plt.show()

fig, ax = plt.subplots()
sns.ecdfplot(df["Lot Area"], ax=ax)
ax.set(title="ECDF of Lot Area (log x-axis)", xlabel="Lot Area, sq ft (log scale)",
       ylabel="Share of houses ≤ x", xscale="log")
plt.show()

# Numbers quoted in the interpretation below
for v in VARS:
    s = df[v]
    print(f"{v:12s}  P5={s.quantile(.05):>10,.0f}  P95={s.quantile(.95):>10,.0f}  "
          f"P99={s.quantile(.99):>10,.0f}  max/median={s.max()/s.median():.1f}x")

SalePrice     P5=    87,500  P95=   335,000  P99=   456,666  max/median=4.7x
Gr Liv Area   P5=       861  P95=     2,463  P99=     2,931  max/median=3.9x
Lot Area      P5=     3,188  P95=    17,131  P99=    32,989  max/median=22.8x

1.4 Interpretation

SalePrice

• Typical value: median ≈ $160 000; the middle 50 % of houses sold between ≈ $129 500 (Q1) and ≈ $213 500 (Q3).
• Spread: IQR ≈ $84 000, std ≈ $80 000. 90 % of prices lie between ≈ $88 000 (P5) and ≈ $335 000 (P95).
• Shape: clearly right-skewed (skewness ≈ 1.74). The density has one peak and a long right tail; the normal curve with the same mean/std fits poorly.
• Unusual values: a tail of expensive houses up to $755 000 and a few extremely cheap sales (≈ $13 000).
• Representative measure: the median — the mean ($180 800) is pulled ≈ $21 000 up by the expensive tail.

Gr Liv Area

• Typical value: median ≈ 1 442 sq ft; half of the houses have 1 126–1 743 sq ft.
• Spread: IQR ≈ 617 sq ft, std ≈ 506 sq ft.
• Shape: moderately right-skewed (skewness ≈ 1.27). The ECDF rises steeply between ≈ 900 and 2 200 sq ft, then flattens: few very large houses.
• Unusual values: five houses above 4 000 sq ft; the smallest is 334 sq ft.
• Representative measure: mean (1 500) and median (1 442) are close, so both are acceptable; the median is safer.

Lot Area

• Typical value: median ≈ 9 437 sq ft (≈ 0.2 acre), IQR 7 440–11 555 sq ft.
• Spread: IQR ≈ 4 115 sq ft, but std ≈ 7 880 sq ft — almost twice the IQR, a sign that a few extreme lots inflate the std.
• Shape: extremely right-skewed (skewness ≈ 12.8). Most lots are standard city lots; a handful are huge. On the log-scale ECDF the distribution looks much more regular.
• Unusual values: max lot 215 245 sq ft (≈ 5 acres), about 23× the median.
• Representative measure: definitely the median.

Part 2. Comparing Mean and Median

interp = {
    "SalePrice": "Mean > median: long right tail of expensive houses",
    "Gr Liv Area": "Small gap: mild right skew, a few very large houses",
    "Lot Area": "Mean > median: a few huge rural lots, extreme right skew",
}
mm = pd.DataFrame({
    "Mean": desc["mean"], "Median": desc["median"],
    "Difference": desc["mean"] - desc["median"],
    "Difference, % of median": (desc["mean"] - desc["median"]) / desc["median"] * 100,
    "Skewness": desc["skewness"],
    "Interpretation": pd.Series(interp),
})
mm

	Mean	Median	Difference	Difference, % of median	Skewness	Interpretation
SalePrice	180,796.06	160,000.00	20,796.06	13.00	1.74	Mean > median: long right tail of expensive ho...
Gr Liv Area	1,499.69	1,442.00	57.69	4.00	1.27	Small gap: mild right skew, a few very large h...
Lot Area	10,147.92	9,436.50	711.42	7.54	12.82	Mean > median: a few huge rural lots, extreme ...

1. Largest difference. In absolute units SalePrice has the largest gap (≈ $20 800). Units differ between variables, so the gap relative to the median is fairer: SalePrice +13 %, Lot Area +7.5 %, Gr Liv Area +4 %.

2. Explanation. The mean uses every value, so a small number of very large values pulls it upward. For SalePrice these are luxury houses in NoRidge / NridgHt / StoneBr (up to $755 000). For Lot Area these are rural lots in Timber and Clear Creek (up to 215 245 sq ft). Prices cannot fall below zero but can rise without a hard limit, so the tail grows only to the right.

3. Skewness. Yes: in all three variables mean > median, which indicates positive (right) skew, and the skewness coefficients confirm it (1.74, 1.27, 12.8). For Lot Area the relative gap is small although skewness is huge: the extreme lots are very few, so they affect the std (3rd-order moment) much more than the mean.

4. Choice of central tendency.

• SalePrice — median (the typical buyer pays ≈ $160 000, not $180 800).
• Gr Liv Area — mean or median; I report the median for consistency.
• Lot Area — median only; the mean and std describe the few big lots, not the typical one.

Part 3. Outlier Investigation (1.5 × IQR rule)

Four variables: the three above plus Total Bsmt SF (basement area, sq ft).

$IQR = Q_3 - Q_1$, lower fence $= Q_1 - 1.5\,IQR$, upper fence $= Q_3 + 1.5\,IQR$.

OUT_VARS = VARS + ["Total Bsmt SF"]

def iqr_fences(s):
    q1, q3 = s.quantile([0.25, 0.75])
    iqr = q3 - q1
    return q1 - 1.5 * iqr, q3 + 1.5 * iqr

rows = []
for v in OUT_VARS:
    s = df[v].dropna()
    lo, hi = iqr_fences(s)
    rows.append({"variable": v, "lower fence": lo, "upper fence": hi,
                 "n below": (s < lo).sum(), "n above": (s > hi).sum(),
                 "% outliers": ((s < lo) | (s > hi)).mean() * 100})
fences = pd.DataFrame(rows).set_index("variable")
fences

	lower fence	upper fence	n below	n above	% outliers
variable
SalePrice	3,500.00	339,500.00	0	137	4.68
Gr Liv Area	200.88	2,667.88	0	75	2.56
Lot Area	1,267.75	17,727.75	0	127	4.33
Total Bsmt SF	29.50	2,065.50	79	44	4.20

Lower fences for SalePrice and Gr Liv Area are close to zero or negative, so almost all flagged points are on the upper side. For every variable 2.5–5 % of houses are flagged. That is too many to be all errors: it is the normal consequence of right skew. The rule marks candidates for investigation, not errors.

3.1 Outlier visualizations

# Outlier plot 1: boxplot + all points, flagged ones in red
fig, axes = plt.subplots(len(OUT_VARS), 1, figsize=(7, 9))
for ax, v in zip(axes, OUT_VARS):
    s = df[v].dropna()
    lo, hi = fences.loc[v, ["lower fence", "upper fence"]]
    flag = (s < lo) | (s > hi)
    sns.boxplot(x=s, ax=ax, color="white", showfliers=False, width=0.5)
    jitter = np.random.default_rng(0).uniform(-0.2, 0.2, len(s))
    ax.scatter(s[~flag], jitter[~flag], s=3, alpha=0.25, color="grey")
    ax.scatter(s[flag], jitter[flag], s=8, color="red", label=f"outliers: {flag.sum()}")
    ax.axvline(hi, color="red", ls="--", lw=0.8)
    ax.set(title=v, xlabel="", yticks=[])
    ax.legend(loc="lower right", fontsize=8)
plt.tight_layout()
plt.show()

# Outlier plot 2: SalePrice vs Gr Liv Area, coloured by sale condition
cases = {
    908154235: "A", 528351010: "B", 916176125: "C", 902207130: "D",
}
fig, ax = plt.subplots(figsize=(7, 5))
sns.scatterplot(data=df, x="Gr Liv Area", y="SalePrice", hue="Sale Condition",
                s=12, alpha=0.6, ax=ax)
for pid, label in cases.items():
    r = df.loc[df.PID == pid].iloc[0]
    ax.annotate(label, (r["Gr Liv Area"], r["SalePrice"]), fontsize=12, fontweight="bold",
                xytext=(6, 6), textcoords="offset points", color="black")
ax.axvline(fences.loc["Gr Liv Area", "upper fence"], color="red", ls="--", lw=0.8)
ax.axhline(fences.loc["SalePrice", "upper fence"], color="red", ls="--", lw=0.8)
ax.set(title="SalePrice vs living area (dashed = IQR upper fences)")
ax.legend(fontsize=7, title="Sale Condition", title_fontsize=8)
plt.show()

# Outlier plot 3: Lot Area vs SalePrice, log axes, by zoning
fig, ax = plt.subplots(figsize=(7, 5))
sns.scatterplot(data=df, x="Lot Area", y="SalePrice", hue="MS Zoning", s=12, alpha=0.6, ax=ax)
ax.set(xscale="log", yscale="log", title="Lot Area vs SalePrice (log–log)")
ax.axvline(fences.loc["Lot Area", "upper fence"], color="red", ls="--", lw=0.8)
r = df.loc[df.PID == 916176125].iloc[0]
ax.annotate("C", (r["Lot Area"], r["SalePrice"]), fontsize=12, fontweight="bold",
            xytext=(-14, 6), textcoords="offset points")
ax.legend(fontsize=7, title="Zoning", title_fontsize=8)
plt.show()

3.2 Investigating selected outliers in the original data

cols = ["PID", "SalePrice", "Gr Liv Area", "Total Bsmt SF", "Lot Area", "Overall Qual",
        "Overall Cond", "Year Built", "Neighborhood", "MS Zoning", "Sale Type",
        "Sale Condition", "Yr Sold"]
sel = df.loc[df.PID.isin(cases), cols].copy()
sel.insert(0, "case", sel.PID.map(cases))
sel.sort_values("case").set_index("case").T

case	A	B	C	D
PID	908154235	528351010	916176125	902207130
SalePrice	160000	755000	375000	12789
Gr Liv Area	5642	4316	2036	832
Total Bsmt SF	6,110.00	2,444.00	2,136.00	678.00
Lot Area	63887	21535	215245	9656
Overall Qual	10	10	7	2
Overall Cond	5	6	5	2
Year Built	2008	1994	1965	1923
Neighborhood	Edwards	NoRidge	Timber	OldTown
MS Zoning	RL	RL	RL	RM
Sale Type	New	WD	WD	WD
Sale Condition	Partial	Normal	Normal	Abnorml
Yr Sold	2008	2007	2009	2010

# Context for case A: the other huge houses in Edwards and their sale conditions
df.loc[df["Gr Liv Area"] > 4000, ["PID", "Gr Liv Area", "SalePrice", "Neighborhood",
                                  "Overall Qual", "Sale Condition", "Yr Sold"]]

	PID	Gr Liv Area	SalePrice	Neighborhood	Overall Qual	Sale Condition	Yr Sold
1498	908154235	5642	160000	Edwards	10	Partial	2008
1760	528320050	4476	745000	NoRidge	10	Abnorml	2007
1767	528351010	4316	755000	NoRidge	10	Normal	2007
2180	908154195	5095	183850	Edwards	10	Partial	2007
2181	908154205	4676	184750	Edwards	10	Partial	2007

# Price per sq ft: case A vs. the rest of the market
ppsf = df.SalePrice / df["Gr Liv Area"]
print(f"median price per sq ft (all):        {ppsf.median():6.1f} USD")
print(f"median price per sq ft (Qual = 10):   {ppsf[df['Overall Qual'] == 10].median():6.1f} USD")
for pid, lab in cases.items():
    print(f"case {lab} (PID {pid}):                 {ppsf[df.PID == pid].iloc[0]:6.1f} USD")

median price per sq ft (all):         120.2 USD
median price per sq ft (Qual = 10):    182.7 USD
case A (PID 908154235):                   28.4 USD
case B (PID 528351010):                  174.9 USD
case C (PID 916176125):                  184.2 USD
case D (PID 902207130):                   15.4 USD

Case	Observation	Possible?	Data-entry error?	Explained by another variable?	Decision
A	PID 908154235: 5 642 sq ft, quality 10, built 2008, but sold for only $160 000 (≈ $28/sq ft vs. ≈ $120 median)	The area is possible; the price does not match the quality	Probably not a typo: Sale Condition = Partial — the house was sold unfinished. The dataset author recommends removing houses > 4 000 sq ft for this reason	Yes: Sale Condition (Partial) and Yr Sold	Investigate further; remove from price models (the price does not reflect the house)
B	PID 528351010: most expensive house, $755 000, 4 316 sq ft, quality 10, NoRidge	Yes — a luxury house in the richest neighbourhood	No	Yes: quality 10, big area, NoRidge; ≈ $175/sq ft, same level as other quality-10 houses	Keep — real part of the market
C	PID 916176125: largest lot, 215 245 sq ft (≈ 5 acres)	Yes	No	Yes: Neighborhood = Timber, Land Contour = Low — rural area on the town edge. The price ($375 000) is high but not extreme	Keep; handle with a log transform
D	PID 902207130: cheapest sale, $12 789 for 832 sq ft	Possible, but very unusual	Maybe — the price is ≈ $15/sq ft	Partly: Overall Cond = 2 (poor), OldTown, Sale Condition = Abnorml (abnormal sale: foreclosure, short sale, trade)	Investigate further; possibly exclude as a non-market transaction

The outliers are not removed automatically. Only case A is a candidate for exclusion, and only in models of market price.

Part 4. Robust Statistics

We compare classical statistics (mean, std) with robust statistics (median, IQR) on the full data and after removing the IQR-flagged observations. This removal is only a demonstration; the data set itself stays unchanged.

rows = []
for v in OUT_VARS:
    s = df[v].dropna()
    lo, hi = iqr_fences(s)
    t = s[(s >= lo) & (s <= hi)]
    rows.append({
        "variable": v,
        "mean (all)": s.mean(), "mean (no outl.)": t.mean(),
        "Δ mean, %": (s.mean() / t.mean() - 1) * 100,
        "median (all)": s.median(), "median (no outl.)": t.median(),
        "Δ median, %": (s.median() / t.median() - 1) * 100,
        "std (all)": s.std(), "std (no outl.)": t.std(),
        "Δ std, %": (s.std() / t.std() - 1) * 100,
        "IQR (all)": s.quantile(.75) - s.quantile(.25),
        "IQR (no outl.)": t.quantile(.75) - t.quantile(.25),
    })
rob = pd.DataFrame(rows).set_index("variable")
rob.T

variable	SalePrice	Gr Liv Area	Lot Area	Total Bsmt SF
mean (all)	180,796.06	1,499.69	10,147.92	1,051.61
mean (no outl.)	169,115.50	1,458.49	9,189.29	1,057.90
Δ mean, %	6.91	2.82	10.43	-0.59
median (all)	160,000.00	1,442.00	9,436.50	990.00
median (no outl.)	157,500.00	1,430.00	9,245.00	994.00
Δ median, %	1.59	0.84	2.07	-0.40
std (all)	79,886.69	505.51	7,880.02	440.62
std (no outl.)	58,989.05	433.17	3,247.45	357.96
Δ std, %	35.43	16.70	142.65	23.09
IQR (all)	84,000.00	616.75	4,115.00	509.00
IQR (no outl.)	74,900.00	601.00	3,863.50	485.50

# Sensitivity demo: add ONE fake mansion to SalePrice and watch the statistics move
s = df.SalePrice
fake = pd.concat([s, pd.Series([10_000_000])], ignore_index=True)
iqr = lambda x: x.quantile(.75) - x.quantile(.25)
pd.DataFrame({
    "original": [s.mean(), s.std(), s.median(), iqr(s)],
    "+ one $10M house": [fake.mean(), fake.std(), fake.median(), iqr(fake)],
}, index=["mean", "std", "median", "IQR"]).assign(**{"change, %": lambda d: (d.iloc[:, 1] / d.iloc[:, 0] - 1) * 100})

	original	+ one $10M house	change, %
mean	180,796.06	184,146.18	1.85
std	79,886.69	198,179.78	148.08
median	160,000.00	160,000.00	0.00
IQR	84,000.00	84,000.00	0.00

# Visual: classical vs robust "centre ± spread" for Lot Area
s = df["Lot Area"]
fig, ax = plt.subplots()
sns.histplot(s[s < 40000], bins=60, color="lightgrey", ax=ax)
ax.axvline(s.mean(), color="red", label="mean")
ax.axvspan(s.mean() - s.std(), s.mean() + s.std(), color="red", alpha=0.12, label="mean ± std")
ax.axvline(s.median(), color="blue", label="median")
ax.axvspan(s.quantile(.25), s.quantile(.75), color="blue", alpha=0.15, label="Q1–Q3 (IQR)")
ax.set(title="Lot Area: classical vs robust summary (lots < 40 000 sq ft shown)",
       xlabel="Lot Area, sq ft")
ax.legend(fontsize=8)
plt.show()

How outliers affect the mean and std. Without outliers the mean of Lot Area falls by ≈ 9 % (10 148 → 9 189), but its std falls by ≈ 59 % (7 880 → 3 247). The std squares each deviation, so a lot 20× the median adds a very large value to the sum. For SalePrice the std falls by ≈ 26 %, the median only by 1.6 %. One fake $10 M house raises the mean by ≈ 1.9 % and the std by ≈ 148 %. The median and the IQR do not change at all.

Why median and IQR are more useful here. They depend only on the order of the middle observations, not on the size of extreme values (breakdown point 50 % / 25 % vs. 0 % for mean/std). In Ames the typical house is in the middle of the distribution. The mean ± std band for Lot Area spans from ≈ 2 300 to ≈ 18 000 sq ft, so it covers many lots that do not exist in the "typical" range. The Q1–Q3 band marks the real bulk of the lots. For skewed variables with extreme values we therefore report median and IQR.

Part 5. Transformation

Variable: SalePrice (skewness ≈ 1.74), the key variable of the course.

Why transform?

• Prices behave multiplicatively: a better house costs "20 % more", not "$20 000 more". A log turns ratios into differences.
• The right tail compresses, so expensive houses no longer dominate the mean, the std and future regression fits.
• Many later methods (correlation, linear regression, t-tests) work better for approximately symmetric data with constant variance.

Transformation: natural logarithm $\ln(\text{SalePrice})$. All prices are > 0, so no shift is needed. For comparison we also show the square root.

df["log_SalePrice"] = np.log(df["SalePrice"])
sqrt_price = np.sqrt(df["SalePrice"])
print(f"skewness: original = {df.SalePrice.skew():.2f}, "
      f"sqrt = {sqrt_price.skew():.2f}, log = {df.log_SalePrice.skew():.2f}")

skewness: original = 1.74, sqrt = 0.88, log = -0.01

# Before/after 1: histograms
fig, axes = plt.subplots(2, 1, figsize=(7, 7))
sns.histplot(df.SalePrice, bins=50, kde=True, ax=axes[0], color="steelblue")
axes[0].set(title=f"Before: SalePrice (skew = {df.SalePrice.skew():.2f})", xlabel="USD")
sns.histplot(df.log_SalePrice, bins=50, kde=True, ax=axes[1], color="seagreen")
axes[1].set(title=f"After: ln(SalePrice) (skew = {df.log_SalePrice.skew():.2f})", xlabel="ln(USD)")
plt.tight_layout()
plt.show()

# Before/after 2: boxplots
fig, axes = plt.subplots(2, 1, figsize=(7, 4.5))
sns.boxplot(x=df.SalePrice, ax=axes[0], color="lightsteelblue", flierprops={"markersize": 3})
axes[0].set(title="Before: SalePrice", xlabel="USD")
sns.boxplot(x=df.log_SalePrice, ax=axes[1], color="palegreen", flierprops={"markersize": 3})
axes[1].set(title="After: ln(SalePrice)", xlabel="ln(USD)")
plt.tight_layout()
plt.show()

# Before/after 3: Q-Q plots against the normal distribution
fig, axes = plt.subplots(1, 2, figsize=(7, 3.6))
stats.probplot(df.SalePrice, dist="norm", plot=axes[0])
axes[0].set_title("Q-Q: SalePrice")
stats.probplot(df.log_SalePrice, dist="norm", plot=axes[1])
axes[1].set_title("Q-Q: ln(SalePrice)")
for ax in axes:
    ax.get_lines()[0].set_markersize(2)
plt.tight_layout()
plt.show()

def summary(s):
    lo, hi = iqr_fences(s)
    return pd.Series({
        "mean": s.mean(), "median": s.median(),
        "(mean-median)/IQR": (s.mean() - s.median()) / (s.quantile(.75) - s.quantile(.25)),
        "std": s.std(), "IQR": s.quantile(.75) - s.quantile(.25),
        "skewness": s.skew(), "kurtosis": s.kurt(),
        "outliers low": (s < lo).sum(), "outliers high": (s > hi).sum(),
    })

cmp = pd.DataFrame({"SalePrice": summary(df.SalePrice), "ln(SalePrice)": summary(df.log_SalePrice)})
cmp["ln → back to USD"] = [np.exp(df.log_SalePrice.mean()), np.exp(df.log_SalePrice.median())] + [np.nan] * 7
cmp

	SalePrice	ln(SalePrice)	ln → back to USD
mean	180,796.06	12.02	166,203.58
median	160,000.00	11.98	160,000.00
(mean-median)/IQR	0.25	0.08	NaN
std	79,886.69	0.41	NaN
IQR	84,000.00	0.50	NaN
skewness	1.74	-0.01	NaN
kurtosis	5.12	1.51	NaN
outliers low	0.00	28.00	NaN
outliers high	137.00	32.00	NaN

Before vs after

• Mean vs median: before, the mean was $20 800 above the median (≈ 0.25 IQR). After the log, mean (12.02) and median (11.98) are close: the gap is only 0.08 IQR. Back in dollars, $e^{\text{mean of log}}$ ≈ $166 000 — the geometric mean, much closer to the median ($160 000) than the arithmetic mean ($180 800).
• Spread: the std (≈ 0.41) is now in log units. It means prices typically differ from the centre by a factor of ≈ $e^{0.41}$ ≈ 1.5, not by ± $80 000.
• Extreme observations: before, all 137 outliers were on the high side. After, only 60 points are flagged: 32 high and 28 low. The very cheap sales (case D, ≈ $13 000) now stand out on the low side, where they really are unusual.
• Shape: skewness 1.74 → ≈ 0. The histogram is bell-shaped, and the Q-Q plot is almost a straight line, except for a slightly heavy left tail (the abnormal cheap sales).

# z-scores of the selected cases before and after the log transform
z = lambda x: (x - x.mean()) / x.std()
zz = pd.DataFrame({"case": df.PID.map(cases), "SalePrice": df.SalePrice,
                   "z (SalePrice)": z(df.SalePrice), "z (ln SalePrice)": z(df.log_SalePrice)})
zz.dropna(subset=["case"]).set_index("case").sort_index()

	SalePrice	z (SalePrice)	z (ln SalePrice)
case
A	160000	-0.26	-0.09
B	755000	7.19	3.71
C	375000	2.43	2.00
D	12789	-2.10	-6.29

Part 6. Interpretation of Transformation

1. Did the transformation reduce skewness? Yes — from 1.74 to ≈ −0.01. The square root reduces it only to ≈ 0.9, so the log is the right choice.
2. Did the distribution become more symmetric? Yes. Mean ≈ median, the box is centred between the whiskers, and the Q-Q plot follows the normal line.
3. Did the influence of extreme observations change? Yes. The $755 000 house was ≈ 7.2 std above the mean on the original scale; on the log scale it is ≈ 3.7 std (see the z-scores above). The expensive tail no longer dominates the mean and std. At the same time the log makes the cheap anomalies more visible, which helps to find the non-market sales (case D).
4. More useful representation? For price analysis — yes. Differences in log price are relative differences (Δ 0.1 ≈ 10 % more expensive). This is how buyers and appraisers think about prices, and it treats a $100 k and a $500 k house on the same relative scale.
5. Use it further? Yes — log_SalePrice will be the target for correlation and regression in the next assignments. Results will be converted back to dollars (via exp) for reporting. For describing typical prices to a reader I still report the median in USD.

Part 7. What Have You Learned?

A1 question revisited — Q1: "What does a typical house cost, and how unequal are prices?" A1 gave a first impression from the mean (≈ $181 000). A2 shows that the mean is not a good answer. Prices are right-skewed, so the typical house costs ≈ $160 000 (median), and the middle half of the market is $129 500–$213 500. Inequality is best expressed on the log scale: prices typically differ by a factor of ≈ 1.5 around the centre. The outlier analysis also shows that the most extreme "prices" are not always market prices: some are partial or abnormal sales. The answer to Q1 therefore depends on filtering by Sale Condition.

New questions from A2

1. How much do Partial and Abnormal sales differ from Normal sales? Should the analysis use only normal sales?
2. Is the relationship between ln(SalePrice) and ln(Gr Liv Area) approximately linear, and what is the price elasticity of area?
3. Does Lot Area still matter for price once we control for neighbourhood (rural Timber / Clear Creek vs. city)?
4. Is the spread of prices (IQR of log price) different between neighbourhoods — are some markets more homogeneous?
5. Did the 2008 financial crisis change the price distribution (compare Yr Sold 2006–2007 vs. 2009–2010)?

Conclusion

• Main distributional findings. All three key size/price variables are right-skewed, with a compact bulk and a long upper tail. None is close to normal on the original scale.
• Most skewed variable. Lot Area (skewness ≈ 12.8) — a few rural lots are 10–23× the typical city lot. SalePrice is the most skewed of the variables that matter for pricing (1.74).
• Important potential outliers. (A) the 5 642 sq ft house sold for $160 000 — a partial sale; (B) the $755 000 luxury house — genuine; (C) the 5-acre lot in Timber — genuine; (D) the $12 789 abnormal sale in poor condition — suspicious.
• Retained or removed? All outliers are retained in the dataset. The IQR rule flags 3–5 % of each variable, and most of them are real, explainable houses. Only the partial sales > 4 000 sq ft (A and its two neighbours) are candidates for exclusion from price models, because their price does not describe the house. Case D will be checked together with the other abnormal sales.
• Robust statistics. Outliers move the mean a little and the std a lot (Lot Area std −59 % without outliers; one fake house +148 %). The median and IQR hardly move. For skewed variables median + IQR describe the typical house better.
• Transformation. The log of SalePrice was clearly useful: skewness → 0, mean ≈ median, a near-normal Q-Q plot, and an interpretation in percent. It will be used as the target in later assignments.
• How my understanding changed. In A1 the dataset looked like a table of houses with "some big values". Now I know that the "big values" are a structural property (multiplicative prices, rural lots), that some records are not ordinary market sales, and that the scale of analysis (USD vs. log) changes what looks unusual.
• Next steps. Relationship between log price and log area (correlation, scatter plots by neighbourhood), effect of Overall Qual and Neighborhood, a comparison of Normal vs. other sale conditions, and a log transform of Lot Area before any modelling.