Contents

2. The Bootstrap

A data scientist is using the data in a random sample to estimate an unknown parameter. She uses the sample to calculate the value of a statistic that she will use as her estimate.

Once she has calculated the observed value of her statistic, she could just present it as her estimate and go on her merry way. But she’s a data scientist. She knows that her random sample is just one of numerous possible random samples, and thus her estimate is just one of numerous plausible estimates.

By how much could those estimates vary? To answer this, it appears as though she needs to draw another sample from the population, and compute a new estimate based on the new sample. But she doesn’t have the resources to go back to the population and draw another sample.

It looks as though the data scientist is stuck.

Fortunately, a brilliant idea called the bootstrap can help her out. Since it is not feasible to generate new samples from the population, the bootstrap generates new random samples by a method called resampling: the new samples are drawn at random from the original sample.

In this section, we will see how and why the bootstrap works. In the rest of the chapter, we will use the bootstrap for inference.

2.1. Employee Compensation in the City of San Francisco

SF OpenData is a website where the City and County of San Francisco make some of their data publicly available. One of the data sets contains compensation data for employees of the City. These include medical professionals at City-run hospitals, police officers, fire fighters, transportation workers, elected officials, and all other employees of the City.

Compensation data for the calendar year 2015 are in the table sf2015.

sf2015 = pd.read_csv(path_data + 'san_francisco_2015.csv')

sf2015.head(10)
Year Type Year Organization Group Code Organization Group Department Code Department Union Code Union Job Family Code Job Family ... Employee Identifier Salaries Overtime Other Salaries Total Salary Retirement Health/Dental Other Benefits Total Benefits Total Compensation
0 Calendar 2015 2 Public Works, Transportation & Commerce WTR PUC Water Department 21.0 Prof & Tech Engineers - Miscellaneous, Local 21 2400 Lab, Pharmacy & Med Techs ... 21538 82146.04 0.00 0.00 82146.04 16942.21 12340.88 6337.73 35620.82 117766.86
1 Calendar 2015 2 Public Works, Transportation & Commerce DPW General Services Agency - Public Works 12.0 Carpet, Linoleum and Soft Tile Workers, Local 12 7300 Journeyman Trade ... 5459 32165.75 973.19 848.96 33987.90 0.00 4587.51 2634.42 7221.93 41209.83
2 Calendar 2015 4 Community Health DPH Public Health 790.0 SEIU - Miscellaneous, Local 1021 1600 Payroll, Billing & Accounting ... 41541 71311.00 5757.98 0.00 77068.98 14697.59 12424.50 6370.06 33492.15 110561.13
3 Calendar 2015 4 Community Health DPH Public Health 351.0 Municipal Executive Association - Miscellaneous 0900 Management ... 26718 28430.25 0.00 763.07 29193.32 0.00 4223.14 5208.51 9431.65 38624.97
4 Calendar 2015 2 Public Works, Transportation & Commerce MTA Municipal Transportation Agency 790.0 SEIU - Miscellaneous, Local 1021 8200 Protection & Apprehension ... 45810 7948.75 0.00 0.00 7948.75 0.00 2873.17 616.24 3489.41 11438.16
5 Calendar 2015 1 Public Protection POL Police 911.0 Police Officers' Association Q000 Police Services ... 32906 2235.00 0.00 0.00 2235.00 490.36 286.72 176.57 953.65 3188.65
6 Calendar 2015 4 Community Health DPH Public Health 791.0 SEIU - Staff and Per Diem Nurses, Local 1021 2300 Nursing ... 7506 187247.00 0.00 11704.06 198951.06 37683.66 12424.50 11221.73 61329.89 260280.95
7 Calendar 2015 2 Public Works, Transportation & Commerce MTA Municipal Transportation Agency 253.0 Transport Workers - Transit Operators, Local 2... 9100 Street Transit ... 36773 66988.54 3512.88 2770.39 73271.81 19127.22 13202.95 5455.10 37785.27 111057.08
8 Calendar 2015 6 General Administration & Finance CAT City Attorney 311.0 Municipal Attorneys' Association 8100 Legal & Court ... 12963 135189.60 0.00 1562.50 136752.10 27501.81 12424.50 10102.97 50029.28 186781.38
9 Calendar 2015 3 Human Welfare & Neighborhood Development DSS Human Services 535.0 SEIU - Human Services, Local 1021 9700 Community Development ... 35179 70474.77 147.28 1647.24 72269.29 14650.28 10696.90 5993.11 31340.29 103609.58

10 rows × 22 columns

There is one row for each of 42,979 employees. There are numerous columns containing information about City departmental affiliation and details of the different parts of the employee’s compensation package. Here is the row correspoding to the late Edward Lee, the Mayor at that time.

sf2015[sf2015['Job'] == 'Mayor']
Year Type Year Organization Group Code Organization Group Department Code Department Union Code Union Job Family Code Job Family ... Employee Identifier Salaries Overtime Other Salaries Total Salary Retirement Health/Dental Other Benefits Total Benefits Total Compensation
3335 Calendar 2015 6 General Administration & Finance MYR Mayor 556.0 Elected Officials 1100 Administrative & Mgmt (Unrep) ... 22433 288963.55 0.0 0.0 288963.55 58117.03 12424.5 20292.95 90834.48 379798.03

1 rows × 22 columns

We are going to study the final column, Total Compensation. That’s the employee’s salary plus the City’s contribution towards his/her retirement and benefit plans.

Financial packages in a calendar year can sometimes be hard to understand as they depend on the date of hire, whether the employee is changing jobs within the City, and so on. For example, the lowest values in the Total Compensation column look a little strange.

sf2015.sort_values(by=['Total Compensation'])
Year Type Year Organization Group Code Organization Group Department Code Department Union Code Union Job Family Code Job Family ... Employee Identifier Salaries Overtime Other Salaries Total Salary Retirement Health/Dental Other Benefits Total Benefits Total Compensation
27308 Calendar 2015 1 Public Protection FIR Fire Department 798.0 Firefighters - Miscellaneous, Local 798 H000 Fire Services ... 43833 0.00 0.00 0.00 0.00 0.00 0.00 -423.76 -423.76 -423.76
15746 Calendar 2015 4 Community Health DPH Public Health 790.0 SEIU - Miscellaneous, Local 1021 9900 Public Service Aide ... 27871 -292.40 0.00 0.00 -292.40 0.00 -95.58 -22.63 -118.21 -410.61
24576 Calendar 2015 1 Public Protection JUV Juvenile Probation 790.0 SEIU - Miscellaneous, Local 1021 8300 Correction & Detention ... 10517 0.00 0.00 0.00 0.00 0.00 0.00 -159.12 -159.12 -159.12
42982 Calendar 2015 6 General Administration & Finance CPC City Planning 21.0 Prof & Tech Engineers - Miscellaneous, Local 21 1000 Information Systems ... 18961 0.00 0.00 0.00 0.00 0.00 0.00 -26.53 -26.53 -26.53
23310 Calendar 2015 6 General Administration & Finance CPC City Planning 21.0 Prof & Tech Engineers - Miscellaneous, Local 21 5200 Professional Engineering ... 19387 0.00 0.00 0.00 0.00 0.00 0.00 -9.51 -9.51 -9.51
... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ...
5171 Calendar 2015 4 Community Health DPH Public Health 351.0 Municipal Executive Association - Miscellaneous 0900 Management ... 1523 256098.01 0.00 82292.31 338390.32 51977.53 11468.77 20963.32 84409.62 422799.94
17805 Calendar 2015 2 Public Works, Transportation & Commerce AIR Airport Commission 351.0 Municipal Executive Association - Miscellaneous 0900 Management ... 17356 326764.01 0.00 0.00 326764.01 65806.33 12424.50 21691.23 99922.06 426686.07
499 Calendar 2015 6 General Administration & Finance ADM General Services Agency - City Admin 164.0 Physicians and Dentists - Miscellaneous 2500 Med Therapy & Auxiliary ... 13746 279311.03 9046.92 56742.56 345100.51 56211.61 12424.50 13482.66 82118.77 427219.28
13194 Calendar 2015 6 General Administration & Finance ADM General Services Agency - City Admin 164.0 Physicians and Dentists - Miscellaneous 2500 Med Therapy & Auxiliary ... 1016 279311.10 3829.36 114433.58 397574.04 56211.64 12424.50 14299.10 82935.24 480509.28
19177 Calendar 2015 6 General Administration & Finance RET Retirement System 351.0 Municipal Executive Association - Miscellaneous 1100 Administrative & Mgmt (Unrep) ... 46881 507831.60 0.00 0.00 507831.60 105052.98 12424.50 23566.16 141043.64 648875.24

42989 rows × 22 columns

For clarity of comparison, we will focus our attention on those who had at least the equivalent of a half-time job for the whole year. At a minimum wage of about $10 per hour, and 20 hours per week for 52 weeks, that’s a salary of about $10,000.

sf2015 = sf2015[sf2015['Salaries'] > 10000]

len(sf2015)

36569

2.1.1. Population and Parameter

Let this table of just over 36,500 rows be our population. Here is a histogram of the total compensations.

sf_bins = np.arange(0, 700000, 25000)

unit = ''

fig, ax = plt.subplots(figsize=(10,5))

ax.hist(sf2015['Total Compensation'], bins = sf_bins, density=True, color='blue', alpha=0.8, ec='white')

#ax.scatter(observed_distance, 0, color='red', s=40, zorder=10).set_clip_on(False)

y_vals = ax.get_yticks()

y_label = 'Percent per ' + (unit if unit else 'unit')

x_label = 'Total Compensation'

ax.set_yticklabels(['{:g}'.format(x * 100) for x in y_vals])

plt.ylabel(y_label)

plt.xlabel(x_label)

plt.xticks(rotation=90)

plt.title('');

plt.show()
../../_images/Bootstrap_14_0.png

While most of the values are below \\(300,000, a few are quite a bit higher. For example, the total compensation of the Chief Investment Officer was almost \\\)650,000. That is why the horizontal axis stretches to \$700,000.

sf2015.sort_values(by=['Total Compensation'], ascending=False).head(2)
Year Type Year Organization Group Code Organization Group Department Code Department Union Code Union Job Family Code Job Family ... Employee Identifier Salaries Overtime Other Salaries Total Salary Retirement Health/Dental Other Benefits Total Benefits Total Compensation
19177 Calendar 2015 6 General Administration & Finance RET Retirement System 351.0 Municipal Executive Association - Miscellaneous 1100 Administrative & Mgmt (Unrep) ... 46881 507831.6 0.00 0.00 507831.60 105052.98 12424.5 23566.16 141043.64 648875.24
13194 Calendar 2015 6 General Administration & Finance ADM General Services Agency - City Admin 164.0 Physicians and Dentists - Miscellaneous 2500 Med Therapy & Auxiliary ... 1016 279311.1 3829.36 114433.58 397574.04 56211.64 12424.5 14299.10 82935.24 480509.28

2 rows × 22 columns

Now let the parameter be the median of the total compensations.

Since we have the luxury of having all of the data from the population, we can simply calculate the parameter:

pop_median = np.percentile(sf2015['Total Compensation'], 50)
pop_median

110305.79

The median total compensation of all employees was just over $110,300.

From a practical perspective, there is no reason for us to draw a sample to estimate this parameter since we simply know its value. But in this section we are going to pretend we don’t know the value, and see how well we can estimate it based on a random sample.

In later sections, we will come down to earth and work in situations where the parameter is unknown. For now, we are all-knowing.

2.1.2. A Random Sample and an Estimate

Let us draw a sample of 500 employees at random without replacement, and let the median total compensation of the sampled employees serve as our estimate of the parameter.

our_sample = sf2015.sample(500, replace=False)

unit = ''

fig, ax = plt.subplots(figsize=(10,5))

ax.hist(our_sample['Total Compensation'], bins = sf_bins, density=True, color='blue', alpha=0.8, ec='white')

y_vals = ax.get_yticks()

y_label = 'Percent per ' + (unit if unit else 'unit')

x_label = 'Total Compensation'

ax.set_yticklabels(['{:g}'.format(x * 100) for x in y_vals])

plt.ylabel(y_label)

plt.xlabel(x_label)

plt.xticks(rotation=90)

plt.title('');

plt.show()
../../_images/Bootstrap_21_0.png 
est_median = np.percentile(our_sample['Total Compensation'], 50)
est_median

111827.425

The sample size is large. By the law of averages, the distribution of the sample resembles that of the population, and consequently the sample median is not very far from the population median (though of course it is not exactly the same).

So now we have one estimate of the parameter. But had the sample come out differently, the estimate would have had a different value. We would like to be able to quantify the amount by which the estimate could vary across samples. That measure of variability will help us measure how accurately we can estimate the parameter.

To see how different the estimate would be if the sample had come out differently, we could just draw another sample from the population, but that would be cheating. We are trying to mimic real life, in which we won’t have all the population data at hand.

Somehow, we have to get another random sample without sampling from the population.

2.1.3. The Bootstrap: Resampling from the Sample

What we do have is a large random sample from the population. As we know, a large random sample is likely to resemble the population from which it is drawn. This observation allows data scientists to lift themselves up by their own bootstraps: the sampling procedure can be replicated by sampling from the sample.

Here are the steps of the bootstrap method for generating another random sample that resembles the population:

  • Treat the original sample as if it were the population.

  • Draw from the sample, at random with replacement, the same number of times as the original sample size.

It is important to resample the same number of times as the original sample size. The reason is that the variability of an estimate depends on the size of the sample. Since our original sample consisted of 500 employees, our sample median was based on 500 values. To see how different the sample could have been, we have to compare it to the median of other samples of size 500.

If we drew 500 times at random without replacement from our sample of size 500, we would just get the same sample back. By drawing with replacement, we create the possibility for the new samples to be different from the original, because some employees might be drawn more than once and others not at all.

Why is this a good idea? By the law of averages, the distribution of the original sample is likely to resemble the population, and the distributions of all the “resamples” are likely to resemble the original sample. So the distributions of all the resamples are likely to resemble the population as well.

Bootstrap

2.1.4. A Resampled Median

Recall that when the sample method is used without specifying a sample size, by default the sample size equals the number of rows of the table from which the sample is drawn. That’s perfect for the bootstrap! Here is one new sample drawn from the original sample, and the corresponding sample median.

resample_1 = our_sample.sample(len(our_sample), replace=True)
resample_1
Year Type Year Organization Group Code Organization Group Department Code Department Union Code Union Job Family Code Job Family ... Employee Identifier Salaries Overtime Other Salaries Total Salary Retirement Health/Dental Other Benefits Total Benefits Total Compensation
31887 Calendar 2015 3 Human Welfare & Neighborhood Development DSS Human Services 535.0 SEIU - Human Services, Local 1021 2900 Human Services ... 12650 97934.04 0.00 96.00 98030.04 20197.44 12424.50 8087.09 40709.03 138739.07
15831 Calendar 2015 4 Community Health DPH Public Health 250.0 SEIU - Health Workers, Local 1021 1400 Clerical, Secretarial & Steno ... 39780 65592.00 4630.88 1716.50 71939.38 13528.23 12424.50 5901.69 31854.42 103793.80
14344 Calendar 2015 4 Community Health DPH Public Health 790.0 SEIU - Miscellaneous, Local 1021 9900 Public Service Aide ... 20240 13626.58 0.00 0.00 13626.58 0.00 3524.25 1118.49 4642.74 18269.32
35647 Calendar 2015 2 Public Works, Transportation & Commerce DPW General Services Agency - Public Works 790.0 SEIU - Miscellaneous, Local 1021 1800 Budget, Admn & Stats Analysis ... 25662 14267.35 0.00 0.00 14267.35 458.05 3013.54 1134.51 4606.10 18873.45
3750 Calendar 2015 6 General Administration & Finance MYR Mayor 2.0 Management Unrepresented Employees 0900 Management ... 32718 88831.00 0.00 0.00 88831.00 18080.75 12424.50 15196.77 45702.02 134533.02
... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ...
27355 Calendar 2015 4 Community Health DPH Public Health 791.0 SEIU - Staff and Per Diem Nurses, Local 1021 2300 Nursing ... 40639 11141.33 0.00 1084.47 12225.80 4283.32 812.37 501.33 5597.02 17822.82
23696 Calendar 2015 1 Public Protection CRT Superior Court 195.0 Court Unrepresented Professionals SCRT SF Superior Court ... 14273 115128.03 0.00 5838.00 120966.03 23309.69 12424.50 9414.65 45148.84 166114.87
34607 Calendar 2015 4 Community Health DPH Public Health 250.0 SEIU - Health Workers, Local 1021 2400 Lab, Pharmacy & Med Techs ... 39306 82717.07 2020.85 2127.50 86865.42 17048.48 12424.50 6528.82 36001.80 122867.22
27338 Calendar 2015 5 Culture & Recreation LIB Public Library 790.0 SEIU - Miscellaneous, Local 1021 3600 Library ... 48079 30464.77 0.00 437.19 30901.96 7414.33 7526.37 2538.13 17478.83 48380.79
36914 Calendar 2015 4 Community Health DPH Public Health 791.0 SEIU - Staff and Per Diem Nurses, Local 1021 2300 Nursing ... 11794 18543.12 0.00 3708.66 22251.78 0.00 0.00 3298.32 3298.32 25550.10

500 rows × 22 columns

unit = ''

fig, ax = plt.subplots(figsize=(10,5))

ax.hist(resample_1['Total Compensation'], bins = sf_bins, density=True, color='blue', alpha=0.8, ec='white')

y_vals = ax.get_yticks()

y_label = 'Percent per ' + (unit if unit else 'unit')

x_label = 'Total Compensation'

ax.set_yticklabels(['{:g}'.format(x * 100) for x in y_vals])

plt.ylabel(y_label)

plt.xlabel(x_label)

plt.xticks(rotation=90)

plt.title('');

plt.show()
../../_images/Bootstrap_28_0.png 
resampled_median_1 = np.percentile(resample_1['Total Compensation'], 50, interpolation='nearest')
resampled_median_1

108745.99

By resampling, we have another estimate of the population median. By resampling again and again, we will get many such estimates, and hence an empirical distribution of the estimates.

resample_2 = our_sample.sample(len(our_sample), replace=True)
resampled_median_2 = np.percentile(resample_2['Total Compensation'], 50, interpolation='nearest')
resampled_median_2

115449.12

2.1.5. Bootstrap Empirical Distribution of the Sample Median

Let us define a function bootstrap_median that takes our original sample, the label of the column containing the variable, and the number of bootstrap samples we want to take, and returns an array of the corresponding resampled medians.

Each time we resample and find the median, we replicate the bootstrap process. So the number of bootstrap samples will be called the number of replications.

def bootstrap_median(original_sample, label, replications):
    """Returns an array of bootstrapped sample medians:
    original_sample: table containing the original sample
    label: label of column containing the variable
    replications: number of bootstrap samples
    """
    just_one_column = original_sample[label]
    medians = np.array([])
    for i in np.arange(replications):
        bootstrap_sample = just_one_column.sample(len(just_one_column), replace=True)
        resampled_median = np.percentile(bootstrap_sample, 50)
        medians = np.append(medians, resampled_median)
        
    return medians

We now replicate the bootstrap process 5,000 times. The array bstrap_medians contains the medians of all 5,000 bootstrap samples. Notice that the code takes longer to run than our previous code. It has a lot of resampling to do!

bstrap_medians = bootstrap_median(our_sample, 'Total Compensation', 5000)

Here is the histogram of the 5000 medians. The red dot is the population parameter: it is the median of the entire population, which we happen to know but did not use in the bootstrap process.

resampled_medians = pd.DataFrame({'Bootstrap Sample Median':bstrap_medians})

unit = ''

fig, ax = plt.subplots(figsize=(10,5))

ax.hist(resampled_medians, density=True, color='blue', alpha=0.8, ec='white')

ax.scatter(pop_median, 0, color='red', s=40, zorder=10).set_clip_on(False)

y_vals = ax.get_yticks()

y_label = 'Percent per ' + (unit if unit else 'unit')

x_label = 'Bootstrap Sample Median'

ax.set_yticklabels(['{:g}'.format(x * 100) for x in y_vals])

plt.ylabel(y_label)

plt.xlabel(x_label)

plt.xticks(rotation=90)

plt.title('');

plt.show()
../../_images/Bootstrap_37_0.png

It is important to remember that the red dot is fixed: it is $110,305.79, the population median. The empirical histogram is the result of random draws, and will be situated randomly relative to the red dot.

Remember also that the point of all these computations is to estimate the population median, which is the red dot. Our estimates are all the randomly generated sampled medians whose histogram you see above. We want those estimates to contain the parameter – it they don’t, then they are off.

2.1.6. Do the Estimates Capture the Parameter?

How often does the empirical histogram of the resampled medians sit firmly over the red dot, and not just brush the dot with its tails? To answer this, we must define “sit firmly”. Let’s take that to mean “the middle 95% of the resampled medians contains the red dot”.

Here are the two ends of the “middle 95%” interval of resampled medians:

left = np.percentile(bstrap_medians, 2.5, interpolation='nearest')
left

105746.455

right = np.percentile(bstrap_medians, 97.5, interpolation='nearest')
right

116399.19

The population median of $110,305 is between these two numbers. The interval and the population median are shown on the histogram below.

resampled_medians = pd.DataFrame({'Bootstrap Sample Median':bstrap_medians})

unit = ''

fig, ax = plt.subplots(figsize=(10,5))

ax.hist(resampled_medians, density=True, color='blue', alpha=0.8, ec='white', zorder=5)

ax.plot(np.array([left, right]), np.array([0,0]), color='yellow', lw=8, zorder=10)

ax.scatter(pop_median, 0, color='red', s=40, zorder=15).set_clip_on(False)

y_vals = ax.get_yticks()

y_label = 'Percent per ' + (unit if unit else 'unit')

x_label = 'Bootstrap Sample Median'

ax.set_yticklabels(['{:g}'.format(x * 100) for x in y_vals])

plt.ylabel(y_label)

plt.xlabel(x_label)

plt.xticks(rotation=90)

plt.title('');

plt.show()
../../_images/Bootstrap_43_0.png

The “middle 95%” interval of estimates captured the parameter in our example. But was that a fluke?

To see how frequently the interval contains the parameter, we have to run the entire process over and over again. Specifically, we will repeat the following process 100 times:

  • Draw an original sample of size 500 from the population.

  • Carry out 5,000 replications of the bootstrap process and generate the “middle 95%” interval of resampled medians.

We will end up with 100 intervals, and count how many of them contain the population median.

Spoiler alert: The statistical theory of the bootstrap says that the number should be around 95. It may be in the low 90s or high 90s, but not much farther off 95 than that.

total_comps = sf2015[['Total Compensation']]
total_comps
Total Compensation
0 117766.86
1 41209.83
2 110561.13
3 38624.97
6 260280.95
... ...
42983 61349.71
42984 132788.81
42986 73295.59
42987 19973.37
42988 55812.90

36569 rows × 1 columns

# THE BIG SIMULATION: This one takes several minutes.

# Generate 100 intervals, in the table intervals

left_ends = np.array([])
right_ends = np.array([])

total_comps = sf2015[['Total Compensation']]

for i in np.arange(100):
    first_sample = total_comps.sample(500, replace=False)
    medians = bootstrap_median(first_sample, 'Total Compensation', 5000)
    left_ends = np.append(left_ends, np.percentile(medians, 2.5))
    right_ends = np.append(right_ends, np.percentile(medians, 97.5))

intervals = pd.DataFrame(
    {'Left':left_ends,
    'Right':right_ends}
)

For each of the 100 replications, we get one interval of estimates of the median.

intervals
Left Right
0 100639.945000 113009.80850
1 103596.780000 113549.15000
2 106650.085125 116872.21000
3 101855.280000 111880.11000
4 98633.755000 113396.61500
... ... ...
95 103042.430000 117401.49500
96 97683.995000 113182.87500
97 111704.995000 121917.01000
98 105761.655000 117376.24225
99 104795.530000 116920.50500

100 rows × 2 columns

The good intervals are those that contain the parameter we are trying to estimate. Typically the parameter is unknown, but in this section we happen to know what the parameter is.

pop_median

110305.79

How many of the 100 intervals contain the population median? That’s the number of intervals where the left end is below the population median and the right end is above.

len(
    intervals[
        (intervals['Left'] < pop_median) & 
              (intervals['Right'] > pop_median)
    ]
)

95

It takes a few minutes to construct all the intervals, but try it again if you have the patience. Most likely, about 95 of the 100 intervals will be good ones: they will contain the parameter.

It’s hard to show you all the intervals on the horizontal axis as they have large overlaps – after all, they are all trying to estimate the same parameter. The graphic below shows each interval on the same axes by stacking them vertically. The vertical axis is simply the number of the replication from which the interval was generated.

The red line is where the parameter is. Good intervals cover the parameter; there are about 95 of these, typically.

If an interval doesn’t cover the parameter, it’s a dud. The duds are the ones where you can see “daylight” around the red line. There are very few of them – about 5, typically – but they do happen.

Any method based on sampling has the possibility of being off. The beauty of methods based on random sampling is that we can quantify how often they are likely to be off.

replication_number = np.arange(1, 101)

replication_number = replication_number.astype(str)

intervals2 = pd.DataFrame(np.array([left_ends, right_ends]), columns=[replication_number])

intervals2

plt.figure(figsize=(8,8))
for i in np.arange(100):
    ends = intervals2.iloc[:,i]
    plt.plot(ends, np.array([i+1, i+1]), color='gold')
plt.plot(np.array([pop_median, pop_median]), np.array([0, 100]), color='red', lw=2)
plt.xlabel('Median (dollars)')
plt.ylabel('Replication')
plt.title('Population Median and Intervals of Estimates');
../../_images/Bootstrap_54_0.png

To summarize what the simulation shows, suppose you are estimating the population median by the following process:

  • Draw a large random sample from the population.

  • Bootstrap your random sample and get an estimate from the new random sample.

  • Repeat the above step thousands of times, and get thousands of estimates.

  • Pick off the “middle 95%” interval of all the estimates.

That gives you one interval of estimates. Now if you repeat the entire process 100 times, ending up with 100 intervals, then about 95 of those 100 intervals will contain the population parameter.

In other words, this process of estimation captures the parameter about 95% of the time.

You can replace 95% by a different value, as long as it’s not 100. Suppose you replace 95% by 80% and keep the sample size fixed at 500. Then your intervals of estimates will be shorter than those we simulated here, because the “middle 80%” is a smaller range than the “middle 95%”. Only about 80% of your intervals will contain the parameter.

1. Percentiles 3. Confidence Intervals