Loading the data

import pandas as pd
import numpy as np

# Importing the dataset
df = pd.read_csv('/Users/lucamartucci/Desktop/Assignment/mldata_0003194553.csv', engine='python', index_col=0)

df.shape

(1000, 21)

We have a dataset of 1000 rows and 21 columns.

df.head()

	label	feature_1	feature_2	feature_3	feature_4	feature_5	feature_6	feature_7	feature_8	feature_9	...	feature_11	feature_12	feature_13	feature_14	feature_15	feature_16	feature_17	feature_18	feature_19	feature_20
0	1	2.400647	0.810467	0.890485	2.876323	-0.201693	1.340662	-0.713567	-1.962509	-2.145410	...	1.249459	-1.740091	-0.796538	-0.097185	-0.290498	-0.842398	-0.260564	0.154766	1.286151	0.464597
1	0	-4.950411	-1.841936	3.113165	-8.725528	1.779361	-4.398690	-4.850836	0.222025	0.593351	...	-2.912149	1.753206	-1.448536	5.706732	-0.923660	-3.782716	3.230223	0.032502	1.242510	-1.667227
2	0	-6.725489	-6.729168	-0.785141	-8.553268	-0.538009	-4.606925	-5.409361	3.050847	2.292932	...	-6.046431	-2.824603	-2.854134	5.976063	-2.757761	0.154910	6.105090	0.404889	-0.484718	-5.106409
3	1	-3.531168	2.486647	-1.226836	0.608371	-1.113568	2.312421	-0.365272	-2.611237	-1.255728	...	0.188368	1.730081	2.393850	-2.271567	-1.779465	3.370979	1.471535	-3.072215	3.134686	-1.100149
4	1	0.290140	-1.006722	-1.269914	2.141340	0.021300	-1.031355	0.418452	-1.443210	-1.012174	...	0.243905	0.967477	2.708935	-1.358529	1.490291	0.436279	-0.141598	-4.003198	1.182402	-0.254557

5 rows × 21 columns

df.describe()

	label	feature_1	feature_2	feature_3	feature_4	feature_5	feature_6	feature_7	feature_8	feature_9	...	feature_11	feature_12	feature_13	feature_14	feature_15	feature_16	feature_17	feature_18	feature_19	feature_20
count	1000.000000	1000.000000	1000.000000	1000.000000	1000.000000	1000.000000	1000.000000	1000.000000	1000.000000	1000.000000	...	1000.000000	1000.000000	1000.000000	1000.000000	1000.000000	1000.000000	1000.000000	1000.000000	1000.000000	1000.000000
mean	0.492000	0.050673	-0.062736	-0.081388	0.640447	-0.333612	0.301935	-0.175986	-0.400607	-0.220434	...	-0.063231	-0.379010	-0.049449	0.173760	0.080598	-0.242342	0.378365	-0.516986	0.446335	0.106811
std	0.500186	2.531997	1.801628	1.664056	3.038165	1.626121	2.022650	1.765217	2.113550	1.786022	...	2.160147	1.569052	3.146255	2.458944	1.767448	1.969696	2.314681	2.634441	1.539008	2.468075
min	0.000000	-7.878319	-6.729168	-7.366695	-9.675188	-5.736812	-6.910966	-5.736090	-6.982050	-5.518066	...	-7.462937	-6.502639	-11.235926	-7.339681	-7.089460	-6.119338	-6.846388	-8.674397	-4.806411	-9.073984
25%	0.000000	-1.647045	-1.204716	-1.192115	-1.436935	-1.407978	-1.072498	-1.370443	-1.924368	-1.412787	...	-1.517236	-1.380533	-1.942196	-1.563310	-1.001913	-1.585309	-1.188314	-2.513739	-0.513291	-1.558812
50%	0.000000	0.164925	-0.037475	-0.011759	0.822677	-0.372698	0.264713	-0.211634	-0.408701	-0.285880	...	-0.006251	-0.308581	-0.307676	0.147119	0.201661	-0.268712	0.415564	-0.592875	0.492140	0.081360
75%	1.000000	1.850736	1.177979	0.999061	2.758884	0.717168	1.539625	0.991165	1.099646	0.997783	...	1.362884	0.629686	1.823826	1.794706	1.187407	1.014912	1.832109	1.412090	1.485073	1.687177
max	1.000000	7.587167	5.786481	5.220784	9.626333	4.555429	8.227681	4.573053	5.646083	5.815049	...	6.055950	4.552465	10.655765	7.012969	4.680725	7.774033	8.489627	7.114982	4.860566	7.925294

8 rows × 21 columns

Most of the features (columns) seem to have a mean around 0.

# Check N/A
df.isnull().any().any()

False

Luckily, there are no missing values.

Features and labels split

We move to separating the labels from the features in the dataset.

# Features
X_raw = df.iloc[:, 1:]

X_raw

	feature_1	feature_2	feature_3	feature_4	feature_5	feature_6	feature_7	feature_8	feature_9	feature_10	feature_11	feature_12	feature_13	feature_14	feature_15	feature_16	feature_17	feature_18	feature_19	feature_20
0	2.400647	0.810467	0.890485	2.876323	-0.201693	1.340662	-0.713567	-1.962509	-2.145410	1.001471	1.249459	-1.740091	-0.796538	-0.097185	-0.290498	-0.842398	-0.260564	0.154766	1.286151	0.464597
1	-4.950411	-1.841936	3.113165	-8.725528	1.779361	-4.398690	-4.850836	0.222025	0.593351	3.258047	-2.912149	1.753206	-1.448536	5.706732	-0.923660	-3.782716	3.230223	0.032502	1.242510	-1.667227
2	-6.725489	-6.729168	-0.785141	-8.553268	-0.538009	-4.606925	-5.409361	3.050847	2.292932	0.081609	-6.046431	-2.824603	-2.854134	5.976063	-2.757761	0.154910	6.105090	0.404889	-0.484718	-5.106409
3	-3.531168	2.486647	-1.226836	0.608371	-1.113568	2.312421	-0.365272	-2.611237	-1.255728	-0.738244	0.188368	1.730081	2.393850	-2.271567	-1.779465	3.370979	1.471535	-3.072215	3.134686	-1.100149
4	0.290140	-1.006722	-1.269914	2.141340	0.021300	-1.031355	0.418452	-1.443210	-1.012174	0.963606	0.243905	0.967477	2.708935	-1.358529	1.490291	0.436279	-0.141598	-4.003198	1.182402	-0.254557
...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...
995	3.197395	3.380767	0.145049	1.046807	-0.489398	1.370536	-0.952867	-2.173577	1.363996	2.438781	1.300450	1.504412	2.218751	-3.900480	-0.257592	0.847030	0.315966	-3.640403	1.051552	-1.892422
996	3.067038	-0.760621	-2.259636	0.210233	-2.197586	3.280090	-1.038396	0.458064	-0.935781	-1.306850	-0.643018	-0.093952	2.644516	-2.919563	0.812125	1.468202	0.630370	-1.408633	-0.692481	-0.200247
997	-1.864331	2.032807	0.469221	0.247139	-1.783519	-1.751897	-1.191711	0.348700	-1.102367	1.404830	0.522812	2.686949	1.921519	0.707830	-1.608232	0.056693	0.452602	-3.770797	3.673332	0.774602
998	1.360212	-1.591831	-2.710063	0.249091	-4.337037	-2.483085	-0.802431	1.454194	-0.207299	0.985949	1.602937	-0.251941	6.339966	-3.515254	-2.325363	3.442479	2.408409	-2.706212	1.985068	-3.487837
999	2.000445	-0.867167	-1.928661	3.790247	-1.047372	-1.800007	1.201010	-0.227720	-0.617714	1.271794	3.051431	1.279707	-3.596284	0.766007	-1.529126	1.068731	-0.110111	-3.747913	-0.487724	2.437824

1000 rows × 20 columns

# Labels
y = df.iloc[:, 0]

y

0      1
1      0
2      0
3      1
4      1
      ..
995    1
996    1
997    1
998    1
999    0
Name: label, Length: 1000, dtype: int64

While features have all float values, labels have a binary value of either 1 or 0, as expected.

Exploratory Data Analysis

We first check that the proportion of ones and zeros in the values of the labels are balanced.

df['label'].value_counts(normalize=True)

label
0    0.508
1    0.492
Name: proportion, dtype: float64

Satisfied with such proportion, we can try to visualize the data in multiple ways.

To do this, it is better to first separate a set on which to test the models built later.

from sklearn.model_selection import train_test_split

# 20% of the data will be used for testing
X_train, X_test, y_train, y_test = train_test_split(X_raw, y, test_size=0.2, random_state=1)

Distributions in the features

We can visualize the distribution in the features using histograms.

import matplotlib.pyplot as plt

X_train.hist(bins=50, figsize=(20,15), color='c', edgecolor='white')
plt.show()

We can simultaneosuly visualize the distribution of our datapoints in each feature in a smooth way by means of Kernel Density Estimation.

import seaborn as sns

sns.displot(data=X_train, palette="YlGnBu", kind="kde")

<seaborn.axisgrid.FacetGrid at 0x144427a40>

Even though the distributions for the raw data look pretty irregular, their means are centered around 0 as already seen.

However we would like to see more uniformity among these "bell-shaped" curves.

Feature scaling

We standardize each feature to have a mean of 0 and a standard deviation of 1, separately on the train and test datasets.

from sklearn.preprocessing import StandardScaler

# Fit the scaler on the training data
scaler = StandardScaler().fit(X_train)

# Transform the training data
X_train = pd.DataFrame(scaler.transform(X_train), columns=X_train.columns)

# Transform the test data using the same scaler
X_test = pd.DataFrame(scaler.transform(X_test), columns=X_test.columns)

Fitting the StandardScaler on the training data and then using the same scaler to transform both the training and test data is generally recommended, since it ensures that the scaling is consistent and prevents data leakage from the test set.

X_train.hist(bins=50, figsize=(20,15), color='c', edgecolor='white')
plt.show()

We notice that the range of values was shrinked to a much smaller interval, for each of the features when scaled, as expected.

sns.displot(data=X_train, palette="YlGnBu", kind="kde")

<seaborn.axisgrid.FacetGrid at 0x146cd3800>

All the distributions observed now follow a similar standard normal distribution.

Correlation

Intuitively, highly correlated features carry redundant information. Since PCA identifies combinations of the original features that capture the most variance in the data and aims to reduce its dimensionality by combining correlated features into the so-called principal components, it is useful to understand the correlation structure of the data.

We can continue working on scaled data since the standardization does not affect correlation between variables, indeed it is defined as: $$\rho_{ij}:=\frac{\text{Cov}(X_i, X_j)}{\sigma_i\sigma_j}$$ and does not depend on the overall mean and standard deviation of the full train set.

correlation = X_train.corr()

We can simultaneously analyze the correlation among all features by means of a heatmap.

plt.figure(figsize=(20, 15))
sns.heatmap(X_train.corr(), cmap="YlGnBu", annot=True)

<Axes: >

Most features are mildly correlated, thus applying PCA may be effective in reducing dimensionality, because there is "(at least) some redundancy" to eliminate.

We can also more evidently visualize this by only considering features with a significant correlation (e.g. more than 1/4 as correlation coefficient in absolute value).

plt.figure(figsize=(16, 12))
sns.heatmap(X_train.corr().mask(np.abs(X_train.corr()) < 0.25), cmap="YlGnBu", annot=True)

<Axes: >

Principal Component Analysis

Initially, we are applying PCA with as many components as features, just to understand how explained variance evolves with the number of components, with the aim of grasping the optimal number of such.

from sklearn.decomposition import PCA

pca_tot = PCA(n_components = X_train.shape[1])

# Fit the "total" PCA model on the training data
pca_tot.fit(X_train)

PCA(n_components=20)

In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook.
On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.

# Plot the ratio of variance explained by each component
components = pca_tot.components_
var = pca_tot.explained_variance_ratio_

plt.figure(figsize=(15, 5))
plt.plot(np.cumsum(var), color='c', linewidth=2)
plt.xlabel("Components")
plt.ylabel("Explained Variance")
plt.title("Explained Variance vs Number of Principal Components")
plt.grid(False)
plt.show()

labels = ["PC"+str(i+1) for i in range(X_train.shape[1])]

plt.figure(figsize=(16,4))
plt.bar(labels, var, color='c')
plt.xlabel('Pricipal Components')
plt.ylabel('Proportion of Variance Explained')
plt.show()

We would like to see an "elbow", i.e. a point after which the histogram of the proportion of explained variance flattens out, or such that in the cumulative explained variance plot the addition of more components starts to provide "diminishing returns". This would give the number of components retaining more information.

Even though we cannot notice an elbow, there is some evidence of "redundancy", since for example the variance explained by the last two components is negligible.

At this point, we can just see the number of components yielding a cumulative explained variance of at least 95%: usually, this approach ensures that the majority of the data's variability is captured while still achieving a substantial reduction in dimensionality.

pca = PCA(n_components=0.95)

# Fit the PCA model on the training data
pca = PCA().fit(X_train)

# Transform the training data
X_train_red = pca.transform(X_train)

# Transform the test data using the same PCA model
X_test_red = pca.transform(X_test)

X_train_red.shape

(800, 20)

The significant proportion of 14 features out of 20 are able to explain 95% of the variance.

Such reduction in dimensionality, even if not drastic, could however still offer benefits such as improved model performance, reduced redundancy, and enhanced interpretability. The key is to evaluate the impact of PCA in our specific case by comparing model performance and understanding the variance captured by the principal components.

Visualization intermezzo

Here we try reducing the dimensionality of the full scaled raw data to just two components, for the sake of fitting the data in a two-dimensional space and try to visualize, if present, any separation between the label classes.

# Standardize the full dataset
scaler = StandardScaler().fit(X_raw)
X = pd.DataFrame(scaler.transform(X_raw), columns = X_raw.columns)

PCA

# Apply 2D PCA to the full scaled dataset
pca_2D = PCA(n_components=2)
red_2D = pca_2D.fit_transform(X)

# Split datapoints according to labels
l0_2D = red_2D[y == 0]
l1_2D = red_2D[y == 1]

fig, ax = plt.subplots(figsize=(8, 8))
ax.set_xlabel('Principal Component 1', fontsize=10)
ax.set_ylabel('Principal Component 2', fontsize=10)
ax.set_title('2D plot after PCA', fontsize=15)

ax.scatter(l0_2D[:, 0], l0_2D[:, 1], color='royalblue', label='Label 0')
ax.scatter(l1_2D[:, 0], l1_2D[:, 1], color='lightgreen', label='Label 1')

ax.legend()

plt.tight_layout()
plt.show()

There is no division clearly visible in two dimensions.

Although such a division will be evident using non-linear dimensionality reduction technqieues such as UMAP and tSNE, it is still valuable to try PCA as a first step. Indeed, it is computationally less intensive and can provide insights into the variance explained by each component, which can be useful for understanding the overall structure of the data.

tSNE

We apply t-distributed Stochastic Neighbour Embedding, which employs a nonlinear approach to embed high-dimensional data into a lower-dimensional space, which in our case we will choose to be two. The core of this method involves minimizing the Kullback-Leibler (KL) divergence $$\text{KL}(P\|Q):=\sum_{i\neq j}p_{ij}\log\frac{p_{ij}}{q_{ij}}$$ between the two probability distributions: $P$ in the original high-dimensional space, such that the similarity of datapoint $x_j$ to $x_i$ is the conditional probability $p_{i|j}$, which is assumed to be a Gaussian, constructed so that similar points are assigned higher probabilities, and $Q$ in the lower-dimensional space, similarly constructed.

The goal of the optimization process is to ensure that similar points in the high-dimensional space remain close to each other in the lower-dimensional representation. This helps preserving the local structure of the data, maintaining the relationships and similarities between points even after the dimensionality reduction.

from sklearn.manifold import TSNE

tsne = TSNE(n_components=2, init='random', random_state=1)
X_tsne = tsne.fit_transform(X)

l0_tsne = X_tsne[y == 0]
l1_tsne = X_tsne[y == 1]

fig, ax = plt.subplots(figsize=(8, 8))
ax.set_title('t-SNE 2D Scatter Plot', fontsize=15)

ax.scatter(l0_tsne[:, 0], l0_tsne[:, 1], color='royalblue', label='Label 0')
ax.scatter(l1_tsne[:, 0], l1_tsne[:, 1], color='lightgreen', label='Label 1')

ax.legend()

plt.tight_layout()
plt.show()

It is important to note that t-SNE performs a nonlinear transformation on the data, unlike PCA. As a result, the meaning of the axes in the t-SNE plot is not straightforward to interpret.

UMAP

We also attempt using UMAP, which stands for Uniform Manifold Approximation and Projection. It is a dimensionality reduction technique operating under the key assumption that the data is uniformly distributed on a Riemannian manifold, which the algorithm endeavors to approximate. By doing so, it effectively captures the intrinsic geometric structure of the data.

We utilize UMAP as an alternative to t-SNE (t-Distributed Stochastic Neighbor Embedding), leveraging its ability to maintain both the local and global structure of the data.

import umap.umap_ as umap

/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html
  from .autonotebook import tqdm as notebook_tqdm

umap = umap.UMAP(n_components = 2)
X_umap = umap.fit_transform(X)

l0_umap = X_umap[y == 0]
l1_umap = X_umap[y == 1]

fig, ax = plt.subplots(figsize=(8, 8))
ax.set_title('UMAP 2D Scatter Plot', fontsize=15)

ax.scatter(l0_umap[:, 0], l0_umap[:, 1], color='royalblue', label='Label 0')
ax.scatter(l1_umap[:, 0], l1_umap[:, 1], color='lightgreen', label='Label 1')

ax.legend()

plt.tight_layout()
plt.show()

As before, the meaning of the axes is not clearly interpretable. However, these plots still provide a useful visual representation of the data's structure.

Random Forest

We now compare the performance of a Random Forest classifier with and without implementing PCA, on the rescaled data.

Without PCA

from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import accuracy_score

rf = RandomForestClassifier(random_state=1, n_jobs=-1)

# Train the Random Forest model on the training data
rf.fit(X_train, y_train)

RandomForestClassifier(n_jobs=-1, random_state=1)

In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook.
On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.

# Predict the labels for the test set
y_pred = rf.predict(X_test)

# Calculate the accuracy of the model
accuracy = accuracy_score(y_test, y_pred)
print(f'Accuracy of Random Forest without PCA: {accuracy:.4f}')

Accuracy of Random Forest without PCA: 0.8700

We now sort features according to their relative "importance", that is the average over all trees in the forest of the total reduction of the criterion brought by each feature, and plot them to look at their distribution

feature_names = [f'feature_{i+1}' for i in range(X_train.shape[1])]
features_importances = [(feature_names[i], rf.feature_importances_[i]) for i in range(len(feature_names))]

sorted(features_importances, key=lambda x: x[1], reverse=True)

[('feature_14', 0.12763747709108203),
 ('feature_6', 0.12249867164863713),
 ('feature_13', 0.1024715435089309),
 ('feature_15', 0.07074594741418183),
 ('feature_7', 0.06443894194143622),
 ('feature_4', 0.061230981342643584),
 ('feature_9', 0.05424055277750717),
 ('feature_18', 0.04273128870159602),
 ('feature_11', 0.035437408118838165),
 ('feature_2', 0.034529457912775355),
 ('feature_16', 0.03410159993018076),
 ('feature_1', 0.03346283971056454),
 ('feature_20', 0.03339973223054457),
 ('feature_3', 0.03206101670458729),
 ('feature_5', 0.03004816414316784),
 ('feature_10', 0.02915472960404653),
 ('feature_8', 0.02526540337596601),
 ('feature_19', 0.023042804873301282),
 ('feature_12', 0.02303508064290121),
 ('feature_17', 0.02046635832711147)]

There is no feature having a more significant relative importance with respect to the others.

fig = plt.figure(figsize=(8, 6))
plt.plot(rf.feature_importances_, color='c', linewidth=2)
plt.ylabel('Relative Importance')
plt.xlabel('Features')
plt.title('Feature Importance using Random Forests')

Text(0.5, 1.0, 'Feature Importance using Random Forests')

Indeed, their distribution does not show any particular pattern.

With PCA

rf_pca = RandomForestClassifier(random_state=1)

rf_pca.fit(X_train_red, y_train)

RandomForestClassifier(random_state=1)

In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook.
On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.

y_pred_pca = rf_pca.predict(X_test_red)

accuracy = accuracy_score(y_test, y_pred_pca)
print(f'Accuracy of Random Forest with PCA: {accuracy:.4f}')

Accuracy of Random Forest with PCA: 0.8950

The accuracy on the PCA-reduced data is slightly improved.

feature_names_red = [f'feature_{i+1}' for i in range(X_train_red.shape[1])]
features_importances_red = [(feature_names[i], rf_pca.feature_importances_[i]) for i in range(len(feature_names_red))]

sorted(features_importances_red, key=lambda x: x[1], reverse=True)

[('feature_1', 0.2067520434237167),
 ('feature_3', 0.12036735670145142),
 ('feature_2', 0.09701881316738317),
 ('feature_6', 0.07763913688770387),
 ('feature_4', 0.06581806027068164),
 ('feature_5', 0.05400724069619528),
 ('feature_7', 0.04946209657796722),
 ('feature_8', 0.04265842326705147),
 ('feature_10', 0.04080963480962326),
 ('feature_11', 0.03667635096577979),
 ('feature_16', 0.0302422685354677),
 ('feature_9', 0.02999099483921723),
 ('feature_18', 0.026628122933274094),
 ('feature_14', 0.025822805126005283),
 ('feature_12', 0.02473102162010316),
 ('feature_15', 0.024524465266156474),
 ('feature_13', 0.023938022122829413),
 ('feature_17', 0.022913142789392826),
 ('feature_19', 0.0),
 ('feature_20', 0.0)]

We see that, first of all, the most relevant features in relative importance were preserved by PCA, and also that in the first couple are the prominent ones.

fig = plt.figure(figsize=(8, 6))
plt.plot(rf_pca.feature_importances_, color='c', linewidth=2)
plt.ylabel('Relative Importance')
plt.xlabel('Features')
plt.title('Feature Importance using Random Forests')

Text(0.5, 1.0, 'Feature Importance using Random Forests')

As expected, here, their distribution follows an almost monotonically decreasing trend.

We can also further compare the results both with and without implementing PCA, by looking at the confusion matrices generated.

from sklearn.metrics import confusion_matrix
from sklearn.metrics import precision_score
from sklearn.metrics import recall_score

# Confusion matrix for both classifiers
cm = confusion_matrix(y_test, y_pred)
red_cm = confusion_matrix(y_test, y_pred_pca)

# Precision and Recall scores 
precision = precision_score(y_test, y_pred, average='weighted')
recall = recall_score(y_test, y_pred, average='weighted')
accuracy = accuracy_score(y_test, y_pred)

red_precision = precision_score(y_test, y_pred_pca)
red_recall = recall_score(y_test, y_pred_pca)
red_accuracy = accuracy_score(y_test, y_pred_pca)

# Output precision and recall scores
print(f"""Original Dataset              Precision: {precision:.4f}, Recall: {recall:.4f}, Accuracy: {accuracy:.4f}
PCA-Transformed Dataset       Precision: {red_precision:.4f}, Recall: {red_recall:.4f}, Accuracy: {red_accuracy:.4f}""")

Original Dataset              Precision: 0.8720, Recall: 0.8700, Accuracy: 0.8700
PCA-Transformed Dataset       Precision: 0.9223, Recall: 0.8796, Accuracy: 0.8950

# Plot the confusion matrices
plt.figure(figsize=(12, 5))

plt.subplot(1, 2, 1)
sns.heatmap(cm, annot=True, fmt="d", cmap="YlGnBu")
plt.title("Original Random Forest Confusion Matrix\nPrecision: {:.2f}, Recall: {:.2f}".format(precision, recall))
plt.xlabel("Predicted")
plt.ylabel("True")

plt.subplot(1, 2, 2)
sns.heatmap(red_cm, annot=True, fmt="d", cmap="YlGnBu")
plt.title("Random Forest with PCA Confusion Matrix\nPrecision: {:.2f}, Recall: {:.2f}".format(red_precision, red_recall))
plt.xlabel("Predicted")
plt.ylabel("True")

plt.tight_layout()
plt.show()

We notice that, even though for a very small size of additional labels correctly predicted with PCA, the overall precision, that is, the ratio of correctly predicted positive observations to the total predicted positives, increases.

Hyperparameter optimization

Once we have noticed that PCA does actually improve the performance of the classifier, we refine our approach.

We want to find the best combination of hyperparameters that maximize our model's performance, namely: the number of trees ('n_estimators'), the maximum depth of the trees ('max_depth'), the minimum number of samples required to split an internal node ('min_samples_split'), the number of features to consider when looking for the best split ('max_features'), whether to use bootstrap or not ('bootstrap'), and the function to measure the quality of a split ('criterion').

To do this, we first use Cross-Validation to make the choice of the range for the number of trees narrower, and then use Random Search, which samples parameter combinations randomly from specified distributions, with cross-validation, to then train the Random Forest model.

from sklearn.model_selection import cross_val_score

# Define a range of values for n_estimators to try
n_estimators_values = [1, 50, 100, 150, 200, 250, 300, 350, 400, 450, 500]

train_scores = []
cv_scores = []

# Iterate over each value of n_estimators
for n_estimators in n_estimators_values:

    rf_classifier = RandomForestClassifier(n_estimators=n_estimators, random_state=1)
    rf_classifier.fit(X_train_red, y_train)
    
    # Compute training score and append to train_scores
    train_score = rf_classifier.score(X_train_red, y_train)
    train_scores.append(train_score)
    
    # Compute cross-validation score and append to cv_scores
    cv_score = np.mean(cross_val_score(rf_classifier, X_train_red, y_train, cv=5))
    cv_scores.append(cv_score)

# Plot the validation curve
plt.figure(figsize=(10, 6))
plt.plot(n_estimators_values, train_scores, label='Training Score', marker='o', color='royalblue')
plt.plot(n_estimators_values, cv_scores, label='Cross-validation Score', marker='o', color='lightgreen')
plt.xlabel('n_estimators')
plt.ylabel('Score')
plt.title('Validation Curve for Random Forest')
plt.legend()
plt.grid(False)
plt.show()

Since the validation score almost stops increasing, and reaches a "pleateau", after approximately 50 'n_estimators', we reduce the choice of range to analyze, during the hyperparameter optimization, in a small neighbourhood around this value.

from sklearn.model_selection import RandomizedSearchCV
from scipy.stats import randint

The choice of the sampling distribution for all the hyperparameters is trivial, since all of them are discrete (i.e. we want to search among integer numbers) and in the range chosen all values seem similarly likely: a discrete uniform distribution will work for us.

# Parameter distribution for the Random Search
param_dist = {
    'n_estimators': randint(40, 60), # narrow interval due to CV
    'max_depth': [None, 10, 20, 30], # we select a relatively shallow depth to avoid overfitting
    'min_samples_split': randint(2, 11),
    'min_samples_leaf': randint(1, 5),
    'max_features': [None, 'sqrt', 'log2'], 
    'bootstrap': [True, False],
    'criterion': ['gini', 'entropy']
}

random_search = RandomizedSearchCV(estimator=rf_pca, param_distributions=param_dist, n_iter=200, cv=5, random_state=1, n_jobs=-1, verbose=2)
random_search.fit(X_train_red, y_train)

Fitting 5 folds for each of 200 candidates, totalling 1000 fits
[CV] END bootstrap=True, criterion=gini, max_depth=10, max_features=None, min_samples_leaf=4, min_samples_split=8, n_estimators=58; total time=   0.5s
[CV] END bootstrap=True, criterion=gini, max_depth=10, max_features=None, min_samples_leaf=4, min_samples_split=8, n_estimators=58; total time=   0.5s
[CV] END bootstrap=True, criterion=gini, max_depth=10, max_features=None, min_samples_leaf=4, min_samples_split=8, n_estimators=58; total time=   0.5s
[CV] END bootstrap=False, criterion=entropy, max_depth=None, max_features=None, min_samples_leaf=4, min_samples_split=7, n_estimators=55; total time=   0.9s
[CV] END bootstrap=False, criterion=entropy, max_depth=None, max_features=None, min_samples_leaf=4, min_samples_split=7, n_estimators=55; total time=   0.9s
[CV] END bootstrap=True, criterion=gini, max_depth=10, max_features=log2, min_samples_leaf=1, min_samples_split=4, n_estimators=44; total time=   0.2s
[CV] END bootstrap=False, criterion=entropy, max_depth=None, max_features=None, min_samples_leaf=4, min_samples_split=7, n_estimators=55; total time=   0.9s
[CV] END bootstrap=False, criterion=entropy, max_depth=None, max_features=None, min_samples_leaf=4, min_samples_split=7, n_estimators=55; total time=   0.9s
[CV] END bootstrap=False, criterion=entropy, max_depth=None, max_features=None, min_samples_leaf=4, min_samples_split=7, n_estimators=55; total time=   0.9s
[CV] END bootstrap=True, criterion=gini, max_depth=10, max_features=log2, min_samples_leaf=1, min_samples_split=4, n_estimators=44; total time=   0.2s
[CV] END bootstrap=True, criterion=gini, max_depth=10, max_features=log2, min_samples_leaf=1, min_samples_split=4, n_estimators=44; total time=   0.2s
[CV] END bootstrap=True, criterion=gini, max_depth=10, max_features=log2, min_samples_leaf=1, min_samples_split=4, n_estimators=44; total time=   0.2s
[CV] END bootstrap=True, criterion=gini, max_depth=10, max_features=log2, min_samples_leaf=1, min_samples_split=4, n_estimators=44; total time=   0.2s
[CV] END bootstrap=True, criterion=gini, max_depth=10, max_features=None, min_samples_leaf=4, min_samples_split=8, n_estimators=58; total time=   0.6s
[CV] END bootstrap=True, criterion=gini, max_depth=10, max_features=None, min_samples_leaf=4, min_samples_split=8, n_estimators=58; total time=   0.6s
[CV] END bootstrap=False, criterion=entropy, max_depth=10, max_features=sqrt, min_samples_leaf=4, min_samples_split=2, n_estimators=53; total time=   0.3s
[CV] END bootstrap=False, criterion=entropy, max_depth=10, max_features=sqrt, min_samples_leaf=4, min_samples_split=2, n_estimators=53; total time=   0.3s
[CV] END bootstrap=False, criterion=entropy, max_depth=10, max_features=sqrt, min_samples_leaf=4, min_samples_split=2, n_estimators=53; total time=   0.3s
[CV] END bootstrap=False, criterion=entropy, max_depth=30, max_features=sqrt, min_samples_leaf=3, min_samples_split=3, n_estimators=40; total time=   0.3s
[CV] END bootstrap=False, criterion=entropy, max_depth=10, max_features=sqrt, min_samples_leaf=4, min_samples_split=2, n_estimators=53; total time=   0.3s
[CV] END bootstrap=False, criterion=entropy, max_depth=10, max_features=sqrt, min_samples_leaf=4, min_samples_split=2, n_estimators=53; total time=   0.3s
[CV] END bootstrap=False, criterion=entropy, max_depth=30, max_features=sqrt, min_samples_leaf=3, min_samples_split=3, n_estimators=40; total time=   0.2s
[CV] END bootstrap=False, criterion=entropy, max_depth=30, max_features=sqrt, min_samples_leaf=3, min_samples_split=3, n_estimators=40; total time=   0.3s
[CV] END bootstrap=False, criterion=entropy, max_depth=30, max_features=sqrt, min_samples_leaf=3, min_samples_split=3, n_estimators=40; total time=   0.3s
[CV] END bootstrap=False, criterion=entropy, max_depth=30, max_features=sqrt, min_samples_leaf=3, min_samples_split=3, n_estimators=40; total time=   0.3s
[CV] END bootstrap=True, criterion=entropy, max_depth=None, max_features=log2, min_samples_leaf=4, min_samples_split=5, n_estimators=46; total time=   0.2s
[CV] END bootstrap=True, criterion=entropy, max_depth=None, max_features=log2, min_samples_leaf=4, min_samples_split=5, n_estimators=46; total time=   0.2s
[CV] END bootstrap=True, criterion=entropy, max_depth=None, max_features=log2, min_samples_leaf=4, min_samples_split=5, n_estimators=46; total time=   0.3s
[CV] END bootstrap=True, criterion=entropy, max_depth=None, max_features=log2, min_samples_leaf=4, min_samples_split=5, n_estimators=46; total time=   0.2s
[CV] END bootstrap=True, criterion=entropy, max_depth=None, max_features=None, min_samples_leaf=2, min_samples_split=5, n_estimators=55; total time=   0.7s
[CV] END bootstrap=True, criterion=entropy, max_depth=None, max_features=log2, min_samples_leaf=4, min_samples_split=5, n_estimators=46; total time=   0.2s
[CV] END bootstrap=True, criterion=entropy, max_depth=None, max_features=None, min_samples_leaf=2, min_samples_split=5, n_estimators=55; total time=   0.6s
[CV] END bootstrap=True, criterion=entropy, max_depth=None, max_features=None, min_samples_leaf=2, min_samples_split=5, n_estimators=55; total time=   0.6s
[CV] END bootstrap=True, criterion=entropy, max_depth=None, max_features=None, min_samples_leaf=2, min_samples_split=5, n_estimators=55; total time=   0.6s
[CV] END bootstrap=True, criterion=entropy, max_depth=None, max_features=None, min_samples_leaf=2, min_samples_split=5, n_estimators=55; total time=   0.6s
[CV] END bootstrap=True, criterion=gini, max_depth=None, max_features=None, min_samples_leaf=2, min_samples_split=5, n_estimators=49; total time=   0.5s[CV] END bootstrap=True, criterion=gini, max_depth=None, max_features=None, min_samples_leaf=2, min_samples_split=5, n_estimators=49; total time=   0.5s

[CV] END bootstrap=True, criterion=gini, max_depth=None, max_features=None, min_samples_leaf=2, min_samples_split=5, n_estimators=49; total time=   0.5s
[CV] END bootstrap=False, criterion=entropy, max_depth=10, max_features=None, min_samples_leaf=1, min_samples_split=3, n_estimators=51; total time=   0.9s
[CV] END bootstrap=False, criterion=entropy, max_depth=10, max_features=None, min_samples_leaf=1, min_samples_split=3, n_estimators=51; total time=   0.9s
[CV] END bootstrap=False, criterion=entropy, max_depth=10, max_features=None, min_samples_leaf=1, min_samples_split=3, n_estimators=51; total time=   0.9s
[CV] END bootstrap=False, criterion=entropy, max_depth=10, max_features=None, min_samples_leaf=1, min_samples_split=3, n_estimators=51; total time=   0.9s
[CV] END bootstrap=False, criterion=entropy, max_depth=10, max_features=None, min_samples_leaf=1, min_samples_split=3, n_estimators=51; total time=   1.0s
[CV] END bootstrap=True, criterion=gini, max_depth=None, max_features=None, min_samples_leaf=2, min_samples_split=5, n_estimators=49; total time=   0.5s
[CV] END bootstrap=True, criterion=gini, max_depth=None, max_features=None, min_samples_leaf=2, min_samples_split=5, n_estimators=49; total time=   0.5s
[CV] END bootstrap=True, criterion=entropy, max_depth=None, max_features=None, min_samples_leaf=2, min_samples_split=4, n_estimators=47; total time=   0.5s
[CV] END bootstrap=True, criterion=entropy, max_depth=None, max_features=log2, min_samples_leaf=2, min_samples_split=5, n_estimators=47; total time=   0.2s
[CV] END bootstrap=True, criterion=entropy, max_depth=None, max_features=None, min_samples_leaf=2, min_samples_split=4, n_estimators=47; total time=   0.5s
[CV] END bootstrap=True, criterion=entropy, max_depth=None, max_features=log2, min_samples_leaf=2, min_samples_split=5, n_estimators=47; total time=   0.2s
[CV] END bootstrap=True, criterion=entropy, max_depth=None, max_features=log2, min_samples_leaf=2, min_samples_split=5, n_estimators=47; total time=   0.2s
[CV] END bootstrap=True, criterion=entropy, max_depth=None, max_features=None, min_samples_leaf=2, min_samples_split=4, n_estimators=47; total time=   0.5s
[CV] END bootstrap=True, criterion=entropy, max_depth=None, max_features=log2, min_samples_leaf=2, min_samples_split=5, n_estimators=47; total time=   0.2s
[CV] END bootstrap=True, criterion=entropy, max_depth=None, max_features=None, min_samples_leaf=2, min_samples_split=4, n_estimators=47; total time=   0.5s
[CV] END bootstrap=True, criterion=entropy, max_depth=None, max_features=log2, min_samples_leaf=2, min_samples_split=5, n_estimators=47; total time=   0.2s
[CV] END bootstrap=True, criterion=entropy, max_depth=None, max_features=None, min_samples_leaf=2, min_samples_split=4, n_estimators=47; total time=   0.5s
[CV] END bootstrap=True, criterion=gini, max_depth=None, max_features=log2, min_samples_leaf=3, min_samples_split=9, n_estimators=55; total time=   0.2s
[CV] END bootstrap=True, criterion=gini, max_depth=None, max_features=log2, min_samples_leaf=3, min_samples_split=9, n_estimators=55; total time=   0.2s
[CV] END bootstrap=True, criterion=gini, max_depth=None, max_features=log2, min_samples_leaf=3, min_samples_split=9, n_estimators=55; total time=   0.2s
[CV] END bootstrap=True, criterion=gini, max_depth=None, max_features=log2, min_samples_leaf=3, min_samples_split=9, n_estimators=55; total time=   0.2s
[CV] END bootstrap=True, criterion=gini, max_depth=None, max_features=log2, min_samples_leaf=3, min_samples_split=9, n_estimators=55; total time=   0.2s
[CV] END bootstrap=True, criterion=entropy, max_depth=10, max_features=None, min_samples_leaf=4, min_samples_split=7, n_estimators=59; total time=   0.6s
[CV] END bootstrap=True, criterion=entropy, max_depth=10, max_features=None, min_samples_leaf=4, min_samples_split=7, n_estimators=59; total time=   0.6s
[CV] END bootstrap=True, criterion=entropy, max_depth=10, max_features=None, min_samples_leaf=4, min_samples_split=7, n_estimators=59; total time=   0.6s
[CV] END bootstrap=True, criterion=entropy, max_depth=10, max_features=None, min_samples_leaf=4, min_samples_split=7, n_estimators=59; total time=   0.6s
[CV] END bootstrap=True, criterion=entropy, max_depth=10, max_features=None, min_samples_leaf=4, min_samples_split=7, n_estimators=59; total time=   0.6s
[CV] END bootstrap=False, criterion=entropy, max_depth=10, max_features=None, min_samples_leaf=3, min_samples_split=2, n_estimators=41; total time=   0.6s
[CV] END bootstrap=False, criterion=entropy, max_depth=10, max_features=None, min_samples_leaf=3, min_samples_split=2, n_estimators=41; total time=   0.6s
[CV] END bootstrap=False, criterion=entropy, max_depth=10, max_features=None, min_samples_leaf=3, min_samples_split=2, n_estimators=41; total time=   0.6s
[CV] END bootstrap=False, criterion=entropy, max_depth=10, max_features=None, min_samples_leaf=3, min_samples_split=2, n_estimators=41; total time=   0.6s
[CV] END bootstrap=False, criterion=entropy, max_depth=10, max_features=None, min_samples_leaf=3, min_samples_split=2, n_estimators=41; total time=   0.7s
[CV] END bootstrap=True, criterion=gini, max_depth=30, max_features=sqrt, min_samples_leaf=4, min_samples_split=9, n_estimators=53; total time=   0.2s
[CV] END bootstrap=False, criterion=entropy, max_depth=30, max_features=None, min_samples_leaf=1, min_samples_split=8, n_estimators=46; total time=   0.7s
[CV] END bootstrap=True, criterion=gini, max_depth=30, max_features=sqrt, min_samples_leaf=4, min_samples_split=9, n_estimators=53; total time=   0.2s
[CV] END bootstrap=False, criterion=entropy, max_depth=30, max_features=None, min_samples_leaf=1, min_samples_split=8, n_estimators=46; total time=   0.7s
[CV] END bootstrap=False, criterion=entropy, max_depth=30, max_features=None, min_samples_leaf=1, min_samples_split=8, n_estimators=46; total time=   0.7s
[CV] END bootstrap=True, criterion=gini, max_depth=30, max_features=sqrt, min_samples_leaf=4, min_samples_split=9, n_estimators=53; total time=   0.2s
[CV] END bootstrap=True, criterion=gini, max_depth=30, max_features=sqrt, min_samples_leaf=4, min_samples_split=9, n_estimators=53; total time=   0.2s
[CV] END bootstrap=True, criterion=gini, max_depth=30, max_features=sqrt, min_samples_leaf=4, min_samples_split=9, n_estimators=53; total time=   0.2s
[CV] END bootstrap=False, criterion=entropy, max_depth=30, max_features=None, min_samples_leaf=1, min_samples_split=8, n_estimators=46; total time=   0.7s
[CV] END bootstrap=False, criterion=entropy, max_depth=None, max_features=sqrt, min_samples_leaf=1, min_samples_split=4, n_estimators=40; total time=   0.2s
[CV] END bootstrap=True, criterion=entropy, max_depth=None, max_features=None, min_samples_leaf=3, min_samples_split=6, n_estimators=47; total time=   0.5s
[CV] END bootstrap=True, criterion=entropy, max_depth=None, max_features=None, min_samples_leaf=3, min_samples_split=6, n_estimators=47; total time=   0.5s
[CV] END bootstrap=True, criterion=entropy, max_depth=None, max_features=None, min_samples_leaf=3, min_samples_split=6, n_estimators=47; total time=   0.5s
[CV] END bootstrap=True, criterion=entropy, max_depth=None, max_features=None, min_samples_leaf=3, min_samples_split=6, n_estimators=47; total time=   0.5s
[CV] END bootstrap=False, criterion=entropy, max_depth=None, max_features=sqrt, min_samples_leaf=1, min_samples_split=4, n_estimators=40; total time=   0.2s
[CV] END bootstrap=False, criterion=entropy, max_depth=None, max_features=sqrt, min_samples_leaf=1, min_samples_split=4, n_estimators=40; total time=   0.2s
[CV] END bootstrap=True, criterion=entropy, max_depth=None, max_features=None, min_samples_leaf=3, min_samples_split=6, n_estimators=47; total time=   0.5s
[CV] END bootstrap=False, criterion=entropy, max_depth=30, max_features=None, min_samples_leaf=1, min_samples_split=8, n_estimators=46; total time=   0.9s
[CV] END bootstrap=False, criterion=entropy, max_depth=None, max_features=sqrt, min_samples_leaf=1, min_samples_split=4, n_estimators=40; total time=   0.2s
[CV] END bootstrap=False, criterion=entropy, max_depth=None, max_features=sqrt, min_samples_leaf=1, min_samples_split=4, n_estimators=40; total time=   0.2s
[CV] END bootstrap=True, criterion=gini, max_depth=None, max_features=None, min_samples_leaf=2, min_samples_split=10, n_estimators=58; total time=   0.5s
[CV] END bootstrap=False, criterion=gini, max_depth=30, max_features=None, min_samples_leaf=2, min_samples_split=9, n_estimators=50; total time=   0.6s
[CV] END bootstrap=False, criterion=gini, max_depth=30, max_features=None, min_samples_leaf=2, min_samples_split=9, n_estimators=50; total time=   0.6s
[CV] END bootstrap=True, criterion=gini, max_depth=None, max_features=None, min_samples_leaf=2, min_samples_split=10, n_estimators=58; total time=   0.5s
[CV] END bootstrap=False, criterion=gini, max_depth=30, max_features=None, min_samples_leaf=2, min_samples_split=9, n_estimators=50; total time=   0.7s
[CV] END bootstrap=True, criterion=gini, max_depth=None, max_features=None, min_samples_leaf=2, min_samples_split=10, n_estimators=58; total time=   0.5s
[CV] END bootstrap=False, criterion=gini, max_depth=30, max_features=None, min_samples_leaf=2, min_samples_split=9, n_estimators=50; total time=   0.7s
[CV] END bootstrap=False, criterion=gini, max_depth=30, max_features=None, min_samples_leaf=2, min_samples_split=9, n_estimators=50; total time=   0.8s
[CV] END bootstrap=True, criterion=gini, max_depth=None, max_features=sqrt, min_samples_leaf=3, min_samples_split=8, n_estimators=46; total time=   0.2s
[CV] END bootstrap=True, criterion=gini, max_depth=None, max_features=sqrt, min_samples_leaf=3, min_samples_split=8, n_estimators=46; total time=   0.1s
[CV] END bootstrap=True, criterion=gini, max_depth=None, max_features=None, min_samples_leaf=2, min_samples_split=10, n_estimators=58; total time=   0.5s
[CV] END bootstrap=True, criterion=gini, max_depth=None, max_features=None, min_samples_leaf=2, min_samples_split=10, n_estimators=58; total time=   0.5s
[CV] END bootstrap=True, criterion=gini, max_depth=None, max_features=sqrt, min_samples_leaf=3, min_samples_split=8, n_estimators=46; total time=   0.2s
[CV] END bootstrap=True, criterion=gini, max_depth=None, max_features=sqrt, min_samples_leaf=3, min_samples_split=8, n_estimators=46; total time=   0.1s
[CV] END bootstrap=False, criterion=entropy, max_depth=20, max_features=None, min_samples_leaf=1, min_samples_split=9, n_estimators=42; total time=   0.6s
[CV] END bootstrap=True, criterion=gini, max_depth=None, max_features=sqrt, min_samples_leaf=3, min_samples_split=8, n_estimators=46; total time=   0.2s
[CV] END bootstrap=False, criterion=entropy, max_depth=20, max_features=None, min_samples_leaf=1, min_samples_split=9, n_estimators=42; total time=   0.7s
[CV] END bootstrap=False, criterion=entropy, max_depth=20, max_features=None, min_samples_leaf=1, min_samples_split=9, n_estimators=42; total time=   0.7s
[CV] END bootstrap=False, criterion=entropy, max_depth=20, max_features=None, min_samples_leaf=1, min_samples_split=9, n_estimators=42; total time=   0.7s
[CV] END bootstrap=False, criterion=entropy, max_depth=20, max_features=None, min_samples_leaf=1, min_samples_split=9, n_estimators=42; total time=   0.7s
[CV] END bootstrap=True, criterion=entropy, max_depth=30, max_features=None, min_samples_leaf=3, min_samples_split=7, n_estimators=57; total time=   0.6s
[CV] END bootstrap=True, criterion=gini, max_depth=20, max_features=None, min_samples_leaf=3, min_samples_split=7, n_estimators=53; total time=   0.4s
[CV] END bootstrap=True, criterion=entropy, max_depth=30, max_features=None, min_samples_leaf=3, min_samples_split=7, n_estimators=57; total time=   0.6s
[CV] END bootstrap=True, criterion=entropy, max_depth=30, max_features=None, min_samples_leaf=3, min_samples_split=7, n_estimators=57; total time=   0.6s
[CV] END bootstrap=True, criterion=gini, max_depth=20, max_features=None, min_samples_leaf=3, min_samples_split=7, n_estimators=53; total time=   0.5s
[CV] END bootstrap=True, criterion=entropy, max_depth=30, max_features=None, min_samples_leaf=3, min_samples_split=7, n_estimators=57; total time=   0.6s
[CV] END bootstrap=True, criterion=entropy, max_depth=30, max_features=None, min_samples_leaf=3, min_samples_split=7, n_estimators=57; total time=   0.6s
[CV] END bootstrap=True, criterion=gini, max_depth=20, max_features=None, min_samples_leaf=3, min_samples_split=7, n_estimators=53; total time=   0.5s
[CV] END bootstrap=False, criterion=gini, max_depth=10, max_features=sqrt, min_samples_leaf=1, min_samples_split=2, n_estimators=47; total time=   0.2s
[CV] END bootstrap=False, criterion=gini, max_depth=10, max_features=sqrt, min_samples_leaf=1, min_samples_split=2, n_estimators=47; total time=   0.2s
[CV] END bootstrap=False, criterion=gini, max_depth=10, max_features=sqrt, min_samples_leaf=1, min_samples_split=2, n_estimators=47; total time=   0.2s
[CV] END bootstrap=False, criterion=gini, max_depth=10, max_features=sqrt, min_samples_leaf=1, min_samples_split=2, n_estimators=47; total time=   0.2s
[CV] END bootstrap=False, criterion=gini, max_depth=10, max_features=sqrt, min_samples_leaf=1, min_samples_split=2, n_estimators=47; total time=   0.3s
[CV] END bootstrap=True, criterion=entropy, max_depth=30, max_features=sqrt, min_samples_leaf=2, min_samples_split=6, n_estimators=51; total time=   0.2s
[CV] END bootstrap=True, criterion=gini, max_depth=20, max_features=None, min_samples_leaf=3, min_samples_split=7, n_estimators=53; total time=   0.5s
[CV] END bootstrap=True, criterion=entropy, max_depth=30, max_features=sqrt, min_samples_leaf=2, min_samples_split=6, n_estimators=51; total time=   0.2s
[CV] END bootstrap=True, criterion=gini, max_depth=20, max_features=None, min_samples_leaf=3, min_samples_split=7, n_estimators=53; total time=   0.5s
[CV] END bootstrap=True, criterion=entropy, max_depth=30, max_features=sqrt, min_samples_leaf=2, min_samples_split=6, n_estimators=51; total time=   0.2s
[CV] END bootstrap=True, criterion=entropy, max_depth=30, max_features=sqrt, min_samples_leaf=2, min_samples_split=6, n_estimators=51; total time=   0.2s
[CV] END bootstrap=True, criterion=entropy, max_depth=30, max_features=sqrt, min_samples_leaf=2, min_samples_split=6, n_estimators=51; total time=   0.2s
[CV] END bootstrap=False, criterion=entropy, max_depth=None, max_features=sqrt, min_samples_leaf=1, min_samples_split=7, n_estimators=48; total time=   0.3s
[CV] END bootstrap=False, criterion=entropy, max_depth=None, max_features=sqrt, min_samples_leaf=1, min_samples_split=7, n_estimators=48; total time=   0.3s
[CV] END bootstrap=False, criterion=entropy, max_depth=None, max_features=sqrt, min_samples_leaf=1, min_samples_split=7, n_estimators=48; total time=   0.2s
[CV] END bootstrap=True, criterion=entropy, max_depth=None, max_features=None, min_samples_leaf=4, min_samples_split=4, n_estimators=47; total time=   0.5s
[CV] END bootstrap=True, criterion=entropy, max_depth=None, max_features=None, min_samples_leaf=4, min_samples_split=4, n_estimators=47; total time=   0.5s
[CV] END bootstrap=True, criterion=entropy, max_depth=None, max_features=None, min_samples_leaf=4, min_samples_split=4, n_estimators=47; total time=   0.5s
[CV] END bootstrap=True, criterion=entropy, max_depth=None, max_features=None, min_samples_leaf=4, min_samples_split=4, n_estimators=47; total time=   0.5s
[CV] END bootstrap=True, criterion=entropy, max_depth=None, max_features=None, min_samples_leaf=4, min_samples_split=4, n_estimators=47; total time=   0.5s
[CV] END bootstrap=False, criterion=entropy, max_depth=None, max_features=sqrt, min_samples_leaf=1, min_samples_split=7, n_estimators=48; total time=   0.3s
[CV] END bootstrap=False, criterion=entropy, max_depth=None, max_features=sqrt, min_samples_leaf=1, min_samples_split=7, n_estimators=48; total time=   0.3s
[CV] END bootstrap=False, criterion=entropy, max_depth=10, max_features=sqrt, min_samples_leaf=4, min_samples_split=10, n_estimators=54; total time=   0.3s
[CV] END bootstrap=False, criterion=entropy, max_depth=10, max_features=sqrt, min_samples_leaf=4, min_samples_split=10, n_estimators=54; total time=   0.3s
[CV] END bootstrap=False, criterion=entropy, max_depth=10, max_features=sqrt, min_samples_leaf=4, min_samples_split=10, n_estimators=54; total time=   0.3s
[CV] END bootstrap=False, criterion=entropy, max_depth=10, max_features=sqrt, min_samples_leaf=4, min_samples_split=10, n_estimators=54; total time=   0.3s
[CV] END bootstrap=False, criterion=entropy, max_depth=10, max_features=sqrt, min_samples_leaf=4, min_samples_split=10, n_estimators=54; total time=   0.3s
[CV] END bootstrap=True, criterion=gini, max_depth=None, max_features=sqrt, min_samples_leaf=4, min_samples_split=5, n_estimators=41; total time=   0.1s
[CV] END bootstrap=True, criterion=gini, max_depth=None, max_features=sqrt, min_samples_leaf=4, min_samples_split=5, n_estimators=41; total time=   0.1s
[CV] END bootstrap=True, criterion=gini, max_depth=None, max_features=sqrt, min_samples_leaf=4, min_samples_split=5, n_estimators=41; total time=   0.1s
[CV] END bootstrap=True, criterion=gini, max_depth=None, max_features=sqrt, min_samples_leaf=4, min_samples_split=5, n_estimators=41; total time=   0.1s
[CV] END bootstrap=True, criterion=gini, max_depth=None, max_features=sqrt, min_samples_leaf=4, min_samples_split=5, n_estimators=41; total time=   0.1s
[CV] END bootstrap=False, criterion=entropy, max_depth=None, max_features=None, min_samples_leaf=3, min_samples_split=5, n_estimators=42; total time=   0.7s
[CV] END bootstrap=False, criterion=entropy, max_depth=None, max_features=None, min_samples_leaf=3, min_samples_split=5, n_estimators=42; total time=   0.7s
[CV] END bootstrap=False, criterion=entropy, max_depth=None, max_features=None, min_samples_leaf=3, min_samples_split=5, n_estimators=42; total time=   0.7s
[CV] END bootstrap=False, criterion=entropy, max_depth=None, max_features=None, min_samples_leaf=3, min_samples_split=5, n_estimators=42; total time=   0.7s
[CV] END bootstrap=False, criterion=entropy, max_depth=None, max_features=None, min_samples_leaf=3, min_samples_split=5, n_estimators=42; total time=   0.7s
[CV] END bootstrap=True, criterion=gini, max_depth=30, max_features=None, min_samples_leaf=3, min_samples_split=2, n_estimators=58; total time=   0.5s
[CV] END bootstrap=False, criterion=gini, max_depth=20, max_features=log2, min_samples_leaf=1, min_samples_split=5, n_estimators=40; total time=   0.2s
[CV] END bootstrap=False, criterion=gini, max_depth=20, max_features=log2, min_samples_leaf=1, min_samples_split=5, n_estimators=40; total time=   0.2s
[CV] END bootstrap=False, criterion=gini, max_depth=20, max_features=log2, min_samples_leaf=1, min_samples_split=5, n_estimators=40; total time=   0.2s
[CV] END bootstrap=True, criterion=gini, max_depth=30, max_features=None, min_samples_leaf=3, min_samples_split=2, n_estimators=58; total time=   0.5s
[CV] END bootstrap=True, criterion=gini, max_depth=30, max_features=None, min_samples_leaf=3, min_samples_split=2, n_estimators=58; total time=   0.6s
[CV] END bootstrap=False, criterion=gini, max_depth=20, max_features=log2, min_samples_leaf=1, min_samples_split=5, n_estimators=40; total time=   0.2s
[CV] END bootstrap=False, criterion=gini, max_depth=20, max_features=log2, min_samples_leaf=1, min_samples_split=5, n_estimators=40; total time=   0.2s
[CV] END bootstrap=True, criterion=gini, max_depth=30, max_features=None, min_samples_leaf=3, min_samples_split=2, n_estimators=58; total time=   0.5s
[CV] END bootstrap=True, criterion=gini, max_depth=10, max_features=log2, min_samples_leaf=4, min_samples_split=2, n_estimators=52; total time=   0.2s
[CV] END bootstrap=True, criterion=gini, max_depth=10, max_features=log2, min_samples_leaf=4, min_samples_split=2, n_estimators=52; total time=   0.2s
[CV] END bootstrap=True, criterion=gini, max_depth=10, max_features=log2, min_samples_leaf=4, min_samples_split=2, n_estimators=52; total time=   0.2s
[CV] END bootstrap=True, criterion=gini, max_depth=10, max_features=log2, min_samples_leaf=4, min_samples_split=2, n_estimators=52; total time=   0.2s
[CV] END bootstrap=True, criterion=gini, max_depth=10, max_features=log2, min_samples_leaf=4, min_samples_split=2, n_estimators=52; total time=   0.2s
[CV] END bootstrap=True, criterion=gini, max_depth=30, max_features=None, min_samples_leaf=3, min_samples_split=2, n_estimators=58; total time=   0.6s
[CV] END bootstrap=False, criterion=gini, max_depth=10, max_features=log2, min_samples_leaf=1, min_samples_split=5, n_estimators=59; total time=   0.3s
[CV] END bootstrap=False, criterion=gini, max_depth=10, max_features=log2, min_samples_leaf=1, min_samples_split=5, n_estimators=59; total time=   0.3s
[CV] END bootstrap=False, criterion=gini, max_depth=10, max_features=log2, min_samples_leaf=1, min_samples_split=5, n_estimators=59; total time=   0.3s
[CV] END bootstrap=False, criterion=gini, max_depth=10, max_features=log2, min_samples_leaf=1, min_samples_split=5, n_estimators=59; total time=   0.3s
[CV] END bootstrap=False, criterion=gini, max_depth=10, max_features=log2, min_samples_leaf=1, min_samples_split=5, n_estimators=59; total time=   0.3s
[CV] END bootstrap=False, criterion=gini, max_depth=30, max_features=log2, min_samples_leaf=4, min_samples_split=5, n_estimators=55; total time=   0.2s
[CV] END bootstrap=False, criterion=gini, max_depth=30, max_features=log2, min_samples_leaf=4, min_samples_split=5, n_estimators=55; total time=   0.2s
[CV] END bootstrap=False, criterion=gini, max_depth=30, max_features=log2, min_samples_leaf=4, min_samples_split=5, n_estimators=55; total time=   0.2s
[CV] END bootstrap=True, criterion=gini, max_depth=30, max_features=log2, min_samples_leaf=1, min_samples_split=6, n_estimators=40; total time=   0.1s
[CV] END bootstrap=False, criterion=gini, max_depth=30, max_features=log2, min_samples_leaf=4, min_samples_split=5, n_estimators=55; total time=   0.2s
[CV] END bootstrap=True, criterion=gini, max_depth=30, max_features=log2, min_samples_leaf=1, min_samples_split=6, n_estimators=40; total time=   0.2s
[CV] END bootstrap=True, criterion=gini, max_depth=30, max_features=log2, min_samples_leaf=1, min_samples_split=6, n_estimators=40; total time=   0.1s
[CV] END bootstrap=True, criterion=gini, max_depth=30, max_features=log2, min_samples_leaf=1, min_samples_split=6, n_estimators=40; total time=   0.1s
[CV] END bootstrap=False, criterion=gini, max_depth=30, max_features=log2, min_samples_leaf=4, min_samples_split=5, n_estimators=55; total time=   0.2s
[CV] END bootstrap=True, criterion=gini, max_depth=30, max_features=log2, min_samples_leaf=1, min_samples_split=6, n_estimators=40; total time=   0.1s
[CV] END bootstrap=False, criterion=entropy, max_depth=None, max_features=sqrt, min_samples_leaf=3, min_samples_split=8, n_estimators=55; total time=   0.3s
[CV] END bootstrap=False, criterion=entropy, max_depth=None, max_features=sqrt, min_samples_leaf=3, min_samples_split=8, n_estimators=55; total time=   0.3s
[CV] END bootstrap=False, criterion=entropy, max_depth=None, max_features=sqrt, min_samples_leaf=3, min_samples_split=8, n_estimators=55; total time=   0.3s
[CV] END bootstrap=False, criterion=entropy, max_depth=None, max_features=sqrt, min_samples_leaf=3, min_samples_split=8, n_estimators=55; total time=   0.3s
[CV] END bootstrap=False, criterion=entropy, max_depth=None, max_features=sqrt, min_samples_leaf=3, min_samples_split=8, n_estimators=55; total time=   0.3s
[CV] END bootstrap=False, criterion=entropy, max_depth=30, max_features=sqrt, min_samples_leaf=4, min_samples_split=2, n_estimators=50; total time=   0.3s
[CV] END bootstrap=False, criterion=entropy, max_depth=30, max_features=sqrt, min_samples_leaf=4, min_samples_split=2, n_estimators=50; total time=   0.3s
[CV] END bootstrap=False, criterion=entropy, max_depth=30, max_features=sqrt, min_samples_leaf=4, min_samples_split=2, n_estimators=50; total time=   0.3s
[CV] END bootstrap=True, criterion=gini, max_depth=None, max_features=log2, min_samples_leaf=3, min_samples_split=3, n_estimators=44; total time=   0.2s
[CV] END bootstrap=True, criterion=gini, max_depth=None, max_features=log2, min_samples_leaf=3, min_samples_split=3, n_estimators=44; total time=   0.1s
[CV] END bootstrap=True, criterion=gini, max_depth=None, max_features=log2, min_samples_leaf=3, min_samples_split=3, n_estimators=44; total time=   0.1s
[CV] END bootstrap=False, criterion=entropy, max_depth=30, max_features=sqrt, min_samples_leaf=4, min_samples_split=2, n_estimators=50; total time=   0.3s
[CV] END bootstrap=True, criterion=gini, max_depth=None, max_features=log2, min_samples_leaf=3, min_samples_split=3, n_estimators=44; total time=   0.2s
[CV] END bootstrap=False, criterion=entropy, max_depth=30, max_features=sqrt, min_samples_leaf=4, min_samples_split=2, n_estimators=50; total time=   0.3s
[CV] END bootstrap=True, criterion=gini, max_depth=None, max_features=log2, min_samples_leaf=3, min_samples_split=3, n_estimators=44; total time=   0.1s
[CV] END bootstrap=True, criterion=gini, max_depth=None, max_features=log2, min_samples_leaf=2, min_samples_split=9, n_estimators=43; total time=   0.2s
[CV] END bootstrap=True, criterion=gini, max_depth=None, max_features=log2, min_samples_leaf=2, min_samples_split=9, n_estimators=43; total time=   0.2s
[CV] END bootstrap=True, criterion=gini, max_depth=None, max_features=log2, min_samples_leaf=2, min_samples_split=9, n_estimators=43; total time=   0.1s
[CV] END bootstrap=True, criterion=gini, max_depth=None, max_features=log2, min_samples_leaf=2, min_samples_split=9, n_estimators=43; total time=   0.1s
[CV] END bootstrap=True, criterion=gini, max_depth=None, max_features=log2, min_samples_leaf=2, min_samples_split=9, n_estimators=43; total time=   0.1s
[CV] END bootstrap=False, criterion=gini, max_depth=20, max_features=log2, min_samples_leaf=2, min_samples_split=8, n_estimators=46; total time=   0.2s
[CV] END bootstrap=False, criterion=gini, max_depth=20, max_features=log2, min_samples_leaf=2, min_samples_split=8, n_estimators=46; total time=   0.2s
[CV] END bootstrap=False, criterion=gini, max_depth=20, max_features=log2, min_samples_leaf=2, min_samples_split=8, n_estimators=46; total time=   0.2s
[CV] END bootstrap=True, criterion=gini, max_depth=20, max_features=log2, min_samples_leaf=1, min_samples_split=8, n_estimators=40; total time=   0.2s
[CV] END bootstrap=True, criterion=gini, max_depth=20, max_features=log2, min_samples_leaf=1, min_samples_split=8, n_estimators=40; total time=   0.1s
[CV] END bootstrap=False, criterion=gini, max_depth=20, max_features=log2, min_samples_leaf=2, min_samples_split=8, n_estimators=46; total time=   0.2s
[CV] END bootstrap=True, criterion=gini, max_depth=20, max_features=log2, min_samples_leaf=1, min_samples_split=8, n_estimators=40; total time=   0.1s
[CV] END bootstrap=False, criterion=gini, max_depth=20, max_features=log2, min_samples_leaf=2, min_samples_split=8, n_estimators=46; total time=   0.2s
[CV] END bootstrap=True, criterion=gini, max_depth=20, max_features=log2, min_samples_leaf=1, min_samples_split=8, n_estimators=40; total time=   0.1s
[CV] END bootstrap=True, criterion=gini, max_depth=20, max_features=log2, min_samples_leaf=1, min_samples_split=8, n_estimators=40; total time=   0.1s
[CV] END bootstrap=True, criterion=entropy, max_depth=None, max_features=sqrt, min_samples_leaf=1, min_samples_split=5, n_estimators=55; total time=   0.2s
[CV] END bootstrap=True, criterion=entropy, max_depth=None, max_features=sqrt, min_samples_leaf=1, min_samples_split=5, n_estimators=55; total time=   0.2s
[CV] END bootstrap=True, criterion=entropy, max_depth=None, max_features=sqrt, min_samples_leaf=1, min_samples_split=5, n_estimators=55; total time=   0.2s
[CV] END bootstrap=True, criterion=entropy, max_depth=None, max_features=sqrt, min_samples_leaf=1, min_samples_split=5, n_estimators=55; total time=   0.2s
[CV] END bootstrap=True, criterion=entropy, max_depth=None, max_features=sqrt, min_samples_leaf=1, min_samples_split=5, n_estimators=55; total time=   0.2s
[CV] END bootstrap=True, criterion=entropy, max_depth=None, max_features=log2, min_samples_leaf=4, min_samples_split=7, n_estimators=54; total time=   0.2s
[CV] END bootstrap=True, criterion=entropy, max_depth=None, max_features=log2, min_samples_leaf=4, min_samples_split=7, n_estimators=54; total time=   0.2s
[CV] END bootstrap=True, criterion=entropy, max_depth=None, max_features=log2, min_samples_leaf=4, min_samples_split=7, n_estimators=54; total time=   0.2s
[CV] END bootstrap=True, criterion=entropy, max_depth=None, max_features=log2, min_samples_leaf=4, min_samples_split=7, n_estimators=54; total time=   0.2s
[CV] END bootstrap=True, criterion=entropy, max_depth=None, max_features=log2, min_samples_leaf=4, min_samples_split=7, n_estimators=54; total time=   0.2s
[CV] END bootstrap=False, criterion=gini, max_depth=20, max_features=sqrt, min_samples_leaf=4, min_samples_split=7, n_estimators=58; total time=   0.2s
[CV] END bootstrap=False, criterion=gini, max_depth=20, max_features=sqrt, min_samples_leaf=4, min_samples_split=7, n_estimators=58; total time=   0.2s
[CV] END bootstrap=False, criterion=gini, max_depth=20, max_features=sqrt, min_samples_leaf=4, min_samples_split=7, n_estimators=58; total time=   0.2s
[CV] END bootstrap=True, criterion=entropy, max_depth=None, max_features=None, min_samples_leaf=1, min_samples_split=8, n_estimators=55; total time=   0.6s
[CV] END bootstrap=True, criterion=entropy, max_depth=None, max_features=None, min_samples_leaf=1, min_samples_split=8, n_estimators=55; total time=   0.6s
[CV] END bootstrap=True, criterion=entropy, max_depth=None, max_features=None, min_samples_leaf=1, min_samples_split=8, n_estimators=55; total time=   0.6s
[CV] END bootstrap=False, criterion=gini, max_depth=20, max_features=sqrt, min_samples_leaf=4, min_samples_split=7, n_estimators=58; total time=   0.2s
[CV] END bootstrap=True, criterion=entropy, max_depth=None, max_features=sqrt, min_samples_leaf=3, min_samples_split=8, n_estimators=52; total time=   0.2s
[CV] END bootstrap=False, criterion=gini, max_depth=20, max_features=sqrt, min_samples_leaf=4, min_samples_split=7, n_estimators=58; total time=   0.3s
[CV] END bootstrap=True, criterion=entropy, max_depth=None, max_features=None, min_samples_leaf=1, min_samples_split=8, n_estimators=55; total time=   0.6s
[CV] END bootstrap=True, criterion=entropy, max_depth=None, max_features=None, min_samples_leaf=1, min_samples_split=8, n_estimators=55; total time=   0.6s
[CV] END bootstrap=True, criterion=entropy, max_depth=None, max_features=sqrt, min_samples_leaf=3, min_samples_split=8, n_estimators=52; total time=   0.2s
[CV] END bootstrap=True, criterion=entropy, max_depth=None, max_features=sqrt, min_samples_leaf=3, min_samples_split=8, n_estimators=52; total time=   0.2s
[CV] END bootstrap=True, criterion=entropy, max_depth=None, max_features=sqrt, min_samples_leaf=3, min_samples_split=8, n_estimators=52; total time=   0.2s
[CV] END bootstrap=True, criterion=gini, max_depth=30, max_features=log2, min_samples_leaf=1, min_samples_split=9, n_estimators=47; total time=   0.2s
[CV] END bootstrap=True, criterion=entropy, max_depth=None, max_features=sqrt, min_samples_leaf=3, min_samples_split=8, n_estimators=52; total time=   0.2s
[CV] END bootstrap=True, criterion=gini, max_depth=30, max_features=log2, min_samples_leaf=1, min_samples_split=9, n_estimators=47; total time=   0.2s
[CV] END bootstrap=True, criterion=gini, max_depth=30, max_features=log2, min_samples_leaf=1, min_samples_split=9, n_estimators=47; total time=   0.2s
[CV] END bootstrap=True, criterion=gini, max_depth=30, max_features=log2, min_samples_leaf=1, min_samples_split=9, n_estimators=47; total time=   0.2s
[CV] END bootstrap=True, criterion=gini, max_depth=30, max_features=log2, min_samples_leaf=1, min_samples_split=9, n_estimators=47; total time=   0.2s
[CV] END bootstrap=False, criterion=entropy, max_depth=None, max_features=log2, min_samples_leaf=3, min_samples_split=2, n_estimators=45; total time=   0.2s
[CV] END bootstrap=False, criterion=entropy, max_depth=None, max_features=log2, min_samples_leaf=3, min_samples_split=2, n_estimators=45; total time=   0.2s
[CV] END bootstrap=False, criterion=entropy, max_depth=None, max_features=log2, min_samples_leaf=3, min_samples_split=2, n_estimators=45; total time=   0.2s
[CV] END bootstrap=False, criterion=entropy, max_depth=None, max_features=log2, min_samples_leaf=3, min_samples_split=2, n_estimators=45; total time=   0.2s
[CV] END bootstrap=True, criterion=entropy, max_depth=30, max_features=log2, min_samples_leaf=1, min_samples_split=7, n_estimators=49; total time=   0.2s
[CV] END bootstrap=False, criterion=entropy, max_depth=None, max_features=log2, min_samples_leaf=3, min_samples_split=2, n_estimators=45; total time=   0.2s
[CV] END bootstrap=True, criterion=entropy, max_depth=30, max_features=log2, min_samples_leaf=1, min_samples_split=7, n_estimators=49; total time=   0.2s
[CV] END bootstrap=True, criterion=entropy, max_depth=30, max_features=log2, min_samples_leaf=1, min_samples_split=7, n_estimators=49; total time=   0.2s
[CV] END bootstrap=True, criterion=entropy, max_depth=30, max_features=log2, min_samples_leaf=1, min_samples_split=7, n_estimators=49; total time=   0.2s
[CV] END bootstrap=True, criterion=entropy, max_depth=30, max_features=log2, min_samples_leaf=1, min_samples_split=7, n_estimators=49; total time=   0.2s
[CV] END bootstrap=True, criterion=entropy, max_depth=30, max_features=sqrt, min_samples_leaf=3, min_samples_split=2, n_estimators=47; total time=   0.2s
[CV] END bootstrap=True, criterion=entropy, max_depth=30, max_features=sqrt, min_samples_leaf=3, min_samples_split=2, n_estimators=47; total time=   0.2s
[CV] END bootstrap=True, criterion=entropy, max_depth=30, max_features=sqrt, min_samples_leaf=3, min_samples_split=2, n_estimators=47; total time=   0.2s
[CV] END bootstrap=True, criterion=entropy, max_depth=30, max_features=sqrt, min_samples_leaf=3, min_samples_split=2, n_estimators=47; total time=   0.2s
[CV] END bootstrap=True, criterion=entropy, max_depth=30, max_features=sqrt, min_samples_leaf=3, min_samples_split=2, n_estimators=47; total time=   0.2s
[CV] END bootstrap=True, criterion=gini, max_depth=20, max_features=None, min_samples_leaf=4, min_samples_split=8, n_estimators=52; total time=   0.4s
[CV] END bootstrap=True, criterion=gini, max_depth=20, max_features=None, min_samples_leaf=4, min_samples_split=8, n_estimators=52; total time=   0.4s
[CV] END bootstrap=True, criterion=gini, max_depth=20, max_features=None, min_samples_leaf=4, min_samples_split=8, n_estimators=52; total time=   0.4s
[CV] END bootstrap=True, criterion=gini, max_depth=20, max_features=None, min_samples_leaf=4, min_samples_split=8, n_estimators=52; total time=   0.5s
[CV] END bootstrap=True, criterion=gini, max_depth=20, max_features=None, min_samples_leaf=4, min_samples_split=8, n_estimators=52; total time=   0.4s
[CV] END bootstrap=True, criterion=entropy, max_depth=20, max_features=log2, min_samples_leaf=3, min_samples_split=7, n_estimators=49; total time=   0.2s
[CV] END bootstrap=True, criterion=entropy, max_depth=20, max_features=log2, min_samples_leaf=3, min_samples_split=7, n_estimators=49; total time=   0.2s
[CV] END bootstrap=True, criterion=entropy, max_depth=20, max_features=log2, min_samples_leaf=3, min_samples_split=7, n_estimators=49; total time=   0.2s
[CV] END bootstrap=False, criterion=gini, max_depth=30, max_features=None, min_samples_leaf=2, min_samples_split=8, n_estimators=53; total time=   0.7s
[CV] END bootstrap=False, criterion=gini, max_depth=30, max_features=None, min_samples_leaf=2, min_samples_split=8, n_estimators=53; total time=   0.7s
[CV] END bootstrap=False, criterion=gini, max_depth=30, max_features=None, min_samples_leaf=2, min_samples_split=8, n_estimators=53; total time=   0.8s
[CV] END bootstrap=True, criterion=entropy, max_depth=20, max_features=log2, min_samples_leaf=3, min_samples_split=7, n_estimators=49; total time=   0.2s
[CV] END bootstrap=True, criterion=entropy, max_depth=20, max_features=log2, min_samples_leaf=3, min_samples_split=7, n_estimators=49; total time=   0.2s
[CV] END bootstrap=False, criterion=gini, max_depth=30, max_features=None, min_samples_leaf=2, min_samples_split=8, n_estimators=53; total time=   0.7s
[CV] END bootstrap=False, criterion=entropy, max_depth=None, max_features=sqrt, min_samples_leaf=3, min_samples_split=6, n_estimators=52; total time=   0.3s
[CV] END bootstrap=False, criterion=entropy, max_depth=None, max_features=sqrt, min_samples_leaf=3, min_samples_split=6, n_estimators=52; total time=   0.3s
[CV] END bootstrap=True, criterion=gini, max_depth=30, max_features=sqrt, min_samples_leaf=1, min_samples_split=2, n_estimators=56; total time=   0.2s
[CV] END bootstrap=False, criterion=entropy, max_depth=None, max_features=sqrt, min_samples_leaf=3, min_samples_split=6, n_estimators=52; total time=   0.3s
[CV] END bootstrap=False, criterion=entropy, max_depth=None, max_features=sqrt, min_samples_leaf=3, min_samples_split=6, n_estimators=52; total time=   0.3s
[CV] END bootstrap=False, criterion=entropy, max_depth=None, max_features=sqrt, min_samples_leaf=3, min_samples_split=6, n_estimators=52; total time=   0.3s
[CV] END bootstrap=False, criterion=gini, max_depth=30, max_features=None, min_samples_leaf=2, min_samples_split=8, n_estimators=53; total time=   0.7s
[CV] END bootstrap=True, criterion=gini, max_depth=30, max_features=sqrt, min_samples_leaf=1, min_samples_split=2, n_estimators=56; total time=   0.2s
[CV] END bootstrap=True, criterion=gini, max_depth=30, max_features=sqrt, min_samples_leaf=1, min_samples_split=2, n_estimators=56; total time=   0.2s
[CV] END bootstrap=True, criterion=gini, max_depth=30, max_features=sqrt, min_samples_leaf=1, min_samples_split=2, n_estimators=56; total time=   0.2s
[CV] END bootstrap=True, criterion=gini, max_depth=30, max_features=sqrt, min_samples_leaf=1, min_samples_split=2, n_estimators=56; total time=   0.2s
[CV] END bootstrap=False, criterion=gini, max_depth=10, max_features=sqrt, min_samples_leaf=1, min_samples_split=2, n_estimators=48; total time=   0.2s
[CV] END bootstrap=False, criterion=gini, max_depth=10, max_features=sqrt, min_samples_leaf=1, min_samples_split=2, n_estimators=48; total time=   0.2s
[CV] END bootstrap=False, criterion=gini, max_depth=10, max_features=sqrt, min_samples_leaf=1, min_samples_split=2, n_estimators=48; total time=   0.2s
[CV] END bootstrap=True, criterion=entropy, max_depth=20, max_features=None, min_samples_leaf=2, min_samples_split=5, n_estimators=49; total time=   0.5s
[CV] END bootstrap=True, criterion=entropy, max_depth=20, max_features=None, min_samples_leaf=2, min_samples_split=5, n_estimators=49; total time=   0.5s
[CV] END bootstrap=True, criterion=gini, max_depth=30, max_features=sqrt, min_samples_leaf=2, min_samples_split=6, n_estimators=44; total time=   0.1s
[CV] END bootstrap=False, criterion=gini, max_depth=10, max_features=sqrt, min_samples_leaf=1, min_samples_split=2, n_estimators=48; total time=   0.2s
[CV] END bootstrap=True, criterion=entropy, max_depth=20, max_features=None, min_samples_leaf=2, min_samples_split=5, n_estimators=49; total time=   0.5s
[CV] END bootstrap=True, criterion=entropy, max_depth=20, max_features=None, min_samples_leaf=2, min_samples_split=5, n_estimators=49; total time=   0.5s
[CV] END bootstrap=False, criterion=gini, max_depth=10, max_features=sqrt, min_samples_leaf=1, min_samples_split=2, n_estimators=48; total time=   0.2s
[CV] END bootstrap=True, criterion=entropy, max_depth=20, max_features=None, min_samples_leaf=2, min_samples_split=5, n_estimators=49; total time=   0.6s
[CV] END bootstrap=True, criterion=gini, max_depth=30, max_features=sqrt, min_samples_leaf=2, min_samples_split=6, n_estimators=44; total time=   0.2s
[CV] END bootstrap=True, criterion=gini, max_depth=30, max_features=sqrt, min_samples_leaf=2, min_samples_split=6, n_estimators=44; total time=   0.2s
[CV] END bootstrap=True, criterion=gini, max_depth=30, max_features=sqrt, min_samples_leaf=2, min_samples_split=6, n_estimators=44; total time=   0.2s
[CV] END bootstrap=True, criterion=gini, max_depth=30, max_features=sqrt, min_samples_leaf=2, min_samples_split=6, n_estimators=44; total time=   0.2s
[CV] END bootstrap=True, criterion=gini, max_depth=10, max_features=log2, min_samples_leaf=4, min_samples_split=5, n_estimators=54; total time=   0.2s
[CV] END bootstrap=True, criterion=gini, max_depth=10, max_features=log2, min_samples_leaf=4, min_samples_split=5, n_estimators=54; total time=   0.2s
[CV] END bootstrap=True, criterion=gini, max_depth=10, max_features=log2, min_samples_leaf=4, min_samples_split=5, n_estimators=54; total time=   0.2s
[CV] END bootstrap=True, criterion=gini, max_depth=10, max_features=log2, min_samples_leaf=4, min_samples_split=5, n_estimators=54; total time=   0.2s
[CV] END bootstrap=True, criterion=gini, max_depth=10, max_features=log2, min_samples_leaf=4, min_samples_split=5, n_estimators=54; total time=   0.2s
[CV] END bootstrap=True, criterion=gini, max_depth=20, max_features=log2, min_samples_leaf=3, min_samples_split=10, n_estimators=56; total time=   0.2s
[CV] END bootstrap=True, criterion=gini, max_depth=20, max_features=log2, min_samples_leaf=3, min_samples_split=10, n_estimators=56; total time=   0.2s
[CV] END bootstrap=True, criterion=gini, max_depth=20, max_features=log2, min_samples_leaf=3, min_samples_split=10, n_estimators=56; total time=   0.2s
[CV] END bootstrap=True, criterion=entropy, max_depth=30, max_features=None, min_samples_leaf=4, min_samples_split=7, n_estimators=42; total time=   0.4s
[CV] END bootstrap=True, criterion=entropy, max_depth=30, max_features=None, min_samples_leaf=4, min_samples_split=7, n_estimators=42; total time=   0.4s
[CV] END bootstrap=True, criterion=entropy, max_depth=30, max_features=None, min_samples_leaf=4, min_samples_split=7, n_estimators=42; total time=   0.5s
[CV] END bootstrap=True, criterion=gini, max_depth=20, max_features=log2, min_samples_leaf=3, min_samples_split=10, n_estimators=56; total time=   0.2s
[CV] END bootstrap=True, criterion=entropy, max_depth=30, max_features=None, min_samples_leaf=4, min_samples_split=7, n_estimators=42; total time=   0.4s
[CV] END bootstrap=True, criterion=gini, max_depth=20, max_features=log2, min_samples_leaf=3, min_samples_split=10, n_estimators=56; total time=   0.2s
[CV] END bootstrap=True, criterion=entropy, max_depth=30, max_features=None, min_samples_leaf=4, min_samples_split=7, n_estimators=42; total time=   0.4s
[CV] END bootstrap=True, criterion=entropy, max_depth=10, max_features=sqrt, min_samples_leaf=1, min_samples_split=8, n_estimators=46; total time=   0.2s
[CV] END bootstrap=True, criterion=entropy, max_depth=10, max_features=sqrt, min_samples_leaf=1, min_samples_split=8, n_estimators=46; total time=   0.2s
[CV] END bootstrap=True, criterion=entropy, max_depth=10, max_features=sqrt, min_samples_leaf=1, min_samples_split=8, n_estimators=46; total time=   0.2s
[CV] END bootstrap=True, criterion=entropy, max_depth=10, max_features=sqrt, min_samples_leaf=1, min_samples_split=8, n_estimators=46; total time=   0.2s
[CV] END bootstrap=True, criterion=entropy, max_depth=10, max_features=sqrt, min_samples_leaf=1, min_samples_split=8, n_estimators=46; total time=   0.2s
[CV] END bootstrap=True, criterion=gini, max_depth=20, max_features=sqrt, min_samples_leaf=3, min_samples_split=6, n_estimators=58; total time=   0.2s
[CV] END bootstrap=True, criterion=gini, max_depth=20, max_features=sqrt, min_samples_leaf=3, min_samples_split=6, n_estimators=58; total time=   0.2s
[CV] END bootstrap=True, criterion=gini, max_depth=20, max_features=sqrt, min_samples_leaf=3, min_samples_split=6, n_estimators=58; total time=   0.2s
[CV] END bootstrap=True, criterion=gini, max_depth=None, max_features=None, min_samples_leaf=3, min_samples_split=3, n_estimators=46; total time=   0.4s
[CV] END bootstrap=True, criterion=gini, max_depth=None, max_features=None, min_samples_leaf=3, min_samples_split=3, n_estimators=46; total time=   0.4s
[CV] END bootstrap=True, criterion=gini, max_depth=None, max_features=None, min_samples_leaf=3, min_samples_split=3, n_estimators=46; total time=   0.4s
[CV] END bootstrap=True, criterion=gini, max_depth=None, max_features=None, min_samples_leaf=3, min_samples_split=3, n_estimators=46; total time=   0.4s
[CV] END bootstrap=True, criterion=gini, max_depth=20, max_features=sqrt, min_samples_leaf=3, min_samples_split=6, n_estimators=58; total time=   0.2s
[CV] END bootstrap=True, criterion=gini, max_depth=None, max_features=None, min_samples_leaf=3, min_samples_split=3, n_estimators=46; total time=   0.4s
[CV] END bootstrap=True, criterion=gini, max_depth=20, max_features=sqrt, min_samples_leaf=3, min_samples_split=6, n_estimators=58; total time=   0.2s
[CV] END bootstrap=True, criterion=entropy, max_depth=10, max_features=None, min_samples_leaf=4, min_samples_split=10, n_estimators=48; total time=   0.5s
[CV] END bootstrap=True, criterion=entropy, max_depth=10, max_features=None, min_samples_leaf=4, min_samples_split=10, n_estimators=48; total time=   0.5s
[CV] END bootstrap=True, criterion=entropy, max_depth=10, max_features=None, min_samples_leaf=4, min_samples_split=10, n_estimators=48; total time=   0.5s
[CV] END bootstrap=False, criterion=gini, max_depth=10, max_features=None, min_samples_leaf=3, min_samples_split=4, n_estimators=58; total time=   0.8s
[CV] END bootstrap=False, criterion=gini, max_depth=10, max_features=None, min_samples_leaf=3, min_samples_split=4, n_estimators=58; total time=   0.7s
[CV] END bootstrap=False, criterion=gini, max_depth=10, max_features=None, min_samples_leaf=3, min_samples_split=4, n_estimators=58; total time=   0.7s
[CV] END bootstrap=False, criterion=gini, max_depth=10, max_features=None, min_samples_leaf=3, min_samples_split=4, n_estimators=58; total time=   0.8s
[CV] END bootstrap=True, criterion=gini, max_depth=20, max_features=log2, min_samples_leaf=1, min_samples_split=2, n_estimators=47; total time=   0.2s
[CV] END bootstrap=False, criterion=gini, max_depth=10, max_features=None, min_samples_leaf=3, min_samples_split=4, n_estimators=58; total time=   0.8s
[CV] END bootstrap=True, criterion=gini, max_depth=20, max_features=log2, min_samples_leaf=1, min_samples_split=2, n_estimators=47; total time=   0.2s
[CV] END bootstrap=True, criterion=gini, max_depth=20, max_features=log2, min_samples_leaf=1, min_samples_split=2, n_estimators=47; total time=   0.2s
[CV] END bootstrap=True, criterion=gini, max_depth=20, max_features=log2, min_samples_leaf=1, min_samples_split=2, n_estimators=47; total time=   0.2s
[CV] END bootstrap=True, criterion=gini, max_depth=20, max_features=log2, min_samples_leaf=1, min_samples_split=2, n_estimators=47; total time=   0.2s
[CV] END bootstrap=True, criterion=entropy, max_depth=20, max_features=sqrt, min_samples_leaf=3, min_samples_split=7, n_estimators=57; total time=   0.2s
[CV] END bootstrap=True, criterion=entropy, max_depth=10, max_features=None, min_samples_leaf=4, min_samples_split=10, n_estimators=48; total time=   0.5s
[CV] END bootstrap=True, criterion=entropy, max_depth=20, max_features=sqrt, min_samples_leaf=3, min_samples_split=7, n_estimators=57; total time=   0.2s
[CV] END bootstrap=True, criterion=entropy, max_depth=10, max_features=None, min_samples_leaf=4, min_samples_split=10, n_estimators=48; total time=   0.5s
[CV] END bootstrap=True, criterion=entropy, max_depth=20, max_features=sqrt, min_samples_leaf=3, min_samples_split=7, n_estimators=57; total time=   0.2s
[CV] END bootstrap=True, criterion=entropy, max_depth=20, max_features=sqrt, min_samples_leaf=3, min_samples_split=7, n_estimators=57; total time=   0.2s
[CV] END bootstrap=True, criterion=entropy, max_depth=20, max_features=sqrt, min_samples_leaf=3, min_samples_split=7, n_estimators=57; total time=   0.2s
[CV] END bootstrap=False, criterion=entropy, max_depth=20, max_features=log2, min_samples_leaf=1, min_samples_split=10, n_estimators=54; total time=   0.3s
[CV] END bootstrap=False, criterion=entropy, max_depth=20, max_features=log2, min_samples_leaf=1, min_samples_split=10, n_estimators=54; total time=   0.3s
[CV] END bootstrap=False, criterion=entropy, max_depth=20, max_features=log2, min_samples_leaf=1, min_samples_split=10, n_estimators=54; total time=   0.3s
[CV] END bootstrap=False, criterion=gini, max_depth=10, max_features=sqrt, min_samples_leaf=3, min_samples_split=10, n_estimators=56; total time=   0.2s
[CV] END bootstrap=False, criterion=entropy, max_depth=20, max_features=log2, min_samples_leaf=1, min_samples_split=10, n_estimators=54; total time=   0.3s
[CV] END bootstrap=False, criterion=gini, max_depth=10, max_features=sqrt, min_samples_leaf=3, min_samples_split=10, n_estimators=56; total time=   0.2s
[CV] END bootstrap=False, criterion=entropy, max_depth=20, max_features=log2, min_samples_leaf=1, min_samples_split=10, n_estimators=54; total time=   0.3s
[CV] END bootstrap=False, criterion=gini, max_depth=10, max_features=sqrt, min_samples_leaf=3, min_samples_split=10, n_estimators=56; total time=   0.2s
[CV] END bootstrap=False, criterion=gini, max_depth=10, max_features=sqrt, min_samples_leaf=3, min_samples_split=10, n_estimators=56; total time=   0.2s
[CV] END bootstrap=False, criterion=gini, max_depth=10, max_features=sqrt, min_samples_leaf=3, min_samples_split=10, n_estimators=56; total time=   0.2s
[CV] END bootstrap=False, criterion=entropy, max_depth=None, max_features=sqrt, min_samples_leaf=2, min_samples_split=2, n_estimators=44; total time=   0.2s
[CV] END bootstrap=False, criterion=entropy, max_depth=None, max_features=sqrt, min_samples_leaf=2, min_samples_split=2, n_estimators=44; total time=   0.2s
[CV] END bootstrap=False, criterion=entropy, max_depth=None, max_features=sqrt, min_samples_leaf=2, min_samples_split=2, n_estimators=44; total time=   0.2s
[CV] END bootstrap=False, criterion=entropy, max_depth=None, max_features=sqrt, min_samples_leaf=2, min_samples_split=2, n_estimators=44; total time=   0.2s
[CV] END bootstrap=True, criterion=gini, max_depth=30, max_features=log2, min_samples_leaf=1, min_samples_split=5, n_estimators=52; total time=   0.2s
[CV] END bootstrap=True, criterion=gini, max_depth=30, max_features=log2, min_samples_leaf=1, min_samples_split=5, n_estimators=52; total time=   0.2s
[CV] END bootstrap=False, criterion=entropy, max_depth=None, max_features=sqrt, min_samples_leaf=2, min_samples_split=2, n_estimators=44; total time=   0.2s
[CV] END bootstrap=True, criterion=gini, max_depth=30, max_features=log2, min_samples_leaf=1, min_samples_split=5, n_estimators=52; total time=   0.2s
[CV] END bootstrap=True, criterion=gini, max_depth=30, max_features=log2, min_samples_leaf=1, min_samples_split=5, n_estimators=52; total time=   0.2s
[CV] END bootstrap=True, criterion=gini, max_depth=30, max_features=log2, min_samples_leaf=1, min_samples_split=5, n_estimators=52; total time=   0.2s
[CV] END bootstrap=True, criterion=gini, max_depth=20, max_features=sqrt, min_samples_leaf=1, min_samples_split=2, n_estimators=51; total time=   0.2s
[CV] END bootstrap=True, criterion=gini, max_depth=20, max_features=sqrt, min_samples_leaf=1, min_samples_split=2, n_estimators=51; total time=   0.2s
[CV] END bootstrap=True, criterion=gini, max_depth=20, max_features=sqrt, min_samples_leaf=1, min_samples_split=2, n_estimators=51; total time=   0.2s
[CV] END bootstrap=True, criterion=gini, max_depth=None, max_features=None, min_samples_leaf=2, min_samples_split=6, n_estimators=50; total time=   0.5s
[CV] END bootstrap=True, criterion=gini, max_depth=20, max_features=sqrt, min_samples_leaf=1, min_samples_split=2, n_estimators=51; total time=   0.2s
[CV] END bootstrap=True, criterion=gini, max_depth=None, max_features=None, min_samples_leaf=2, min_samples_split=6, n_estimators=50; total time=   0.5s
[CV] END bootstrap=True, criterion=gini, max_depth=None, max_features=None, min_samples_leaf=2, min_samples_split=6, n_estimators=50; total time=   0.5s
[CV] END bootstrap=True, criterion=gini, max_depth=20, max_features=sqrt, min_samples_leaf=1, min_samples_split=2, n_estimators=51; total time=   0.2s
[CV] END bootstrap=True, criterion=gini, max_depth=None, max_features=None, min_samples_leaf=2, min_samples_split=6, n_estimators=50; total time=   0.5s
[CV] END bootstrap=True, criterion=gini, max_depth=None, max_features=None, min_samples_leaf=2, min_samples_split=6, n_estimators=50; total time=   0.5s
[CV] END bootstrap=True, criterion=entropy, max_depth=30, max_features=None, min_samples_leaf=1, min_samples_split=7, n_estimators=59; total time=   0.6s
[CV] END bootstrap=True, criterion=entropy, max_depth=30, max_features=None, min_samples_leaf=1, min_samples_split=7, n_estimators=59; total time=   0.6s
[CV] END bootstrap=True, criterion=entropy, max_depth=30, max_features=None, min_samples_leaf=1, min_samples_split=7, n_estimators=59; total time=   0.6s
[CV] END bootstrap=True, criterion=entropy, max_depth=30, max_features=None, min_samples_leaf=1, min_samples_split=7, n_estimators=59; total time=   0.6s
[CV] END bootstrap=True, criterion=entropy, max_depth=30, max_features=None, min_samples_leaf=1, min_samples_split=7, n_estimators=59; total time=   0.6s
[CV] END bootstrap=False, criterion=entropy, max_depth=None, max_features=None, min_samples_leaf=4, min_samples_split=4, n_estimators=42; total time=   0.6s
[CV] END bootstrap=False, criterion=entropy, max_depth=None, max_features=None, min_samples_leaf=4, min_samples_split=4, n_estimators=42; total time=   0.6s
[CV] END bootstrap=False, criterion=entropy, max_depth=None, max_features=None, min_samples_leaf=4, min_samples_split=4, n_estimators=42; total time=   0.7s
[CV] END bootstrap=True, criterion=gini, max_depth=20, max_features=log2, min_samples_leaf=3, min_samples_split=2, n_estimators=57; total time=   0.2s
[CV] END bootstrap=True, criterion=gini, max_depth=20, max_features=log2, min_samples_leaf=3, min_samples_split=2, n_estimators=57; total time=   0.2s
[CV] END bootstrap=True, criterion=gini, max_depth=20, max_features=log2, min_samples_leaf=3, min_samples_split=2, n_estimators=57; total time=   0.2s
[CV] END bootstrap=True, criterion=gini, max_depth=20, max_features=log2, min_samples_leaf=3, min_samples_split=2, n_estimators=57; total time=   0.2s
[CV] END bootstrap=True, criterion=gini, max_depth=20, max_features=log2, min_samples_leaf=3, min_samples_split=2, n_estimators=57; total time=   0.2s
[CV] END bootstrap=True, criterion=entropy, max_depth=20, max_features=log2, min_samples_leaf=4, min_samples_split=7, n_estimators=48; total time=   0.2s
[CV] END bootstrap=True, criterion=entropy, max_depth=20, max_features=log2, min_samples_leaf=4, min_samples_split=7, n_estimators=48; total time=   0.2s
[CV] END bootstrap=True, criterion=entropy, max_depth=20, max_features=log2, min_samples_leaf=4, min_samples_split=7, n_estimators=48; total time=   0.2s
[CV] END bootstrap=True, criterion=entropy, max_depth=20, max_features=log2, min_samples_leaf=4, min_samples_split=7, n_estimators=48; total time=   0.2s
[CV] END bootstrap=True, criterion=entropy, max_depth=20, max_features=log2, min_samples_leaf=4, min_samples_split=7, n_estimators=48; total time=   0.2s
[CV] END bootstrap=False, criterion=entropy, max_depth=None, max_features=None, min_samples_leaf=4, min_samples_split=4, n_estimators=42; total time=   0.7s
[CV] END bootstrap=True, criterion=entropy, max_depth=10, max_features=log2, min_samples_leaf=2, min_samples_split=3, n_estimators=59; total time=   0.2s
[CV] END bootstrap=True, criterion=entropy, max_depth=10, max_features=log2, min_samples_leaf=2, min_samples_split=3, n_estimators=59; total time=   0.2s
[CV] END bootstrap=False, criterion=entropy, max_depth=None, max_features=None, min_samples_leaf=4, min_samples_split=4, n_estimators=42; total time=   0.7s
[CV] END bootstrap=True, criterion=entropy, max_depth=10, max_features=log2, min_samples_leaf=2, min_samples_split=3, n_estimators=59; total time=   0.3s
[CV] END bootstrap=True, criterion=entropy, max_depth=10, max_features=log2, min_samples_leaf=2, min_samples_split=3, n_estimators=59; total time=   0.2s
[CV] END bootstrap=False, criterion=gini, max_depth=30, max_features=log2, min_samples_leaf=2, min_samples_split=5, n_estimators=43; total time=   0.2s
[CV] END bootstrap=True, criterion=entropy, max_depth=10, max_features=log2, min_samples_leaf=2, min_samples_split=3, n_estimators=59; total time=   0.3s
[CV] END bootstrap=False, criterion=gini, max_depth=30, max_features=log2, min_samples_leaf=2, min_samples_split=5, n_estimators=43; total time=   0.2s
[CV] END bootstrap=False, criterion=gini, max_depth=30, max_features=log2, min_samples_leaf=2, min_samples_split=5, n_estimators=43; total time=   0.2s
[CV] END bootstrap=False, criterion=gini, max_depth=30, max_features=log2, min_samples_leaf=2, min_samples_split=5, n_estimators=43; total time=   0.3s
[CV] END bootstrap=False, criterion=gini, max_depth=30, max_features=log2, min_samples_leaf=2, min_samples_split=5, n_estimators=43; total time=   0.2s
[CV] END bootstrap=False, criterion=entropy, max_depth=20, max_features=log2, min_samples_leaf=4, min_samples_split=8, n_estimators=50; total time=   0.3s
[CV] END bootstrap=False, criterion=entropy, max_depth=20, max_features=log2, min_samples_leaf=4, min_samples_split=8, n_estimators=50; total time=   0.2s
[CV] END bootstrap=False, criterion=entropy, max_depth=20, max_features=log2, min_samples_leaf=4, min_samples_split=8, n_estimators=50; total time=   0.3s
[CV] END bootstrap=False, criterion=gini, max_depth=10, max_features=None, min_samples_leaf=1, min_samples_split=7, n_estimators=40; total time=   0.6s
[CV] END bootstrap=False, criterion=gini, max_depth=10, max_features=None, min_samples_leaf=1, min_samples_split=7, n_estimators=40; total time=   0.5s
[CV] END bootstrap=False, criterion=gini, max_depth=10, max_features=None, min_samples_leaf=1, min_samples_split=7, n_estimators=40; total time=   0.6s
[CV] END bootstrap=False, criterion=gini, max_depth=10, max_features=None, min_samples_leaf=1, min_samples_split=7, n_estimators=40; total time=   0.6s
[CV] END bootstrap=True, criterion=gini, max_depth=30, max_features=sqrt, min_samples_leaf=3, min_samples_split=6, n_estimators=47; total time=   0.2s
[CV] END bootstrap=False, criterion=gini, max_depth=10, max_features=None, min_samples_leaf=1, min_samples_split=7, n_estimators=40; total time=   0.6s
[CV] END bootstrap=False, criterion=entropy, max_depth=20, max_features=log2, min_samples_leaf=4, min_samples_split=8, n_estimators=50; total time=   0.3s
[CV] END bootstrap=False, criterion=entropy, max_depth=20, max_features=log2, min_samples_leaf=4, min_samples_split=8, n_estimators=50; total time=   0.3s
[CV] END bootstrap=True, criterion=gini, max_depth=30, max_features=sqrt, min_samples_leaf=3, min_samples_split=6, n_estimators=47; total time=   0.2s
[CV] END bootstrap=True, criterion=gini, max_depth=30, max_features=sqrt, min_samples_leaf=3, min_samples_split=6, n_estimators=47; total time=   0.2s
[CV] END bootstrap=True, criterion=gini, max_depth=30, max_features=sqrt, min_samples_leaf=3, min_samples_split=6, n_estimators=47; total time=   0.2s
[CV] END bootstrap=True, criterion=gini, max_depth=30, max_features=sqrt, min_samples_leaf=3, min_samples_split=6, n_estimators=47; total time=   0.2s
[CV] END bootstrap=True, criterion=gini, max_depth=10, max_features=log2, min_samples_leaf=4, min_samples_split=3, n_estimators=47; total time=   0.2s
[CV] END bootstrap=True, criterion=gini, max_depth=10, max_features=log2, min_samples_leaf=4, min_samples_split=3, n_estimators=47; total time=   0.2s
[CV] END bootstrap=True, criterion=gini, max_depth=10, max_features=log2, min_samples_leaf=4, min_samples_split=3, n_estimators=47; total time=   0.2s
[CV] END bootstrap=True, criterion=gini, max_depth=10, max_features=log2, min_samples_leaf=4, min_samples_split=3, n_estimators=47; total time=   0.2s
[CV] END bootstrap=True, criterion=gini, max_depth=10, max_features=log2, min_samples_leaf=4, min_samples_split=3, n_estimators=47; total time=   0.2s
[CV] END bootstrap=True, criterion=gini, max_depth=30, max_features=None, min_samples_leaf=3, min_samples_split=8, n_estimators=45; total time=   0.4s
[CV] END bootstrap=True, criterion=gini, max_depth=30, max_features=None, min_samples_leaf=3, min_samples_split=8, n_estimators=45; total time=   0.4s
[CV] END bootstrap=True, criterion=gini, max_depth=30, max_features=None, min_samples_leaf=3, min_samples_split=8, n_estimators=45; total time=   0.4s
[CV] END bootstrap=False, criterion=entropy, max_depth=None, max_features=None, min_samples_leaf=4, min_samples_split=2, n_estimators=52; total time=   0.8s
[CV] END bootstrap=False, criterion=entropy, max_depth=None, max_features=None, min_samples_leaf=4, min_samples_split=2, n_estimators=52; total time=   0.8s
[CV] END bootstrap=False, criterion=entropy, max_depth=None, max_features=None, min_samples_leaf=4, min_samples_split=2, n_estimators=52; total time=   0.8s
[CV] END bootstrap=False, criterion=entropy, max_depth=10, max_features=log2, min_samples_leaf=1, min_samples_split=6, n_estimators=53; total time=   0.3s
[CV] END bootstrap=False, criterion=entropy, max_depth=None, max_features=None, min_samples_leaf=4, min_samples_split=2, n_estimators=52; total time=   0.8s
[CV] END bootstrap=False, criterion=entropy, max_depth=None, max_features=None, min_samples_leaf=4, min_samples_split=2, n_estimators=52; total time=   0.8s
[CV] END bootstrap=True, criterion=gini, max_depth=30, max_features=None, min_samples_leaf=3, min_samples_split=8, n_estimators=45; total time=   0.4s
[CV] END bootstrap=True, criterion=gini, max_depth=30, max_features=None, min_samples_leaf=3, min_samples_split=8, n_estimators=45; total time=   0.4s
[CV] END bootstrap=False, criterion=entropy, max_depth=10, max_features=log2, min_samples_leaf=1, min_samples_split=6, n_estimators=53; total time=   0.3s
[CV] END bootstrap=True, criterion=gini, max_depth=20, max_features=sqrt, min_samples_leaf=2, min_samples_split=4, n_estimators=57; total time=   0.2s
[CV] END bootstrap=True, criterion=gini, max_depth=20, max_features=sqrt, min_samples_leaf=2, min_samples_split=4, n_estimators=57; total time=   0.2s
[CV] END bootstrap=True, criterion=gini, max_depth=20, max_features=sqrt, min_samples_leaf=2, min_samples_split=4, n_estimators=57; total time=   0.2s
[CV] END bootstrap=False, criterion=entropy, max_depth=10, max_features=log2, min_samples_leaf=1, min_samples_split=6, n_estimators=53; total time=   0.3s
[CV] END bootstrap=False, criterion=entropy, max_depth=10, max_features=log2, min_samples_leaf=1, min_samples_split=6, n_estimators=53; total time=   0.3s
[CV] END bootstrap=False, criterion=entropy, max_depth=10, max_features=log2, min_samples_leaf=1, min_samples_split=6, n_estimators=53; total time=   0.3s
[CV] END bootstrap=True, criterion=gini, max_depth=20, max_features=sqrt, min_samples_leaf=2, min_samples_split=4, n_estimators=57; total time=   0.2s
[CV] END bootstrap=True, criterion=gini, max_depth=20, max_features=sqrt, min_samples_leaf=2, min_samples_split=4, n_estimators=57; total time=   0.2s
[CV] END bootstrap=True, criterion=entropy, max_depth=10, max_features=sqrt, min_samples_leaf=2, min_samples_split=5, n_estimators=55; total time=   0.2s
[CV] END bootstrap=True, criterion=entropy, max_depth=10, max_features=sqrt, min_samples_leaf=2, min_samples_split=5, n_estimators=55; total time=   0.2s
[CV] END bootstrap=True, criterion=entropy, max_depth=10, max_features=sqrt, min_samples_leaf=2, min_samples_split=5, n_estimators=55; total time=   0.2s
[CV] END bootstrap=True, criterion=entropy, max_depth=10, max_features=sqrt, min_samples_leaf=2, min_samples_split=5, n_estimators=55; total time=   0.2s
[CV] END bootstrap=True, criterion=entropy, max_depth=10, max_features=sqrt, min_samples_leaf=2, min_samples_split=5, n_estimators=55; total time=   0.2s
[CV] END bootstrap=False, criterion=gini, max_depth=None, max_features=sqrt, min_samples_leaf=4, min_samples_split=5, n_estimators=55; total time=   0.2s
[CV] END bootstrap=False, criterion=gini, max_depth=None, max_features=sqrt, min_samples_leaf=4, min_samples_split=5, n_estimators=55; total time=   0.2s
[CV] END bootstrap=False, criterion=gini, max_depth=None, max_features=sqrt, min_samples_leaf=4, min_samples_split=5, n_estimators=55; total time=   0.2s
[CV] END bootstrap=True, criterion=gini, max_depth=20, max_features=log2, min_samples_leaf=4, min_samples_split=7, n_estimators=41; total time=   0.1s
[CV] END bootstrap=True, criterion=gini, max_depth=20, max_features=log2, min_samples_leaf=4, min_samples_split=7, n_estimators=41; total time=   0.2s
[CV] END bootstrap=False, criterion=gini, max_depth=None, max_features=sqrt, min_samples_leaf=4, min_samples_split=5, n_estimators=55; total time=   0.2s
[CV] END bootstrap=False, criterion=gini, max_depth=None, max_features=sqrt, min_samples_leaf=4, min_samples_split=5, n_estimators=55; total time=   0.2s
[CV] END bootstrap=True, criterion=gini, max_depth=20, max_features=log2, min_samples_leaf=4, min_samples_split=7, n_estimators=41; total time=   0.1s
[CV] END bootstrap=True, criterion=gini, max_depth=20, max_features=log2, min_samples_leaf=4, min_samples_split=7, n_estimators=41; total time=   0.2s
[CV] END bootstrap=True, criterion=gini, max_depth=20, max_features=log2, min_samples_leaf=4, min_samples_split=7, n_estimators=41; total time=   0.1s
[CV] END bootstrap=False, criterion=entropy, max_depth=30, max_features=log2, min_samples_leaf=3, min_samples_split=5, n_estimators=50; total time=   0.3s
[CV] END bootstrap=False, criterion=entropy, max_depth=30, max_features=log2, min_samples_leaf=3, min_samples_split=5, n_estimators=50; total time=   0.3s
[CV] END bootstrap=False, criterion=entropy, max_depth=30, max_features=log2, min_samples_leaf=3, min_samples_split=5, n_estimators=50; total time=   0.3s
[CV] END bootstrap=False, criterion=entropy, max_depth=30, max_features=log2, min_samples_leaf=3, min_samples_split=5, n_estimators=50; total time=   0.3s
[CV] END bootstrap=True, criterion=entropy, max_depth=30, max_features=log2, min_samples_leaf=1, min_samples_split=2, n_estimators=57; total time=   0.3s
[CV] END bootstrap=False, criterion=entropy, max_depth=30, max_features=log2, min_samples_leaf=3, min_samples_split=5, n_estimators=50; total time=   0.3s
[CV] END bootstrap=True, criterion=entropy, max_depth=30, max_features=log2, min_samples_leaf=1, min_samples_split=2, n_estimators=57; total time=   0.2s
[CV] END bootstrap=True, criterion=entropy, max_depth=30, max_features=log2, min_samples_leaf=1, min_samples_split=2, n_estimators=57; total time=   0.2s
[CV] END bootstrap=True, criterion=entropy, max_depth=30, max_features=log2, min_samples_leaf=1, min_samples_split=2, n_estimators=57; total time=   0.2s
[CV] END bootstrap=True, criterion=entropy, max_depth=30, max_features=log2, min_samples_leaf=1, min_samples_split=2, n_estimators=57; total time=   0.2s
[CV] END bootstrap=False, criterion=entropy, max_depth=None, max_features=log2, min_samples_leaf=1, min_samples_split=2, n_estimators=41; total time=   0.3s
[CV] END bootstrap=False, criterion=entropy, max_depth=None, max_features=log2, min_samples_leaf=1, min_samples_split=2, n_estimators=41; total time=   0.3s
[CV] END bootstrap=False, criterion=entropy, max_depth=None, max_features=log2, min_samples_leaf=1, min_samples_split=2, n_estimators=41; total time=   0.3s
[CV] END bootstrap=True, criterion=gini, max_depth=10, max_features=log2, min_samples_leaf=3, min_samples_split=7, n_estimators=44; total time=   0.1s
[CV] END bootstrap=False, criterion=entropy, max_depth=None, max_features=log2, min_samples_leaf=1, min_samples_split=2, n_estimators=41; total time=   0.2s
[CV] END bootstrap=False, criterion=entropy, max_depth=None, max_features=log2, min_samples_leaf=1, min_samples_split=2, n_estimators=41; total time=   0.2s
[CV] END bootstrap=True, criterion=gini, max_depth=10, max_features=log2, min_samples_leaf=3, min_samples_split=7, n_estimators=44; total time=   0.2s
[CV] END bootstrap=True, criterion=gini, max_depth=10, max_features=log2, min_samples_leaf=3, min_samples_split=7, n_estimators=44; total time=   0.2s
[CV] END bootstrap=True, criterion=gini, max_depth=10, max_features=log2, min_samples_leaf=3, min_samples_split=7, n_estimators=44; total time=   0.2s
[CV] END bootstrap=False, criterion=gini, max_depth=20, max_features=None, min_samples_leaf=1, min_samples_split=3, n_estimators=58; total time=   0.9s
[CV] END bootstrap=True, criterion=gini, max_depth=10, max_features=log2, min_samples_leaf=3, min_samples_split=7, n_estimators=44; total time=   0.2s
[CV] END bootstrap=False, criterion=gini, max_depth=20, max_features=None, min_samples_leaf=1, min_samples_split=3, n_estimators=58; total time=   1.1s
[CV] END bootstrap=False, criterion=gini, max_depth=20, max_features=None, min_samples_leaf=1, min_samples_split=3, n_estimators=58; total time=   1.1s
[CV] END bootstrap=False, criterion=gini, max_depth=20, max_features=None, min_samples_leaf=1, min_samples_split=3, n_estimators=58; total time=   1.1s
[CV] END bootstrap=False, criterion=gini, max_depth=20, max_features=None, min_samples_leaf=1, min_samples_split=3, n_estimators=58; total time=   1.1s
[CV] END bootstrap=False, criterion=gini, max_depth=20, max_features=log2, min_samples_leaf=2, min_samples_split=6, n_estimators=52; total time=   0.3s
[CV] END bootstrap=False, criterion=gini, max_depth=20, max_features=log2, min_samples_leaf=2, min_samples_split=6, n_estimators=52; total time=   0.3s
[CV] END bootstrap=False, criterion=gini, max_depth=20, max_features=log2, min_samples_leaf=2, min_samples_split=6, n_estimators=52; total time=   0.2s
[CV] END bootstrap=False, criterion=gini, max_depth=20, max_features=log2, min_samples_leaf=2, min_samples_split=6, n_estimators=52; total time=   0.3s
[CV] END bootstrap=True, criterion=entropy, max_depth=None, max_features=sqrt, min_samples_leaf=1, min_samples_split=9, n_estimators=47; total time=   0.2s
[CV] END bootstrap=False, criterion=gini, max_depth=20, max_features=log2, min_samples_leaf=2, min_samples_split=6, n_estimators=52; total time=   0.2s
[CV] END bootstrap=True, criterion=entropy, max_depth=None, max_features=sqrt, min_samples_leaf=1, min_samples_split=9, n_estimators=47; total time=   0.2s
[CV] END bootstrap=True, criterion=entropy, max_depth=None, max_features=sqrt, min_samples_leaf=1, min_samples_split=9, n_estimators=47; total time=   0.3s
[CV] END bootstrap=True, criterion=entropy, max_depth=None, max_features=sqrt, min_samples_leaf=1, min_samples_split=9, n_estimators=47; total time=   0.3s
[CV] END bootstrap=True, criterion=entropy, max_depth=None, max_features=sqrt, min_samples_leaf=1, min_samples_split=9, n_estimators=47; total time=   0.3s
[CV] END bootstrap=True, criterion=entropy, max_depth=20, max_features=log2, min_samples_leaf=1, min_samples_split=7, n_estimators=47; total time=   0.3s
[CV] END bootstrap=True, criterion=entropy, max_depth=20, max_features=log2, min_samples_leaf=1, min_samples_split=7, n_estimators=47; total time=   0.3s
[CV] END bootstrap=True, criterion=entropy, max_depth=20, max_features=log2, min_samples_leaf=1, min_samples_split=7, n_estimators=47; total time=   0.3s
[CV] END bootstrap=True, criterion=entropy, max_depth=20, max_features=log2, min_samples_leaf=1, min_samples_split=7, n_estimators=47; total time=   0.3s
[CV] END bootstrap=True, criterion=entropy, max_depth=20, max_features=log2, min_samples_leaf=1, min_samples_split=7, n_estimators=47; total time=   0.3s
[CV] END bootstrap=True, criterion=gini, max_depth=10, max_features=log2, min_samples_leaf=1, min_samples_split=7, n_estimators=58; total time=   0.3s
[CV] END bootstrap=True, criterion=gini, max_depth=10, max_features=log2, min_samples_leaf=1, min_samples_split=7, n_estimators=58; total time=   0.3s
[CV] END bootstrap=True, criterion=gini, max_depth=10, max_features=log2, min_samples_leaf=1, min_samples_split=7, n_estimators=58; total time=   0.2s
[CV] END bootstrap=True, criterion=gini, max_depth=10, max_features=log2, min_samples_leaf=1, min_samples_split=7, n_estimators=58; total time=   0.2s
[CV] END bootstrap=True, criterion=gini, max_depth=10, max_features=log2, min_samples_leaf=1, min_samples_split=7, n_estimators=58; total time=   0.3s
[CV] END bootstrap=False, criterion=entropy, max_depth=None, max_features=log2, min_samples_leaf=2, min_samples_split=8, n_estimators=41; total time=   0.3s
[CV] END bootstrap=False, criterion=entropy, max_depth=None, max_features=log2, min_samples_leaf=2, min_samples_split=8, n_estimators=41; total time=   0.3s
[CV] END bootstrap=False, criterion=entropy, max_depth=None, max_features=log2, min_samples_leaf=2, min_samples_split=8, n_estimators=41; total time=   0.2s
[CV] END bootstrap=False, criterion=gini, max_depth=30, max_features=None, min_samples_leaf=4, min_samples_split=9, n_estimators=41; total time=   0.6s
[CV] END bootstrap=False, criterion=gini, max_depth=30, max_features=None, min_samples_leaf=4, min_samples_split=9, n_estimators=41; total time=   0.6s
[CV] END bootstrap=False, criterion=gini, max_depth=30, max_features=None, min_samples_leaf=4, min_samples_split=9, n_estimators=41; total time=   0.6s
[CV] END bootstrap=False, criterion=entropy, max_depth=None, max_features=log2, min_samples_leaf=2, min_samples_split=8, n_estimators=41; total time=   0.2s
[CV] END bootstrap=False, criterion=gini, max_depth=30, max_features=None, min_samples_leaf=4, min_samples_split=9, n_estimators=41; total time=   0.6s
[CV] END bootstrap=False, criterion=entropy, max_depth=None, max_features=log2, min_samples_leaf=2, min_samples_split=8, n_estimators=41; total time=   0.2s
[CV] END bootstrap=False, criterion=gini, max_depth=30, max_features=None, min_samples_leaf=4, min_samples_split=9, n_estimators=41; total time=   0.7s
[CV] END bootstrap=False, criterion=gini, max_depth=30, max_features=sqrt, min_samples_leaf=2, min_samples_split=4, n_estimators=46; total time=   0.2s
[CV] END bootstrap=False, criterion=gini, max_depth=30, max_features=sqrt, min_samples_leaf=2, min_samples_split=4, n_estimators=46; total time=   0.2s
[CV] END bootstrap=False, criterion=gini, max_depth=30, max_features=sqrt, min_samples_leaf=2, min_samples_split=4, n_estimators=46; total time=   0.2s
[CV] END bootstrap=False, criterion=gini, max_depth=30, max_features=sqrt, min_samples_leaf=2, min_samples_split=4, n_estimators=46; total time=   0.2s
[CV] END bootstrap=False, criterion=gini, max_depth=30, max_features=sqrt, min_samples_leaf=2, min_samples_split=4, n_estimators=46; total time=   0.2s
[CV] END bootstrap=True, criterion=entropy, max_depth=20, max_features=log2, min_samples_leaf=4, min_samples_split=2, n_estimators=50; total time=   0.2s
[CV] END bootstrap=True, criterion=entropy, max_depth=20, max_features=log2, min_samples_leaf=4, min_samples_split=2, n_estimators=50; total time=   0.2s
[CV] END bootstrap=True, criterion=entropy, max_depth=20, max_features=log2, min_samples_leaf=4, min_samples_split=2, n_estimators=50; total time=   0.2s
[CV] END bootstrap=True, criterion=entropy, max_depth=20, max_features=log2, min_samples_leaf=4, min_samples_split=2, n_estimators=50; total time=   0.2s
[CV] END bootstrap=True, criterion=entropy, max_depth=20, max_features=log2, min_samples_leaf=4, min_samples_split=2, n_estimators=50; total time=   0.2s
[CV] END bootstrap=True, criterion=gini, max_depth=None, max_features=sqrt, min_samples_leaf=1, min_samples_split=2, n_estimators=49; total time=   0.2s
[CV] END bootstrap=True, criterion=gini, max_depth=None, max_features=sqrt, min_samples_leaf=1, min_samples_split=2, n_estimators=49; total time=   0.2s
[CV] END bootstrap=True, criterion=gini, max_depth=None, max_features=sqrt, min_samples_leaf=1, min_samples_split=2, n_estimators=49; total time=   0.2s
[CV] END bootstrap=True, criterion=gini, max_depth=None, max_features=sqrt, min_samples_leaf=1, min_samples_split=2, n_estimators=49; total time=   0.2s
[CV] END bootstrap=True, criterion=entropy, max_depth=10, max_features=log2, min_samples_leaf=4, min_samples_split=6, n_estimators=40; total time=   0.1s
[CV] END bootstrap=True, criterion=entropy, max_depth=10, max_features=None, min_samples_leaf=2, min_samples_split=10, n_estimators=51; total time=   0.6s
[CV] END bootstrap=True, criterion=gini, max_depth=None, max_features=sqrt, min_samples_leaf=1, min_samples_split=2, n_estimators=49; total time=   0.2s
[CV] END bootstrap=True, criterion=entropy, max_depth=10, max_features=None, min_samples_leaf=2, min_samples_split=10, n_estimators=51; total time=   0.6s
[CV] END bootstrap=True, criterion=entropy, max_depth=10, max_features=None, min_samples_leaf=2, min_samples_split=10, n_estimators=51; total time=   0.6s
[CV] END bootstrap=True, criterion=entropy, max_depth=10, max_features=None, min_samples_leaf=2, min_samples_split=10, n_estimators=51; total time=   0.6s
[CV] END bootstrap=True, criterion=entropy, max_depth=10, max_features=None, min_samples_leaf=2, min_samples_split=10, n_estimators=51; total time=   0.6s
[CV] END bootstrap=True, criterion=entropy, max_depth=10, max_features=log2, min_samples_leaf=4, min_samples_split=6, n_estimators=40; total time=   0.2s
[CV] END bootstrap=True, criterion=entropy, max_depth=10, max_features=log2, min_samples_leaf=4, min_samples_split=6, n_estimators=40; total time=   0.2s
[CV] END bootstrap=True, criterion=entropy, max_depth=10, max_features=log2, min_samples_leaf=4, min_samples_split=6, n_estimators=40; total time=   0.2s
[CV] END bootstrap=True, criterion=entropy, max_depth=10, max_features=log2, min_samples_leaf=4, min_samples_split=6, n_estimators=40; total time=   0.2s
[CV] END bootstrap=False, criterion=entropy, max_depth=None, max_features=log2, min_samples_leaf=4, min_samples_split=2, n_estimators=48; total time=   0.3s
[CV] END bootstrap=False, criterion=entropy, max_depth=None, max_features=log2, min_samples_leaf=4, min_samples_split=2, n_estimators=48; total time=   0.3s
[CV] END bootstrap=False, criterion=entropy, max_depth=None, max_features=log2, min_samples_leaf=4, min_samples_split=2, n_estimators=48; total time=   0.3s
[CV] END bootstrap=False, criterion=gini, max_depth=20, max_features=sqrt, min_samples_leaf=4, min_samples_split=9, n_estimators=53; total time=   0.2s
[CV] END bootstrap=False, criterion=gini, max_depth=20, max_features=sqrt, min_samples_leaf=4, min_samples_split=9, n_estimators=53; total time=   0.3s
[CV] END bootstrap=False, criterion=entropy, max_depth=None, max_features=log2, min_samples_leaf=4, min_samples_split=2, n_estimators=48; total time=   0.3s
[CV] END bootstrap=False, criterion=gini, max_depth=20, max_features=sqrt, min_samples_leaf=4, min_samples_split=9, n_estimators=53; total time=   0.3s
[CV] END bootstrap=False, criterion=entropy, max_depth=None, max_features=log2, min_samples_leaf=4, min_samples_split=2, n_estimators=48; total time=   0.4s
[CV] END bootstrap=False, criterion=gini, max_depth=20, max_features=sqrt, min_samples_leaf=4, min_samples_split=9, n_estimators=53; total time=   0.3s
[CV] END bootstrap=False, criterion=gini, max_depth=20, max_features=sqrt, min_samples_leaf=4, min_samples_split=9, n_estimators=53; total time=   0.2s
[CV] END bootstrap=False, criterion=entropy, max_depth=None, max_features=sqrt, min_samples_leaf=3, min_samples_split=5, n_estimators=40; total time=   0.2s
[CV] END bootstrap=False, criterion=entropy, max_depth=None, max_features=sqrt, min_samples_leaf=3, min_samples_split=5, n_estimators=40; total time=   0.2s
[CV] END bootstrap=False, criterion=entropy, max_depth=None, max_features=sqrt, min_samples_leaf=3, min_samples_split=5, n_estimators=40; total time=   0.2s
[CV] END bootstrap=False, criterion=entropy, max_depth=None, max_features=sqrt, min_samples_leaf=3, min_samples_split=5, n_estimators=40; total time=   0.2s
[CV] END bootstrap=False, criterion=entropy, max_depth=None, max_features=sqrt, min_samples_leaf=3, min_samples_split=5, n_estimators=40; total time=   0.2s
[CV] END bootstrap=True, criterion=gini, max_depth=None, max_features=log2, min_samples_leaf=3, min_samples_split=2, n_estimators=57; total time=   0.2s
[CV] END bootstrap=True, criterion=gini, max_depth=None, max_features=log2, min_samples_leaf=3, min_samples_split=2, n_estimators=57; total time=   0.2s
[CV] END bootstrap=True, criterion=gini, max_depth=None, max_features=log2, min_samples_leaf=3, min_samples_split=2, n_estimators=57; total time=   0.2s
[CV] END bootstrap=True, criterion=gini, max_depth=None, max_features=log2, min_samples_leaf=3, min_samples_split=2, n_estimators=57; total time=   0.2s
[CV] END bootstrap=True, criterion=gini, max_depth=30, max_features=sqrt, min_samples_leaf=1, min_samples_split=10, n_estimators=48; total time=   0.2s
[CV] END bootstrap=True, criterion=gini, max_depth=None, max_features=log2, min_samples_leaf=3, min_samples_split=2, n_estimators=57; total time=   0.2s
[CV] END bootstrap=True, criterion=gini, max_depth=30, max_features=sqrt, min_samples_leaf=1, min_samples_split=10, n_estimators=48; total time=   0.2s
[CV] END bootstrap=True, criterion=gini, max_depth=30, max_features=sqrt, min_samples_leaf=1, min_samples_split=10, n_estimators=48; total time=   0.2s
[CV] END bootstrap=True, criterion=gini, max_depth=30, max_features=sqrt, min_samples_leaf=1, min_samples_split=10, n_estimators=48; total time=   0.2s
[CV] END bootstrap=True, criterion=gini, max_depth=30, max_features=sqrt, min_samples_leaf=1, min_samples_split=10, n_estimators=48; total time=   0.2s
[CV] END bootstrap=True, criterion=gini, max_depth=20, max_features=log2, min_samples_leaf=1, min_samples_split=8, n_estimators=44; total time=   0.2s
[CV] END bootstrap=True, criterion=gini, max_depth=20, max_features=log2, min_samples_leaf=1, min_samples_split=8, n_estimators=44; total time=   0.2s
[CV] END bootstrap=True, criterion=gini, max_depth=20, max_features=log2, min_samples_leaf=1, min_samples_split=8, n_estimators=44; total time=   0.2s
[CV] END bootstrap=True, criterion=gini, max_depth=20, max_features=log2, min_samples_leaf=1, min_samples_split=8, n_estimators=44; total time=   0.2s
[CV] END bootstrap=True, criterion=gini, max_depth=20, max_features=log2, min_samples_leaf=1, min_samples_split=8, n_estimators=44; total time=   0.2s
[CV] END bootstrap=False, criterion=gini, max_depth=None, max_features=log2, min_samples_leaf=2, min_samples_split=7, n_estimators=41; total time=   0.2s
[CV] END bootstrap=False, criterion=gini, max_depth=None, max_features=log2, min_samples_leaf=2, min_samples_split=7, n_estimators=41; total time=   0.2s
[CV] END bootstrap=False, criterion=gini, max_depth=None, max_features=log2, min_samples_leaf=2, min_samples_split=7, n_estimators=41; total time=   0.2s
[CV] END bootstrap=False, criterion=gini, max_depth=None, max_features=log2, min_samples_leaf=2, min_samples_split=7, n_estimators=41; total time=   0.2s
[CV] END bootstrap=False, criterion=gini, max_depth=None, max_features=log2, min_samples_leaf=2, min_samples_split=7, n_estimators=41; total time=   0.2s
[CV] END bootstrap=False, criterion=gini, max_depth=10, max_features=sqrt, min_samples_leaf=4, min_samples_split=9, n_estimators=45; total time=   0.2s
[CV] END bootstrap=False, criterion=gini, max_depth=10, max_features=sqrt, min_samples_leaf=4, min_samples_split=9, n_estimators=45; total time=   0.2s
[CV] END bootstrap=False, criterion=gini, max_depth=10, max_features=sqrt, min_samples_leaf=4, min_samples_split=9, n_estimators=45; total time=   0.2s
[CV] END bootstrap=False, criterion=gini, max_depth=10, max_features=sqrt, min_samples_leaf=4, min_samples_split=9, n_estimators=45; total time=   0.2s
[CV] END bootstrap=False, criterion=gini, max_depth=10, max_features=sqrt, min_samples_leaf=4, min_samples_split=9, n_estimators=45; total time=   0.2s
[CV] END bootstrap=True, criterion=entropy, max_depth=20, max_features=log2, min_samples_leaf=4, min_samples_split=7, n_estimators=46; total time=   0.2s
[CV] END bootstrap=True, criterion=entropy, max_depth=20, max_features=log2, min_samples_leaf=4, min_samples_split=7, n_estimators=46; total time=   0.2s
[CV] END bootstrap=True, criterion=entropy, max_depth=20, max_features=log2, min_samples_leaf=4, min_samples_split=7, n_estimators=46; total time=   0.2s
[CV] END bootstrap=True, criterion=entropy, max_depth=20, max_features=log2, min_samples_leaf=4, min_samples_split=7, n_estimators=46; total time=   0.2s
[CV] END bootstrap=True, criterion=entropy, max_depth=20, max_features=log2, min_samples_leaf=4, min_samples_split=7, n_estimators=46; total time=   0.2s
[CV] END bootstrap=True, criterion=entropy, max_depth=None, max_features=None, min_samples_leaf=3, min_samples_split=6, n_estimators=40; total time=   0.4s
[CV] END bootstrap=True, criterion=entropy, max_depth=None, max_features=None, min_samples_leaf=3, min_samples_split=6, n_estimators=40; total time=   0.4s
[CV] END bootstrap=True, criterion=entropy, max_depth=None, max_features=None, min_samples_leaf=3, min_samples_split=6, n_estimators=40; total time=   0.4s
[CV] END bootstrap=True, criterion=entropy, max_depth=None, max_features=None, min_samples_leaf=3, min_samples_split=6, n_estimators=40; total time=   0.4s
[CV] END bootstrap=True, criterion=entropy, max_depth=None, max_features=None, min_samples_leaf=3, min_samples_split=6, n_estimators=40; total time=   0.4s
[CV] END bootstrap=True, criterion=gini, max_depth=30, max_features=None, min_samples_leaf=3, min_samples_split=7, n_estimators=56; total time=   0.5s
[CV] END bootstrap=True, criterion=gini, max_depth=30, max_features=None, min_samples_leaf=3, min_samples_split=7, n_estimators=56; total time=   0.5s
[CV] END bootstrap=True, criterion=gini, max_depth=30, max_features=None, min_samples_leaf=3, min_samples_split=7, n_estimators=56; total time=   0.5s
[CV] END bootstrap=False, criterion=gini, max_depth=10, max_features=sqrt, min_samples_leaf=3, min_samples_split=3, n_estimators=48; total time=   0.2s
[CV] END bootstrap=False, criterion=gini, max_depth=10, max_features=sqrt, min_samples_leaf=3, min_samples_split=3, n_estimators=48; total time=   0.2s
[CV] END bootstrap=False, criterion=gini, max_depth=10, max_features=sqrt, min_samples_leaf=3, min_samples_split=3, n_estimators=48; total time=   0.2s
[CV] END bootstrap=False, criterion=gini, max_depth=10, max_features=sqrt, min_samples_leaf=3, min_samples_split=3, n_estimators=48; total time=   0.2s
[CV] END bootstrap=False, criterion=gini, max_depth=10, max_features=sqrt, min_samples_leaf=3, min_samples_split=3, n_estimators=48; total time=   0.2s
[CV] END bootstrap=True, criterion=entropy, max_depth=None, max_features=sqrt, min_samples_leaf=4, min_samples_split=3, n_estimators=54; total time=   0.2s
[CV] END bootstrap=True, criterion=gini, max_depth=30, max_features=None, min_samples_leaf=3, min_samples_split=7, n_estimators=56; total time=   0.5s
[CV] END bootstrap=True, criterion=entropy, max_depth=None, max_features=sqrt, min_samples_leaf=4, min_samples_split=3, n_estimators=54; total time=   0.2s
[CV] END bootstrap=True, criterion=entropy, max_depth=None, max_features=sqrt, min_samples_leaf=4, min_samples_split=3, n_estimators=54; total time=   0.2s
[CV] END bootstrap=True, criterion=gini, max_depth=30, max_features=None, min_samples_leaf=3, min_samples_split=7, n_estimators=56; total time=   0.5s
[CV] END bootstrap=True, criterion=entropy, max_depth=None, max_features=sqrt, min_samples_leaf=4, min_samples_split=3, n_estimators=54; total time=   0.2s
[CV] END bootstrap=True, criterion=entropy, max_depth=None, max_features=sqrt, min_samples_leaf=4, min_samples_split=3, n_estimators=54; total time=   0.2s
[CV] END bootstrap=False, criterion=entropy, max_depth=10, max_features=sqrt, min_samples_leaf=1, min_samples_split=7, n_estimators=58; total time=   0.3s
[CV] END bootstrap=False, criterion=entropy, max_depth=10, max_features=sqrt, min_samples_leaf=1, min_samples_split=7, n_estimators=58; total time=   0.3s
[CV] END bootstrap=False, criterion=entropy, max_depth=10, max_features=sqrt, min_samples_leaf=1, min_samples_split=7, n_estimators=58; total time=   0.3s
[CV] END bootstrap=False, criterion=entropy, max_depth=10, max_features=sqrt, min_samples_leaf=1, min_samples_split=7, n_estimators=58; total time=   0.3s
[CV] END bootstrap=False, criterion=entropy, max_depth=10, max_features=sqrt, min_samples_leaf=1, min_samples_split=7, n_estimators=58; total time=   0.3s
[CV] END bootstrap=False, criterion=gini, max_depth=30, max_features=None, min_samples_leaf=2, min_samples_split=5, n_estimators=45; total time=   0.5s
[CV] END bootstrap=False, criterion=gini, max_depth=30, max_features=None, min_samples_leaf=2, min_samples_split=5, n_estimators=45; total time=   0.6s
[CV] END bootstrap=False, criterion=gini, max_depth=30, max_features=None, min_samples_leaf=2, min_samples_split=5, n_estimators=45; total time=   0.6s
[CV] END bootstrap=False, criterion=gini, max_depth=30, max_features=None, min_samples_leaf=2, min_samples_split=5, n_estimators=45; total time=   0.6s
[CV] END bootstrap=False, criterion=gini, max_depth=10, max_features=None, min_samples_leaf=4, min_samples_split=10, n_estimators=46; total time=   0.5s
[CV] END bootstrap=False, criterion=gini, max_depth=30, max_features=None, min_samples_leaf=2, min_samples_split=5, n_estimators=45; total time=   0.6s
[CV] END bootstrap=False, criterion=gini, max_depth=10, max_features=None, min_samples_leaf=4, min_samples_split=10, n_estimators=46; total time=   0.6s
[CV] END bootstrap=False, criterion=gini, max_depth=None, max_features=log2, min_samples_leaf=1, min_samples_split=6, n_estimators=46; total time=   0.2s
[CV] END bootstrap=False, criterion=gini, max_depth=10, max_features=None, min_samples_leaf=4, min_samples_split=10, n_estimators=46; total time=   0.6s
[CV] END bootstrap=False, criterion=gini, max_depth=None, max_features=log2, min_samples_leaf=1, min_samples_split=6, n_estimators=46; total time=   0.2s
[CV] END bootstrap=False, criterion=gini, max_depth=None, max_features=log2, min_samples_leaf=1, min_samples_split=6, n_estimators=46; total time=   0.2s
[CV] END bootstrap=False, criterion=gini, max_depth=None, max_features=log2, min_samples_leaf=1, min_samples_split=6, n_estimators=46; total time=   0.2s
[CV] END bootstrap=False, criterion=gini, max_depth=None, max_features=log2, min_samples_leaf=1, min_samples_split=6, n_estimators=46; total time=   0.2s
[CV] END bootstrap=False, criterion=gini, max_depth=10, max_features=None, min_samples_leaf=4, min_samples_split=10, n_estimators=46; total time=   0.5s
[CV] END bootstrap=False, criterion=gini, max_depth=10, max_features=None, min_samples_leaf=4, min_samples_split=10, n_estimators=46; total time=   0.6s
[CV] END bootstrap=False, criterion=gini, max_depth=30, max_features=sqrt, min_samples_leaf=2, min_samples_split=3, n_estimators=46; total time=   0.2s
[CV] END bootstrap=False, criterion=gini, max_depth=30, max_features=sqrt, min_samples_leaf=2, min_samples_split=3, n_estimators=46; total time=   0.2s
[CV] END bootstrap=False, criterion=gini, max_depth=10, max_features=None, min_samples_leaf=2, min_samples_split=5, n_estimators=40; total time=   0.5s
[CV] END bootstrap=False, criterion=gini, max_depth=10, max_features=None, min_samples_leaf=2, min_samples_split=5, n_estimators=40; total time=   0.6s
[CV] END bootstrap=False, criterion=gini, max_depth=10, max_features=None, min_samples_leaf=2, min_samples_split=5, n_estimators=40; total time=   0.6s
[CV] END bootstrap=False, criterion=gini, max_depth=30, max_features=sqrt, min_samples_leaf=2, min_samples_split=3, n_estimators=46; total time=   0.2s
[CV] END bootstrap=False, criterion=gini, max_depth=30, max_features=sqrt, min_samples_leaf=2, min_samples_split=3, n_estimators=46; total time=   0.3s
[CV] END bootstrap=False, criterion=gini, max_depth=30, max_features=sqrt, min_samples_leaf=2, min_samples_split=3, n_estimators=46; total time=   0.3s
[CV] END bootstrap=False, criterion=gini, max_depth=10, max_features=None, min_samples_leaf=2, min_samples_split=5, n_estimators=40; total time=   0.6s
[CV] END bootstrap=True, criterion=gini, max_depth=None, max_features=log2, min_samples_leaf=2, min_samples_split=10, n_estimators=52; total time=   0.2s
[CV] END bootstrap=True, criterion=gini, max_depth=None, max_features=log2, min_samples_leaf=2, min_samples_split=10, n_estimators=52; total time=   0.3s
[CV] END bootstrap=True, criterion=gini, max_depth=None, max_features=log2, min_samples_leaf=2, min_samples_split=10, n_estimators=52; total time=   0.3s
[CV] END bootstrap=True, criterion=gini, max_depth=None, max_features=log2, min_samples_leaf=2, min_samples_split=10, n_estimators=52; total time=   0.3s
[CV] END bootstrap=False, criterion=gini, max_depth=10, max_features=None, min_samples_leaf=2, min_samples_split=5, n_estimators=40; total time=   0.8s
[CV] END bootstrap=True, criterion=gini, max_depth=None, max_features=log2, min_samples_leaf=2, min_samples_split=10, n_estimators=52; total time=   0.3s
[CV] END bootstrap=True, criterion=entropy, max_depth=30, max_features=log2, min_samples_leaf=4, min_samples_split=7, n_estimators=46; total time=   0.2s
[CV] END bootstrap=True, criterion=entropy, max_depth=30, max_features=log2, min_samples_leaf=4, min_samples_split=7, n_estimators=46; total time=   0.2s
[CV] END bootstrap=True, criterion=entropy, max_depth=30, max_features=log2, min_samples_leaf=4, min_samples_split=7, n_estimators=46; total time=   0.2s
[CV] END bootstrap=True, criterion=entropy, max_depth=30, max_features=log2, min_samples_leaf=4, min_samples_split=7, n_estimators=46; total time=   0.2s
[CV] END bootstrap=True, criterion=gini, max_depth=20, max_features=log2, min_samples_leaf=2, min_samples_split=2, n_estimators=45; total time=   0.2s
[CV] END bootstrap=True, criterion=entropy, max_depth=30, max_features=log2, min_samples_leaf=4, min_samples_split=7, n_estimators=46; total time=   0.2s
[CV] END bootstrap=True, criterion=gini, max_depth=20, max_features=log2, min_samples_leaf=2, min_samples_split=2, n_estimators=45; total time=   0.2s
[CV] END bootstrap=True, criterion=gini, max_depth=20, max_features=log2, min_samples_leaf=2, min_samples_split=2, n_estimators=45; total time=   0.2s
[CV] END bootstrap=True, criterion=gini, max_depth=20, max_features=log2, min_samples_leaf=2, min_samples_split=2, n_estimators=45; total time=   0.2s
[CV] END bootstrap=True, criterion=gini, max_depth=20, max_features=log2, min_samples_leaf=2, min_samples_split=2, n_estimators=45; total time=   0.2s
[CV] END bootstrap=False, criterion=gini, max_depth=20, max_features=log2, min_samples_leaf=3, min_samples_split=2, n_estimators=51; total time=   0.2s
[CV] END bootstrap=False, criterion=gini, max_depth=20, max_features=log2, min_samples_leaf=3, min_samples_split=2, n_estimators=51; total time=   0.3s
[CV] END bootstrap=False, criterion=gini, max_depth=20, max_features=log2, min_samples_leaf=3, min_samples_split=2, n_estimators=51; total time=   0.3s
[CV] END bootstrap=False, criterion=gini, max_depth=20, max_features=log2, min_samples_leaf=3, min_samples_split=2, n_estimators=51; total time=   0.4s
[CV] END bootstrap=False, criterion=gini, max_depth=20, max_features=log2, min_samples_leaf=3, min_samples_split=2, n_estimators=51; total time=   0.4s
[CV] END bootstrap=False, criterion=entropy, max_depth=10, max_features=sqrt, min_samples_leaf=3, min_samples_split=5, n_estimators=58; total time=   0.4s
[CV] END bootstrap=False, criterion=entropy, max_depth=10, max_features=sqrt, min_samples_leaf=3, min_samples_split=5, n_estimators=58; total time=   0.4s
[CV] END bootstrap=False, criterion=entropy, max_depth=10, max_features=sqrt, min_samples_leaf=3, min_samples_split=5, n_estimators=58; total time=   0.4s
[CV] END bootstrap=False, criterion=entropy, max_depth=10, max_features=sqrt, min_samples_leaf=3, min_samples_split=5, n_estimators=58; total time=   0.4s
[CV] END bootstrap=False, criterion=entropy, max_depth=20, max_features=sqrt, min_samples_leaf=2, min_samples_split=4, n_estimators=54; total time=   0.3s
[CV] END bootstrap=False, criterion=entropy, max_depth=10, max_features=sqrt, min_samples_leaf=3, min_samples_split=5, n_estimators=58; total time=   0.4s
[CV] END bootstrap=False, criterion=entropy, max_depth=20, max_features=sqrt, min_samples_leaf=2, min_samples_split=4, n_estimators=54; total time=   0.3s
[CV] END bootstrap=False, criterion=entropy, max_depth=20, max_features=sqrt, min_samples_leaf=2, min_samples_split=4, n_estimators=54; total time=   0.3s
[CV] END bootstrap=False, criterion=entropy, max_depth=20, max_features=sqrt, min_samples_leaf=2, min_samples_split=4, n_estimators=54; total time=   0.3s
[CV] END bootstrap=False, criterion=entropy, max_depth=20, max_features=sqrt, min_samples_leaf=2, min_samples_split=4, n_estimators=54; total time=   0.3s
[CV] END bootstrap=True, criterion=entropy, max_depth=10, max_features=None, min_samples_leaf=2, min_samples_split=9, n_estimators=59; total time=   0.8s
[CV] END bootstrap=True, criterion=entropy, max_depth=10, max_features=None, min_samples_leaf=2, min_samples_split=9, n_estimators=59; total time=   0.8s
[CV] END bootstrap=True, criterion=entropy, max_depth=10, max_features=None, min_samples_leaf=2, min_samples_split=9, n_estimators=59; total time=   0.7s
[CV] END bootstrap=True, criterion=entropy, max_depth=10, max_features=None, min_samples_leaf=2, min_samples_split=9, n_estimators=59; total time=   0.7s
[CV] END bootstrap=True, criterion=entropy, max_depth=10, max_features=None, min_samples_leaf=2, min_samples_split=9, n_estimators=59; total time=   0.8s
[CV] END bootstrap=False, criterion=gini, max_depth=20, max_features=sqrt, min_samples_leaf=4, min_samples_split=10, n_estimators=48; total time=   0.2s
[CV] END bootstrap=False, criterion=gini, max_depth=20, max_features=sqrt, min_samples_leaf=4, min_samples_split=10, n_estimators=48; total time=   0.2s
[CV] END bootstrap=False, criterion=gini, max_depth=20, max_features=sqrt, min_samples_leaf=4, min_samples_split=10, n_estimators=48; total time=   0.2s
[CV] END bootstrap=False, criterion=entropy, max_depth=10, max_features=None, min_samples_leaf=3, min_samples_split=3, n_estimators=54; total time=   0.9s
[CV] END bootstrap=False, criterion=entropy, max_depth=10, max_features=None, min_samples_leaf=3, min_samples_split=3, n_estimators=54; total time=   0.9s
[CV] END bootstrap=False, criterion=entropy, max_depth=10, max_features=None, min_samples_leaf=3, min_samples_split=3, n_estimators=54; total time=   0.9s
[CV] END bootstrap=False, criterion=gini, max_depth=20, max_features=sqrt, min_samples_leaf=4, min_samples_split=10, n_estimators=48; total time=   0.2s
[CV] END bootstrap=False, criterion=gini, max_depth=20, max_features=sqrt, min_samples_leaf=4, min_samples_split=10, n_estimators=48; total time=   0.3s
[CV] END bootstrap=False, criterion=entropy, max_depth=None, max_features=log2, min_samples_leaf=2, min_samples_split=10, n_estimators=50; total time=   0.3s
[CV] END bootstrap=False, criterion=entropy, max_depth=None, max_features=log2, min_samples_leaf=2, min_samples_split=10, n_estimators=50; total time=   0.3s
[CV] END bootstrap=False, criterion=entropy, max_depth=None, max_features=log2, min_samples_leaf=2, min_samples_split=10, n_estimators=50; total time=   0.4s
[CV] END bootstrap=False, criterion=entropy, max_depth=None, max_features=log2, min_samples_leaf=2, min_samples_split=10, n_estimators=50; total time=   0.3s
[CV] END bootstrap=False, criterion=entropy, max_depth=None, max_features=log2, min_samples_leaf=2, min_samples_split=10, n_estimators=50; total time=   0.4s
[CV] END bootstrap=False, criterion=gini, max_depth=10, max_features=log2, min_samples_leaf=3, min_samples_split=7, n_estimators=55; total time=   0.3s
[CV] END bootstrap=False, criterion=entropy, max_depth=10, max_features=None, min_samples_leaf=3, min_samples_split=3, n_estimators=54; total time=   1.0s
[CV] END bootstrap=False, criterion=gini, max_depth=10, max_features=log2, min_samples_leaf=3, min_samples_split=7, n_estimators=55; total time=   0.3s
[CV] END bootstrap=False, criterion=entropy, max_depth=10, max_features=None, min_samples_leaf=3, min_samples_split=3, n_estimators=54; total time=   1.0s
[CV] END bootstrap=False, criterion=gini, max_depth=10, max_features=log2, min_samples_leaf=3, min_samples_split=7, n_estimators=55; total time=   0.3s
[CV] END bootstrap=False, criterion=gini, max_depth=10, max_features=log2, min_samples_leaf=3, min_samples_split=7, n_estimators=55; total time=   0.2s
[CV] END bootstrap=False, criterion=gini, max_depth=10, max_features=log2, min_samples_leaf=3, min_samples_split=7, n_estimators=55; total time=   0.3s
[CV] END bootstrap=True, criterion=gini, max_depth=10, max_features=sqrt, min_samples_leaf=2, min_samples_split=4, n_estimators=53; total time=   0.2s
[CV] END bootstrap=True, criterion=gini, max_depth=10, max_features=sqrt, min_samples_leaf=2, min_samples_split=4, n_estimators=53; total time=   0.2s
[CV] END bootstrap=True, criterion=gini, max_depth=10, max_features=sqrt, min_samples_leaf=2, min_samples_split=4, n_estimators=53; total time=   0.2s
[CV] END bootstrap=True, criterion=gini, max_depth=10, max_features=sqrt, min_samples_leaf=2, min_samples_split=4, n_estimators=53; total time=   0.2s
[CV] END bootstrap=True, criterion=gini, max_depth=10, max_features=sqrt, min_samples_leaf=2, min_samples_split=4, n_estimators=53; total time=   0.2s
[CV] END bootstrap=False, criterion=entropy, max_depth=None, max_features=sqrt, min_samples_leaf=3, min_samples_split=10, n_estimators=57; total time=   0.3s
[CV] END bootstrap=False, criterion=entropy, max_depth=None, max_features=sqrt, min_samples_leaf=3, min_samples_split=10, n_estimators=57; total time=   0.3s
[CV] END bootstrap=True, criterion=entropy, max_depth=20, max_features=log2, min_samples_leaf=3, min_samples_split=3, n_estimators=48; total time=   0.2s
[CV] END bootstrap=True, criterion=entropy, max_depth=20, max_features=log2, min_samples_leaf=3, min_samples_split=3, n_estimators=48; total time=   0.2s
[CV] END bootstrap=False, criterion=entropy, max_depth=None, max_features=sqrt, min_samples_leaf=3, min_samples_split=10, n_estimators=57; total time=   0.4s
[CV] END bootstrap=False, criterion=entropy, max_depth=None, max_features=sqrt, min_samples_leaf=3, min_samples_split=10, n_estimators=57; total time=   0.3s
[CV] END bootstrap=True, criterion=entropy, max_depth=20, max_features=log2, min_samples_leaf=3, min_samples_split=3, n_estimators=48; total time=   0.2s
[CV] END bootstrap=False, criterion=entropy, max_depth=None, max_features=sqrt, min_samples_leaf=3, min_samples_split=10, n_estimators=57; total time=   0.3s
[CV] END bootstrap=True, criterion=entropy, max_depth=20, max_features=log2, min_samples_leaf=3, min_samples_split=3, n_estimators=48; total time=   0.2s
[CV] END bootstrap=True, criterion=entropy, max_depth=20, max_features=log2, min_samples_leaf=3, min_samples_split=3, n_estimators=48; total time=   0.3s
[CV] END bootstrap=False, criterion=entropy, max_depth=None, max_features=sqrt, min_samples_leaf=3, min_samples_split=2, n_estimators=40; total time=   0.3s
[CV] END bootstrap=False, criterion=entropy, max_depth=None, max_features=sqrt, min_samples_leaf=3, min_samples_split=2, n_estimators=40; total time=   0.3s
[CV] END bootstrap=False, criterion=entropy, max_depth=None, max_features=sqrt, min_samples_leaf=3, min_samples_split=2, n_estimators=40; total time=   0.3s
[CV] END bootstrap=False, criterion=entropy, max_depth=None, max_features=sqrt, min_samples_leaf=3, min_samples_split=2, n_estimators=40; total time=   0.3s
[CV] END bootstrap=True, criterion=entropy, max_depth=30, max_features=sqrt, min_samples_leaf=4, min_samples_split=7, n_estimators=51; total time=   0.2s
[CV] END bootstrap=False, criterion=entropy, max_depth=None, max_features=sqrt, min_samples_leaf=3, min_samples_split=2, n_estimators=40; total time=   0.3s
[CV] END bootstrap=True, criterion=entropy, max_depth=30, max_features=sqrt, min_samples_leaf=4, min_samples_split=7, n_estimators=51; total time=   0.3s
[CV] END bootstrap=True, criterion=entropy, max_depth=30, max_features=sqrt, min_samples_leaf=4, min_samples_split=7, n_estimators=51; total time=   0.3s
[CV] END bootstrap=True, criterion=gini, max_depth=10, max_features=sqrt, min_samples_leaf=4, min_samples_split=4, n_estimators=50; total time=   0.2s
[CV] END bootstrap=True, criterion=entropy, max_depth=30, max_features=sqrt, min_samples_leaf=4, min_samples_split=7, n_estimators=51; total time=   0.3s
[CV] END bootstrap=True, criterion=gini, max_depth=10, max_features=sqrt, min_samples_leaf=4, min_samples_split=4, n_estimators=50; total time=   0.2s
[CV] END bootstrap=True, criterion=entropy, max_depth=30, max_features=sqrt, min_samples_leaf=4, min_samples_split=7, n_estimators=51; total time=   0.3s
[CV] END bootstrap=True, criterion=gini, max_depth=10, max_features=sqrt, min_samples_leaf=4, min_samples_split=4, n_estimators=50; total time=   0.2s
[CV] END bootstrap=True, criterion=gini, max_depth=10, max_features=sqrt, min_samples_leaf=4, min_samples_split=4, n_estimators=50; total time=   0.2s
[CV] END bootstrap=True, criterion=gini, max_depth=10, max_features=sqrt, min_samples_leaf=4, min_samples_split=4, n_estimators=50; total time=   0.2s
[CV] END bootstrap=False, criterion=entropy, max_depth=20, max_features=log2, min_samples_leaf=2, min_samples_split=10, n_estimators=57; total time=   0.4s
[CV] END bootstrap=True, criterion=gini, max_depth=None, max_features=log2, min_samples_leaf=1, min_samples_split=7, n_estimators=55; total time=   0.2s
[CV] END bootstrap=False, criterion=entropy, max_depth=20, max_features=log2, min_samples_leaf=2, min_samples_split=10, n_estimators=57; total time=   0.3s
[CV] END bootstrap=False, criterion=entropy, max_depth=20, max_features=log2, min_samples_leaf=2, min_samples_split=10, n_estimators=57; total time=   0.3s
[CV] END bootstrap=False, criterion=entropy, max_depth=20, max_features=log2, min_samples_leaf=2, min_samples_split=10, n_estimators=57; total time=   0.3s
[CV] END bootstrap=True, criterion=gini, max_depth=None, max_features=log2, min_samples_leaf=1, min_samples_split=7, n_estimators=55; total time=   0.2s
[CV] END bootstrap=True, criterion=gini, max_depth=None, max_features=log2, min_samples_leaf=1, min_samples_split=7, n_estimators=55; total time=   0.3s
[CV] END bootstrap=False, criterion=entropy, max_depth=20, max_features=log2, min_samples_leaf=2, min_samples_split=10, n_estimators=57; total time=   0.3s
[CV] END bootstrap=True, criterion=gini, max_depth=None, max_features=log2, min_samples_leaf=1, min_samples_split=7, n_estimators=55; total time=   0.2s
[CV] END bootstrap=True, criterion=entropy, max_depth=30, max_features=log2, min_samples_leaf=4, min_samples_split=4, n_estimators=40; total time=   0.2s
[CV] END bootstrap=True, criterion=gini, max_depth=None, max_features=log2, min_samples_leaf=1, min_samples_split=7, n_estimators=55; total time=   0.2s
[CV] END bootstrap=True, criterion=entropy, max_depth=30, max_features=log2, min_samples_leaf=4, min_samples_split=4, n_estimators=40; total time=   0.2s
[CV] END bootstrap=True, criterion=entropy, max_depth=30, max_features=log2, min_samples_leaf=4, min_samples_split=4, n_estimators=40; total time=   0.2s
[CV] END bootstrap=True, criterion=entropy, max_depth=30, max_features=log2, min_samples_leaf=4, min_samples_split=4, n_estimators=40; total time=   0.2s
[CV] END bootstrap=True, criterion=entropy, max_depth=30, max_features=log2, min_samples_leaf=4, min_samples_split=4, n_estimators=40; total time=   0.2s
[CV] END bootstrap=False, criterion=gini, max_depth=None, max_features=sqrt, min_samples_leaf=1, min_samples_split=7, n_estimators=42; total time=   0.2s
[CV] END bootstrap=False, criterion=gini, max_depth=None, max_features=sqrt, min_samples_leaf=1, min_samples_split=7, n_estimators=42; total time=   0.2s
[CV] END bootstrap=False, criterion=gini, max_depth=None, max_features=sqrt, min_samples_leaf=1, min_samples_split=7, n_estimators=42; total time=   0.2s
[CV] END bootstrap=False, criterion=gini, max_depth=None, max_features=sqrt, min_samples_leaf=1, min_samples_split=7, n_estimators=42; total time=   0.2s
[CV] END bootstrap=False, criterion=gini, max_depth=None, max_features=sqrt, min_samples_leaf=1, min_samples_split=7, n_estimators=42; total time=   0.2s
[CV] END bootstrap=True, criterion=entropy, max_depth=20, max_features=None, min_samples_leaf=1, min_samples_split=7, n_estimators=55; total time=   0.6s
[CV] END bootstrap=True, criterion=entropy, max_depth=20, max_features=None, min_samples_leaf=1, min_samples_split=7, n_estimators=55; total time=   0.7s
[CV] END bootstrap=True, criterion=entropy, max_depth=20, max_features=None, min_samples_leaf=1, min_samples_split=7, n_estimators=55; total time=   0.7s
[CV] END bootstrap=True, criterion=entropy, max_depth=20, max_features=None, min_samples_leaf=1, min_samples_split=7, n_estimators=55; total time=   0.7s
[CV] END bootstrap=False, criterion=gini, max_depth=30, max_features=sqrt, min_samples_leaf=2, min_samples_split=3, n_estimators=57; total time=   0.4s
[CV] END bootstrap=True, criterion=entropy, max_depth=20, max_features=None, min_samples_leaf=1, min_samples_split=7, n_estimators=55; total time=   0.8s
[CV] END bootstrap=False, criterion=gini, max_depth=30, max_features=sqrt, min_samples_leaf=2, min_samples_split=3, n_estimators=57; total time=   0.4s
[CV] END bootstrap=False, criterion=gini, max_depth=30, max_features=sqrt, min_samples_leaf=2, min_samples_split=3, n_estimators=57; total time=   0.4s
[CV] END bootstrap=False, criterion=gini, max_depth=30, max_features=sqrt, min_samples_leaf=2, min_samples_split=3, n_estimators=57; total time=   0.4s
[CV] END bootstrap=True, criterion=entropy, max_depth=10, max_features=log2, min_samples_leaf=3, min_samples_split=2, n_estimators=47; total time=   0.3s
[CV] END bootstrap=True, criterion=entropy, max_depth=10, max_features=log2, min_samples_leaf=3, min_samples_split=2, n_estimators=47; total time=   0.3s
[CV] END bootstrap=False, criterion=gini, max_depth=30, max_features=sqrt, min_samples_leaf=2, min_samples_split=3, n_estimators=57; total time=   0.4s
[CV] END bootstrap=True, criterion=entropy, max_depth=10, max_features=log2, min_samples_leaf=3, min_samples_split=2, n_estimators=47; total time=   0.3s
[CV] END bootstrap=True, criterion=entropy, max_depth=10, max_features=log2, min_samples_leaf=3, min_samples_split=2, n_estimators=47; total time=   0.3s
[CV] END bootstrap=True, criterion=entropy, max_depth=10, max_features=log2, min_samples_leaf=3, min_samples_split=2, n_estimators=47; total time=   0.3s
[CV] END bootstrap=False, criterion=gini, max_depth=None, max_features=None, min_samples_leaf=2, min_samples_split=4, n_estimators=42; total time=   0.7s
[CV] END bootstrap=False, criterion=gini, max_depth=None, max_features=None, min_samples_leaf=2, min_samples_split=4, n_estimators=42; total time=   0.7s
[CV] END bootstrap=False, criterion=gini, max_depth=None, max_features=None, min_samples_leaf=2, min_samples_split=4, n_estimators=42; total time=   0.6s
[CV] END bootstrap=False, criterion=entropy, max_depth=None, max_features=None, min_samples_leaf=4, min_samples_split=4, n_estimators=48; total time=   0.9s
[CV] END bootstrap=False, criterion=entropy, max_depth=None, max_features=None, min_samples_leaf=4, min_samples_split=4, n_estimators=48; total time=   0.8s
[CV] END bootstrap=False, criterion=entropy, max_depth=None, max_features=None, min_samples_leaf=4, min_samples_split=4, n_estimators=48; total time=   0.9s
[CV] END bootstrap=False, criterion=entropy, max_depth=None, max_features=None, min_samples_leaf=4, min_samples_split=4, n_estimators=48; total time=   0.8s
[CV] END bootstrap=False, criterion=entropy, max_depth=None, max_features=None, min_samples_leaf=4, min_samples_split=4, n_estimators=48; total time=   0.8s
[CV] END bootstrap=False, criterion=entropy, max_depth=10, max_features=sqrt, min_samples_leaf=3, min_samples_split=6, n_estimators=51; total time=   0.3s
[CV] END bootstrap=False, criterion=entropy, max_depth=10, max_features=sqrt, min_samples_leaf=3, min_samples_split=6, n_estimators=51; total time=   0.2s
[CV] END bootstrap=False, criterion=entropy, max_depth=10, max_features=sqrt, min_samples_leaf=3, min_samples_split=6, n_estimators=51; total time=   0.2s
[CV] END bootstrap=False, criterion=entropy, max_depth=10, max_features=sqrt, min_samples_leaf=3, min_samples_split=6, n_estimators=51; total time=   0.2s
[CV] END bootstrap=False, criterion=entropy, max_depth=10, max_features=sqrt, min_samples_leaf=3, min_samples_split=6, n_estimators=51; total time=   0.2s
[CV] END bootstrap=False, criterion=entropy, max_depth=None, max_features=sqrt, min_samples_leaf=1, min_samples_split=4, n_estimators=48; total time=   0.3s
[CV] END bootstrap=False, criterion=gini, max_depth=None, max_features=None, min_samples_leaf=2, min_samples_split=4, n_estimators=42; total time=   0.5s
[CV] END bootstrap=False, criterion=entropy, max_depth=None, max_features=sqrt, min_samples_leaf=1, min_samples_split=4, n_estimators=48; total time=   0.3s
[CV] END bootstrap=False, criterion=entropy, max_depth=None, max_features=sqrt, min_samples_leaf=1, min_samples_split=4, n_estimators=48; total time=   0.3s
[CV] END bootstrap=False, criterion=entropy, max_depth=None, max_features=sqrt, min_samples_leaf=1, min_samples_split=4, n_estimators=48; total time=   0.3s
[CV] END bootstrap=False, criterion=entropy, max_depth=None, max_features=sqrt, min_samples_leaf=1, min_samples_split=4, n_estimators=48; total time=   0.3s
[CV] END bootstrap=False, criterion=gini, max_depth=None, max_features=None, min_samples_leaf=2, min_samples_split=4, n_estimators=42; total time=   0.6s
[CV] END bootstrap=False, criterion=gini, max_depth=30, max_features=sqrt, min_samples_leaf=2, min_samples_split=4, n_estimators=59; total time=   0.3s
[CV] END bootstrap=False, criterion=gini, max_depth=30, max_features=sqrt, min_samples_leaf=2, min_samples_split=4, n_estimators=59; total time=   0.3s
[CV] END bootstrap=False, criterion=gini, max_depth=30, max_features=sqrt, min_samples_leaf=2, min_samples_split=4, n_estimators=59; total time=   0.3s
[CV] END bootstrap=False, criterion=gini, max_depth=30, max_features=sqrt, min_samples_leaf=2, min_samples_split=4, n_estimators=59; total time=   0.3s
[CV] END bootstrap=False, criterion=gini, max_depth=30, max_features=sqrt, min_samples_leaf=2, min_samples_split=4, n_estimators=59; total time=   0.3s
[CV] END bootstrap=True, criterion=gini, max_depth=30, max_features=None, min_samples_leaf=2, min_samples_split=9, n_estimators=46; total time=   0.4s
[CV] END bootstrap=True, criterion=gini, max_depth=30, max_features=None, min_samples_leaf=1, min_samples_split=4, n_estimators=59; total time=   0.6s
[CV] END bootstrap=True, criterion=gini, max_depth=30, max_features=None, min_samples_leaf=1, min_samples_split=4, n_estimators=59; total time=   0.6s
[CV] END bootstrap=True, criterion=gini, max_depth=30, max_features=None, min_samples_leaf=2, min_samples_split=9, n_estimators=46; total time=   0.4s
[CV] END bootstrap=True, criterion=gini, max_depth=30, max_features=None, min_samples_leaf=1, min_samples_split=4, n_estimators=59; total time=   0.6s
[CV] END bootstrap=True, criterion=gini, max_depth=30, max_features=None, min_samples_leaf=1, min_samples_split=4, n_estimators=59; total time=   0.6s
[CV] END bootstrap=True, criterion=gini, max_depth=30, max_features=None, min_samples_leaf=1, min_samples_split=4, n_estimators=59; total time=   0.6s
[CV] END bootstrap=True, criterion=gini, max_depth=30, max_features=None, min_samples_leaf=2, min_samples_split=9, n_estimators=46; total time=   0.4s
[CV] END bootstrap=True, criterion=entropy, max_depth=None, max_features=log2, min_samples_leaf=4, min_samples_split=10, n_estimators=58; total time=   0.2s
[CV] END bootstrap=True, criterion=gini, max_depth=30, max_features=None, min_samples_leaf=2, min_samples_split=9, n_estimators=46; total time=   0.4s
[CV] END bootstrap=True, criterion=gini, max_depth=30, max_features=None, min_samples_leaf=2, min_samples_split=9, n_estimators=46; total time=   0.4s
[CV] END bootstrap=True, criterion=entropy, max_depth=20, max_features=None, min_samples_leaf=2, min_samples_split=3, n_estimators=50; total time=   0.5s
[CV] END bootstrap=True, criterion=entropy, max_depth=None, max_features=log2, min_samples_leaf=4, min_samples_split=10, n_estimators=58; total time=   0.2s
[CV] END bootstrap=True, criterion=entropy, max_depth=20, max_features=None, min_samples_leaf=2, min_samples_split=3, n_estimators=50; total time=   0.5s
[CV] END bootstrap=True, criterion=entropy, max_depth=20, max_features=None, min_samples_leaf=2, min_samples_split=3, n_estimators=50; total time=   0.5s
[CV] END bootstrap=True, criterion=entropy, max_depth=20, max_features=None, min_samples_leaf=2, min_samples_split=3, n_estimators=50; total time=   0.5s
[CV] END bootstrap=True, criterion=entropy, max_depth=20, max_features=None, min_samples_leaf=2, min_samples_split=3, n_estimators=50; total time=   0.5s
[CV] END bootstrap=True, criterion=entropy, max_depth=None, max_features=log2, min_samples_leaf=4, min_samples_split=10, n_estimators=58; total time=   0.2s
[CV] END bootstrap=True, criterion=entropy, max_depth=None, max_features=log2, min_samples_leaf=4, min_samples_split=10, n_estimators=58; total time=   0.2s
[CV] END bootstrap=True, criterion=entropy, max_depth=None, max_features=log2, min_samples_leaf=4, min_samples_split=10, n_estimators=58; total time=   0.2s
[CV] END bootstrap=True, criterion=entropy, max_depth=30, max_features=sqrt, min_samples_leaf=3, min_samples_split=6, n_estimators=51; total time=   0.2s
[CV] END bootstrap=True, criterion=entropy, max_depth=30, max_features=sqrt, min_samples_leaf=3, min_samples_split=6, n_estimators=51; total time=   0.2s
[CV] END bootstrap=True, criterion=entropy, max_depth=30, max_features=sqrt, min_samples_leaf=3, min_samples_split=6, n_estimators=51; total time=   0.2s
[CV] END bootstrap=True, criterion=entropy, max_depth=30, max_features=sqrt, min_samples_leaf=3, min_samples_split=6, n_estimators=51; total time=   0.2s
[CV] END bootstrap=True, criterion=entropy, max_depth=30, max_features=sqrt, min_samples_leaf=3, min_samples_split=6, n_estimators=51; total time=   0.3s
[CV] END bootstrap=True, criterion=entropy, max_depth=10, max_features=log2, min_samples_leaf=3, min_samples_split=6, n_estimators=51; total time=   0.3s
[CV] END bootstrap=True, criterion=entropy, max_depth=10, max_features=log2, min_samples_leaf=3, min_samples_split=6, n_estimators=51; total time=   0.4s
[CV] END bootstrap=True, criterion=entropy, max_depth=10, max_features=log2, min_samples_leaf=3, min_samples_split=6, n_estimators=51; total time=   0.3s
[CV] END bootstrap=True, criterion=gini, max_depth=10, max_features=log2, min_samples_leaf=4, min_samples_split=7, n_estimators=42; total time=   0.2s
[CV] END bootstrap=True, criterion=entropy, max_depth=10, max_features=log2, min_samples_leaf=3, min_samples_split=6, n_estimators=51; total time=   0.4s
[CV] END bootstrap=True, criterion=gini, max_depth=10, max_features=log2, min_samples_leaf=4, min_samples_split=7, n_estimators=42; total time=   0.2s
[CV] END bootstrap=True, criterion=entropy, max_depth=10, max_features=log2, min_samples_leaf=3, min_samples_split=6, n_estimators=51; total time=   0.4s
[CV] END bootstrap=True, criterion=gini, max_depth=10, max_features=log2, min_samples_leaf=4, min_samples_split=7, n_estimators=42; total time=   0.2s
[CV] END bootstrap=True, criterion=gini, max_depth=10, max_features=log2, min_samples_leaf=4, min_samples_split=7, n_estimators=42; total time=   0.2s
[CV] END bootstrap=True, criterion=gini, max_depth=10, max_features=log2, min_samples_leaf=4, min_samples_split=7, n_estimators=42; total time=   0.2s
[CV] END bootstrap=True, criterion=gini, max_depth=20, max_features=None, min_samples_leaf=3, min_samples_split=10, n_estimators=40; total time=   0.5s
[CV] END bootstrap=True, criterion=gini, max_depth=20, max_features=None, min_samples_leaf=3, min_samples_split=10, n_estimators=40; total time=   0.5s
[CV] END bootstrap=True, criterion=gini, max_depth=20, max_features=None, min_samples_leaf=3, min_samples_split=10, n_estimators=40; total time=   0.5s
[CV] END bootstrap=True, criterion=gini, max_depth=20, max_features=None, min_samples_leaf=3, min_samples_split=10, n_estimators=40; total time=   0.5s
[CV] END bootstrap=True, criterion=gini, max_depth=20, max_features=None, min_samples_leaf=3, min_samples_split=10, n_estimators=40; total time=   0.4s
[CV] END bootstrap=True, criterion=gini, max_depth=10, max_features=sqrt, min_samples_leaf=1, min_samples_split=5, n_estimators=45; total time=   0.2s
[CV] END bootstrap=True, criterion=gini, max_depth=10, max_features=sqrt, min_samples_leaf=1, min_samples_split=5, n_estimators=45; total time=   0.2s
[CV] END bootstrap=True, criterion=gini, max_depth=10, max_features=sqrt, min_samples_leaf=1, min_samples_split=5, n_estimators=45; total time=   0.2s
[CV] END bootstrap=True, criterion=gini, max_depth=10, max_features=sqrt, min_samples_leaf=1, min_samples_split=5, n_estimators=45; total time=   0.2s
[CV] END bootstrap=True, criterion=gini, max_depth=10, max_features=sqrt, min_samples_leaf=1, min_samples_split=5, n_estimators=45; total time=   0.2s
[CV] END bootstrap=False, criterion=gini, max_depth=20, max_features=None, min_samples_leaf=4, min_samples_split=10, n_estimators=59; total time=   0.8s
[CV] END bootstrap=False, criterion=gini, max_depth=20, max_features=None, min_samples_leaf=4, min_samples_split=10, n_estimators=59; total time=   0.9s
[CV] END bootstrap=False, criterion=gini, max_depth=20, max_features=None, min_samples_leaf=4, min_samples_split=10, n_estimators=59; total time=   0.9s
[CV] END bootstrap=False, criterion=entropy, max_depth=20, max_features=sqrt, min_samples_leaf=3, min_samples_split=2, n_estimators=47; total time=   0.3s
[CV] END bootstrap=False, criterion=gini, max_depth=20, max_features=None, min_samples_leaf=4, min_samples_split=10, n_estimators=59; total time=   0.9s
[CV] END bootstrap=False, criterion=entropy, max_depth=20, max_features=sqrt, min_samples_leaf=3, min_samples_split=2, n_estimators=47; total time=   0.4s
[CV] END bootstrap=False, criterion=entropy, max_depth=20, max_features=sqrt, min_samples_leaf=3, min_samples_split=2, n_estimators=47; total time=   0.4s
[CV] END bootstrap=False, criterion=gini, max_depth=None, max_features=sqrt, min_samples_leaf=1, min_samples_split=3, n_estimators=44; total time=   0.3s
[CV] END bootstrap=False, criterion=entropy, max_depth=20, max_features=sqrt, min_samples_leaf=3, min_samples_split=2, n_estimators=47; total time=   0.4s
[CV] END bootstrap=False, criterion=entropy, max_depth=20, max_features=sqrt, min_samples_leaf=3, min_samples_split=2, n_estimators=47; total time=   0.4s
[CV] END bootstrap=False, criterion=gini, max_depth=None, max_features=sqrt, min_samples_leaf=1, min_samples_split=3, n_estimators=44; total time=   0.3s
[CV] END bootstrap=False, criterion=gini, max_depth=20, max_features=None, min_samples_leaf=4, min_samples_split=10, n_estimators=59; total time=   1.1s
[CV] END bootstrap=False, criterion=gini, max_depth=None, max_features=sqrt, min_samples_leaf=1, min_samples_split=3, n_estimators=44; total time=   0.3s
[CV] END bootstrap=False, criterion=gini, max_depth=None, max_features=sqrt, min_samples_leaf=1, min_samples_split=3, n_estimators=44; total time=   0.3s
[CV] END bootstrap=True, criterion=gini, max_depth=10, max_features=log2, min_samples_leaf=2, min_samples_split=5, n_estimators=54; total time=   0.2s
[CV] END bootstrap=False, criterion=gini, max_depth=None, max_features=sqrt, min_samples_leaf=1, min_samples_split=3, n_estimators=44; total time=   0.3s
[CV] END bootstrap=True, criterion=gini, max_depth=10, max_features=log2, min_samples_leaf=2, min_samples_split=5, n_estimators=54; total time=   0.2s
[CV] END bootstrap=True, criterion=gini, max_depth=10, max_features=log2, min_samples_leaf=2, min_samples_split=5, n_estimators=54; total time=   0.2s
[CV] END bootstrap=True, criterion=gini, max_depth=10, max_features=log2, min_samples_leaf=2, min_samples_split=5, n_estimators=54; total time=   0.2s
[CV] END bootstrap=True, criterion=gini, max_depth=10, max_features=log2, min_samples_leaf=2, min_samples_split=5, n_estimators=54; total time=   0.2s
[CV] END bootstrap=True, criterion=entropy, max_depth=30, max_features=None, min_samples_leaf=1, min_samples_split=6, n_estimators=46; total time=   0.5s
[CV] END bootstrap=True, criterion=entropy, max_depth=30, max_features=None, min_samples_leaf=1, min_samples_split=6, n_estimators=46; total time=   0.5s
[CV] END bootstrap=True, criterion=entropy, max_depth=30, max_features=None, min_samples_leaf=1, min_samples_split=6, n_estimators=46; total time=   0.5s
[CV] END bootstrap=False, criterion=gini, max_depth=30, max_features=None, min_samples_leaf=2, min_samples_split=10, n_estimators=59; total time=   0.7s
[CV] END bootstrap=False, criterion=gini, max_depth=30, max_features=None, min_samples_leaf=2, min_samples_split=10, n_estimators=59; total time=   0.8s
[CV] END bootstrap=False, criterion=gini, max_depth=30, max_features=None, min_samples_leaf=2, min_samples_split=10, n_estimators=59; total time=   0.9s
[CV] END bootstrap=False, criterion=gini, max_depth=30, max_features=None, min_samples_leaf=2, min_samples_split=10, n_estimators=59; total time=   0.9s
[CV] END bootstrap=False, criterion=gini, max_depth=30, max_features=None, min_samples_leaf=2, min_samples_split=10, n_estimators=59; total time=   0.8s
[CV] END bootstrap=True, criterion=entropy, max_depth=30, max_features=None, min_samples_leaf=1, min_samples_split=6, n_estimators=46; total time=   0.6s
[CV] END bootstrap=True, criterion=entropy, max_depth=30, max_features=None, min_samples_leaf=1, min_samples_split=6, n_estimators=46; total time=   0.6s
[CV] END bootstrap=False, criterion=gini, max_depth=30, max_features=None, min_samples_leaf=3, min_samples_split=8, n_estimators=44; total time=   0.7s
[CV] END bootstrap=False, criterion=gini, max_depth=30, max_features=None, min_samples_leaf=3, min_samples_split=8, n_estimators=44; total time=   0.7s
[CV] END bootstrap=False, criterion=gini, max_depth=30, max_features=None, min_samples_leaf=3, min_samples_split=8, n_estimators=44; total time=   0.6s
[CV] END bootstrap=False, criterion=gini, max_depth=30, max_features=None, min_samples_leaf=3, min_samples_split=8, n_estimators=44; total time=   0.7s
[CV] END bootstrap=False, criterion=gini, max_depth=30, max_features=None, min_samples_leaf=3, min_samples_split=8, n_estimators=44; total time=   0.7s
[CV] END bootstrap=False, criterion=gini, max_depth=30, max_features=None, min_samples_leaf=3, min_samples_split=4, n_estimators=49; total time=   0.8s
[CV] END bootstrap=False, criterion=gini, max_depth=30, max_features=None, min_samples_leaf=3, min_samples_split=4, n_estimators=49; total time=   0.7s
[CV] END bootstrap=False, criterion=gini, max_depth=30, max_features=None, min_samples_leaf=3, min_samples_split=4, n_estimators=49; total time=   0.7s
[CV] END bootstrap=True, criterion=gini, max_depth=None, max_features=log2, min_samples_leaf=2, min_samples_split=2, n_estimators=40; total time=   0.1s
[CV] END bootstrap=False, criterion=gini, max_depth=30, max_features=None, min_samples_leaf=3, min_samples_split=4, n_estimators=49; total time=   0.8s
[CV] END bootstrap=True, criterion=entropy, max_depth=10, max_features=None, min_samples_leaf=3, min_samples_split=3, n_estimators=58; total time=   0.6s
[CV] END bootstrap=True, criterion=entropy, max_depth=10, max_features=None, min_samples_leaf=3, min_samples_split=3, n_estimators=58; total time=   0.6s
[CV] END bootstrap=True, criterion=gini, max_depth=None, max_features=log2, min_samples_leaf=2, min_samples_split=2, n_estimators=40; total time=   0.2s
[CV] END bootstrap=True, criterion=entropy, max_depth=10, max_features=None, min_samples_leaf=3, min_samples_split=3, n_estimators=58; total time=   0.6s
[CV] END bootstrap=True, criterion=gini, max_depth=None, max_features=log2, min_samples_leaf=2, min_samples_split=2, n_estimators=40; total time=   0.1s
[CV] END bootstrap=False, criterion=gini, max_depth=30, max_features=None, min_samples_leaf=3, min_samples_split=4, n_estimators=49; total time=   0.8s
[CV] END bootstrap=True, criterion=gini, max_depth=None, max_features=log2, min_samples_leaf=2, min_samples_split=2, n_estimators=40; total time=   0.1s
[CV] END bootstrap=True, criterion=gini, max_depth=None, max_features=log2, min_samples_leaf=2, min_samples_split=2, n_estimators=40; total time=   0.1s
[CV] END bootstrap=True, criterion=entropy, max_depth=10, max_features=None, min_samples_leaf=3, min_samples_split=3, n_estimators=58; total time=   0.6s
[CV] END bootstrap=True, criterion=entropy, max_depth=10, max_features=None, min_samples_leaf=3, min_samples_split=3, n_estimators=58; total time=   0.6s
[CV] END bootstrap=True, criterion=gini, max_depth=20, max_features=None, min_samples_leaf=3, min_samples_split=3, n_estimators=45; total time=   0.4s
[CV] END bootstrap=True, criterion=gini, max_depth=20, max_features=None, min_samples_leaf=3, min_samples_split=3, n_estimators=45; total time=   0.4s
[CV] END bootstrap=True, criterion=gini, max_depth=20, max_features=None, min_samples_leaf=3, min_samples_split=3, n_estimators=45; total time=   0.4s
[CV] END bootstrap=True, criterion=gini, max_depth=20, max_features=None, min_samples_leaf=3, min_samples_split=3, n_estimators=45; total time=   0.4s
[CV] END bootstrap=True, criterion=gini, max_depth=20, max_features=None, min_samples_leaf=3, min_samples_split=3, n_estimators=45; total time=   0.4s
[CV] END bootstrap=False, criterion=entropy, max_depth=10, max_features=log2, min_samples_leaf=3, min_samples_split=9, n_estimators=46; total time=   0.2s
[CV] END bootstrap=False, criterion=entropy, max_depth=10, max_features=log2, min_samples_leaf=3, min_samples_split=9, n_estimators=46; total time=   0.2s
[CV] END bootstrap=False, criterion=gini, max_depth=20, max_features=None, min_samples_leaf=2, min_samples_split=4, n_estimators=46; total time=   0.6s
[CV] END bootstrap=False, criterion=entropy, max_depth=10, max_features=log2, min_samples_leaf=3, min_samples_split=9, n_estimators=46; total time=   0.3s
[CV] END bootstrap=False, criterion=gini, max_depth=20, max_features=None, min_samples_leaf=2, min_samples_split=4, n_estimators=46; total time=   0.7s
[CV] END bootstrap=False, criterion=gini, max_depth=20, max_features=None, min_samples_leaf=2, min_samples_split=4, n_estimators=46; total time=   0.6s
[CV] END bootstrap=False, criterion=entropy, max_depth=10, max_features=log2, min_samples_leaf=3, min_samples_split=9, n_estimators=46; total time=   0.3s
[CV] END bootstrap=False, criterion=entropy, max_depth=10, max_features=log2, min_samples_leaf=3, min_samples_split=9, n_estimators=46; total time=   0.2s
[CV] END bootstrap=False, criterion=gini, max_depth=20, max_features=None, min_samples_leaf=2, min_samples_split=4, n_estimators=46; total time=   0.6s
[CV] END bootstrap=False, criterion=gini, max_depth=10, max_features=sqrt, min_samples_leaf=4, min_samples_split=4, n_estimators=59; total time=   0.2s
[CV] END bootstrap=False, criterion=gini, max_depth=10, max_features=sqrt, min_samples_leaf=4, min_samples_split=4, n_estimators=59; total time=   0.2s
[CV] END bootstrap=False, criterion=gini, max_depth=10, max_features=sqrt, min_samples_leaf=4, min_samples_split=4, n_estimators=59; total time=   0.3s
[CV] END bootstrap=False, criterion=gini, max_depth=20, max_features=None, min_samples_leaf=2, min_samples_split=4, n_estimators=46; total time=   0.7s
[CV] END bootstrap=False, criterion=gini, max_depth=10, max_features=sqrt, min_samples_leaf=4, min_samples_split=4, n_estimators=59; total time=   0.3s
[CV] END bootstrap=False, criterion=gini, max_depth=10, max_features=sqrt, min_samples_leaf=4, min_samples_split=4, n_estimators=59; total time=   0.3s
[CV] END bootstrap=False, criterion=gini, max_depth=10, max_features=sqrt, min_samples_leaf=2, min_samples_split=10, n_estimators=59; total time=   0.3s
[CV] END bootstrap=False, criterion=gini, max_depth=10, max_features=sqrt, min_samples_leaf=2, min_samples_split=10, n_estimators=59; total time=   0.3s
[CV] END bootstrap=False, criterion=gini, max_depth=10, max_features=sqrt, min_samples_leaf=2, min_samples_split=10, n_estimators=59; total time=   0.3s
[CV] END bootstrap=False, criterion=gini, max_depth=10, max_features=sqrt, min_samples_leaf=2, min_samples_split=10, n_estimators=59; total time=   0.3s
[CV] END bootstrap=False, criterion=gini, max_depth=10, max_features=sqrt, min_samples_leaf=2, min_samples_split=10, n_estimators=59; total time=   0.3s
[CV] END bootstrap=True, criterion=entropy, max_depth=None, max_features=None, min_samples_leaf=1, min_samples_split=6, n_estimators=54; total time=   0.6s
[CV] END bootstrap=True, criterion=entropy, max_depth=None, max_features=None, min_samples_leaf=1, min_samples_split=6, n_estimators=54; total time=   0.6s
[CV] END bootstrap=True, criterion=gini, max_depth=20, max_features=None, min_samples_leaf=2, min_samples_split=9, n_estimators=50; total time=   0.5s
[CV] END bootstrap=True, criterion=gini, max_depth=20, max_features=None, min_samples_leaf=2, min_samples_split=9, n_estimators=50; total time=   0.5s
[CV] END bootstrap=True, criterion=entropy, max_depth=None, max_features=None, min_samples_leaf=1, min_samples_split=6, n_estimators=54; total time=   0.6s
[CV] END bootstrap=True, criterion=entropy, max_depth=None, max_features=None, min_samples_leaf=1, min_samples_split=6, n_estimators=54; total time=   0.6s
[CV] END bootstrap=True, criterion=gini, max_depth=20, max_features=None, min_samples_leaf=2, min_samples_split=9, n_estimators=50; total time=   0.5s
[CV] END bootstrap=True, criterion=entropy, max_depth=None, max_features=None, min_samples_leaf=1, min_samples_split=6, n_estimators=54; total time=   0.6s
[CV] END bootstrap=False, criterion=entropy, max_depth=20, max_features=log2, min_samples_leaf=2, min_samples_split=10, n_estimators=55; total time=   0.3s
[CV] END bootstrap=False, criterion=gini, max_depth=30, max_features=sqrt, min_samples_leaf=2, min_samples_split=8, n_estimators=42; total time=   0.2s
[CV] END bootstrap=False, criterion=entropy, max_depth=20, max_features=log2, min_samples_leaf=2, min_samples_split=10, n_estimators=55; total time=   0.3s
[CV] END bootstrap=False, criterion=entropy, max_depth=20, max_features=log2, min_samples_leaf=2, min_samples_split=10, n_estimators=55; total time=   0.3s
[CV] END bootstrap=False, criterion=entropy, max_depth=20, max_features=log2, min_samples_leaf=2, min_samples_split=10, n_estimators=55; total time=   0.3s
[CV] END bootstrap=True, criterion=gini, max_depth=20, max_features=None, min_samples_leaf=2, min_samples_split=9, n_estimators=50; total time=   0.5s
[CV] END bootstrap=True, criterion=gini, max_depth=20, max_features=None, min_samples_leaf=2, min_samples_split=9, n_estimators=50; total time=   0.5s
[CV] END bootstrap=False, criterion=entropy, max_depth=20, max_features=log2, min_samples_leaf=2, min_samples_split=10, n_estimators=55; total time=   0.3s
[CV] END bootstrap=False, criterion=gini, max_depth=30, max_features=sqrt, min_samples_leaf=2, min_samples_split=8, n_estimators=42; total time=   0.2s
[CV] END bootstrap=False, criterion=gini, max_depth=30, max_features=sqrt, min_samples_leaf=2, min_samples_split=8, n_estimators=42; total time=   0.2s
[CV] END bootstrap=False, criterion=gini, max_depth=30, max_features=sqrt, min_samples_leaf=2, min_samples_split=8, n_estimators=42; total time=   0.2s
[CV] END bootstrap=False, criterion=gini, max_depth=30, max_features=sqrt, min_samples_leaf=2, min_samples_split=8, n_estimators=42; total time=   0.2s
[CV] END bootstrap=True, criterion=gini, max_depth=10, max_features=log2, min_samples_leaf=3, min_samples_split=8, n_estimators=56; total time=   0.2s
[CV] END bootstrap=True, criterion=gini, max_depth=10, max_features=log2, min_samples_leaf=3, min_samples_split=8, n_estimators=56; total time=   0.2s
[CV] END bootstrap=True, criterion=gini, max_depth=10, max_features=log2, min_samples_leaf=3, min_samples_split=8, n_estimators=56; total time=   0.2s
[CV] END bootstrap=True, criterion=gini, max_depth=10, max_features=log2, min_samples_leaf=3, min_samples_split=8, n_estimators=56; total time=   0.2s
[CV] END bootstrap=True, criterion=gini, max_depth=10, max_features=log2, min_samples_leaf=3, min_samples_split=8, n_estimators=56; total time=   0.2s
[CV] END bootstrap=True, criterion=entropy, max_depth=30, max_features=log2, min_samples_leaf=2, min_samples_split=2, n_estimators=58; total time=   0.2s
[CV] END bootstrap=False, criterion=entropy, max_depth=20, max_features=None, min_samples_leaf=1, min_samples_split=5, n_estimators=45; total time=   0.7s
[CV] END bootstrap=False, criterion=entropy, max_depth=20, max_features=None, min_samples_leaf=1, min_samples_split=5, n_estimators=45; total time=   0.7s
[CV] END bootstrap=False, criterion=entropy, max_depth=20, max_features=None, min_samples_leaf=1, min_samples_split=5, n_estimators=45; total time=   0.7s
[CV] END bootstrap=True, criterion=entropy, max_depth=30, max_features=log2, min_samples_leaf=2, min_samples_split=2, n_estimators=58; total time=   0.2s
[CV] END bootstrap=True, criterion=entropy, max_depth=30, max_features=log2, min_samples_leaf=2, min_samples_split=2, n_estimators=58; total time=   0.3s
[CV] END bootstrap=False, criterion=entropy, max_depth=20, max_features=None, min_samples_leaf=1, min_samples_split=5, n_estimators=45; total time=   0.8s
[CV] END bootstrap=False, criterion=entropy, max_depth=20, max_features=None, min_samples_leaf=1, min_samples_split=5, n_estimators=45; total time=   0.8s
[CV] END bootstrap=True, criterion=entropy, max_depth=30, max_features=log2, min_samples_leaf=2, min_samples_split=2, n_estimators=58; total time=   0.3s
[CV] END bootstrap=True, criterion=entropy, max_depth=30, max_features=log2, min_samples_leaf=2, min_samples_split=2, n_estimators=58; total time=   0.3s
[CV] END bootstrap=True, criterion=entropy, max_depth=10, max_features=None, min_samples_leaf=4, min_samples_split=4, n_estimators=55; total time=   0.5s
[CV] END bootstrap=True, criterion=entropy, max_depth=10, max_features=None, min_samples_leaf=4, min_samples_split=4, n_estimators=55; total time=   0.6s
[CV] END bootstrap=True, criterion=entropy, max_depth=10, max_features=None, min_samples_leaf=4, min_samples_split=4, n_estimators=55; total time=   0.5s
[CV] END bootstrap=True, criterion=entropy, max_depth=10, max_features=None, min_samples_leaf=4, min_samples_split=4, n_estimators=55; total time=   0.5s
[CV] END bootstrap=True, criterion=entropy, max_depth=10, max_features=None, min_samples_leaf=4, min_samples_split=4, n_estimators=55; total time=   0.6s
[CV] END bootstrap=False, criterion=entropy, max_depth=10, max_features=None, min_samples_leaf=4, min_samples_split=5, n_estimators=41; total time=   0.6s
[CV] END bootstrap=False, criterion=entropy, max_depth=10, max_features=None, min_samples_leaf=4, min_samples_split=5, n_estimators=41; total time=   0.7s
[CV] END bootstrap=False, criterion=gini, max_depth=30, max_features=sqrt, min_samples_leaf=4, min_samples_split=5, n_estimators=48; total time=   0.2s
[CV] END bootstrap=False, criterion=gini, max_depth=30, max_features=sqrt, min_samples_leaf=4, min_samples_split=5, n_estimators=48; total time=   0.2s
[CV] END bootstrap=False, criterion=entropy, max_depth=10, max_features=None, min_samples_leaf=4, min_samples_split=5, n_estimators=41; total time=   0.6s
[CV] END bootstrap=False, criterion=gini, max_depth=30, max_features=sqrt, min_samples_leaf=4, min_samples_split=5, n_estimators=48; total time=   0.2s
[CV] END bootstrap=False, criterion=gini, max_depth=30, max_features=sqrt, min_samples_leaf=4, min_samples_split=5, n_estimators=48; total time=   0.2s
[CV] END bootstrap=False, criterion=gini, max_depth=30, max_features=sqrt, min_samples_leaf=4, min_samples_split=5, n_estimators=48; total time=   0.2s
[CV] END bootstrap=True, criterion=entropy, max_depth=None, max_features=sqrt, min_samples_leaf=3, min_samples_split=10, n_estimators=54; total time=   0.2s
[CV] END bootstrap=True, criterion=entropy, max_depth=None, max_features=sqrt, min_samples_leaf=3, min_samples_split=10, n_estimators=54; total time=   0.2s
[CV] END bootstrap=True, criterion=entropy, max_depth=None, max_features=sqrt, min_samples_leaf=3, min_samples_split=10, n_estimators=54; total time=   0.2s
[CV] END bootstrap=True, criterion=entropy, max_depth=None, max_features=sqrt, min_samples_leaf=3, min_samples_split=10, n_estimators=54; total time=   0.2s
[CV] END bootstrap=True, criterion=entropy, max_depth=None, max_features=sqrt, min_samples_leaf=3, min_samples_split=10, n_estimators=54; total time=   0.2s
[CV] END bootstrap=False, criterion=entropy, max_depth=10, max_features=None, min_samples_leaf=4, min_samples_split=5, n_estimators=41; total time=   0.7s
[CV] END bootstrap=False, criterion=entropy, max_depth=10, max_features=None, min_samples_leaf=4, min_samples_split=5, n_estimators=41; total time=   0.7s
[CV] END bootstrap=False, criterion=gini, max_depth=10, max_features=log2, min_samples_leaf=1, min_samples_split=5, n_estimators=59; total time=   0.3s
[CV] END bootstrap=False, criterion=gini, max_depth=10, max_features=log2, min_samples_leaf=1, min_samples_split=5, n_estimators=59; total time=   0.3s
[CV] END bootstrap=False, criterion=gini, max_depth=10, max_features=log2, min_samples_leaf=1, min_samples_split=5, n_estimators=59; total time=   0.3s
[CV] END bootstrap=False, criterion=gini, max_depth=10, max_features=log2, min_samples_leaf=1, min_samples_split=5, n_estimators=59; total time=   0.3s
[CV] END bootstrap=False, criterion=gini, max_depth=10, max_features=log2, min_samples_leaf=1, min_samples_split=5, n_estimators=59; total time=   0.3s
[CV] END bootstrap=True, criterion=entropy, max_depth=20, max_features=sqrt, min_samples_leaf=4, min_samples_split=8, n_estimators=50; total time=   0.2s
[CV] END bootstrap=True, criterion=entropy, max_depth=20, max_features=sqrt, min_samples_leaf=4, min_samples_split=8, n_estimators=50; total time=   0.2s
[CV] END bootstrap=True, criterion=entropy, max_depth=20, max_features=sqrt, min_samples_leaf=4, min_samples_split=8, n_estimators=50; total time=   0.2s
[CV] END bootstrap=True, criterion=entropy, max_depth=20, max_features=sqrt, min_samples_leaf=4, min_samples_split=8, n_estimators=50; total time=   0.2s
[CV] END bootstrap=True, criterion=entropy, max_depth=20, max_features=sqrt, min_samples_leaf=4, min_samples_split=8, n_estimators=50; total time=   0.2s
[CV] END bootstrap=True, criterion=gini, max_depth=30, max_features=None, min_samples_leaf=3, min_samples_split=9, n_estimators=48; total time=   0.4s
[CV] END bootstrap=False, criterion=entropy, max_depth=10, max_features=None, min_samples_leaf=3, min_samples_split=7, n_estimators=55; total time=   0.9s
[CV] END bootstrap=False, criterion=entropy, max_depth=10, max_features=None, min_samples_leaf=3, min_samples_split=7, n_estimators=55; total time=   0.8s
[CV] END bootstrap=False, criterion=entropy, max_depth=10, max_features=None, min_samples_leaf=3, min_samples_split=7, n_estimators=55; total time=   0.9s
[CV] END bootstrap=False, criterion=entropy, max_depth=10, max_features=None, min_samples_leaf=3, min_samples_split=7, n_estimators=55; total time=   0.9s
[CV] END bootstrap=True, criterion=gini, max_depth=30, max_features=None, min_samples_leaf=3, min_samples_split=9, n_estimators=48; total time=   0.4s
[CV] END bootstrap=False, criterion=entropy, max_depth=10, max_features=None, min_samples_leaf=3, min_samples_split=7, n_estimators=55; total time=   0.9s
[CV] END bootstrap=True, criterion=gini, max_depth=30, max_features=None, min_samples_leaf=3, min_samples_split=9, n_estimators=48; total time=   0.5s
[CV] END bootstrap=False, criterion=entropy, max_depth=20, max_features=log2, min_samples_leaf=3, min_samples_split=5, n_estimators=56; total time=   0.3s
[CV] END bootstrap=False, criterion=entropy, max_depth=20, max_features=log2, min_samples_leaf=3, min_samples_split=5, n_estimators=56; total time=   0.3s
[CV] END bootstrap=False, criterion=entropy, max_depth=20, max_features=log2, min_samples_leaf=3, min_samples_split=5, n_estimators=56; total time=   0.3s
[CV] END bootstrap=True, criterion=gini, max_depth=30, max_features=None, min_samples_leaf=3, min_samples_split=9, n_estimators=48; total time=   0.5s
[CV] END bootstrap=True, criterion=gini, max_depth=30, max_features=None, min_samples_leaf=3, min_samples_split=9, n_estimators=48; total time=   0.5s
[CV] END bootstrap=False, criterion=entropy, max_depth=20, max_features=log2, min_samples_leaf=3, min_samples_split=5, n_estimators=56; total time=   0.3s
[CV] END bootstrap=False, criterion=entropy, max_depth=20, max_features=log2, min_samples_leaf=3, min_samples_split=5, n_estimators=56; total time=   0.3s
[CV] END bootstrap=True, criterion=gini, max_depth=None, max_features=None, min_samples_leaf=3, min_samples_split=6, n_estimators=42; total time=   0.4s
[CV] END bootstrap=True, criterion=gini, max_depth=None, max_features=None, min_samples_leaf=3, min_samples_split=6, n_estimators=42; total time=   0.4s
[CV] END bootstrap=True, criterion=gini, max_depth=None, max_features=None, min_samples_leaf=3, min_samples_split=6, n_estimators=42; total time=   0.4s
[CV] END bootstrap=True, criterion=gini, max_depth=None, max_features=None, min_samples_leaf=3, min_samples_split=6, n_estimators=42; total time=   0.4s
[CV] END bootstrap=True, criterion=gini, max_depth=None, max_features=None, min_samples_leaf=3, min_samples_split=6, n_estimators=42; total time=   0.4s
[CV] END bootstrap=False, criterion=entropy, max_depth=20, max_features=log2, min_samples_leaf=4, min_samples_split=10, n_estimators=57; total time=   0.3s
[CV] END bootstrap=False, criterion=entropy, max_depth=20, max_features=log2, min_samples_leaf=4, min_samples_split=10, n_estimators=57; total time=   0.4s
[CV] END bootstrap=False, criterion=entropy, max_depth=20, max_features=log2, min_samples_leaf=4, min_samples_split=10, n_estimators=57; total time=   0.3s
[CV] END bootstrap=False, criterion=entropy, max_depth=20, max_features=log2, min_samples_leaf=4, min_samples_split=10, n_estimators=57; total time=   0.3s
[CV] END bootstrap=False, criterion=entropy, max_depth=30, max_features=None, min_samples_leaf=4, min_samples_split=6, n_estimators=56; total time=   0.9s
[CV] END bootstrap=False, criterion=entropy, max_depth=30, max_features=None, min_samples_leaf=4, min_samples_split=6, n_estimators=56; total time=   0.9s
[CV] END bootstrap=False, criterion=entropy, max_depth=30, max_features=None, min_samples_leaf=4, min_samples_split=6, n_estimators=56; total time=   0.9s
[CV] END bootstrap=False, criterion=entropy, max_depth=20, max_features=log2, min_samples_leaf=4, min_samples_split=10, n_estimators=57; total time=   0.3s
[CV] END bootstrap=False, criterion=entropy, max_depth=30, max_features=None, min_samples_leaf=4, min_samples_split=6, n_estimators=56; total time=   0.9s
[CV] END bootstrap=False, criterion=entropy, max_depth=30, max_features=None, min_samples_leaf=4, min_samples_split=6, n_estimators=56; total time=   0.8s

RandomizedSearchCV(cv=5, estimator=RandomForestClassifier(random_state=1),
                   n_iter=200, n_jobs=-1,
                   param_distributions={'bootstrap': [True, False],
                                        'criterion': ['gini', 'entropy'],
                                        'max_depth': [None, 10, 20, 30],
                                        'max_features': [None, 'sqrt', 'log2'],
                                        'min_samples_leaf': <scipy.stats._distn_infrastructure.rv_discrete_frozen object at 0x157708500>,
                                        'min_samples_split': <scipy.stats._distn_infrastructure.rv_discrete_frozen object at 0x157cec2c0>,
                                        'n_estimators': <scipy.stats._distn_infrastructure.rv_discrete_frozen object at 0x156c01b20>},
                   random_state=1, verbose=2)

In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook.
On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.

# Get the best parameters from the search
best_params = random_search.best_params_
print("Best parameters found:", best_params)

Best parameters found: {'bootstrap': False, 'criterion': 'gini', 'max_depth': 20, 'max_features': 'log2', 'min_samples_leaf': 1, 'min_samples_split': 5, 'n_estimators': 40}

The criterion selected to assess the quality of a split was the Gini impurity index $$I_G(p):=\sum_i p_i(1-p_i)=1-\sum_i p_i^2$$ which measures how often a randomly chosen element from the set would be incorrectly labeled if it were randomly labeled according to the distribution of labels in the subset, and where indeed $p_i$ is the probability of choosing an item with label $i$.

# Train and evaluate the model with the best parameters
rf_opt = RandomForestClassifier(**best_params, random_state=1)
rf_opt.fit(X_train_red, y_train)

RandomForestClassifier(bootstrap=False, max_depth=20, max_features='log2',
                       min_samples_split=5, n_estimators=40, random_state=1)

In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook.
On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.

# Predict on the PCA-reduced test data
y_pred_opt = rf_opt.predict(X_test_red)

# Plot the confusion matrix
plt.figure(figsize=(12, 5))

sns.heatmap(confusion_matrix(y_test, y_pred_opt), annot=True, fmt="d", cmap="YlGnBu")
plt.title("Optimized Random Forest Confusion Matrix")
plt.xlabel("Predicted")
plt.ylabel("True")

Text(120.72222222222221, 0.5, 'True')

We now have a look at the final classification report and accuracy score for the optimized classifier.

from sklearn.metrics import classification_report

print(classification_report(y_test, y_pred_opt))
print("Accuracy score: ", accuracy_score(y_test, y_pred_opt))

              precision    recall  f1-score   support

           0       0.86      0.91      0.88        92
           1       0.92      0.87      0.90       108

    accuracy                           0.89       200
   macro avg       0.89      0.89      0.89       200
weighted avg       0.89      0.89      0.89       200

Accuracy score:  0.89

With the optimal hyperparameters found by Random Search, our Random Forest classifier almost stands still, at an accuracy score of 89%, of which we shall be satisfied. Indeed, we consider this a more "authentic" result than the previous one without optimized parameters, of which the 0.5% higher accuracy could of course be linked to the fortuitous way of splitting of our data between test and train.

Moreover, also the other metrics outline a more satisfying performance, for instance both the precision and the recall, the ratio of correctly predicted positive observations to all observations in the actual class, both stand around 90%, and of course so does their weighted average, the f1-score.

In praticular, class 1 has a slightly higher precision but lower recall compared to Class 0, meaning it makes fewer false positives but misses more actual positives for class 1.

Predictions

We have seen that the PCA slightly enhances the performance of our Random Forest Classifier, bringing the accuracy from 87% up to 89%, when passing from the original dataset to the reduced one and when optimized using Random Search. Hence, we are going to use the latter to our test dataset.

test = pd.read_csv('/Users/lucamartucci/Desktop/Assignment/mldata_0003194553.TEST_FEATURES.csv', engine='python', index_col=0)
test

	feature_1	feature_2	feature_3	feature_4	feature_5	feature_6	feature_7	feature_8	feature_9	feature_10	feature_11	feature_12	feature_13	feature_14	feature_15	feature_16	feature_17	feature_18	feature_19	feature_20
id
0	-3.169079	0.476954	-0.205536	-1.998103	0.258794	0.185094	-1.172078	-0.552032	-0.903555	-1.839311	-3.459635	1.321482	-0.871417	-0.740335	0.809934	-1.551334	-0.093306	-0.416684	0.358708	-0.317423
1	-1.658675	2.350997	1.492960	-3.760101	1.254247	-3.294714	2.789122	0.100599	-1.914627	2.392324	3.053127	1.009401	-0.843474	-1.257097	-2.504218	0.069982	-3.329727	0.365409	1.170961	-1.089895
2	-4.568093	-1.716479	-0.178482	-4.717965	-1.045755	-5.980412	1.262953	0.039024	-1.437658	4.331904	-0.635809	-1.062635	-3.636461	2.424329	-3.956190	-2.323679	2.066404	-1.847628	1.668435	-2.248526
3	-0.642175	2.942497	1.781474	-1.312740	0.781662	2.929671	-1.343054	-0.846422	1.426521	-0.871360	0.278109	3.150546	0.860363	-0.109239	0.852702	0.835139	-4.463065	3.424975	-1.261658	1.822430
4	-3.213704	-1.417102	0.714174	1.841850	1.621388	0.180564	-0.740836	-2.882465	5.330997	-4.662478	-1.366435	-2.458569	-2.689017	2.316703	1.948322	-2.290263	-1.512436	1.130158	-0.726419	-0.026087
...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...
995	-3.351507	-0.227752	0.840096	-0.206810	0.472872	-0.526747	0.674494	-1.783052	-1.902254	-1.268285	2.116374	0.614230	-2.479640	0.485984	-1.367704	-0.350791	-1.956390	1.068145	1.307559	2.536949
996	3.505611	-0.628739	-2.984615	3.939552	-1.741247	1.095613	-1.105055	-0.283217	-2.251992	-1.768783	0.305321	-1.112875	1.910792	-3.480918	2.121035	1.819193	-0.051809	-3.098418	-0.895142	-1.487822
997	1.267481	-0.447051	0.094762	3.729275	-1.961422	-0.450547	1.908331	-1.743704	0.068433	1.229349	2.018320	0.467066	-1.969064	1.037036	-0.362378	-1.873542	0.491163	-1.284261	0.603708	3.522902
998	-0.143402	-1.608293	-1.161044	-0.422607	-1.271535	-1.355502	2.016635	1.705046	-3.382848	2.029733	-0.431555	-2.275189	-0.170876	-0.418104	1.364641	-2.029354	2.421128	-1.467155	-0.454694	-0.072440
999	-3.929308	-2.248856	2.524760	4.425537	1.973793	0.551257	2.122711	-3.418377	0.459158	-0.232480	0.893146	-0.643976	-4.032756	3.465283	1.996857	-0.958340	-4.484022	-1.247471	1.348738	5.794755

1000 rows × 20 columns

# Standardize the new data using the same scaler fitted on the training data
test_scaled = scaler.transform(test)

# Apply PCA to the standardized new data using the same PCA transformation fitted on the training data
new_data_pca = pca.transform(test)

# Use the optimized Random Forest model to make predictions on the PCA-transformed new data
predictions = rf_opt.predict(new_data_pca)

print(predictions)

[0 0 0 1 0 0 0 1 1 0 1 1 0 0 0 0 0 0 1 0 0 1 1 0 0 1 1 0 1 1 0 1 1 0 0 1 0
 0 1 1 0 0 0 0 0 1 1 0 1 0 0 0 0 0 0 1 0 0 1 0 1 0 1 0 0 0 1 0 1 1 0 0 0 0
 1 0 1 0 1 1 1 0 1 1 0 1 1 0 0 1 1 1 0 1 0 1 0 0 0 1 0 0 1 1 1 0 1 0 0 0 1
 1 1 0 0 1 1 1 0 1 0 1 1 0 0 1 1 0 1 1 0 1 1 1 1 0 0 0 0 1 0 1 1 1 0 0 0 0
 1 0 0 0 1 1 1 1 1 0 0 1 0 0 0 1 0 0 1 1 0 0 1 1 0 1 1 1 0 1 0 1 1 0 1 0 0
 0 0 1 0 0 1 1 1 1 1 1 0 0 0 0 0 0 0 1 1 0 1 0 1 0 0 0 0 1 0 0 1 0 0 1 0 0
 1 0 1 1 0 0 1 1 1 0 0 0 0 1 1 1 0 1 0 1 1 0 1 0 1 0 1 0 0 1 1 1 1 0 0 0 0
 0 1 0 0 1 1 1 0 0 0 0 1 1 0 0 0 1 1 0 0 0 1 0 1 0 0 0 0 1 1 1 1 0 0 1 1 0
 1 1 0 0 1 0 0 1 1 1 1 1 1 0 0 0 0 1 0 0 0 1 1 1 0 1 1 0 1 1 1 0 0 0 1 0 0
 0 1 1 0 1 1 1 1 0 0 1 0 0 1 1 1 1 0 1 1 1 1 1 1 1 1 0 0 0 1 0 1 0 1 1 0 1
 0 0 0 1 1 1 1 1 0 1 0 1 1 0 1 1 0 1 0 0 1 1 1 1 1 0 1 1 1 0 0 1 1 0 1 0 1
 1 0 0 1 1 1 0 0 0 0 0 1 1 0 0 1 1 1 1 1 1 0 0 0 1 0 1 0 0 1 1 1 1 0 1 0 1
 0 1 0 0 1 1 1 0 0 1 0 1 0 0 1 1 0 0 1 0 1 1 1 0 1 1 0 1 0 1 0 1 0 0 0 0 0
 1 1 1 1 1 1 0 1 0 1 0 1 0 1 0 0 0 0 0 1 0 0 0 0 0 1 0 0 0 0 1 0 1 0 0 1 1
 0 0 1 0 1 1 0 0 0 0 1 1 0 1 0 1 1 1 1 1 1 1 0 0 0 1 1 0 0 0 1 0 0 0 1 1 1
 1 1 1 1 0 0 0 0 0 0 1 0 1 0 0 1 1 0 1 1 1 0 1 1 1 1 1 0 0 0 0 0 0 1 1 0 1
 1 1 1 1 1 0 0 0 0 0 0 0 0 1 1 0 1 0 0 1 1 0 0 1 0 1 1 0 0 0 1 0 0 0 0 1 1
 1 1 1 0 0 1 0 0 1 0 1 0 0 0 0 1 1 0 1 0 1 1 0 0 1 1 0 1 1 1 0 1 1 1 0 0 1
 0 0 0 1 0 0 0 0 1 0 1 0 0 0 0 0 1 1 0 0 1 1 0 1 0 1 1 0 1 1 0 1 1 1 1 0 1
 0 0 0 0 1 1 0 1 0 0 0 1 0 1 1 0 0 0 1 0 1 1 0 1 1 1 1 0 0 1 0 0 1 1 1 0 0
 0 1 0 0 1 0 1 1 0 1 0 1 1 0 0 0 1 1 0 0 0 0 0 0 1 1 0 1 1 0 0 0 1 0 1 1 1
 1 1 1 1 0 0 1 1 0 1 0 0 0 1 0 1 1 0 0 1 1 1 0 1 0 1 0 0 1 0 1 1 1 0 0 0 1
 0 1 0 0 1 1 0 1 0 1 0 1 0 1 0 0 1 1 0 0 1 0 0 1 1 1 1 0 1 1 0 1 0 1 0 0 1
 1 0 1 1 0 1 0 1 0 1 0 0 1 1 0 1 1 0 1 1 0 0 1 1 1 0 1 0 0 1 1 1 1 1 1 0 0
 0 1 1 0 0 1 0 0 1 0 1 0 0 0 1 1 1 0 1 1 0 1 0 1 1 1 1 0 1 0 0 0 0 1 0 0 1
 0 0 0 0 0 1 0 0 1 1 1 0 0 0 0 1 0 1 0 1 0 1 0 0 1 0 0 1 1 0 1 1 0 0 1 1 0
 1 0 1 1 0 1 1 0 1 0 0 1 0 0 1 0 1 0 0 1 0 0 0 1 1 0 1 0 0 0 1 1 1 0 1 0 1
 0]

filename = 'test_predictions.txt'

with open(filename, 'w') as file:
    for number in predictions:
        file.write(f'{number}\n')

print(f'Array has been written to {filename}')

Array has been written to test_predictions.txt

# Identify the unique predicted labels and their frequency in the predictions.
labels, frequencies = np.unique(predictions, return_counts=True)

for value, count in zip(labels, frequencies):
    print(f"Number: {value}, Frequency: {count}")

Number: 0, Frequency: 509
Number: 1, Frequency: 491

Conclusion

We first loaded the data, containing 1000 samples for each of the 20 features present, split it into labels and features and explored visually, by means of both separate and "simultaneous" histograms, the distribution of the features, and their "correlation structure". The depth of such structure observed, emboldened us to implement PCA for feature selection: we first tried to notice if an optimal number of principal components could be grasped using the usual "elbow" method, and once this failed, we decided to proceed by selecting 95% explained variance, resulting in 14 components out of 20. However small, this change resulted in a noticeable improvement in the performance of our Random Forest classifier, thus justifying the implementation of Principal Component Analysis, reaching up to an accuracy of 89%, with a Random Search to optimize the hyperparameters of the classifier, based on the results of a previous Cross Validation to narrow down the number of trees.

In perspective, we must admit that our approach was a bit more naive than needed, and one may find some lack of rigorousness, for instance in the choice of the narrow margin of estimators on which we run RandomSearch, done by only relying on the visual intuition yielded by the plot of the CrossValidation score. Also, the number of validations and searches attempted with different "grids" was relatively limited, and the results presented only contain the best performances obtained.

Because of the overall more satisfying performance, we decided to use the Random Forest classifier, with PCA and optimized hyperparameters, to make predictions on the test dataset.