主要目标
了解数据集的一些基本情况，如缺失值，异常值，数据集大小，原始特征维度等了解变量间的相互关系、变量与预测值之间存在的关系观察变量的分布情况，为特征工程做准备初始化
import pandas as pdimport numpy as npimport matplotlib.pyplot as pltimport seaborn as snsimport datetimeimport warningsimport pandas.util.testing as tmwarnings.filterwarnings('ignore')# 文件目录，绝对路径INPUT_PATH = 'G:\代码\数据挖掘实践\\'
读取文件
data_train = pd.read_csv(INPUT_PATH +'train.csv')data_test_a = pd.read_csv(INPUT_PATH +'testA.csv')
分块读取
当需要读取的数据文件特别大时，可以采用分块读取的方式，分块读取返回的类型为 TextFileReader，是一个可迭代的对象。
#设置chunksize参数，来控制每次读取数据的大小chunker = pd.read_csv(INPUT_PATH +"train.csv",chunksize=100000,iterator=True)temp = chunker.get_chunk(200000) # 读取指定长度的数据len(temp)
200000
# 对每一块进行迭代for item in chunker: # 每个item是一个长度为chunksize的DataFrameprint(type(item))print(len(item))print(item['loanAmnt'].mean())
<class 'pandas.core.frame.DataFrame'>10000014405.10325<class 'pandas.core.frame.DataFrame'>10000014449.50525<class 'pandas.core.frame.DataFrame'>10000014368.34225<class 'pandas.core.frame.DataFrame'>10000014422.87375<class 'pandas.core.frame.DataFrame'>10000014426.4795<class 'pandas.core.frame.DataFrame'>10000014417.336
统计分析
查看数据集的样本个数和原始特征维度:
print(data_train.shape)print(data_train.columns)
(800000, 47)Index(['id', 'loanAmnt', 'term', 'interestRate', 'installment', 'grade','subGrade', 'employmentTitle', 'employmentLength', 'homeOwnership','annualIncome', 'verificationStatus', 'issueDate', 'isDefault','purpose', 'postCode', 'regionCode', 'dti', 'delinquency_2years','ficoRangeLow', 'ficoRangeHigh', 'openAcc', 'pubRec','pubRecBankruptcies', 'revolBal', 'revolUtil', 'totalAcc','initialListStatus', 'applicationType', 'earliesCreditLine', 'title','policyCode', 'n0', 'n1', 'n2', 'n3', 'n4', 'n5', 'n6', 'n7', 'n8','n9', 'n10', 'n11', 'n12', 'n13', 'n14'],dtype='object')
查看数据类型以及是否有空值：
data_train.info()
<class 'pandas.core.frame.DataFrame'>RangeIndex: 800000 entries, 0 to 799999Data columns (total 47 columns):# Column Non-Null Count Dtype --- ------ -------------- ----- 0 id 800000 non-null int64 1 loanAmnt 800000 non-null float642 term800000 non-null int64 3 interestRate 800000 non-null float644 installment 800000 non-null float645 grade800000 non-null object 6 subGrade 800000 non-null object 7 employmentTitle799999 non-null float648 employmentLength 753201 non-null object 9 homeOwnership 800000 non-null int64 10 annualIncome 800000 non-null float6411 verificationStatus 800000 non-null int64 12 issueDate 800000 non-null object 13 isDefault 800000 non-null int64 14 purpose 800000 non-null int64 15 postCode 799999 non-null float6416 regionCode800000 non-null int64 17 dti 799761 non-null float6418 delinquency_2years 800000 non-null float6419 ficoRangeLow 800000 non-null float6420 ficoRangeHigh 800000 non-null float6421 openAcc 800000 non-null float6422 pubRec 800000 non-null float6423 pubRecBankruptcies 799595 non-null float6424 revolBal 800000 non-null float6425 revolUtil 799469 non-null float6426 totalAcc 800000 non-null float6427 initialListStatus 800000 non-null int64 28 applicationType800000 non-null int64 29 earliesCreditLine 800000 non-null object 30 title799999 non-null float6431 policyCode800000 non-null float6432 n0 759730 non-null float6433 n1 759730 non-null float6434 n2 759730 non-null float6435 n3 759730 non-null float6436 n4 766761 non-null float6437 n5 759730 non-null float6438 n6 759730 non-null float6439 n7 759730 non-null float6440 n8 759729 non-null float6441 n9 759730 non-null float6442 n10 766761 non-null float6443 n11 730248 non-null float6444 n12 759730 non-null float6445 n13 759730 non-null float6446 n14 759730 non-null float64dtypes: float64(33), int64(9), object(5)memory usage: 286.9+ MB
查看数据的统计描述：
data_train.describe()
8 rows × 42 columns
查看存在缺失值的情况：
print(f'有{data_train.isnull().any().sum()}列数据存在缺失值') # .any()表示任意一列中存在缺失值
有22列数据存在缺失值
查看缺失特征中缺失率大于50%的特征：
have_null_fea_dict = (data_train.isnull().sum()/len(data_train)).to_dict()fea_null_moreThanHalf = {}for key,value in have_null_fea_dict.items():if value > 0.5:fea_null_moreThanHalf[key] = valuefea_null_moreThanHalf
{}
具体的查看缺失特征及缺失率：
missing = data_train.isnull().sum()/len(data_train)missing = missing[missing > 0]missing.sort_values(inplace=True)missing.plot.bar()
如果缺失值的数量较小一般选择填充，如使用 lgb 等树模型可以直接填充空缺，让树自己去优化。如果存在的缺失值过多，则可以考虑删掉该特征。
统计取值都相同的特征：
one_value_fea = [col for col in data_train.columns if data_train[col].nunique() <= 1] one_value_fea
['policyCode']
查看特征的数据类型：
特征一般都是由类别型特征和数值型特征组成。类别型特征有时具有非数值关系，有时也具有数值关系。比如’grade’中的等级 A，B，C 等，是否只是单纯的分类，还是A优于其他要结合业务判断。数值型特征一般是可以直接入模的，但风控人员往往还要对其做分箱、转化为 WOE 编码进而做标准评分卡等操作。从模型效果上来看，特征分箱主要是为了降低变量的复杂性，减少变量噪音对模型的影响，提高自变量和因变量的相关度。从而使模型更加稳定。numerical_fea = list(data_train.select_dtypes(exclude=['object']).columns)category_fea = list(filter(lambda x: x not in numerical_fea,list(data_train.columns)))print('数值型特征')print(numerical_fea)print('类别型特征')print(category_fea)
数值型特征['id', 'loanAmnt', 'term', 'interestRate', 'installment', 'employmentTitle', 'homeOwnership', 'annualIncome', 'verificationStatus', 'isDefault', 'purpose', 'postCode', 'regionCode', 'dti', 'delinquency_2years', 'ficoRangeLow', 'ficoRangeHigh', 'openAcc', 'pubRec', 'pubRecBankruptcies', 'revolBal', 'revolUtil', 'totalAcc', 'initialListStatus', 'applicationType', 'title', 'policyCode', 'n0', 'n1', 'n2', 'n3', 'n4', 'n5', 'n6', 'n7', 'n8', 'n9', 'n10', 'n11', 'n12', 'n13', 'n14']类别型特征['grade', 'subGrade', 'employmentLength', 'issueDate', 'earliesCreditLine']
对数值型变量进行分析
判断数值特征是离散型还是连续型# 过滤数值型类别特征def get_numerical_serial_fea(data,feas):numerical_serial_fea = []numerical_noserial_fea = []for fea in feas:temp = data[fea].nunique()if temp <= 10: # 取值少于10个的特征视为离散型特征，取值多于10个的视为连续性特征numerical_noserial_fea.append(fea)continuenumerical_serial_fea.append(fea)return numerical_serial_fea,numerical_noserial_feanumerical_serial_fea,numerical_noserial_fea = get_numerical_serial_fea(data_train,numerical_fea)print('连续性数值特征')print(numerical_serial_fea)print('离散型数值特征')print(numerical_noserial_fea)
连续性数值特征['id', 'loanAmnt', 'interestRate', 'installment', 'employmentTitle', 'annualIncome', 'purpose', 'postCode', 'regionCode', 'dti', 'delinquency_2years', 'ficoRangeLow', 'ficoRangeHigh', 'openAcc', 'pubRec', 'pubRecBankruptcies', 'revolBal', 'revolUtil', 'totalAcc', 'title', 'n0', 'n1', 'n2', 'n3', 'n4', 'n5', 'n6', 'n7', 'n8', 'n9', 'n10', 'n13', 'n14']离散型数值特征['term', 'homeOwnership', 'verificationStatus', 'isDefault', 'initialListStatus', 'applicationType', 'policyCode', 'n11', 'n12']
查看离散型数值特征的类别数
# 查看离散型数值特征的类别数print('term')print(data_train['term'].value_counts())print('\nhomeOwnership')print(data_train['homeOwnership'].value_counts())print('\nverificationStatus')print(data_train['verificationStatus'].value_counts())print('\ninitialListStatus')print(data_train['initialListStatus'].value_counts())print('\napplicationType')print(data_train['applicationType'].value_counts())print('\npolicyCode')print(data_train['policyCode'].value_counts())print('\nn11')print(data_train['n11'].value_counts())print('\nn12')print(data_train['n12'].value_counts())
term3 6069025 193098Name: term, dtype: int64homeOwnership0 3957321 3176602863093 1855 814 33Name: homeOwnership, dtype: int64verificationStatus1 3098102 2489680 241222Name: verificationStatus, dtype: int64initialListStatus0 4664381 333562Name: initialListStatus, dtype: int64applicationType0 784586115414Name: applicationType, dtype: int64policyCode1.0 800000Name: policyCode, dtype: int64n110.0 7296821.0 5402.0 244.0 13.0 1Name: n11, dtype: int64n120.0 7573151.022812.0 1153.0 164.0 3Name: n12, dtype: int64
存在取值全部相同的情况，如’policyCode’，这在实际分析中是没有价值的。此外，有些变量存在取值数量相差较大的情况，如’n11’,‘n12’.
连续型数值变量可视化
# 每个数值特征的分布可视化f = pd.melt(data_train, value_vars=numerical_serial_fea) # pd.melt 列转行g = sns.FacetGrid(f, col="variable", col_wrap=5, sharex=False, sharey=False) # FacetGrid 是sns的库的一个类，在初始化的时候只需要传入一个DataFrame将其示例化，实例化之后就可以直接使用这个类绘制需要的图形g = g.map(sns.distplot, "value")
查看某一个数值型变量的分布，查看变量是否符合正态分布，如果不符合正态分布的变量可以取对数后再观察是否符合正态分布。取对数的作用相当于标准化，可以将变化范围较大的数据限制在较小的变化范围内。
# 画出 loanAmnt 与 log(loanAmnt) 的分布plt.figure(figsize=(16,12))plt.suptitle('Transaction Values Distribution', fontsize=22)plt.subplot(221)sub_plot_1 = sns.distplot(data_train['loanAmnt'])sub_plot_1.set_title("loanAmnt Distribuition", fontsize=18)sub_plot_1.set_xlabel("")sub_plot_1.set_ylabel("Probability", fontsize=15)plt.subplot(222)sub_plot_2 = sns.distplot(np.log(data_train['loanAmnt']))sub_plot_2.set_title("loanAmnt (Log) Distribuition", fontsize=18)sub_plot_2.set_xlabel("")sub_plot_2.set_ylabel("Probability", fontsize=15)
对类别型特征进行分析
print('grade')print(data_train['grade'].value_counts()) print('\nsubGrade')print(data_train['subGrade'].value_counts())print('\nemploymentLength')print(data_train['employmentLength'].value_counts())print('\nissueDate')print(data_train['issueDate'].value_counts())print('\nearliesCreditLine')print(data_train['earliesCreditLine'].value_counts())print('\nisDefault')print(data_train['isDefault'].value_counts())
gradeB 233690C 227118A 139661D 119453E55661F19053G5364Name: grade, dtype: int64subGradeC1 50763B4 49516B5 48965B3 48600C2 47068C3 44751C4 44272B2 44227B1 42382C5 40264A5 38045A4 30928D1 30538D2 26528A1 25909D3 23410A3 22655A2 22124D4 21139D5 17838E1 14064E2 12746E3 10925E49273E58653F15925F24340F33577F42859F52352G11759G21231G3978G4751G5645Name: subGrade, dtype: int64employmentLength10+ years 2627532 years 72358< 1 year642373 years 641521 year 524895 years 501024 years 479856 years 372548 years 361927 years 354079 years 30272Name: employmentLength, dtype: int64issueDate-03-01 29066-10-01 25525-07-01 24496-12-01 23245-10-01 21461... -08-01 23-07-01 21-09-01 19-09-01 7-06-01 1Name: issueDate, Length: 139, dtype: int64earliesCreditLineAug-2001 5567Aug-2002 5403Sep- 5403Oct-2001 5258Aug-2000 5246... Feb-1960 1Jul-1955 1Oct-1957 1Aug-1946 1Aug-1958 1Name: earliesCreditLine, Length: 720, dtype: int64isDefault0 6403901 159610Name: isDefault, dtype: int64
变量分布可视化
单一变量分布可视化
plt.figure(figsize=(8, 8))sns.barplot(data_train["employmentLength"].value_counts(dropna=False)[:20],data_train["employmentLength"].value_counts(dropna=False).keys()[:20])plt.show()
可视化某个特征x在不同y值上的分布
查看类别型变量在不同y值上的分布train_loan_fr = data_train.loc[data_train['isDefault'] == 1]train_loan_nofr = data_train.loc[data_train['isDefault'] == 0]fig, ((ax1, ax2), (ax3, ax4)) = plt.subplots(2, 2, figsize=(15, 8))train_loan_fr.groupby('grade')['grade'].count().plot(kind='barh', ax=ax1, title='Count of grade fraud')train_loan_nofr.groupby('grade')['grade'].count().plot(kind='barh', ax=ax2, title='Count of grade non-fraud')train_loan_fr.groupby('employmentLength')['employmentLength'].count().plot(kind='barh',ax=ax3, title='Count of employmentLength fraud')train_loan_nofr.groupby('employmentLength')['employmentLength'].count().plot(kind='barh',ax=ax4, title='Count of employmentLength non-fraud')plt.show()
查看连续型变量在不同y值上的分布
fig, ((ax1, ax2)) = plt.subplots(1, 2, figsize=(15, 6))data_train.loc[data_train['isDefault'] == 1] \['loanAmnt'].apply(np.log) \.plot(kind='hist',bins=100,title='Log Loan Amt - Fraud',color='r',xlim=(-3, 10),ax= ax1)data_train.loc[data_train['isDefault'] == 0] \['loanAmnt'].apply(np.log) \.plot(kind='hist',bins=100,title='Log Loan Amt - Not Fraud',color='b',xlim=(-3, 10),ax=ax2)
total = len(data_train)total_amt = data_train.groupby(['isDefault'])['loanAmnt'].sum().sum()plt.figure(figsize=(12,5))plt.subplot(121) # 1代表行，2代表列，所以一共有2个图，1代表此时绘制第一个图。plot_tr = sns.countplot(x='isDefault',data=data_train) plot_tr.set_title("Fraud Loan Distribution \n 0: good user | 1: bad user", fontsize=14)plot_tr.set_xlabel("Is fraud by count", fontsize=16)plot_tr.set_ylabel('Count', fontsize=16)for p in plot_tr.patches:height = p.get_height()plot_tr.text(p.get_x()+p.get_width()/2.,height + 3,'{:1.2f}%'.format(height/total*100),ha="center", fontsize=15)percent_amt = (data_train.groupby(['isDefault'])['loanAmnt'].sum())percent_amt = percent_amt.reset_index()plt.subplot(122)plot_tr_2 = sns.barplot(x='isDefault', y='loanAmnt', dodge=True, data=percent_amt)plot_tr_2.set_title("Total Amount in loanAmnt \n 0: good user | 1: bad user", fontsize=14)plot_tr_2.set_xlabel("Is fraud by percent", fontsize=16)plot_tr_2.set_ylabel('Total Loan Amount Scalar', fontsize=16)for p in plot_tr_2.patches:height = p.get_height()plot_tr_2.text(p.get_x()+p.get_width()/2.,height + 3,'{:1.2f}%'.format(height/total_amt * 100),ha="center", fontsize=15)
时间格式数据处理及查看
#转化成时间格式data_train['issueDate'] = pd.to_datetime(data_train['issueDate'],format='%Y-%m-%d')startdate = datetime.datetime.strptime('-06-01', '%Y-%m-%d')data_train['issueDateDT'] = data_train['issueDate'].apply(lambda x: x-startdate).dt.daysdata_test_a['issueDate'] = pd.to_datetime(data_train['issueDate'],format='%Y-%m-%d')startdate = datetime.datetime.strptime('-06-01', '%Y-%m-%d')data_test_a['issueDateDT'] = data_test_a['issueDate'].apply(lambda x: x-startdate).dt.daysplt.hist(data_train['issueDateDT'], label='train');plt.hist(data_test_a['issueDateDT'], label='test');plt.legend();plt.title('Distribution of issueDateDT dates');# train 和 test issueDateDT 日期有重叠，所以使用基于时间的分割进行验证是不明智的
透视图
# 透视图的索引可以有多个，'columns'是可选的，聚合函数 aggfunc 最后是被应用到了变量'values'中所列举的项目上pivot = pd.pivot_table(data_train, index=['grade'], columns=['issueDateDT'], values=['loanAmnt'], aggfunc=np.sum)pivot
7 rows × 139 columns
用 pandas_profiling 生成数据报告
import pandas_profilingpfr = pandas_profiling.ProfileReport(data_train)pfr.to_file("example.html")
参考资料
关于特征分箱可以参考：特征的分箱
