[구글 클라우드] Estimator API 사용해서 AI 모델 구현 (TensorFlow)

 

(source : GCP qwiklabs)

 

AI Platform Notebooks 생성

 

1. Navigation Menu > AI Platform > Notebooks 클릭

 

2. 인스턴스 생성 : NEW INSTANCE 클릭 

- Tensorflow Enterprise 1.15 > Without GPUs 선택

 

3. 팝업창이 뜨면 딥러닝 VM 이름을 입력/확인 하고 Create 클릭

 

(VM 생성까지 2~3분 정도 소요될 수 있음)

 

4. Open JupyterLab 클릭 > JupyterLab Window가 새창으로 열림

 

Datalab instance에 학습용 repo 복제

- training-data-analyst 를 내 JupyterLab 인스턴스에 복제

 

1. JupyterLab 에서 터미널 아이콘(Terminal icon) 클릭하여 새 터미널 창을 오픈

 

2. 커맨드라인에서 아래 명령어를 입력하고 Enter

git clone https://github.com/GoogleCloudPlatform/training-data-analyst

 

실행 결과 : 

 

3. training-data-anlyst 디렉토리를 더블 클릭하여 컨텐츠가 제대로 복제되었는지 확인

 

 

tf.estimator 를 이용한 머신러닝

 

1. 노트북 패널에서 training-data-analyst > courses > machine_learning > deepdive > 03_tensorflow > labs > b_estimator.ipynb 선택

 

- Jupyter Notebook 실습 코드 : 모델

b_estimator.html

0.28MB

 

- Jupyter Notebook 실습 코드 : 정답 포함

b_estimator_solution.html

0.39MB

 

2. Train and eval 입력 기능 : Pandas Dataframe 읽기

def make_train_input_fn(df, num_epochs):
  return tf.estimator.inputs.pandas_input_fn(
    x = df,
    y = df[LABEL],
    batch_size = 128,
    num_epochs = num_epochs,
    shuffle = True,
    queue_capacity = 1000
  )

def make_eval_input_fn(df):
  return tf.estimator.inputs.pandas_input_fn(
    x = df,
    y = df[LABEL],
    batch_size = 128,
    shuffle = False,
    queue_capacity = 1000
  )

def make_prediction_input_fn(df):
  return tf.estimator.inputs.pandas_input_fn(
    x = df,
    y = None,
    batch_size = 128,
    shuffle = False,
    queue_capacity = 1000
  )

 

 

3. estimator 를 위한 feature columns 생성 

def make_feature_cols():
  input_columns = [tf.feature_column.numeric_column(k) for k in FEATURES]
  return input_columns

 

 

 

4. Linear Regression (선형회귀) with tf.Estimator framework 

tf.logging.set_verbosity(tf.logging.INFO)

OUTDIR = 'taxi_trained'
shutil.rmtree(OUTDIR, ignore_errors = True) # start fresh each time

model = tf.estimator.LinearRegressor(
      feature_columns = make_feature_cols(), model_dir = OUTDIR)

model.train(input_fn = make_train_input_fn(df_train, num_epochs = 10))

predictions = model.predict(input_fn = make_prediction_input_fn(df_test))
for items in predictions:
  print(items)

 

5. Deep Neural Network regression 

tf.logging.set_verbosity(tf.logging.INFO)
shutil.rmtree(OUTDIR, ignore_errors = True) # start fresh each time
model = tf.estimator.DNNRegressor(hidden_units = [32, 8, 2],
      feature_columns = make_feature_cols(), model_dir = OUTDIR)
model.train(input_fn = make_train_input_fn(df_train, num_epochs = 100));
print_rmse(model, df_valid)

 

실행 결과 : 

INFO:tensorflow:Using default config.
INFO:tensorflow:Using config: {'_experimental_distribute': None, '_session_config': allow_soft_placement: true
graph_options {
  rewrite_options {
    meta_optimizer_iterations: ONE
  }
}
, '_global_id_in_cluster': 0, '_tf_random_seed': None, '_service': None, '_num_ps_replicas': 0, '_evaluation_master': '', '_log_step_count_steps': 100, '_save_checkpoints_steps': None, '_task_id': 0, '_device_fn': None, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x7f15d7d567f0>, '_eval_distribute': None, '_keep_checkpoint_max': 5, '_master': '', '_keep_checkpoint_every_n_hours': 10000, '_save_summary_steps': 100, '_protocol': None, '_task_type': 'worker', '_is_chief': True, '_save_checkpoints_secs': 600, '_model_dir': 'taxi_trained', '_num_worker_replicas': 1, '_train_distribute': None}
INFO:tensorflow:Calling model_fn.
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Create CheckpointSaverHook.
INFO:tensorflow:Graph was finalized.
INFO:tensorflow:Running local_init_op.
INFO:tensorflow:Done running local_init_op.
INFO:tensorflow:Saving checkpoints for 0 into taxi_trained/model.ckpt.
INFO:tensorflow:loss = 467570.2, step = 1
INFO:tensorflow:global_step/sec: 100.82
INFO:tensorflow:loss = 20871.418, step = 101 (0.994 sec)
INFO:tensorflow:global_step/sec: 108.186
INFO:tensorflow:loss = 19313.43, step = 201 (0.926 sec)
INFO:tensorflow:global_step/sec: 112.618
INFO:tensorflow:loss = 26053.674, step = 301 (0.890 sec)
INFO:tensorflow:global_step/sec: 94.7783
INFO:tensorflow:loss = 24630.527, step = 401 (1.054 sec)
INFO:tensorflow:global_step/sec: 123.391
INFO:tensorflow:loss = 20816.373, step = 501 (0.811 sec)
INFO:tensorflow:global_step/sec: 125.027
INFO:tensorflow:loss = 19305.867, step = 601 (0.800 sec)
INFO:tensorflow:global_step/sec: 100.336
INFO:tensorflow:loss = 17916.236, step = 701 (0.997 sec)
INFO:tensorflow:global_step/sec: 123.813
INFO:tensorflow:loss = 16324.613, step = 801 (0.808 sec)
INFO:tensorflow:global_step/sec: 96.5849
INFO:tensorflow:loss = 25742.564, step = 901 (1.034 sec)
INFO:tensorflow:global_step/sec: 126.786
INFO:tensorflow:loss = 20716.746, step = 1001 (0.789 sec)
INFO:tensorflow:global_step/sec: 102.184
INFO:tensorflow:loss = 17350.438, step = 1101 (0.979 sec)
INFO:tensorflow:global_step/sec: 129.286
INFO:tensorflow:loss = 23090.242, step = 1201 (0.773 sec)
INFO:tensorflow:global_step/sec: 119.124
INFO:tensorflow:loss = 14777.045, step = 1301 (0.840 sec)
INFO:tensorflow:global_step/sec: 96.1406
INFO:tensorflow:loss = 20134.01, step = 1401 (1.040 sec)
INFO:tensorflow:global_step/sec: 112.909
INFO:tensorflow:loss = 15141.786, step = 1501 (0.882 sec)
INFO:tensorflow:global_step/sec: 102.4
INFO:tensorflow:loss = 17473.344, step = 1601 (0.980 sec)
INFO:tensorflow:global_step/sec: 127.155
INFO:tensorflow:loss = 20703.318, step = 1701 (0.787 sec)
INFO:tensorflow:global_step/sec: 118.505
INFO:tensorflow:loss = 20141.992, step = 1801 (0.844 sec)
INFO:tensorflow:global_step/sec: 104.505
INFO:tensorflow:loss = 18941.66, step = 1901 (0.957 sec)
INFO:tensorflow:global_step/sec: 121.39
INFO:tensorflow:loss = 10826.924, step = 2001 (0.820 sec)
INFO:tensorflow:global_step/sec: 103.776
INFO:tensorflow:loss = 15104.154, step = 2101 (0.967 sec)
INFO:tensorflow:global_step/sec: 117.015
INFO:tensorflow:loss = 16825.898, step = 2201 (0.855 sec)
INFO:tensorflow:global_step/sec: 125.782
INFO:tensorflow:loss = 17469.46, step = 2301 (0.794 sec)
INFO:tensorflow:global_step/sec: 106.035
INFO:tensorflow:loss = 15310.082, step = 2401 (0.944 sec)
INFO:tensorflow:global_step/sec: 121.658
INFO:tensorflow:loss = 22304.357, step = 2501 (0.818 sec)
INFO:tensorflow:global_step/sec: 99.4088
INFO:tensorflow:loss = 19509.504, step = 2601 (1.006 sec)
INFO:tensorflow:global_step/sec: 114.029
INFO:tensorflow:loss = 24601.023, step = 2701 (0.877 sec)
INFO:tensorflow:global_step/sec: 55.0054
INFO:tensorflow:loss = 21849.145, step = 2801 (1.817 sec)
INFO:tensorflow:global_step/sec: 128.72
INFO:tensorflow:loss = 11309.27, step = 2901 (0.777 sec)
INFO:tensorflow:global_step/sec: 93.5308
INFO:tensorflow:loss = 21104.457, step = 3001 (1.069 sec)
INFO:tensorflow:global_step/sec: 96.7103
INFO:tensorflow:loss = 16993.004, step = 3101 (1.034 sec)
INFO:tensorflow:global_step/sec: 91.3205
INFO:tensorflow:loss = 18189.752, step = 3201 (1.098 sec)
INFO:tensorflow:global_step/sec: 112.614
INFO:tensorflow:loss = 14859.451, step = 3301 (0.888 sec)
INFO:tensorflow:global_step/sec: 90.5019
INFO:tensorflow:loss = 12333.189, step = 3401 (1.105 sec)
INFO:tensorflow:global_step/sec: 104.302
INFO:tensorflow:loss = 5995.971, step = 3501 (0.959 sec)
INFO:tensorflow:global_step/sec: 92.9583
INFO:tensorflow:loss = 10282.813, step = 3601 (1.075 sec)
INFO:tensorflow:global_step/sec: 104.575
INFO:tensorflow:loss = 9950.309, step = 3701 (0.957 sec)
INFO:tensorflow:global_step/sec: 88.9567
INFO:tensorflow:loss = 16426.35, step = 3801 (1.124 sec)
INFO:tensorflow:global_step/sec: 100.309
INFO:tensorflow:loss = 22024.291, step = 3901 (0.997 sec)
INFO:tensorflow:global_step/sec: 105.657
INFO:tensorflow:loss = 14616.185, step = 4001 (0.947 sec)
INFO:tensorflow:global_step/sec: 100.456
INFO:tensorflow:loss = 13213.275, step = 4101 (0.996 sec)
INFO:tensorflow:global_step/sec: 113.571
INFO:tensorflow:loss = 12385.185, step = 4201 (0.877 sec)
INFO:tensorflow:global_step/sec: 93.7462
INFO:tensorflow:loss = 9375.332, step = 4301 (1.067 sec)
INFO:tensorflow:global_step/sec: 112.179
INFO:tensorflow:loss = 14900.719, step = 4401 (0.894 sec)
INFO:tensorflow:global_step/sec: 93.2297
INFO:tensorflow:loss = 12472.631, step = 4501 (1.071 sec)
INFO:tensorflow:global_step/sec: 106.669
INFO:tensorflow:loss = 15485.025, step = 4601 (0.937 sec)
INFO:tensorflow:global_step/sec: 92.3307
INFO:tensorflow:loss = 18741.918, step = 4701 (1.083 sec)
INFO:tensorflow:global_step/sec: 106.823
INFO:tensorflow:loss = 13244.169, step = 4801 (0.936 sec)
INFO:tensorflow:global_step/sec: 87.4236
INFO:tensorflow:loss = 10333.318, step = 4901 (1.147 sec)
INFO:tensorflow:global_step/sec: 127.855
INFO:tensorflow:loss = 15173.955, step = 5001 (0.782 sec)
INFO:tensorflow:global_step/sec: 112.028
INFO:tensorflow:loss = 12940.822, step = 5101 (0.890 sec)
INFO:tensorflow:global_step/sec: 100.607
INFO:tensorflow:loss = 10854.685, step = 5201 (0.997 sec)
INFO:tensorflow:global_step/sec: 121.355
INFO:tensorflow:loss = 14676.209, step = 5301 (0.822 sec)
INFO:tensorflow:global_step/sec: 95.9606
INFO:tensorflow:loss = 13156.73, step = 5401 (1.043 sec)
INFO:tensorflow:global_step/sec: 118.042
INFO:tensorflow:loss = 17359.934, step = 5501 (0.847 sec)
INFO:tensorflow:global_step/sec: 109.939
INFO:tensorflow:loss = 8592.682, step = 5601 (0.910 sec)
INFO:tensorflow:global_step/sec: 104.666
INFO:tensorflow:loss = 11916.723, step = 5701 (0.956 sec)
INFO:tensorflow:global_step/sec: 110.305
INFO:tensorflow:loss = 17452.98, step = 5801 (0.906 sec)
INFO:tensorflow:global_step/sec: 91.3437
INFO:tensorflow:loss = 20825.383, step = 5901 (1.095 sec)
INFO:tensorflow:global_step/sec: 121.12
INFO:tensorflow:loss = 4669.6084, step = 6001 (0.825 sec)
INFO:tensorflow:Saving checkpoints for 6071 into taxi_trained/model.ckpt.
INFO:tensorflow:Loss for final step: 4455.771.
INFO:tensorflow:Calling model_fn.
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Starting evaluation at 2019-07-01T20:22:40Z
INFO:tensorflow:Graph was finalized.
INFO:tensorflow:Restoring parameters from taxi_trained/model.ckpt-6071
INFO:tensorflow:Running local_init_op.
INFO:tensorflow:Done running local_init_op.
INFO:tensorflow:Finished evaluation at 2019-07-01-20:22:41
INFO:tensorflow:Saving dict for global step 6071: average_loss = 139.01192, global_step = 6071, label/mean = 11.666427, loss = 16532.488, prediction/mean = 6.1680436
INFO:tensorflow:Saving 'checkpoint_path' summary for global step 6071: taxi_trained/model.ckpt-6071
RMSE on dataset = 11.790331840515137