오류 메세지 : Transport endpoint is not connected (error 107)

원인 : 특정 Kudu테이블에 동시에 혹은 연속적으로 많은 update/insert쿼리가 수행되는 경우 

조치 : Kudu tablet server의 configuration에서 --rpc_max_message_size를 200MB 혹은 250MB정도로 증가 시키고 서비스를 restart해주면 오류를 해소 혹은 완화 할 수 있다.

======================== Coordinator 역할 impalad log file ========

node_idcp01.gooper.com INFO November 21, 2023 10:50 AM cc:678 

ExecState: query id=11454a6d71dc7db7:7201b9e000000000 finstance=11454a6d71dc7db7:7201b9e000000013 on host=node_datp09.gooper.com:22000 (EXECUTING -> ERROR) status=Unable to advance iterator for node with id '0' for Kudu table 'impala::db_name.table_name': Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)

======================== Executor 약할의 impalad log file===========

10:20:08.959 AM INFO cc:306 

9a497afbb0dcd55d:7ab21eed00000448] DeregisterRecvr(): fragment_instance_id=9a497afbb0dcd55d:7ab21eed00000448, node=241

10:20:09.011 AM INFO cc:337 

9a497afbb0dcd55d:7ab21eed00000448] cancelling active streams for fragment_instance_id=9a497afbb0dcd55d:7ab21eed00000448

10:20:09.011 AM INFO cc:734 

9a497afbb0dcd55d:7ab21eed00000448] Instance completed. instance_id=9a497afbb0dcd55d:7ab21eed00000448 #in-flight=56 status=OK

10:20:09.112 AM INFO cc:306 

9a497afbb0dcd55d:7ab21eed00000459] DeregisterRecvr(): fragment_instance_id=9a497afbb0dcd55d:7ab21eed00000459, node=243

10:20:15.563 AM INFO cc:72 

61400b1a312547d7:e3cc9f570000000a] Serialized kudu.tserver.ScanRequestPB (134219283 bytes) is larger than the maximum configured RPC message size (52428800 bytes). Sending anyway, but peer may reject the data.

10:20:15.641 AM INFO cc:664 

client connection to 11.111.111.39:7050 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)

10:20:15.641 AM INFO cc:1337 

f44fa39abb1a8044:2ddb9a720000000b] marking tablet server 52bf6df6c61a442c8847fbd228672bc9 (node_datp19.gooper.com:7050) as failed

10:20:15.641 AM INFO cc:1337 

61400b1a312547d7:e3cc9f570000000a] marking tablet server 52bf6df6c61a442c8847fbd228672bc9 (node_datp19.gooper.com:7050) as failed

10:20:15.641 AM INFO cc:288 

f44fa39abb1a8044:2ddb9a720000000b] tablet 4561250e99ea415c8adf1ed18f4eefcc: replica 52bf6df6c61a442c8847fbd228672bc9 (node_datp19.gooper.com:7050) has failed: Network error: TS failed: recv error from unknown peer: Transport endpoint is not connected (error 107)

10:20:15.659 AM INFO cc:72 

61400b1a312547d7:e3cc9f570000000a] Serialized kudu.tserver.ScanRequestPB (134219283 bytes) is larger than the maximum configured RPC message size (52428800 bytes). Sending anyway, but peer may reject the data.

10:20:15.653 AM INFO cc:129 

f44fa39abb1a8044:2ddb9a720000000b] Unable to advance iterator for node with id '0' for Kudu table 'impala::db_name.table_name': Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)

    @           0xc1fe33

    @          0x1620fa1

    @          0x1624595

    @          0x15878a9

    @          0x15891f8

    @          0x144f531

    @          0x14509aa

    @          0x1ca1101

    @     0x7f512a6c62dd

    @     0x7f5127183e82

===================== kudu_tserver log file 내용 =====================

 628292 I1121 10:20:13.968837 3471434 maintenance_manager.cc:382] P 9d84c6e1acd245f99d242449521b64cc: Scheduling UndoDeltaBlockGCOp(07db2efadf1b47a6936aaf0808de1339): 51660 bytes on disk

 628293 I1121 10:20:13.969128 3471142 maintenance_manager.cc:603] P 9d84c6e1acd245f99d242449521b64cc: UndoDeltaBlockGCOp(07db2efadf1b47a6936aaf0808de1339) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":4,"lbm_read_time_us":45,"lbm_reads_lt_1ms":16}

 628294 I1121 10:20:14.153570 3471434 maintenance_manager.cc:382] P 9d84c6e1acd245f99d242449521b64cc: Scheduling CompactRowSetsOp(3275654fd0a8404fa28d667d664135a3): perf score=0.205744

 628295 I1121 10:20:14.853590 3471141 maintenance_manager.cc:603] P 9d84c6e1acd245f99d242449521b64cc: CompactRowSetsOp(07db2efadf1b47a6936aaf0808de1339) complete. Timing: real 4.044s  user 3.590s sys 0.290s Metrics: {"bytes_written":154984322,"cfile_cache_hit":392,"cfile_cache_hit_bytes":14304480,"cfile_cache_miss":2608,"cfile_cache_miss_bytes":257784062,"cfile_init":200,"delta_iterators_relevant":8,"dirs.queue_time_us":342915,"dirs.run_cpu_time_us":33830,"d irs.run_wall_time_us":37009,"drs_written":4,"fdatasync":127,"fdatasync_us":51589,"lbm_read_time_us":111171,"lbm_reads_lt_1ms":3408,"lbm_write_time_us":102309,"lbm_writes_lt_1ms":10049,"mutex_wait_us":18,"num_input_rowsets":4,"rows_written":859031,"spinlock_wait_cycles":56704,"thread_start_us":777,"threads_started":25,"wal-append.queue_time_us":71}

 628296 I1121 10:20:15.031082 3471434 maintenance_manager.cc:382] P 9d84c6e1acd245f99d242449521b64cc: Scheduling CompactRowSetsOp(3275654fd0a8404fa28d667d664135a3): perf score=0.195746

 628297 W1121 10:20:16.467859 3471134 connection.cc:664] server connection from 11.111.111.35:36948 recv error: Network error: RPC frame had a length of 134219341, but we only support messages up to 134217728 bytes long.

 628298 W1121 10:20:16.467888 3471134 connection.cc:295] Shutting down server connection from 11.111.111.35:36948 with pending inbound data (4/134219341 bytes received, last active 0 ns ago, status=Network error: RPC frame had a length of 134219341, but we only support messages up to 134217728 bytes long.)

 628299 W1121 10:20:16.559211 3471137 connection.cc:664] server connection from 11.111.111.33:51706 recv error: Network error: RPC frame had a length of 134219341, but we only support messages up to 134217728 bytes long.

 628300 W1121 10:20:16.559245 3471137 connection.cc:295] Shutting down server connection from 11.111.111.33:51706 with pending inbound data (4/134219341 bytes received, last active 0 ns ago, status=Network error: RPC frame had a length of 134219341, but we only support messages up to 134217728 bytes long.)

 628301 W1121 10:20:17.182857 3471134 connection.cc:664] server connection from 11.111.111.35:38176 recv error: Network error: RPC frame had a length of 134219340, but we only support messages up to 134217728 bytes long.

 628302 W1121 10:20:17.182881 3471134 connection.cc:295] Shutting down server connection from 11.111.111.35:38176 with pending inbound data (4/134219340 bytes received, last active 0 ns ago, status=Network error: RPC frame had a length of 134219340, but we only support messages up to 134217728 bytes long.)

 628303 W1121 10:20:17.292801 3471134 connection.cc:664] server connection from 11.111.111.33:52788 recv error: Network error: RPC frame had a length of 134219340, but we only support messages up to 134217728 bytes long.

 628304 W1121 10:20:17.292827 3471134 connection.cc:295] Shutting down server connection from 11.111.111.33:52788 with pending inbound data (4/134219340 bytes received, last active 0 ns ago, status=Network error: RPC frame had a length of 134219340, but we only support messages up to 134217728 bytes long.)

 628305 I1121 10:20:17.481701 3471139 maintenance_manager.cc:603] P 9d84c6e1acd245f99d242449521b64cc: CompactRowSetsOp(eb28422960104df2a553161f4385d726) complete. Timing: real 18.869s user 6.304s sys 0.412s Metrics: {"bytes_written": 40193385,"cfile_cache_hit":362,"cfile_cache_hit_bytes":4987939,"cfile_cache_miss":4974,"cfile_cache_miss_bytes":217966365,"cfile_init":208,"delta_iterators_relevant":4,"dirs.queue_time_us":5963,"dirs.run_cpu_time_us":4649,"dirs.run_wall_time_us":6201,"drs_written":4,"fdatasync":123,"fdatasync_us":12074,"lbm_read_time_us":11465204,"lbm_reads_1-10_ms":109,"lbm_reads_10-100_ms":209,"lbm_reads_gt_100_ms":21,"lbm_reads_lt_1ms":5467,"lbm_write_time_us":152850,"lbm_writes_1-10_ms":1,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":23230,"mutex_wait_us":382,"num_input_rowsets":4,"rows_written":2189548,"spinlock_wait_cycles":85888,"thread_start_us":893,"threads_started":28,"wal-append.queue_time_us":75}