spark中遇到mysql链接超时的问题解决

背景最近使用scala写spark的运行jar，不小心遇到一个比较傻的坑，主要是mysql方面的报错，报错如下2021-09-13 10:26:12.426INFO 22591 --- [launcher-proc-4] o.a.s.launcher.app.Application: Exception in thread "main" com.mysql.jdbc.exceptions.jdbc

涟漪海洋

1662人浏览 · 2021-09-13 10:41:46

涟漪海洋 · 2021-09-13 10:41:46 发布

背景

最近使用scala写spark的运行jar，不小心遇到一个比较傻的坑，主要是mysql方面的报错，

报错如下

2021-09-13 10:26:12.426  INFO 22591 --- [launcher-proc-4] o.a.s.launcher.app.Application   : Exception in thread "main" com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure
  2021-09-13 10:26:12.427  INFO 22591 --- [launcher-proc-4] o.a.s.launcher.app.Application   :
  2021-09-13 10:26:12.427  INFO 22591 --- [launcher-proc-4] o.a.s.launcher.app.Application   : The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.
  2021-09-13 10:26:12.427  INFO 22591 --- [launcher-proc-4] o.a.s.launcher.app.Application   :    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
  2021-09-13 10:26:12.427  INFO 22591 --- [launcher-proc-4] o.a.s.launcher.app.Application   :    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
  2021-09-13 10:26:12.427  INFO 22591 --- [launcher-proc-4] o.a.s.launcher.app.Application   :    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
  2021-09-13 10:26:12.427  INFO 22591 --- [launcher-proc-4] o.a.s.launcher.app.Application   :    at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
  2021-09-13 10:26:12.427  INFO 22591 --- [launcher-proc-4] o.a.s.launcher.app.Application   :    at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
  2021-09-13 10:26:12.427  INFO 22591 --- [launcher-proc-4] o.a.s.launcher.app.Application   :    at com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:1129)
  2021-09-13 10:26:12.427  INFO 22591 --- [launcher-proc-4] o.a.s.launcher.app.Application   :    at com.mysql.jdbc.MysqlIO.<init>(MysqlIO.java:358)
  2021-09-13 10:26:12.427  INFO 22591 --- [launcher-proc-4] o.a.s.launcher.app.Application   :    at com.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:2498)
  2021-09-13 10:26:12.427  INFO 22591 --- [launcher-proc-4] o.a.s.launcher.app.Application   :    at com.mysql.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:2535)
  2021-09-13 10:26:12.427  INFO 22591 --- [launcher-proc-4] o.a.s.launcher.app.Application   :    at com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2320)
  2021-09-13 10:26:12.427  INFO 22591 --- [launcher-proc-4] o.a.s.launcher.app.Application   :    at com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:834)
  2021-09-13 10:26:12.427  INFO 22591 --- [launcher-proc-4] o.a.s.launcher.app.Application   :    at com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:46)
  2021-09-13 10:26:12.427  INFO 22591 --- [launcher-proc-4] o.a.s.launcher.app.Application   :    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
  2021-09-13 10:26:12.427  INFO 22591 --- [launcher-proc-4] o.a.s.launcher.app.Application   :    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
  2021-09-13 10:26:12.427  INFO 22591 --- [launcher-proc-4] o.a.s.launcher.app.Application   :    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
  2021-09-13 10:26:12.427  INFO 22591 --- [launcher-proc-4] o.a.s.launcher.app.Application   :    at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
  2021-09-13 10:26:12.427  INFO 22591 --- [launcher-proc-4] o.a.s.launcher.app.Application   :    at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
  2021-09-13 10:26:12.427  INFO 22591 --- [launcher-proc-4] o.a.s.launcher.app.Application   :    at com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:416)
  2021-09-13 10:26:12.427  INFO 22591 --- [launcher-proc-4] o.a.s.launcher.app.Application   :    at com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:347)
  2021-09-13 10:26:12.427  INFO 22591 --- [launcher-proc-4] o.a.s.launcher.app.Application   :    at java.sql.DriverManager.getConnection(DriverManager.java:664)
  2021-09-13 10:26:12.428  INFO 22591 --- [launcher-proc-4] o.a.s.launcher.app.Application   :    at java.sql.DriverManager.getConnection(DriverManager.java:247)
  2021-09-13 10:26:12.428  INFO 22591 --- [launcher-proc-4] o.a.s.launcher.app.Application   :    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  2021-09-13 10:26:12.428  INFO 22591 --- [launcher-proc-4] o.a.s.launcher.app.Application   :    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  2021-09-13 10:26:12.428  INFO 22591 --- [launcher-proc-4] o.a.s.launcher.app.Application   :    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  2021-09-13 10:26:12.428  INFO 22591 --- [launcher-proc-4] o.a.s.launcher.app.Application   :    at java.lang.reflect.Method.invoke(Method.java:498)
  2021-09-13 10:26:12.428  INFO 22591 --- [launcher-proc-4] o.a.s.launcher.app.Application   :    at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
  2021-09-13 10:26:12.428  INFO 22591 --- [launcher-proc-4] o.a.s.launcher.app.Application   :    at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:951)
  2021-09-13 10:26:12.428  INFO 22591 --- [launcher-proc-4] o.a.s.launcher.app.Application   :    at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180)
  2021-09-13 10:26:12.428  INFO 22591 --- [launcher-proc-4] o.a.s.launcher.app.Application   :    at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)
  2021-09-13 10:26:12.428  INFO 22591 --- [launcher-proc-4] o.a.s.launcher.app.Application   :    at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)
  2021-09-13 10:26:12.428  INFO 22591 --- [launcher-proc-4] o.a.s.launcher.app.Application   :    at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1039)
  2021-09-13 10:26:12.428  INFO 22591 --- [launcher-proc-4] o.a.s.launcher.app.Application   :    at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1048)
  2021-09-13 10:26:12.428  INFO 22591 --- [launcher-proc-4] o.a.s.launcher.app.Application   :    at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
  2021-09-13 10:26:12.428  INFO 22591 --- [launcher-proc-4] o.a.s.launcher.app.Application   : Caused by: java.net.ConnectException: 连接超时 (Connection timed out)
  2021-09-13 10:26:12.428  INFO 22591 --- [launcher-proc-4] o.a.s.launcher.app.Application   :    at java.net.PlainSocketImpl.socketConnect(Native Method)
  2021-09-13 10:26:12.428  INFO 22591 --- [launcher-proc-4] o.a.s.launcher.app.Application   :    at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
  2021-09-13 10:26:12.428  INFO 22591 --- [launcher-proc-4] o.a.s.launcher.app.Application   :    at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
  2021-09-13 10:26:12.428  INFO 22591 --- [launcher-proc-4] o.a.s.launcher.app.Application   :    at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
  2021-09-13 10:26:12.428  INFO 22591 --- [launcher-proc-4] o.a.s.launcher.app.Application   :    at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
  2021-09-13 10:26:12.428  INFO 22591 --- [launcher-proc-4] o.a.s.launcher.app.Application   :    at java.net.Socket.connect(Socket.java:589)
  2021-09-13 10:26:12.429  INFO 22591 --- [launcher-proc-4] o.a.s.launcher.app.Application   :    at java.net.Socket.connect(Socket.java:538)
  2021-09-13 10:26:12.429  INFO 22591 --- [launcher-proc-4] o.a.s.launcher.app.Application   :    at java.net.Socket.<init>(Socket.java:434)
  2021-09-13 10:26:12.429  INFO 22591 --- [launcher-proc-4] o.a.s.launcher.app.Application   :    at java.net.Socket.<init>(Socket.java:244)
  2021-09-13 10:26:12.429  INFO 22591 --- [launcher-proc-4] o.a.s.launcher.app.Application   :    at com.mysql.jdbc.StandardSocketFactory.connect(StandardSocketFactory.java:256)
  2021-09-13 10:26:12.429  INFO 22591 --- [launcher-proc-4] o.a.s.launcher.app.Application   :    at com.mysql.jdbc.MysqlIO.<init>(MysqlIO.java:308)
  2021-09-13 10:26:12.429  INFO 22591 --- [launcher-proc-4] o.a.s.launcher.app.Application   :    ... 29 more
  2021-09-13 10:26:12.555  INFO 22591 --- [launcher-proc-4] c.a.n.e.s.impl.SparkSubmitServiceImpl    : stateChanged:FAILED

原因分析

根据错误提示，大致应该是使用mysql连接不上，或者是连接超时

编者的真实错误原因

因为做了开发环境和测试环境的配置，且，测试环境和开发环境的网络不通，不小心使用的是测试环境的mysq数据库配置，知道是这个原因，把配置文件改成开发环境的即可

原因总结

1、需要连接的目标mysql服务器是否存在(或者是配置信息是否有误)

2、mysql的链接是否真的是超时的

和链接或者是连接池中的配置做比较

# 查看mysql的链接超时时间(单位是秒)
mysql> show global variables like 'wait_timeout';

可直接使用命令修改

mysql> set global wait_timeout=1814400;

比较搞笑的错误，耽误了这么老久，记录下。。。。。

当然，如果是设置了超时间之后仍然超时，那就可能是代码或者是网络的问题了

开放原子开发者工作坊

开放原子开发者工作坊旨在鼓励更多人参与开源活动，与志同道合的开发者们相互交流开发经验、分享开发心得、获取前沿技术趋势。工作坊有多种形式的开发者活动，如meetup、训练营等，主打技术交流，干货满满，真诚地邀请各位开发者共同参与！

更多推荐

开源成为金融领域创新发展的新动力引擎

开放原子开发者工作坊

开源成为推动城市数字化转型的核心动力

城市，不仅是人们工作与生活的基本载体，更是承载着亿万人民的希望与梦想。近年来，我国智慧城市建设如火如荼，开源技术的深度融合为其发展注入了新的活力。在民生、环保、公共安全、城市服务等多个领域，开源技术的应用不仅为民众带来了切实的便捷，而且促进了政府决策的科学化，显著提升了城市综合治理能力。

开放原子开发者工作坊

OpenLoong项目通过技术监督委员会（TOC）评审

开放原子开发者工作坊

所有评论(0)

查看更多评论

涟漪海洋

@hu10131013

已为社区贡献2条内容