kudu-python安装
对kudu数据库进行CUID操作可以通过impala或者直接连接kudu库,前者有impyla的python包,但速度较慢,后者有kudu-python包,速度我目前还不清楚,但猜测应该会快。impyla的使用较简单,最近想通过kudu-python进行连接,但仅仅是完成了kudu-python的安装,并没有完成连接,有待继续研究。官方给的安装说明中,貌似是将kudu服务端、客户端、kudu-..
对kudu数据库进行CRUD操作可以通过impala或者直接连接kudu库,前者有impyla的python包,但速度较慢,后者有kudu-python包,速度我目前还不清楚,但猜测应该会快。
impyla的使用较简单,最近想通过kudu-python进行连接,但仅仅是完成了kudu-python的安装,并没有完成连接,有待继续研究。
官方给的安装说明中,貌似是将kudu服务端、客户端、kudu-python安装在了同一台服务器上,但我希望通过远程连接的方式进行连接,本机的操作系统是Ubuntu 18.04,故以下安装仅适用Ubuntu系统,CentOS系统安装过程类似。
首先安装kudu-client0和kudu-client-devel,但貌似apt方式安装需要加入源,比较麻烦,因而下载deb包进行安装。
deb下载地址:http://archive.cloudera.com/kudu/ubuntu/xenial/amd64/kudu/pool/contrib/k/kudu/
rpm下载地址:https://archive.cloudera.com/cdh5/redhat/6/x86_64/cdh/5/RPMS/x86_64/
(貌似deb的版本比rpm的版本要低不少)
我下载的是:
libkuduclient-dev_1.4.0+cdh5.12.2+0-1.cdh5.12.2.p0.8_xenial-kudu5.12.2_amd64.deb
libkuduclient0_1.4.0+cdh5.12.2+0-1.cdh5.12.2.p0.8_xenial-kudu5.12.2_amd64.deb
需要根据自己kudu版本和cdh版本进行选择。这里顺带提一下查看kudu和cdh版本的方法:
安装:
>>> sudo dpkg -i libkuduclient0_1.4.0+cdh5.12.2+0-1.cdh5.12.2.p0.8_xenial-kudu5.12.2_amd64.deb
Selecting previously unselected package libkuduclient0.
(Reading database ... 240290 files and directories currently installed.)
Preparing to unpack libkuduclient0_1.4.0+cdh5.12.2+0-1.cdh5.12.2.p0.8_xenial-kudu5.12.2_amd64.deb ...
Unpacking libkuduclient0 (1.4.0+cdh5.12.2+0-1.cdh5.12.2.p0.8~xenial-kudu5.12.2) ...
Setting up libkuduclient0 (1.4.0+cdh5.12.2+0-1.cdh5.12.2.p0.8~xenial-kudu5.12.2) ...
Processing triggers for libc-bin (2.27-3ubuntu1) ...
>>> sudo dpkg -i libkuduclient-dev_1.4.0+cdh5.12.2+0-1.cdh5.12.2.p0.8_xenial-kudu5.12.2_amd64.deb
Selecting previously unselected package libkuduclient-dev.
(Reading database ... 240297 files and directories currently installed.)
Preparing to unpack libkuduclient-dev_1.4.0+cdh5.12.2+0-1.cdh5.12.2.p0.8_xenial-kudu5.12.2_amd64.deb ...
Unpacking libkuduclient-dev (1.4.0+cdh5.12.2+0-1.cdh5.12.2.p0.8~xenial-kudu5.12.2) ...
Setting up libkuduclient-dev (1.4.0+cdh5.12.2+0-1.cdh5.12.2.p0.8~xenial-kudu5.12.2) ...
>>> pip3 install kudu-python==1.2.0
Defaulting to user installation because normal site-packages is not writeable
Looking in indexes: https://pypi.tuna.tsinghua.edu.cn/simple
Collecting kudu-python==1.2.0
Downloading https://pypi.tuna.tsinghua.edu.cn/packages/fa/8b/673e181f8d88c615cb1d57b2c49496294d5d800425507729dfae34bd184b/kudu-python-1.2.0.tar.gz (301 kB)
|████████████████████████████████| 301 kB 208 kB/s
Requirement already satisfied: cython>=0.21 in /home/xiang/.local/lib/python3.6/site-packages (from kudu-python==1.2.0) (0.29.13)
Requirement already satisfied: pytz in /home/xiang/.local/lib/python3.6/site-packages (from kudu-python==1.2.0) (2019.3)
Requirement already satisfied: six in /home/xiang/.local/lib/python3.6/site-packages (from kudu-python==1.2.0) (1.14.0)
Building wheels for collected packages: kudu-python
Building wheel for kudu-python (setup.py) ... done
Created wheel for kudu-python: filename=kudu_python-1.2.0-cp36-cp36m-linux_x86_64.whl size=1238719 sha256=b83a54da8c7461291dfddde0a1352d2564595b80e15aa5c33688ee7bd82afc3c
Stored in directory: /home/xiang/.cache/pip/wheels/a9/28/41/ae247f6bfbe92a5e3a4ab0f042f10559c4184a7d0e7c86a47d
Successfully built kudu-python
Installing collected packages: kudu-python
Successfully installed kudu-python-1.2.0
这里需要注意版本的问题
安装最新版本会报错:
>>> pip3 install kudu-python
Defaulting to user installation because normal site-packages is not writeable
Looking in indexes: https://pypi.tuna.tsinghua.edu.cn/simple
Collecting kudu-python
Using cached https://pypi.tuna.tsinghua.edu.cn/packages/b6/79/8c7b1d16ed419f2193a890473326779b249053ec170f4b865bc31fcef7ed/kudu-python-1.11.1.tar.gz (375 kB)
ERROR: Command errored out with exit status 1:
command: /usr/bin/python3 -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-mjbu7fzz/kudu-python/setup.py'"'"'; __file__='"'"'/tmp/pip-install-mjbu7fzz/kudu-python/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-install-mjbu7fzz/kudu-python/pip-egg-info
cwd: /tmp/pip-install-mjbu7fzz/kudu-python/
Complete output (8 lines):
Building from system prefix /usr
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/tmp/pip-install-mjbu7fzz/kudu-python/setup.py", line 144, in <module>
generate_config_pxi(INCLUDE_PATHS)
File "/tmp/pip-install-mjbu7fzz/kudu-python/setup.py", line 88, in generate_config_pxi
raise Exception("could not find int128.h in Kudu include dirs")
Exception: could not find int128.h in Kudu include dirs
----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
查看所有可用版本:
>>> pip3 install kudu-python==99.99
Defaulting to user installation because normal site-packages is not writeable
Looking in indexes: https://pypi.tuna.tsinghua.edu.cn/simple
ERROR: Could not find a version that satisfies the requirement kudu-python==99.99 (from versions: 0.1.0, 0.1.1, 0.2.0, 0.3.0, 1.1.0, 1.2.0, 1.7.0, 1.7.1, 1.8.0, 1.9.0, 1.10.0, 1.10.1, 1.11.0, 1.11.1)
ERROR: No matching distribution found for kudu-python==99.99
通过尝试发现kudu 1.4版需安装1.2.0版kudu-python
参考:https://community.cloudera.com/t5/Support-Questions/can-not-install-kudu-python/td-p/67496
连接kudu:
>>> import kudu
>>> client = kudu.connect(host='xx.xx.xx.xx', port=7051)
---------------------------------------------------------------------------
KuduBadStatus Traceback (most recent call last)
<ipython-input-7-319aa08e55f5> in <module>
----> 1 client = kudu.connect(host='xx.xx.xx.xx', port=7051)
~/.local/lib/python3.6/site-packages/kudu/__init__.py in connect(host, port, admin_timeout_ms, rpc_timeout_ms)
90
91 return Client(addresses, admin_timeout_ms=admin_timeout_ms,
---> 92 rpc_timeout_ms=rpc_timeout_ms)
93
94
~/.local/lib/python3.6/site-packages/kudu/client.pyx in kudu.client.Client.__cinit__()
~/.local/lib/python3.6/site-packages/kudu/errors.pyx in kudu.errors.check_status()
KuduBadStatus: b'Timed out: Could not connect to the cluster: ConnectToClusterRpc(addrs: xx.xx.xx.xx:7051, num_attempts: 242) passed its deadline: Network error: Client connection negotiation failed: client connection to xx.xx.xx.xx:7051: connect: Connection refused (error 111)'
连接超时
貌似是因为需要对kudu服务端进行一些修改
参考:https://www.jianshu.com/p/dc27bcb3125d
参考:Apache官方安装文档https://kudu.apache.org/docs/installation.html
参考:https://www.libinx.com/2018/how-to-deploy-kudu-and-use-kudu-python-client/
下一步尝试模拟安装kudu并修改相关参数,再使用kudu-python进行连接,目前就先这样吧。
开放原子开发者工作坊旨在鼓励更多人参与开源活动,与志同道合的开发者们相互交流开发经验、分享开发心得、获取前沿技术趋势。工作坊有多种形式的开发者活动,如meetup、训练营等,主打技术交流,干货满满,真诚地邀请各位开发者共同参与!
更多推荐
所有评论(0)