hdfscli 操作 HDFS 出现 ConnectionError, ConnectionRefusedError 的错误

背景:

最近在学习数据预处理相关的东西, 原来爬虫爬的数据都是存在 MongoDB, 数据越来越多, 现在公司想要将把已经拿到的数据放到 Hadoop 集群上. 因为是新手, 又不能直接上集群, 所以就先在本地用虚拟机搭建了一个 Hadoop 伪分布式集群, 用来练手. 至于创建虚拟机及伪分布式搭建教程, 网上有一堆, 这里就不再赘述, 大家可以自行百度或谷歌.

开发环境:

操作:

此时我虚拟机上的 Hadoop 伪分布式已经搭建完成, 开始通过 hdfs 包的接口操作集群里的数据

注意: pip 安装 hdfs 包, 需要新建 ~/.hdfscli.cfg, 并配置, 详情见官方文档

from hdfs import InsecureClient, Config
client = InsecureClient("http://192.168.2.166:9870")

# 查看连接的地址是否正确
print(Config().get_client())                                                                                                                                                                  
# <InsecureClient(url='http://192.168.2.166:9870')>

# 尝试查看集群上 /test 下的文件
print(client.list("/test"))                                                                                                                                                                   
# ['input', 'output', 'test.txt']

# 读取文件内容, 坑就出现在这里
with client.read("/test/test.txt") as reader:
    print(reader.read()) 

错误详情:

C:\Users\Administrator\.virtualenvs\python_test-8pdC7ThV\Scripts\python.exe D:/python_test/65_hdfs.py
Traceback (most recent call last):
  File "C:\Users\Administrator\.virtualenvs\python_test-8pdC7ThV\lib\site-packages\urllib3\connection.py", line 159, in _new_conn
    (self._dns_host, self.port), self.timeout, **extra_kw)
  File "C:\Users\Administrator\.virtualenvs\python_test-8pdC7ThV\lib\site-packages\urllib3\util\connection.py", line 80, in create_connection
    raise err
  File "C:\Users\Administrator\.virtualenvs\python_test-8pdC7ThV\lib\site-packages\urllib3\util\connection.py", line 70, in create_connection
    sock.connect(sa)
ConnectionRefusedError: [WinError 10061] 由于目标计算机积极拒绝,无法连接。

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\Administrator\.virtualenvs\python_test-8pdC7ThV\lib\site-packages\urllib3\connectionpool.py", line 600, in urlopen
    chunked=chunked)
  File "C:\Users\Administrator\.virtualenvs\python_test-8pdC7ThV\lib\site-packages\urllib3\connectionpool.py", line 354, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "c:\users\administrator\appdata\local\programs\python\python36\Lib\http\client.py", line 1239, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "c:\users\administrator\appdata\local\programs\python\python36\Lib\http\client.py", line 1285, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "c:\users\administrator\appdata\local\programs\python\python36\Lib\http\client.py", line 1234, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "c:\users\administrator\appdata\local\programs\python\python36\Lib\http\client.py", line 1026, in _send_output
    self.send(msg)
  File "c:\users\administrator\appdata\local\programs\python\python36\Lib\http\client.py", line 964, in send
    self.connect()
  File "C:\Users\Administrator\.virtualenvs\python_test-8pdC7ThV\lib\site-packages\urllib3\connection.py", line 181, in connect
    conn = self._new_conn()
  File "C:\Users\Administrator\.virtualenvs\python_test-8pdC7ThV\lib\site-packages\urllib3\connection.py", line 168, in _new_conn
    self, "Failed to establish a new connection: %s" % e)
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x0000016861CE29B0>: Failed to establish a new connection: [WinError 10061] 由于目标计算机积极拒绝,无法连接。

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\Administrator\.virtualenvs\python_test-8pdC7ThV\lib\site-packages\requests\adapters.py", line 449, in send
    timeout=timeout
  File "C:\Users\Administrator\.virtualenvs\python_test-8pdC7ThV\lib\site-packages\urllib3\connectionpool.py", line 638, in urlopen
    _stacktrace=sys.exc_info()[2])
  File "C:\Users\Administrator\.virtualenvs\python_test-8pdC7ThV\lib\site-packages\urllib3\util\retry.py", line 399, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='localhost', port=9864): Max retries exceeded with url: /webhdfs/v1/test/test.txt?op=OPEN&user.name=Administrator&namenoderpcaddress=hadoop:9001&offset=0 (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x0000016861CE29B0>: Failed to establish a new connection: [WinError 10061] 由于目标计算机积极拒绝,无法连接。',))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "D:/python_test/65_hdfs.py", line 18, in <module>
    with client.read("/test/test.txt") as reader:
  File "c:\users\administrator\appdata\local\programs\python\python36\Lib\contextlib.py", line 81, in __enter__
    return next(self.gen)
  File "C:\Users\Administrator\.virtualenvs\python_test-8pdC7ThV\lib\site-packages\hdfs\client.py", line 687, in read
    buffersize=buffer_size,
  File "C:\Users\Administrator\.virtualenvs\python_test-8pdC7ThV\lib\site-packages\hdfs\client.py", line 125, in api_handler
    raise err
  File "C:\Users\Administrator\.virtualenvs\python_test-8pdC7ThV\lib\site-packages\hdfs\client.py", line 107, in api_handler
    **self.kwargs
  File "C:\Users\Administrator\.virtualenvs\python_test-8pdC7ThV\lib\site-packages\hdfs\client.py", line 214, in _request
    **kwargs
  File "C:\Users\Administrator\.virtualenvs\python_test-8pdC7ThV\lib\site-packages\requests\sessions.py", line 533, in request
    resp = self.send(prep, **send_kwargs)
  File "C:\Users\Administrator\.virtualenvs\python_test-8pdC7ThV\lib\site-packages\requests\sessions.py", line 668, in send
    history = [resp for resp in gen] if allow_redirects else []
  File "C:\Users\Administrator\.virtualenvs\python_test-8pdC7ThV\lib\site-packages\requests\sessions.py", line 668, in <listcomp>
    history = [resp for resp in gen] if allow_redirects else []
  File "C:\Users\Administrator\.virtualenvs\python_test-8pdC7ThV\lib\site-packages\requests\sessions.py", line 247, in resolve_redirects
    **adapter_kwargs
  File "C:\Users\Administrator\.virtualenvs\python_test-8pdC7ThV\lib\site-packages\requests\sessions.py", line 646, in send
    r = adapter.send(request, **kwargs)
  File "C:\Users\Administrator\.virtualenvs\python_test-8pdC7ThV\lib\site-packages\requests\adapters.py", line 516, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='localhost', port=9864): Max retries exceeded with url: /webhdfs/v1/test/test.txt?op=OPEN&user.name=Administrator&namenoderpcaddress=hadoop:9001&offset=0 (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x0000016861CE29B0>: Failed to establish a new connection: [WinError 10061] 由于目标计算机积极拒绝,无法连接。',))

Process finished with exit code 1

解析:

为什么会出现这种错误呢? 看到错误详情的最后, 发现请求的是 localhost, 我在 Windows 开发, 而 Windows 又没有参与到集群中, 在 Windows 中请求 localhost 肯定是说不通的. 看看虚拟机的 9864 端口具体是什么

[root@localhost ~]# netstat -tnulp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    
tcp        0      0 0.0.0.0:41510           0.0.0.0:*               LISTEN      60673/java          
tcp        0      0 0.0.0.0:9864            0.0.0.0:*               LISTEN      73421/java          
tcp        0      0 0.0.0.0:8040            0.0.0.0:*               LISTEN      60673/java          
tcp        0      0 192.168.2.166:9001      0.0.0.0:*               LISTEN      64111/java          
tcp        0      0 127.0.0.1:27017         0.0.0.0:*               LISTEN      4988/mongod         
tcp        0      0 0.0.0.0:9866            0.0.0.0:*               LISTEN      73421/java          
tcp        0      0 0.0.0.0:8042            0.0.0.0:*               LISTEN      60673/java 

[root@localhost ~]# ps -aux | grep 73421
root      73421  0.2  2.6 3164768 144660 pts/2  Sl   14:20   0:05 /opt/java/bin/java -Dproc_datanode -Djava.net.preferIPv4Stack=true -Dhadoop.security.logger=ERROR,RFAS -Dyarn.log.dir=/opt/hadoop/logs -Dyarn.log.file=hadoop-root-datanode-localhost.log -Dyarn.home.dir=/opt/hadoop -Dyarn.root.logger=INFO,console -Djava.library.path=/opt/hadoop/lib/native -Dhadoop.log.dir=/opt/hadoop/logs -Dhadoop.log.file=hadoop-root-datanode-localhost.log -Dhadoop.home.dir=/opt/hadoop -Dhadoop.id.str=root -Dhadoop.root.logger=INFO,RFA -Dhadoop.policy.file=hadoop-policy.xml org.apache.hadoop.hdfs.server.datanode.DataNode

发现是集群的 Datanode 节点, 于是乎想到的是可能是 Datanode 的地址没有绑定, 默认被帮到 localhost 在浏览器里输入 http://192.168.2.166:9870/dfshealth.html#tab-datanode 查看 Datanode 信息 Datanode 可以看到地址绑定的都是 localhost, 而 Datanode 的配置是在 hdfs-site.xml 里, 查看Hadoop 官方文档 , 并根据文档添加配置

注意: 根据版本号查看官方文档, 我查看的是 3.2.0 文档

hdfs-site.xml 基础上添加以下代码

<property>
    <name>dfs.datanode.http.address</name>
    <value>192.168.2.166:9864</value>
</property>

再次请求 http://192.168.2.166:9870/dfshealth.html#tab-datanode, 发现地址依旧是 localhost Datanode2 修改虚拟机主机名为 hadoop, 并将 hadoop 也解析到虚拟机 IP 192.168.2.166

# 修改主机名并退出
[root@localhost ~]# hostnamectl set-hostname hadoop
[root@localhost ~]# exit
# 重新登录, 主机名已修改
[root@hadoop ~]#
# 修改 `hosts` 文件
    127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
    ::1         localhost localhost.localdomain localhost6 localhost6.localdomain6

    192.168.2.166  hadoop  # 添加这一行

修改 Windows 的 host 位置在 C:\Windows\System32\drivers\etc, 解析 hadoop 为虚拟机 IP 192.168.2.166

Windows host 再重新启动 Datanode 节点

hdfs --daemon stop datanode
hdfs --daemon start datanode

再次查看 Datanode 节点信息, 发现地址已经改变 Datanode3

再次运行程序, 是希望中结果

with client.read("/test/test.txt") as reader:
    print(reader.read())
# b'Hello HDFS'