How to access the hadoop distributed environment built in the internal network through the external network - Codes Helper - Programming Question Answer

How to access the hadoop distributed environment built in the internal network through the external network

1. In the hadoop environment built, datanode uses private network ip (172.16.1.142172.16.1.148), and only 142nodes activate external network ip

.

2. The data on the HDFS is obtained remotely through Java, and the datanode address returned to the Java program is that the intranet ip,Java cannot be connected. What should I do?

3. The parameter configuration.set ("dfs.client.use.datanode.hostname", "true") is set. The error message is as follows:

2018-08-17 14:22:38,670 INFO [org.apache.hadoop.hdfs.DFSClient] - Exception in createBlockOutputStream
java.io.IOException: Got error, status message , ack with firstBadLink as 172.16.1.148:50010
    at org.apache.hadoop.hdfs.protocol.datatransfer.DataTransferProtoUtil.checkBlockOpStatus(DataTransferProtoUtil.java:140)
    at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.createBlockOutputStream(DFSOutputStream.java:1363)
    at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1266)
    at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:449)
2018-08-17 14:22:38,673 INFO [org.apache.hadoop.hdfs.DFSClient] - Abandoning BP-1760557445-172.16.1.142-1534486349045:blk_1073741829_1005
2018-08-17 14:22:38,701 INFO [org.apache.hadoop.hdfs.DFSClient] - Excluding datanode DatanodeInfoWithStorage[172.16.1.148:50010,DS-dd301dfb-ae4e-4adc-8152-e11bbf0880e0,DISK]-sharp-sharp-sharp

Java hadoop

Apr.11,2021

hadoop connections are special, according to the hostname, and every node must be accessible, so you can't solve it with normal port mapping.

you must use the VPN solution to connect, and if you think the VPN is too heavy and just temporarily test the connection, you can also consider the sshuttle scheme. Take a look at my blog: https://www.jianshu.com/p/edc.

Previous: Mongodb transaction problem

Next: The problem of utf-8' reporting an error I dont know what is wrong

How to access the hadoop distributed environment built in the internal network through the external network
in the hadoop environment built by the company, datanode uses the private network ip,. Only the master node has enabled the external network ip, now the data on the HDFS is remotely obtained through Java, and the datanode address returned by master to t...

Remote java hadoop

Mar.05,2021
Hadoop3.1 startup error report
< H1 > hadoop startup error < H1 > configuration files are all written and then started [root@master logs]-sharp start-dfs.sh Starting namenodes on [master] Last login: Wed Jul 4 16:30:19 CST 2018 on pts 0 bash v3.2+ is required. Sorry. Starting da...

Linux java hadoop

Mar.23,2021
A problem with hive query
there is a partition table containing json files when querying, an error is always reported Failed with exception java.io.IOException:org.apache.hadoop.hdfs.BlockMissingException: Could not obtain block: BP-32616997-172.16.0.195-1533624415828:blk_107...

Java hadoop hive

Oct.29,2021

MySQL Query : SELECT * FROM `codeshelper`.`v9_news` WHERE status=99 AND catid='6' ORDER BY rand() LIMIT 5
MySQL Error : Disk full (/tmp/#sql-temptable-64f5-1b33260-2be69.MAI); waiting for someone to free some space... (errno: 28 "No space left on device")
MySQL Errno : 1021
Message : Disk full (/tmp/#sql-temptable-64f5-1b33260-2be69.MAI); waiting for someone to free some space... (errno: 28 "No space left on device")
Need Help?