Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Beginners question: Cannot connect to cluster #58

Open
Para7etamol opened this issue Nov 17, 2024 · 0 comments
Open

Beginners question: Cannot connect to cluster #58

Para7etamol opened this issue Nov 17, 2024 · 0 comments

Comments

@Para7etamol
Copy link

Hi there,

I'm running a hadoop cluster (v3.2.1) using https://github.com/big-data-europe/docker-hadoop

I can run a Java program to test the existence of an file on the hdfs of the cluster.

But I cannot do this using Elly.jl:

using Elly

hdfs = HDFSClient("localhost", 9000,  UserGroupInformation())

exists(hdfs, "/")

yields

namenode         | java.lang.NullPointerException
namenode         | 	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.logAuditEvent(FSNamesystem.java:405)
namenode         | 	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.logAuditEvent(FSNamesystem.java:377)
namenode         | 	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.logAuditEvent(FSNamesystem.java:371)
namenode         | 	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getFileInfo(FSNamesystem.java:3197)
namenode         | 	at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getFileInfo(NameNodeRpcServer.java:1173)
namenode         | 	at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getFileInfo(ClientNamenodeProtocolServerSideTranslatorPB.java:979)
namenode         | 	at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
namenode         | 	at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:528)
namenode         | 	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1070)
namenode         | 	at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:999)
namenode         | 	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2917)

I'm using Julia 1.10.4 and Elly 0.5.1

To access hdfs from JAVA I had to add some dependencies (hadoop-common, hadoop-hdfs, hadoop-hdfs-client) and copied core-site.xml and hdfs-site.xml from the container into the resources dir of the java application. Optionally I added lib/native from the containers /opt/hadoop dir to LD_LIBRARY_PATH to prevent the warning
WARN NativeCodeLoader:60 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

I suppose I have to perform at least some of the above "Java"-steps when using Elly.jl ... but which and how?

Please help, I would LOVE to use hadoop from Julia.

Greetings
Para

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant