Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

I successfully created a storm cluster,but don't find supervisor process #47

Open
hackereye opened this issue Sep 23, 2013 · 1 comment
Open

I successfully created a storm cluster,but don't find supervisor process #47

hackereye opened this issue Sep 23, 2013 · 1 comment

Comments

@hackereye
Copy link

yarn log
13/09/23 16:28:58 INFO yarn.MasterServer: Starting the AM!!!!
13/09/23 16:28:59 INFO service.AbstractService: Service:org.apache.hadoop.yarn.client.AMRMClientImpl is inited.
13/09/23 16:28:59 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
13/09/23 16:29:00 INFO service.AbstractService: Service:org.apache.hadoop.yarn.client.AMRMClientImpl is started.
13/09/23 16:29:00 INFO yarn.StormMasterServerHandler: Storm master host:127.0.1.1
13/09/23 16:29:00 INFO yarn.MasterServer: launch nimbus
13/09/23 16:29:00 INFO yarn.StormMasterServerHandler: starting nimbus...
13/09/23 16:29:00 INFO yarn.MasterServer: launch ui
13/09/23 16:29:00 INFO yarn.StormMasterServerHandler: starting UI...
13/09/23 16:29:00 INFO yarn.MasterServer: launch 1 supervisors
13/09/23 16:29:00 INFO yarn.StormMasterServerHandler: adding 1 supervisors...
13/09/23 16:29:00 INFO yarn.StormAMRMClient: Added 1 supervisors, but not requesting containers now.
13/09/23 16:29:00 INFO yarn.StormMasterServerHandler: Running: java -server -Dstorm.home=./storm/storm-0.9.0-wip19/ -Djava.library.path=/usr/local/lib:/opt/local/lib:/usr/lib -Dstorm.conf.file=storm.yaml -cp conf:./storm/storm-0.9.0-wip19/:./storm/storm-0.9.0-wip19/:./storm/storm-0.9.0-wip19/lib/ -Xmx1024m -Dlogback.configurationFile=./storm/storm-0.9.0-wip19/logback/cluster.xml -Dlogfile.name=nimbus.log backtype.storm.daemon.nimbus
13/09/23 16:29:00 INFO yarn.StormMasterServerHandler: Running: java -server -Dstorm.home=./storm/storm-0.9.0-wip19/ -Djava.library.path=/usr/local/lib:/opt/local/lib:/usr/lib -Dstorm.conf.file=storm.yaml -cp conf:./storm/storm-0.9.0-wip19/:./storm/storm-0.9.0-wip19/:./storm/storm-0.9.0-wip19/lib/ -Xmx768m -Dlogback.configurationFile=./storm/storm-0.9.0-wip19/logback/cluster.xml -Dlogfile.name=ui.log backtype.storm.ui.core
13/09/23 16:29:00 INFO yarn.MasterServer: Got a registration response minimumCapability {, memory: 1024, }, maximumCapability {, memory: 8192, },
13/09/23 16:29:00 INFO yarn.MasterServer: Max Capability memory: 8192
13/09/23 16:29:00 INFO yarn.StormAMRMClient: Max Capability is now memory: 8192
13/09/23 16:29:00 INFO yarn.MasterServer: Starting HB thread
13/09/23 16:29:00 INFO yarn.MasterServer: Starting launcher
13/09/23 16:29:00 INFO yarn.StormAMRMClient: Creating new ContainerRequest with memory: 8192 and 1
13/09/23 16:29:00 INFO yarn.StormAMRMClient: Requested container ask: Capability[memory: 8192]Priority[0]ContainerCount[1]
13/09/23 16:29:00 INFO yarn.MasterServer: Starting Master Thrift Server

storm log
2013-09-23 16:30:12 o.a.z.ClientCnxn [INFO] Opening socket connection to server localhost/127.0.0.1:60000
2013-09-23 16:30:12 o.a.z.ClientCnxn [WARN] Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[na:1.7.0_40]
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:735) ~[na:1.7.0_40]
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1119) ~[zookeeper-3.3.3.jar:3.3.3-1073969]
2013-09-23 16:30:14 o.a.z.ClientCnxn [INFO] Opening socket connection to server localhost/127.0.0.1:60000
2013-09-23 16:30:14 o.a.z.ClientCnxn [WARN] Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[na:1.7.0_40]
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:735) ~[na:1.7.0_40]
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1119) ~[zookeeper-3.3.3.jar:3.3.3-1073969]

zookeeper log

2013-09-23 16:19:56,405 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn$Factory@251] - Accepted socket connection from /127.0.0.1:58588
2013-09-23 16:19:56,406 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@777] - Client attempting to establish new session at /127.0.0.1:58588
2013-09-23 16:19:56,420 - INFO [SyncThread:0:NIOServerCnxn@1580] - Established session 0x14149e0d3560001 with negotiated timeout 20000 for client /127.0.0.1:58588
2013-09-23 16:20:49,758 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn$Factory@251] - Accepted socket connection from /127.0.0.1:58590
2013-09-23 16:20:49,761 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@777] - Client attempting to establish new session at /127.0.0.1:58590
2013-09-23 16:20:49,784 - INFO [SyncThread:0:NIOServerCnxn@1580] - Established session 0x14149e0d3560002 with negotiated timeout 20000 for client /127.0.0.1:58590
2013-09-23 16:20:49,811 - INFO [ProcessThread:-1:PrepRequestProcessor@387] - Processed session termination for sessionid: 0x14149e0d3560002
2013-09-23 16:20:49,817 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1435] - Closed socket connection for client /127.0.0.1:58590 which had sessionid 0x14149e0d3560002
2013-09-23 16:20:49,825 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn$Factory@251] - Accepted socket connection from /127.0.0.1:58591
2013-09-23 16:20:49,829 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@777] - Client attempting to establish new session at /127.0.0.1:58591
2013-09-23 16:20:49,896 - INFO [SyncThread:0:NIOServerCnxn@1580] - Established session 0x14149e0d3560003 with negotiated timeout 20000 for client /127.0.0.1:58591
2013-09-23 16:28:36,046 - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@634] - EndOfStreamException: Unable to read additional data from client sessionid 0x14149e0d3560003, likely client has closed socket
2013-09-23 16:28:36,048 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1435] - Closed socket connection for client /127.0.0.1:58591 which had sessionid 0x14149e0d3560003
2013-09-23 16:28:39,379 - INFO [ProcessThread:-1:PrepRequestProcessor@387] - Processed session termination for sessionid: 0x14149e0d3560001
2013-09-23 16:28:39,399 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1435] - Closed socket connection for client /127.0.0.1:58588 which had sessionid 0x14149e0d3560001
2013-09-23 16:28:56,000 - INFO [SessionTracker:ZooKeeperServer@314] - Expiring session 0x14149e0d3560003, timeout of 20000ms exceeded
2013-09-23 16:28:56,001 - INFO [ProcessThread:-1:PrepRequestProcessor@387] - Processed session termination for sessionid: 0x14149e0d3560003
@anfeng
Copy link
Contributor

anfeng commented Sep 23, 2013

There are two ways to access supervisor logs:

  • During Storm cluster execution Š Find supervisor log URI from stdout, and
    access via HTTP client

    * Please look for a line of the following format in stout:

    * 13/09/19 01:48:23 INFO yarn.StormAMRMClient: Supervisor log:
    http://Š/supervisor.log

  • After storm cluster shutdown Š via "yarn logs".
    Please note supervisor logs will become accessible via "yarn logs" only
    after the application is terminated.
    Here is the basic sequence of your ops:
  • storm-yarn launch Š launch Storm cluster
  • storm-yarn getStormConfig Š retrieve storm.yaml
  • storm jar Š launch storm topology
  • storm kill Š kill storm topology
  • yarn application -kill APPID Š terminate storm cluster
  • yarn logs -applicationId APPID Š view all Storm logs (nimbus, UI,
    supervisor)

From: hackereye [email protected]
Reply-To: yahoo/storm-yarn
<reply+i-19897314-af86c70f780c29cbd8d4de98ebece65d9b1c6e38-2708344@reply.git
hub.com>
Date: Monday, September 23, 2013 1:44 AM
To: yahoo/storm-yarn [email protected]
Subject: [storm-yarn] I successfully created a storm cluster,but don't
find supervisor process (#47)

yarn log
13/09/23 16:28:58 INFO yarn.MasterServer: Starting the AM!!!!
13/09/23 16:28:59 INFO service.AbstractService:
Service:org.apache.hadoop.yarn.client.AMRMClientImpl is inited.
13/09/23 16:28:59 WARN util.NativeCodeLoader: Unable to load native-hadoop
library for your platform... using builtin-java classes where applicable
13/09/23 16:29:00 INFO service.AbstractService:
Service:org.apache.hadoop.yarn.client.AMRMClientImpl is started.
13/09/23 16:29:00 INFO yarn.StormMasterServerHandler: Storm master
host:127.0.1.1
13/09/23 16:29:00 INFO yarn.MasterServer: launch nimbus
13/09/23 16:29:00 INFO yarn.StormMasterServerHandler: starting nimbus...
13/09/23 16:29:00 INFO yarn.MasterServer: launch ui
13/09/23 16:29:00 INFO yarn.StormMasterServerHandler: starting UI...
13/09/23 16:29:00 INFO yarn.MasterServer: launch 1 supervisors
13/09/23 16:29:00 INFO yarn.StormMasterServerHandler: adding 1
supervisors...
13/09/23 16:29:00 INFO yarn.StormAMRMClient: Added 1 supervisors, but not
requesting containers now.
13/09/23 16:29:00 INFO yarn.StormMasterServerHandler: Running: java -server
-Dstorm.home=./storm/storm-0.9.0-wip19/
-Djava.library.path=/usr/local/lib:/opt/local/lib:/usr/lib
-Dstorm.conf.file=storm.yaml -cp
conf:./storm/storm-0.9.0-wip19/:./storm/storm-0.9.0-wip19/:./storm/storm-0.
9.0-wip19/lib/ -Xmx1024m
-Dlogback.configurationFile=./storm/storm-0.9.0-wip19/logback/cluster.xml
-Dlogfile.name=nimbus.log backtype.storm.daemon.nimbus
13/09/23 16:29:00 INFO yarn.StormMasterServerHandler: Running: java -server
-Dstorm.home=./storm/storm-0.9.0-wip19/
-Djava.library.path=/usr/local/lib:/opt/local/lib:/usr/lib
-Dstorm.conf.file=storm.yaml -cp
conf:./storm/storm-0.9.0-wip19/:./storm/storm-0.9.0-wip19/:./storm/storm-0.
9.0-wip19/lib/ -Xmx768m
-Dlogback.configurationFile=./storm/storm-0.9.0-wip19/logback/cluster.xml
-Dlogfile.name=ui.log backtype.storm.ui.core
13/09/23 16:29:00 INFO yarn.MasterServer: Got a registration response
minimumCapability {, memory: 1024, }, maximumCapability {, memory: 8192, },
13/09/23 16:29:00 INFO yarn.MasterServer: Max Capability memory: 8192
13/09/23 16:29:00 INFO yarn.StormAMRMClient: Max Capability is now memory:
8192
13/09/23 16:29:00 INFO yarn.MasterServer: Starting HB thread
13/09/23 16:29:00 INFO yarn.MasterServer: Starting launcher
13/09/23 16:29:00 INFO yarn.StormAMRMClient: Creating new ContainerRequest
with memory: 8192 and 1
13/09/23 16:29:00 INFO yarn.StormAMRMClient: Requested container ask:
Capability[memory: 8192]Priority[0]ContainerCount[1]
13/09/23 16:29:00 INFO yarn.MasterServer: Starting Master Thrift Server

storm log
2013-09-23 16:30:12 o.a.z.ClientCnxn [INFO] Opening socket connection to
server localhost/127.0.0.1:60000
2013-09-23 16:30:12 o.a.z.ClientCnxn [WARN] Session 0x0 for server null,
unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[na:1.7.0_40]
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:735)
~[na:1.7.0_40]
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1119)
~[zookeeper-3.3.3.jar:3.3.3-1073969]
2013-09-23 16:30:14 o.a.z.ClientCnxn [INFO] Opening socket connection to
server localhost/127.0.0.1:60000
2013-09-23 16:30:14 o.a.z.ClientCnxn [WARN] Session 0x0 for server null,
unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[na:1.7.0_40]
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:735)
~[na:1.7.0_40]
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1119)
~[zookeeper-3.3.3.jar:3.3.3-1073969]

zookeeper log

2013-09-23 16:19:56,405 - INFO
[NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn$Factory@251] -
Accepted socket connection from /127.0.0.1:58588
2013-09-23 16:19:56,406 - INFO
[NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@777] - Client
attempting to establish new session at /127.0.0.1:58588
2013-09-23 16:19:56,420 - INFO [SyncThread:0:NIOServerCnxn@1580] -
Established session 0x14149e0d3560001 with negotiated timeout 20000 for
client /127.0.0.1:58588
2013-09-23 16:20:49,758 - INFO
[NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn$Factory@251] -
Accepted socket connection from /127.0.0.1:58590
2013-09-23 16:20:49,761 - INFO
[NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@777] - Client
attempting to establish new session at /127.0.0.1:58590
2013-09-23 16:20:49,784 - INFO [SyncThread:0:NIOServerCnxn@1580] -
Established session 0x14149e0d3560002 with negotiated timeout 20000 for
client /127.0.0.1:58590
2013-09-23 16:20:49,811 - INFO [ProcessThreadPrepRequestProcessor@387] -
Processed session termination for sessionid: 0x14149e0d3560002
2013-09-23 16:20:49,817 - INFO
[NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1435] - Closed
socket connection for client /127.0.0.1:58590 which had sessionid
0x14149e0d3560002
2013-09-23 16:20:49,825 - INFO
[NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn$Factory@251] -
Accepted socket connection from /127.0.0.1:58591
2013-09-23 16:20:49,829 - INFO
[NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@777] - Client
attempting to establish new session at /127.0.0.1:58591
2013-09-23 16:20:49,896 - INFO [SyncThread:0:NIOServerCnxn@1580] -
Established session 0x14149e0d3560003 with negotiated timeout 20000 for
client /127.0.0.1:58591
2013-09-23 16:28:36,046 - WARN
[NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@634] -
EndOfStreamException: Unable to read additional data from client sessionid
0x14149e0d3560003, likely client has closed socket
2013-09-23 16:28:36,048 - INFO
[NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1435] - Closed
socket connection for client /127.0.0.1:58591 which had sessionid
0x14149e0d3560003
2013-09-23 16:28:39,379 - INFO [ProcessThreadPrepRequestProcessor@387] -
Processed session termination for sessionid: 0x14149e0d3560001
2013-09-23 16:28:39,399 - INFO
[NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1435] - Closed
socket connection for client /127.0.0.1:58588 which had sessionid
0x14149e0d3560001
2013-09-23 16:28:56,000 - INFO [SessionTracker:ZooKeeperServer@314] -
Expiring session 0x14149e0d3560003, timeout of 20000ms exceeded
2013-09-23 16:28:56,001 - INFO [ProcessThreadPrepRequestProcessor@387] -
Processed session termination for sessionid: 0x14149e0d3560003

‹
Reply to this email directly or view it on GitHub
#47 .

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@anfeng @hackereye