java.lang.NoSuchMethodError when running in flume1.8.0-cdh6.0.1

[xander-wang]

Trying to implement [MySQL->HBase, Kafka] in CDH6.0.1
Flume config file:

# For mattress streaming data processing:

agent.sources = sqlSource
agent.sinks = kafkaSink hbaseSink
agent.channels = memChannel1 memChannel2

agent.sources.sqlSource.channels = memChannel1 memChannel2
agent.sources.sqlSource.type = org.keedio.flume.source.SQLSource
agent.sources.sqlSource.hibernate.connection.url = jdbc:mysql://192.168.85.208:3306/WebServer?useSSL=false
agent.sources.sqlSource.hibernate.connection.autocommit = true
agent.sources.sqlSource.hibernate.connection.user = root
agent.sources.sqlSource.run.query.delay=10000
agent.sources.sqlSource.hibernate.connection.password = 123456
agent.sources.sqlSource.table = mattress
agent.sources.sqlSource.status.file.path = /var/log/flume-ng
agent.sources.sqlSource.status.file.name = sqlSource.status
agent.sources.sqlSource.hibernate.connection.driver_class = com.mysql.jdbc.Driver
agent.sources.sqlSource.hibernate.dialect = org.hibernate.dialect.MySQLDialect
agent.sources.sqlSource.hibernate.connection.autocommit = true
agent.sources.sqlSource.hibernate.connection.provider_class = org.hibernate.connection.C3P0ConnectionProvider
agent.sources.sqlSource.hibernate.c3p0.min_size=1
agent.sources.sqlSource.hibernate.c3p0.max_size=10

agent.channels.memChannel1.type = memory
agent.channels.memChannel1.capacity = 10000
agent.channels.memChannel1.transactionCapacity = 1000
agent.channels.memChannel2.type = memory
agent.channels.memChannel2.capacity = 10000
agent.channels.memChannel2.transactionCapacity = 1000

agent.sinks.hbaseSink.channel = memChannel1
agent.sinks.hbaseSink.type = hbase
agent.sinks.hbaseSink.table = mattress
agent.sinks.hbaseSink.columnFamily = mattress
agent.sinks.hbaseSink.serializer = org.apache.flume.sink.hbase.SimpleAsyncHbaseEventSerializer
agent.sinks.kafkaSink.type = org.apache.flume.sink.kafka.KafkaSink
agent.sinks.kafkaSink.kafka.bootstrap.servers = cdh-worker0:9092,cdh-worker1:9092,cdh-worker2:9092,cdh-worker3:9092,cdh-worker4:9092,cdh-worker5:9092,cdh-worker6:9092,cdh-worker7:9092
agent.sinks.kafkaSink.kafka.topic = mattress
agent.sinks.kafkaSink.channel = memChannel2

Got following once init the flume service:

java.lang.NoSuchMethodError: org.apache.flume.Context.getSubProperties(Ljava/lang/String;)Lcom/google/common/collect/ImmutableMap;
	at org.keedio.flume.source.HibernateHelper.<init>(HibernateHelper.java:49)
	at org.keedio.flume.source.SQLSource.configure(SQLSource.java:85)
	at org.apache.flume.conf.Configurables.configure(Configurables.java:41)
	at org.apache.flume.node.AbstractConfigurationProvider.loadSources(AbstractConfigurationProvider.java:326)
	at org.apache.flume.node.AbstractConfigurationProvider.getConfiguration(AbstractConfigurationProvider.java:101)
	at org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:145)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

I tried to recompile the source code or use other 'sql-source' releases, all in vain.
It seems to be a DB connection problem after I checked the specific code, and I'm not sure about whether the method 'org.apache.flume.Context.getSubProperties' exists or not in flume1.8.
Or maybe there's error in the config file, which I'm not sure as well...
Just a beginner, help is appreciated!

[qintian95]

@xander-wang Try putting all the packages into the classpath of Flume

[BeerBee]

package your jar with flume-1.8.0-cdh6.0.1，not flume-1.8.0，with maven:


org.apache.flume
flume-ng-core
1.8.0-cdh6.0.1
provided

[BeerBee]

in flume-1.8.0-cdh6.0.1, org.apache.flume.Context.getSubProperties return java.util.Map, not com.google.common.collect.ImmutableMap.

public Map<String, String> getSubProperties(String prefix) {
Preconditions.checkArgument(prefix.endsWith("."),
"The given prefix does not end with a period (" + prefix + ")");
Map<String, String> result = Maps.newHashMap();
synchronized (parameters) {
for (String key : parameters.keySet()) {
if (key.startsWith(prefix)) {
String name = key.substring(prefix.length());
result.put(name, parameters.get(key));
}
}
}
return ImmutableMap.copyOf(result);
}