Can not stream the newly inserted data

[FFYYang]

Hi,
I'm trying to stream data from mysql table into logger.
But I can't stream the newly inserted data in mysql table.

I'm using flume-release-1.8.0, flume-ng-sql-source-1.5.0

My flume.conf looks like this,
agent.sources = sql-source
agent.channels = ch1
agent.sinks = k1

agent.sources = sql-source
agent.channels = ch1
agent.sinks = k1

agent.sources.sql-source.type = org.keedio.flume.source.SQLSource

agent.sources.sql-source.hibernate.connection.url = jdbc:mysql://localhost:3306/test1
agent.sources.sql-source.hibernate.connection.user = root
agent.sources.sql-source.hibernate.connection.password = 123456
agent.sources.sqlSource.hibernate.connection.autocommit = true
agent.sources.sql-source.table = stu

agent.sources.sql-source.run.query.delay = 2000
agent.sources.sql-source.status.file.path = /home/yasinqiu/flume/mysql-test
agent.sources.sql-source.status.file.name = sql-source.status

agent.sources.sql-source.custom.query = SELECT * FROM stu WHERE id > $@$

agent.sources.sql-source.hibernate.dialect = org.hibernate.dialect.MySQL5Dialect
agent.sources.sql-source.hibernate.connection.driver_class = com.mysql.jdbc.Driver

agent.sinks.k1.type = logger

agent.channels.ch1.type = memory
agent.channels.ch1.capacity = 1000
agent.channels.ch1.transactionCapacity = 100

agent.sources.sql-source.channels = ch1
agent.sinks.k1.channel = ch1

=================================================
when i insert a new row in mysql table, i cannot see the new inserted data from logger.

the console's information is shown below:
Info: Sourcing environment configuration script /home/yasinqiu/flume/apache-flume-1.8.0-bin/conf/flume-env.sh
Info: Including Hive libraries found via () for Hive access

• exec /home/yasinqiu/java/jdk1.8.0_171/bin/java -Xmx20m -Dflume.root.logger=INFO,console -cp '/home/yasinqiu/flume/apache-flume-1.8.0-bin/conf:/home/yasinqiu/flume/apache-flume-1.8.0-bin/lib/:/home/yasinqiu/flume/apache-flume-1.8.0-bin/plugins.d/sql-source/lib/:/home/yasinqiu/flume/apache-flume-1.8.0-bin/plugins.d/sql-source/libext/:/lib/' -Djava.library.path= org.apache.flume.node.Application -n agent -f conf/mysql.conf
  2018-07-08 15:32:30,370 (lifecycleSupervisor-1-0) [INFO - org.apache.flume.node.PollingPropertiesFileConfigurationProvider.start(PollingPropertiesFileConfigurationProvider.java:62)] Configuration provider starting
  2018-07-08 15:32:30,373 (conf-file-poller-0) [INFO - org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:134)] Reloading configuration file:conf/mysql.conf
  2018-07-08 15:32:30,377 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:930)] Added sinks: k1 Agent: agent
  2018-07-08 15:32:30,377 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:1016)] Processing:k1
  2018-07-08 15:32:30,377 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:1016)] Processing:k1
  2018-07-08 15:32:30,391 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration.validateConfiguration(FlumeConfiguration.java:140)] Post-validation flume configuration contains configuration for agents: [agent]
  2018-07-08 15:32:30,391 (conf-file-poller-0) [INFO - org.apache.flume.node.AbstractConfigurationProvider.loadChannels(AbstractConfigurationProvider.java:147)] Creating channels
  2018-07-08 15:32:30,395 (conf-file-poller-0) [INFO - org.apache.flume.channel.DefaultChannelFactory.create(DefaultChannelFactory.java:42)] Creating instance of channel ch1 type memory
  2018-07-08 15:32:30,398 (conf-file-poller-0) [INFO - org.apache.flume.node.AbstractConfigurationProvider.loadChannels(AbstractConfigurationProvider.java:201)] Created channel ch1
  2018-07-08 15:32:30,398 (conf-file-poller-0) [INFO - org.apache.flume.source.DefaultSourceFactory.create(DefaultSourceFactory.java:41)] Creating instance of source sql-source, type org.keedio.flume.source.SQLSource
  2018-07-08 15:32:30,400 (conf-file-poller-0) [INFO - org.keedio.flume.source.SQLSource.configure(SQLSource.java:72)] Reading and processing configuration values for source sql-source
  2018-07-08 15:32:30,538 (conf-file-poller-0) [INFO - org.hibernate.annotations.common.reflection.java.JavaReflectionManager.(JavaReflectionManager.java:66)] HCANN000001: Hibernate Commons Annotations {4.0.5.Final}
  2018-07-08 15:32:30,542 (conf-file-poller-0) [INFO - org.hibernate.Version.logVersion(Version.java:54)] HHH000412: Hibernate Core {4.3.10.Final}
  2018-07-08 15:32:30,544 (conf-file-poller-0) [INFO - org.hibernate.cfg.Environment.(Environment.java:239)] HHH000206: hibernate.properties not found
  2018-07-08 15:32:30,545 (conf-file-poller-0) [INFO - org.hibernate.cfg.Environment.buildBytecodeProvider(Environment.java:346)] HHH000021: Bytecode provider name : javassist
  2018-07-08 15:32:30,564 (conf-file-poller-0) [INFO - org.keedio.flume.source.HibernateHelper.establishSession(HibernateHelper.java:63)] Opening hibernate session
  2018-07-08 15:32:30,646 (conf-file-poller-0) [WARN - org.hibernate.engine.jdbc.connections.internal.DriverManagerConnectionProviderImpl.configure(DriverManagerConnectionProviderImpl.java:93)] HHH000402: Using Hibernate built-in connection pool (not for production use!)
  Loading class com.mysql.jdbc.Driver'. This is deprecated. The new driver class is com.mysql.cj.jdbc.Driver'. The driver is automatically registered via the SPI and manual loading of the driver class is generally unnecessary.
  2018-07-08 15:32:30,652 (conf-file-poller-0) [INFO - org.hibernate.engine.jdbc.connections.internal.DriverManagerConnectionProviderImpl.buildCreator(DriverManagerConnectionProviderImpl.java:166)] HHH000401: using driver [com.mysql.jdbc.Driver] at URL [jdbc:mysql://localhost:3306/test1]
  2018-07-08 15:32:30,653 (conf-file-poller-0) [INFO - org.hibernate.engine.jdbc.connections.internal.DriverManagerConnectionProviderImpl.buildCreator(DriverManagerConnectionProviderImpl.java:175)] HHH000046: Connection properties: {user=root, password=****}
  2018-07-08 15:32:30,653 (conf-file-poller-0) [INFO - org.hibernate.engine.jdbc.connections.internal.DriverManagerConnectionProviderImpl.buildCreator(DriverManagerConnectionProviderImpl.java:180)] HHH000006: Autocommit mode: false
  2018-07-08 15:32:30,654 (conf-file-poller-0) [INFO - org.hibernate.engine.jdbc.connections.internal.DriverManagerConnectionProviderImpl.configure(DriverManagerConnectionProviderImpl.java:102)] HHH000115: Hibernate connection pool size: 20 (min=1)
  2018-07-08 15:32:30,948 (conf-file-poller-0) [INFO - org.hibernate.dialect.Dialect.(Dialect.java:145)] HHH000400: Using dialect: org.hibernate.dialect.MySQL5Dialect
  2018-07-08 15:32:30,997 (conf-file-poller-0) [INFO - org.hibernate.engine.transaction.internal.TransactionFactoryInitiator.initiateService(TransactionFactoryInitiator.java:62)] HHH000399: Using default transaction strategy (direct JDBC transactions)
  2018-07-08 15:32:31,000 (conf-file-poller-0) [INFO - org.hibernate.hql.internal.ast.ASTQueryTranslatorFactory.(ASTQueryTranslatorFactory.java:47)] HHH000397: Using ASTQueryTranslatorFactory
  2018-07-08 15:32:31,089 (conf-file-poller-0) [INFO - org.apache.flume.sink.DefaultSinkFactory.create(DefaultSinkFactory.java:42)] Creating instance of sink: k1, type: logger
  2018-07-08 15:32:31,090 (conf-file-poller-0) [INFO - org.apache.flume.node.AbstractConfigurationProvider.getConfiguration(AbstractConfigurationProvider.java:116)] Channel ch1 connected to [sql-source, k1]
  2018-07-08 15:32:31,093 (conf-file-poller-0) [INFO - org.apache.flume.node.Application.startAllComponents(Application.java:137)] Starting new configuration:{ sourceRunners:{sql-source=PollableSourceRunner: { source:org.keedio.flume.source.SQLSource{name:sql-source,state:IDLE} counterGroup:{ name:null counters:{} } }} sinkRunners:{k1=SinkRunner: { policy:org.apache.flume.sink.DefaultSinkProcessor@77e7c26b counterGroup:{ name:null counters:{} } }} channels:{ch1=org.apache.flume.channel.MemoryChannel{name: ch1}} }
  2018-07-08 15:32:31,094 (conf-file-poller-0) [INFO - org.apache.flume.node.Application.startAllComponents(Application.java:144)] Starting Channel ch1
  2018-07-08 15:32:31,094 (conf-file-poller-0) [INFO - org.apache.flume.node.Application.startAllComponents(Application.java:159)] Waiting for channel: ch1 to start. Sleeping for 500 ms
  2018-07-08 15:32:31,125 (lifecycleSupervisor-1-0) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.register(MonitoredCounterGroup.java:119)] Monitored counter group for type: CHANNEL, name: ch1: Successfully registered new MBean.
  2018-07-08 15:32:31,125 (lifecycleSupervisor-1-0) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.start(MonitoredCounterGroup.java:95)] Component type: CHANNEL, name: ch1 started
  2018-07-08 15:32:31,595 (conf-file-poller-0) [INFO - org.apache.flume.node.Application.startAllComponents(Application.java:171)] Starting Sink k1
  2018-07-08 15:32:31,596 (conf-file-poller-0) [INFO - org.apache.flume.node.Application.startAllComponents(Application.java:182)] Starting Source sql-source
  2018-07-08 15:32:31,596 (lifecycleSupervisor-1-4) [INFO - org.keedio.flume.source.SQLSource.start(SQLSource.java:129)] Starting sql source sql-source ...
  2018-07-08 15:32:31,597 (lifecycleSupervisor-1-4) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.register(MonitoredCounterGroup.java:119)] Monitored counter group for type: SOURCE, name: SOURCESQL.sql-source: Successfully registered new MBean.
  2018-07-08 15:32:31,597 (lifecycleSupervisor-1-4) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.start(MonitoredCounterGroup.java:95)] Component type: SOURCE, name: SOURCESQL.sql-source started

[FFYYang]

I wrongly spelled the source name.......
agent.sources.sqlSource.hibernate.connection.autocommit = true
the correct should be :
agent.sources.sql-source.hibernate.connection.autocommit = true

[lazaromedina]

Hi YasinQiu,
i cannot see error on you log, but in your config you are using custom query property. Please can you try in your environment flume-sql-source-1.5.2-SNAPSHOT or wait until next release with a fix for a bug-#57 that may be the problem your report.

best