Performance scaling: key column not taken to decide input spilit

[poruspurush]

Hi,

Original issue posted here,
https://groups.google.com/forum/#!topic/lingual-user/ixaDrCxT_DI

We are exploring JDBC tap to pull the database from RDBMS. we are finding that basic select statements built by cascading JDBC API is not utilizing indexed columns (i.e. primary columns).

Is there anything we are missing that could influence the JDBC tap to utilize the PK index?

Thanks,
Purush

[fs111]

The problem you are facing has to do with the InputFormat used by cascading jdbc. It seems that the sqoop folks have a more clever implementation called DataDrivenInputFormat. Something like that would be very useful in cascading-jdbc as well.

That being said for your concrete case I would propose you create a sub-class of OracleDBInputFormat that matches your data better: https://github.com/Cascading/cascading-jdbc/blob/wip-2.7/cascading-jdbc-oracle/src/main/java/cascading/jdbc/db/OracleDBInputFormat.java and use that isntead of the one that we ship.

If you can come up with an easy way to roll your changes into a portalbe/generic code, I am more than happy to integrate that into the project.