As an experiment, try this and see what happens.
In your kernel config, try this:
Make sure you high resolution timer is enabled
CONFIG_CLKSRC_SH_OSTM=y
CONFIG_HIGH_RES_TIMERS=y
Set you "Timer tick handling" to "Periodic timer ticks"
CONFIG_HZ_PERIODIC=y
Change your jiffies timer for 1024 ticks per second
CONFIG_SHMOBILE_TIMER_HZ=1024
I'm not sure if it will help or not, but it should be an easy experiment.
Another thing I found is that the USB block as 8KB of RAM for FIFOs. The current driver breaks that up and pre-allocates 512 bytes for each PIPE (whether you use them all or not). But, if your camera is really only using 2 PIPEs (BULK-IN, BULK-OUT), then maybe you can hack the USB driver and have it use 2KB per PIPE (the max) and that would streamline your data flow and cut the number of interrupts down and maybe you can pick up some speed that way as well.
If you look in drivers/usb/host/r8a66597.h, you will see:
#define R8A66597_BUF_BSIZE 8
(8 x 64 bytes = 512 bytes)
In theory, just by changing that 8 to a 32 might improve your performance, but I haven't tried it.
You just have to see how many BULK IN/OUT endpoints your camera enumerates....but I would assume just one for each direction.