what cause the oil go into the coolant system?

It could caused by the bad radiator. There are two compartments in the radiator, one for cooling the for antifreeze and another for transmission fluid. The the wall breaches, trans fluid and antifreeze mixes. Just like you have oil in the antifreeze tank, I am sure trans oil also has antifreeze mixed in it. It could also causes bad trans shifting.