Yes, that is the layout I was describing. Connecting a single I2C port on the Arduino to both extenders should be fine because the extenders are using standard I2C on that side, which is open drain.
I am not sure what you mean by "breakout board". If you mean the I2C extender, its datasheet mentions that the extender was tested at 100kHz (at 150ft) but does not really elaborate on how or if it can communicate any faster. If 400kHz operation is important to you, let me know and we can look into setting up a quick test to see if it works.
Also, it's not explicitly labeled in your diagram, but be sure that there are common grounds between each I2C extender and the Arduino or sensor it is connected to.