I don't think the water would be deep enough at the Amable du Fond Rapids sites for swimming.  The only Amable du Fond River site that is good is the one closest to the portage to Manitou.   My wife and I stayed there one night on the August long weekend -- it is a nice open campsite in a hemlock/pine grove with the river flowing by on one side.   We didn't do any swimming as we got there late in the evening and had to get to Biggar Lake the next day but I would expect it would be ok for swimming.  It would be a lot better though if you could get your crew to continue on to Manitou as that is a really nice lake with good campsites.

Thanks for the input!  The kids completely punked out and decided they wanted to camp in our Boler so we ended up doing two nights at Oastler Lake Provincial Park, then taking them and the camper to my parents place in Haliburton, so the hubby and I could do a night of canoe camping.  I ended up with a bad cold on the first day of the trip or we would have gone for longer, as it was, we ended up just going to Little Island Lake.  I had really wanted to go in at an access we hadn't been to yet, but time wise, it made more sense to put in along Highway 60, and with the lakes being so booked up, Little Island was on a very short list of possibilities.

I'm definitely going to check out both of the suggested lakes soon though.  We really got our portaging strategy down on this short trip and feel feel more confident about tackling longer ones and going deeper into the park.