Out of my experience about SEO, I feel page speed does not affect much when you are trying to rank of Search Engine. In overall you may have impacted by visitor leaving the page early, if page does not get loaded properly. Here reduction in visitors may affect your SE positions.
