In any given 24-hour period, the Fermi LAT detects photons with energies of order 100 GeV not associated with any known VHE emitter. The RA/Dec of these photons frequently falls into the field of view of VERITAS the following evening. A single photon in the LAT can potentially mean many detectable photons on the ground due to the superior collection area of the IACT method. During the 2011-2012 observing season, VERITAS implemented a new target of opportunity (ToO) program to trigger VERITAS observations on a small sample of the highest energy LAT photons. Each photon that triggered observation was detected by the LAT within the previous 24-hour period and was not spatially associated with a known VHE emitter. We present here the results of these ToO observations, a discussion of the method used, and our future plans.