GRB 051103 is considered to be a candidate soft gamma repeater (SGR) extragalactic giant magnetar flare by virtue of its proximity on the sky to M81/M82, as well as its time history, localization, and energy spectrum. We have derived a refined interplanetary network localization for this burst which reduces the size of the error box by over a factor of two. We examine its time history for evidence of a periodic component, which would be one signature of an SGR giant flare, and conclude that this component is neither detected nor detectable under reasonable assumptions. We analyze the time-resolved energy spectra of this event with improved time- and energy resolution, and conclude that although the spectrum is very hard, its temporal evolution at late times cannot be determined, which further complicates the giant flare association. We also present new optical observations reaching limiting magnitudes of R > 24.5, about 4 magnitudes deeper than previously reported. In tandem with serendipitous observations of M81 taken immediately before and one month after the burst, these place strong constraints on any rapidly variable sources in the region of the refined error ellipse proximate to M81. We do not find any convincing afterglow candidates from either background galaxies or sources in M81, although within the refined error region we do locate two UV bright star forming regions which may host SGRs. A supernova remnant (SNR) within the error ellipse could provide further support for an SGR giant flare association, but we were unable to identify any SNR within the error ellipse. These data still do not allow strong constraints on the nature of the GRB 051103 progenitor, and suggest that candidate extragalactic SGR giant flares will be difficult, although not impossible, to confirm.