MAJOR RA motion failure

Tuesday, January 9, 2024 7:04 AM

Problem(s) Encountered:

About halfway through the night, the telescopebegan to lose guiding, with occasional small shifts in(mainly) RA. The target was reacquired by moving thetelescope using the paddle, mostly E. During about 10minutes, there were several more of the smaller shifts,followed by a larger jump of several degrees.
It was unclear why this happened, but afterreacquiring the target and settling stably, the jumpshappened again, followed by a larger set of jumps(some oscillating). As far as we could tell, the telescope wasn'tslewing. It was unclear why the telescope wasmoving by such large amounts, and so quickly. At this point,the telescope began moving to large (W) hour angle.(Note our target was around 0 HA, +42 deg).
We tried to stop the motion, by using either the GUI stopbutton or by sending a command back to the target position,turning off tracking, but the telescope went to thelimit and tripped the horizon limit.
Since it happened after midnight with no staff available,there was little we could do to recover from thissituation so we homed and closed the dome and shutdown for the night.

Solution:

Ultimately, the issue was that we lost the HA smart motor encoder.  With this having failed, the telescope slewed uncontrolled since the encoder was trying to zero a delta between where it was and where it thought it needed to be.  Unfortunately it’s not smart enough to recognize that it cannot update it’s positioning (when failed) and as such, that delta never changes.  So the motor just runs and runs.  So why did it stop?  I believe it stopped when the mercury relays (the “safety” horizon limit stop) did their job and cut power to the motors.  Interestingly, I found that the emergency stop switch (located in the dome on the north wall) is wired in series with the horizon limit system.  So if that button is pressed, then the GUI indicates the telescope as being in the horizon limit, even if it isn’t.  There is a way to bypass the horizon limit (I’ll keep that secret to myself for now) but it also understandably bypasses the emergency stop, since it is wired in series.  So ultimately, I believe that the mercury switches did their job and when the telescope was far enough over, it killed all power to the telescope.  What was fairly alarming was that those mercury switches, with the telescope in that orientation, did not activate until the HA was in excess of 8 hours west!  It may be good to think about how to limit travel a bit more, although I admit that I haven’t yet thought of any simple way to implement such an idea.  Perhaps a trip wire similar to the protection at the 1.3m re: contact with the pier?In the end, if this problem occurs again during nighttime operations, the user should be able to kill the motors either by flipping the ‘drives’ switch down (off) OR hitting the ALL STOP button, both located on the MDC panel in the computer room.  Alternately, running out to the dome and hitting the emergency stop plunger switch should also stop motion.  At such a point, the observer should consider the telescope dead in the water and NOT try to reenergize the motors.This likely should be added to day-1 training of all observers.