Problem(s) Encountered:

Had trouble with the dome encoders multiple times throughout the night, and eventually disabled the auto dome entirely. Some details:

- Periodically while tracking, the dome encoders would get lost. First the dome would be moving, but the  Dome Azimuth  field on the TCS would not be changing.

Then, if left long enough, the  Dome Encoders Not Responding  error message would appear.

The dome would then continue to rotate until both the Auto Dome switch was turned off AND the yellow button in the control room was disabled.

If we moved the dome manually to some azimuth and reset the encoders, we could turn the auto dome back on and it would work for a time, but eventually it got lost again.

The times the problem occurred and target coordinates are copied below.

- First occurrence was UT 3:45, early in the evening.

This happened while we were pointing near zenith, and we accidentally set the RA/Dec encoders to a value by the horizon.

The dome started moving to account for the new coordinates, but we then reset the RA/Dec encoders to the correct values, and the dome encoders stopped responding.

We reset the dome encoders, and it worked fine for a few hours.

- At UT 7:26:00, right after finally acquiring a target through the clouds and starting an exposure, the encoders stopped responding again.

This time, the auto dome switch on the TCS control panel did not stop the dome moving; only turning it off in the server room made a difference.

We reset the encoders to Az of 180, then turned the auto dome back on.

- At UT 7:40:00, shortly after restarting the dome, it lost the encoders again.

Target Alt/Az=72/170, RA/Dec=12:59:49.4/14:08:19. We left the auto dome off until the exposure completed.

- At UT 8:04:00, roughly 1 minute after restarting the auto dome, the encoders were lost again.

Target Alt/Az=72/188, RA/Dec=12:59:49.4/14:08:19. We decided to leave the auto dome off until the current target was finished.

- At UT 8:50:00 we moved to a new target and turned the auto dome back on. 5 minutes later (8:55:00) it stopped responding again.

We disabled the auto dome again.

After the fifth time the dome encoders stopped responding, we disabled the auto dome and left it off until the end of the night.

Solutions:

Telescope/dome motions were started early this morning, around 0800 local time.  

Dome was found to be at the home position this morning, but the readout on the TCS display showed it was roughly 200 (120 degrees off from the expected 320).  When autodome was turned on, the dome started turning.  Autodome was turned off, stopping the dome.  The dome was then hand set to the home position.  When an attempt was made to “reset dome az encoder” in the xtcs GUI, the GUI gave an error that it could not communicate with the TCS.  xtcs was closed, restarted and reinitialized.  After this, there were no further issues with the dome tracking.  Tests have been performed once again throughout the day without issue.  Besides simply slewing the telescope around, we also let it track, ensuring that the dome caught up with the telescope as it tracked westward.  

Another thought was that there might be issue with the optical transmitter on the encoder (I believe these are IR LED illuminated).  Perhaps with stray light in the dome during the day, there is enough light for it to work properly, but once it is dark, it might fail.  Night was simulated with dark cloth.   Considering we have a fairly bright moon currently, I think I could get it at least ballpark equivalent to the night sky.  Tests performed with the encoder assembly shrouded showed no issues.  

I pulled the encoder device and thoroughly cleaned the transmitters and receivers as well as the toothed wheel.  Lastly, the encoder chipset was replaced.  This had been done less than a month ago, but with chips that have an unknown history.  While I don’t have any reason to believe that these chips were faulty, the current chipset is with all new chips.  Still, it is of my opinion that the chipset was fine.  

Everything is back together again.  As this is typed up, the telescope/dome is in a tracking state (currently in the SE).  It is working as expected.  The only portion of this system that hasn’t been checked is where it goes through the TCS computer (C/T #1, C/T #2, A/D boards).  I’m extremely hesitant to simply start throwing new hardware at the TCS PC as we do not have a working spare currently.  Besides, when the system is run during the day, it works fine.

My only recommendation to the observer will be to close the xtcs GUI and restart/reinitialize if this happens again.  Other than that, enough variables have been added to ensure that if there really was a problem that has been solved, it will remain a mystery what it was.