Problem(s) Encountered:

During the night, the disperser showed problems every time when we changed the disperser wheel from Blue (#1) to Open (#2).  We encountered this problem about 8 times until 2AM and we could solve the problems following the troubleshooting in manual.  However, after 2pm, turning off the IE and restarting Prospero did not help anymore. We turned down IC and it seemed to solve the problem. However, again, the disp shows the same problem.  We tried same procedure again, again, and again,

and finally gave up. I''m not sure we can observe tomorrow.  

Solutions:

Exchanged email with Eric and logged in remotely this morning.  Found that the disperser wheel microlynx controller was not responding to remote serial port commands.

Quit the OSMOS IE program, and verified that the Comtrol serial port server was releasing all the ports normally.  Then restarted the IE program, verified that it connected to the Comtrol normally, and confirmed at the disperser microlynx was still not communicating.

Used low-level IE commands to power cycle just the disperser microlynx

controller and reset communications.  This restored serial communication with the disperser wheel controller.

I ran a few diagnostics and had no problems, so I attempted to repeat the reported problems by running a Prospero script that did a repeated series of 200 moves between disperser position 1 and 2 (the positions where the observers reported problems), with a pause between.  All moves completed without faults.

On repeating this sequence, I got an out-of-position fault indication after about 30 moves.  The disperser wheel correctly reset on a "disp reset" command from Prospero, and after running some quick checks, was able to run another 200 moves as before without problems.

I did not get any repetition of the communication time out fault reported by the observers.

After consultation with Eric, I left OSMOS in a working state, and instructed the observers to proceed tonight.  I send the lead observer an email asking that if they have problems before 9pm MST/11pm EST, to email me immediately and not take any corrective actions until I can login and see what state the disperser wheel is in.  If the problems occur after I go to bed, I gave the observers some special instructions for recovery procedures that may give us useful information.

On Monday once Eric is back on the mountain, we will run some more tests, and may decide to swap the disperser controller to one of the spare controllers.  These tests require a person to make a visual inspection of the fault LEDs on the microlynx unit, as if we again get into a state where the microlynx drops serial communications, we cannot use remote commands to read the error buffer.