MDM Trouble Report for 2008 Sep 11

Telescope: 2.4m
Observer(s): Stoll/Yee (OSU)
Instrument: TIFKAM

Problem(s) Encountered:
In Brief:
New data are not written to the specified filename. Instead prospero/caliban tries to write data to 080909test.0027 or 080909test.0028, decides that is not a unique filename and then writes to 080909H2.00t or 080909H2.00u. Visual inspection of these images shows that they are not the same as the image displayed on the TIFKAM monitor. They appear to be images taken the previous night (2008 Sept 09).

Verbose:
Last night we took some images with the names 080909test.0001 through 080909test.0027.

We did a lightning shutdown.
We were briefly on generator power.
The weather cleared and we restarted the system and opened the dome.

We moved the previous night's images to the directory tifkam080909/.

For the first image we took tonight, we forgot to change the filename, so it was saved as 080909test.0028. We did not look at this image at the time. It has since been moved to the directory tifkam080910/. 

We then closed for a few hours due to high humidity.

When we reopened and attempted to take data, we noticed the problem.

We entered a new filename using the 'filename' command. We took an exposure and received this message: 

 *** Exposure Started ***
 STATUS: Reading out... RPPPreMode= 0
 [CB> WARNING: FITS file '/lhome/data/080909test.0027.fits' already exists, writing as '/lhome/data/080909H2.00t.fits' instead]
 STATUS: Finished  1 of  1 exposures.
 DONE: Exposures completed.
 *** Exposure Completed ***

Note that '080909test.0027' is not the filename we entered. We tried a variety of different filenames including '080910test', 'squeegee', and 'trblsht'. The problem is not that we tried to use a filename already in existence. Either we received the above message or its partner:

 *** Exposure Started ***
 STATUS: Reading out... RPPPreMode= 0
 [CB> WARNING: FITS file '/lhome/data/080909test.0028.fits' already exists, writing as '/lhome/data/080909H2.00u.fits' instead]
 STATUS: Finished  1 of  1 exposures.
 DONE: Exposures completed.
 *** Exposure Completed ***

Inspecting the images '080909H2.00*.fits' and their headers shows that they are likely to be cached data taken last night. However, we believe that TIFKAM is indeed taking new exposures because we see new images show up on the TIFKAM monitor.

Interestingly, the IC monitor in the Computer room shows the filename set to the filename that we entered e.g. 'squeegee'. Additionally, the prospero status window shows the filename that we entered and the command 'nextfile' also returns this filename.

Things we tried to resolve the problem:
1. Close and restart the ISIS, MDM TCS, Caliban, and Prospero Windows.
2. Restart the Caliban computer.
3. Resetting the filename with the 'filename' command.
4. Deleting or moving the files '080909test.002*' and '080909H2.00*'. When we do this, it will write to 080909test.0027, 080909test.0028, and then try again to write to 080909test.0027 (with the result that we get the warning and it writes to 080909H2.00t). However, these images are not the new exposures. They are the cached data. Prospero seems to be stuck in a loop of writing to 080909test.0027 and .0028.
5. Restart hiltner.

The directory /lhome/data/Trouble contains this report and also the images in question, what appears on the IC monitor in the computer room, prospero, and +verbose +debug output from caliban. The root filename that we set for these images was 'trblsht'.


------------------------------

Submitted on 2008 Sep 11 [4:10:12]