0) What do those all those sounds mean?
per our sound-effects man, bfl:
a) MeepMeep: fringes
b) aooghaa aooghaa: data recording has stopped - must restart the gui
c) "various instances of "you genius"": acquisition failure, sid home
error, fringe-tracker error (typically not enough light)
1) Sequencer doesn't automatically switch to next
star
Be sure that you entered an observing time in the Create_Star_List window
when copying selected stars to the star list. Without an explicit observing
time, it defaults to 86400 seconds.
2) Data not recorded
Wish we knew what caused this. You recover by stopping observing, quitting
and restarting the gui - you don't have to hibernate first. In the very
rare event that this does not fix the problem, stop observing, hibernate
the sequencer, exit the qui, reboot the sequencer, restart the gui, and
reload the star list.
You should check periodically that data is being written by checking
that disk space is being used up with du -k in /home/pti/bin/data. There
is also a daemon that starts up when the gui starts. If you hear periodic
beeps, it may be the daemon warning you of a data recording failure; recover
as above.
3) GUI pop-up message: "Insertion failed - memory
buffer is full"
Need to add the line:
Text.MaxDocumentsSize: 0
to end of your .Xdefaults file.
Exit or kill the gui; logout, log back in, restart gui.
4) Gui won't exit
From the window where you started it, use "jobs" to get the job_number
(number in brackets), then "kill -9 %job_number". Or from anywhere, use
"psg gui" to get the process_id (first number after the owner's name),
then "kill -9 process_id". You can also log in to harbinger from
another machine and do this.
5) Error messages in observing window text box
a) S_ftFiltWheel_MOVE_ERROR
If this comes from FT #0, it means the ratio filter wheel is in manual
mode - switch to computer mode (single star only). If it comes from FT
#1 , or for FT0 in dual-star mode, ignore.
b) S_ftTaskFast_BUF_ERROR, and others when switching between stars
These are normally caused by the system becoming momentarily busy when
switching modes between stars. Repeated messages while on the same star
usually indicate a problem.
c) S_stStateMachine_X_SERVO_OUT_OF_RANGE, and other st messages
These are ok if they occur during initial star acquisition; if they continue,
it means the star tracker is unhappy, possible with not enough photons, or a failed channel.
See below under the star-tracker section.
d) DL1 METROLOGY DROPOUT
This error message is normally a false alarm, and can be cleared by clicking
on it. However, if you get this message and you're not finding fringes,
you should probably do a fidSet.
5a) No error messages in observing window text box
Treat like 2) (data not recorded), described above.
6) Gui is frozen
Sometimes this is caused by heavy message traffic from the star trackers
when in dual-star mode; normally doesn't happen in single-star mode. Bring
their windows up and click them off (even if off button appears clicked)
and wait a bit to see if life has returned. With monitor switch on position
A, check that sequencer isn't doing anything bad, or typing out error messages.
If the sequencer is not responsive, log into the siderostat, type "stow"
to stow the siderostats (there's also a "stop" shell command), and then
DoSystemHiberate to hibernate the system. Kill the gui process. Reboot
the sequencer with a ^x from its terminal next to the racks. If sequencer
doesn't respond, power cycle the left cardcage following the instructions
in reboot.txt.
7) Subsystem is frozen
For acq, see below under acquisition section. Typically, the rest of this
applies to only the fringe tracker, siderostat, and star tracker.
Symptoms are that state never changes and time doesn't increment. This
is sometimes caused by a task panic. On a console or rlogin window, type
schedShow to list tasks. You may need to hit ^C (not ^X) to restart the
shell. schedShow may show a task with the work PANIC next to it. To restart
the panicked task or tasks, type schedTaskEnable "ftTaskFast" or whatever
the task name is.
If the subsystem doesn't provide a prompt, hibernate and reboot that
subsystem with a ^X.
Depending on which subsystem you reboot, you will have to repeat some
calibrations:
st: chop calibrate
acq: laser boresight
sid: home
ft: ftCalLow, ftCalHigh
dl: fidSet, stroke Set
8) Sequencer halts in WAITING_ON_DL_TRACK or WAITING_FOR_NEW_STAR
See below under fringe tracking.
9) Message "Unable to connect..." when launching
gui.
Ignore the SAO image error. Try exiting the gui, waiting a minute, and
trying again. If no luck, you will need to exit gui and reboot the sequencer
(see above under "gui is frozen").
10) Can't open Create_Star_List window
This should be fixed in the latest GUI version.
This is caused by minimizing the observing window with a subwindow open.
Just exit and restart the gui to fix.
11 Pop-up box announces "200 MB", "100 MB", "50 MB",
"20 MB" left
Just click OK. When you get down to 20 MB, you need to stop, quit the gui,
run beginNight again to create a new directory, restart the gui, and restart
primary observe. Please note before quiting and after restarting that you've
run out of disk space and that this is the new directory, respectively.
12 Window manager is nonresponsive.
Some things to try:
You can go to the control bar on the bottom, select
the configuration icon, and change something that requires a window manager
restart (like mouse focus (under window)). Let it restart the window manager
when prompted.
Log into harbinger from horatio or hardcore and find the pid
(first number after the owner) for the window
manager task (ps -elf | grep wm) (likely dtwm or mwm - see which
one is owned by you). Try kill pid#. It should prompt on harbinger:
quit session? Click cancel, and it may work.
If this doesn't work, do again, don't click cancel.
If this doesn't work, do a kill -9 pid#. This should log you out.
Do ps -elf | grep dtlogin to get its pid (get the one oned by you);
try kill -9 pid#.
Still dead? become su, sync;reboot
There's probably a window manager restart command. If someone knows,
let me know.
13 Gui doesn't read in starlist
Make sure you enter the full path. Also, no blank lines are allowed at
the head of the file.
14 Subsystem Mode Indicators appear frozen in Observing Window
You will also see:
XView warning: Object 0x2f2a20, invalid object (embedding seal incorrect), xv_set
printed out in the window where you launched the GUI.
This problem should go away when the subsytem changes mode again. You could rapidly
switch it to another mode (e.g. "unknown" and then "off") to force the update,
or just wait for the sequencer. If it gets too annoying, try restarting the GUI.
15 You get warning "WARNING: GUI MAY NOT BE
RECORDING DATA" when the gui is not running
The script dskchk.sh runs automatically when you start the gui, and
normally terminates when it discovers the gui is not running. However,
if there is a dead gui process still in the process list, the script
will never terminated. To clean up, exit the current gui. Do a ps -elf |
grep gui to identify any old gui processes; kill them as superuser with
kill -9 pid. Then do a ps -elf | grep dskchk, and kill -9 any instances
of these you find.
16 You need or wish to (re)verify the current CD
burn
From /home/pti/bin, execute ./cdCheck data. This, and other cd
commands, are described in /home/pti/writeup/cdBurning.txt.
ACQUISITION PROBLEMS
1a) Acquisition seems to work, but fails to find stars
- returns PAQ_ERROR or similar
Try in order:
a) On acquisition window, try turning off, then Primary_Acquire (this
automatically resets the siderostat offset). Or just go on to next star.
b) Stow, and then home the siderostats, and try again.
c) Check for clouds (really).
d) Reset the cameras and interface (see end of this section).
e) Stow the siderostats, hibernate, reboot sid, and re-home siderostats.
f) Hibernate, reboot acq, and redo laser boresight.
1a) _One_ acquisition system fails to find stars
This could be due to an out of date model on the siderostat. If you are
sure there is nothing else wrong, you can do a simple search by varying
the sid model feedbeam parameters in the _siderostat_ constants window.
First, put the acquisition system into reset, which will cause it to
run in a loop taking pictures (NB: doing this will subsequently generate
an error message saying that the laser boresight has not been done: you
can ignore this message if you'd like.) Record the initial values of
feedbeam Az and El. Then execute a grid search using 0.1 deg steps,
which cause ~50% shifts in the field displayed on the acq monitor, i.e.,
try adding to the initial values: {+0.1, 0}, {-0.1, 0}, {0, +0.1}, {0,
-0.1}, {+0.1, +0.1}, {+0.1, -0.1}, {-0.1, -0.1}, {-0.1, +0.1}; wait for
an image update before moving to the next step. It's unlikely that the
pointing error would be larger than the search space above. If you find
the image, leave the new feedbeam value in place, put the acquistion system
into primary_acquire, and you should be back in business. You should
inform the appropriate person, however, that a new siderostat model
should be done. If you don't get an image this way, something else is
probably wrong, so restore the feedbeam angles to their original
values.
2) Acquisition system is frozen, or always in camera
busy state
Try in order:
a) Reset the cameras and interface (see below).
b) Hibernate, reboot acq, and redo laser boresight.
c) This happens only infrequently, but if it still doesn't work, or
you're getting ACQ_BUS_ERROR messages on the gui, set
the switchbox to display acq and look for messages like
Bus error
Program Counter: 0x0070f68a [BTW: = ib1onl, line GBIBin]
...
Task: 0x76f064
...
This typically requires a reset of the outside cardcages. Record the
message, please. Stow the siderostats; hibernate; cycle the power of the
outside cardcages (off, wait 5 sec, on) with the remote cardcage power
switch, then reset the outside the cardcages (on, wait 5 sec, off) with
the remote cardcage reset switch, then cycle the camera power (off, wait
5 sec, on) with the remote camera power switch. Rlogin into sid, acq,
and sid, and do a ^x in each. Redo
calibrate, and laser boresight.
Note that the problem may also be associated with the Bus Link cards.
If it freezes, check if the error light is illuminated on the Bus Link
card and note if it is. In that case you may need to swap cards.
However, the card sometimes gets better of its own accord. Thermal issue?
d) Very unlikely: Card-cage off reboot. See reboot.txt
3) Acquisition completes without error, but there's
no star there; (often centroid position is at corner of screen)
Try in order:
a) Reset the cameras and interface (see end of this section).
b) Hibernate, reboot acq, and redo laser boresight.
4) Acquisition behaves strangely, e.g., display doesn't
change even though it appears to be working, laser boresight checkbox is
suddenly unchecked and/or message LASER_BORESIGHT_NOT_YET_DONE even though
you did a laser boresight.
This can happen if you click reset after doing your boresights; if that
is the case, just redo the boresights. Otherwise, hibernate, reboot acq, and redo laser boresight.
5) "Light level too low", even though you see images
on the acq monitor
You can increase the max acceptable integration time ("Max Exposure(deciseconds)")
in the constants window. Try increasing it to 500 ds.
If you're getting false acquisitions on hot pixels, you may need to
increase "Low Light Level" slightly (don't increase it too much, though,
or else you defeat the increase in integration time). You could also try
decreasing the main window size to exclude more pixels. To make the main
window size 200x200, change X Orig, X size from 1, 574 to 189, 200; and
Y Orig, Y size from 1, 382 to 93, 200; this assumes that the laser boresight
is well centered. If not, if laser boresight is, say 300, 200; you would
do 200, 200 and 100, 200 for a 200x200 window centered on the actual boresight.
For very faint stars, you may need to increase the star tracker coadd,
and decrease its bandwidth; do this for all stars to avoid introducing
biases.
6) Star is too faint to acquire on.
See under 5), above. You can also force the system to skip the acquisition
step, and just rely on the quality of the pointing model. See barnard.txt
for more info. You can also look at acqWithTwoStars.txt.
7) System if being confused by a second bright star
in the field.
See under 5) about decreasing the window size.
8) Laser boresight image only present in one camera.
First, be sure the problem isn't the acquisition camera having difficulties,
as described above. If the camera is working correctly, verify that you
haven't left the alignment corner cube on the DSM in place. Also, verify
that the ratio filter wheel is open. If it's blocked, turn to manual, select
"open", and then turn back to computer mode.
9) Camera is overtemperature (or at least warmer than normal).
Note first that the TEC mode gets set by sofware. If you cycle camera power, but don't run
the acquision software, this mode won't get set, and the camera won't run as cool as normal.
Usually, overtemp is caused by the reciruculating cooler running out of antifreeze. The cooler is
in the bottom of the outside rack, on top of the camera controller. Inspect the resevoir level
through the slot in the side; if it appears low or empty, turn of the unit, remove the cover
(held by two captive screws, unscrew the top of the resevoir, and fill with 50/50 antifreeze/water
(should be some premixed in the chem cabinet). Put things back together, and turn the unit back on.
I've noted that it sometimes takes 12 hours for the unit to start working normally again after this.
X) Resetting cameras and interface
Put both acq systems to off; power cycle the camera power using the remote
switches. Wait about 1 minute. Then click hardwareReset in the detail part
of the acquisition window. Takes about 10 seconds to complete.
[ PS, if the cameras were wedged, and you logged into the acquisition
system, you would see every 30 secs or so:
** ERROR ** in < (*ibwait)(TIMO|SRQI) & (ERR|TIMO) >
** Ibsta = 16680 = 0x4128
** < TIMO CMPL CIC TACS >
** Iberr = 6 ** EABO: I/O operation aborted
** Ibcnt = 1
## ERROR ## in < dvwait() >
## The Star I Cam device has return an invalid serial poll response
byte
## Status Byte = 0x0
]
FRINGE-TRACKING PROBLEMS
1) No fringes during night (got them earlier)
a) The star is too big to track.
b) The star is too faint.
c) The delay line has lost position: Go in the lab and check the metrology
signal; if it looks iffy, send the delay lines to the back and and realign.
Otherwise, do a fidCheck on both delay lines. If the position error displayed
is larger than a few tens of microns, the metrology has dropped out. Do
fidSets on both delay lines. (Doing a fidCheck first, rather then just
doing a fidSet directly, tells you if this is really the problem). If this
happens a second time, send the delay lines to the back and realign.
d) Jitter problem: leave sequencing on, turn the fringe tracker off,
and check the delay-line jitter. It should be 10-20 nm. BTW, a higher jitter
during fringe search is normal, as the delay line doesn't follow the search
steps perfectly. If it's high, disable the stroke, and check the jitter
again. If it's now ok, follow the procedure below under WAITING_ON_DL_TRACK.
2) No fringes at start of night or after a reboot
a) If you're not using the sequencer, be sure fringe tracker is in Bright
Primary mode, and stroke and fringe-tracker check boxes are clicked on
delay line (this is all automatic if commanded by sequencer).
b) Be sure you have both sent a stroke with stroke send, and set a stroke
with stroke set.
c) Did you do a chop calibrate: white-light on, laser off?
d) If in 10-ms mode, be sure you sent a 100 Hz stroke (normal value
3260); If in 20-ms mode, be sure you sent a 50 Hz stroke.
e)Be sure you did a fidSet on the long active and long passive delay
lines, and that you aligned or checked the metrology signal with the delay
lines at the back.
f) Do a fidCheck on both delay lines. If the position error displayed
is larger than a few tens of microns, it means either a fidSet wasn't done
or the metrology dropped out. Verify that that the metrology signal is
ok, and then do fidSets on both delay lines.
g) If you were doing dual-star the night before, be sure you fidSet
the small delay line (after doing it, turn its power off (bottom back of
central rack); be sure its power is on before doing the fidSet).
h) Leave sequencing on, turn the fringe tracker off, and check the delay-line
jitter. It should be 10-20 nm. BTW, a higher jitter during fringe search
is normal, as the delay line doesn't follow the search steps perfectly.
If jitter is ok, go to next step. If it's high, disable the stroke, and
check the jitter again. If it's now ok, follow the procedure below under
WAITING_ON_DL_TRACK.
i) Redo chop calibrate. (i.e., maybe images aren't overlapping). When
chop calibrate completes, the N and S photon counts should be equal to
within a factor of 2 or so. A larger discrepancy may indicate an alignment
problem on the star tracker. See below under star-tracker problems for
more info.
j) Reboot the fringe tracker and delay line, and redo calLow, calHigh,
strokeSend, strokeSet, fidSet.
3) Fringe tracker is frozen
See above under frozen subsystem.
4) Sequencer halts in WAITNG_ON_DL_TRACK or WAITING_FOR_NEW_STAR
ii) Check the state of the delay lines. Long passive should be in slew:done,
long active in track:track. If either is in stab, it may be because the
metrology dropped and they hit a limit switch; redo fidSet.
i) Record the 5-sec jitter and HP jitter in the log.
a) Turn FT off from FT gui; record the 5-sec jitter and HP jitter in
the log.
b) Disable stroke check box on DL gui.
c) Verify that jitter is low (10-20 nm). (If jitter is not low, there's
something not right with the delay line: try stopping and restarting primary
observe; if it persists, hibernate and reboot the delay line (and then
redo the stroke)). Record the values in the log.
d) Do a strokeCheck from the DL gui. Please record the value in the
log. If the meter is hooked up to the stroke output, set to ac volts and
record its value also.
e) Resend the stroke and do a strokeSet; renable the stroke check box.
f) If this doesn't fix the problem, hibernate and reboot the delay line,
and then redo the stroke.
5) Low light level at start of night
For single-star mode, check that dual-star beamsplitters are not in place.If
so, remove them carefully (they're on magnetic kinematic bases). Also,
verify that the dual-star polarizers are not in the beam.
6) Error or beeps during calLow, calHigh
For light level too low, too high errors: verify that the light level is
ok; no light for cal low, and 500-1000 for each a, b, c, d for calHigh.
You can also get an error when doing a calHigh if the spectrometer is misaligned
and isn't getting any light. Typically, there should be a 2.5:1 ratio of
WL to total spectrometer displayed. To retry, first redo calLow, then calHigh.
Remember that you can't do two calLows or calHighs in a row without first
going to another mode like ftOff.
If you're still having trouble, you may have have a bad pixel. See fringePixel.txt
for details on moving the spots around.
7) Fringe tracker tracks noise on bright stars, or
getting lots of semilocks
Try increasing the fringe tracker SNR thresholds: if you're getting a lot
of semilocks on faint stars, you can increase the fringe-tracker constant
SNR Thresh: Search-Semilock. If you're getting a lot of semilocks on bright
stars, you can increase the SNR Thresh slope (NB: the slope times the photon
rate plus the threshold is the actual threshold use for search-to-semilock
transition.
8) Low visibilities
This is usually just seeing, but things to check:
a) Verify that the stroke and frequency programmed on the delay line
match the stroke and frequency displayed on the fringe tracker window.
I.e., make sure you send a 50Hz stroke if you're 20ms mode, etc.
b) Be sure chop calibrate was done correctly: white light on; laser
OFF, total count less than 500k/sec (~1000 for each channel per 10 ms).
c) In single-star mode, be sure polarizers are out of beam. The polarizers
reduce visibility is dual star mode.
d) Be sure you aligned the fold mirrors to the outside, and that the
the extension tubes aren't vignetting the beam (note that the bottom of
the boresight laser beam is ordinarily vignetted by the metrology injection
mirror - this is not in the starlight path).
9)No or weird counts on meter when aligning
fiber
Make sure ft1 (secondary fringe tracker) is not in ftABCD mode (they
share the same d/a channel)
DELAY LINE PROBLEMS
-1) Star tracker can't find the star after a laser
boresight.
This is likely just due to the offset between the laser and star boresights,
which gets updated after the first star is fringe tracked (although
there is a limit to how much it can move per step). Be patient:
it can take several minutes the first time.
0) One of the star trackers won't lock - had been
working before.
This may be caused by a bad photon rate being passed from the acquisition
system due to star color. Try
a) Go on to the next star and see if it works better.
b) Check "sum low lmt" in the detailed constants window on the
problematic star tracker and compare it to the other, working, star tracker.
If it seems high, or anyways, try decreasing it by a factor of two, then off
and track on the star tracer to restart. See below (1a) on how to make this
type of change permanent.
c) See 2), below.
d) Verify that the acquisition system properly acquired the
problematic star, with a reasonable centroid position. If this seems
wrong after trying primary acquire again, primary observe off, hibernate,
reboot acq, reset camera, redo laser boresight, and try again.
1a) Photon count too low message
If you're in single-star mode, first verify that the extra ND filters used
in dual-star or phase-referencing mode are not still in place. They
mount on a kinematic base in front of the star-tracker head.
There is a threshold in the star-tracker constants window: "Sum Low Lmt",
which is the minimum number for photons per frame for the
star tracker to be happy. The low value is based on a scaled value of
the intensity measured by the acquisition system, via the star tracker
constant "Acq2St Counts Scale". Sometimes this value comes out too
low, depending on star color or seeing. You can adjust this for
an individual star by setting "Sum Low Lmt" in the st constants window (upper right corner).
You can also lower the threshold for all (subsequent) stars by decreasing the
the scale factor (a factor of 2 is usually safe; if you set it much too
low, you can track noise).
Please note in the log if you change this value.
(For very faint stars, you may want to reduce the gains ("X Gain, Y Gain")).
1b) Photon count too high
A second value in the star-tracker constants window is "Sum High
Lmt." It is fixed, and defaults to a
value which warns when the star tracker detector is starting to saturate. This
is especially important when chop calibrating, as too much light will
introduce errors in the centroid locations. When tracking too bright
stars, you'll get warning messages (it will keep tracking, but there is
a danger of systematic biases). You can increase the value to get rid
of the warning messages, but that doesn't change the system behavior.
2) Star tracker just won't stay locked
Many possible causes:
-
This may just be caused by clouds.
- Check that the problem isn't light level (see 1a, above).
Usually, the problem
is that the star is falling below Sum Low Lmt.
- Verify there's no vignetting in the system, going out to sids.
Also, check vignetting within the star tracker by uncovering the st corner
cubes and tracing the light through the system.
In particular, make sure the alignment corner cubes on the outside have
been removed, that the air pipes are properly centered, that there's no
insulation between the air pipe and the window, and that the vaccuum
pipe is properly centered on its support (the N one can be slid back
and forth about 1/2").
- Rarely it may be that the active or passive SDL (short delay line)
is misaligned such that the beam is hitting the edge of the mirror. In this case the boresight will still be visible out to the huts (if somewhat faint),
but the ST photon counts will fluctuate wildly and it will not hold lock.
Check by turning on the boresight and looking at where the spot hits the
flat at the SDL focus. If near edge, you can fix by adjusting the
appropriate micrometer by 1-3 ticks.
- If you are getting "X_SERVO_OUT_OF_RANGE" or "Y_SERVO_OUT_OF_RANGE"
ON ONE AXIS only, then you may have had a failure in one of the
DAC/AMP/PZT channels. You can move the FSM X or Y channel to an
unused (secondary) channel and see if that fixes the problem.
- You may wish to check the telescope focus. If the star tracker
is unable to stay locked, and it drops out by among other things
complaining that X(and Y)_SERVO_OUT_OF_RANGE, and the FT counts
are unusually low, then it may be a defocus issue.
- Still? It's possible the star tracker is in need of a full alignment;
see the separate writeup.
3) Large intensity difference between star trackers
The intensity numbers are displayed on the observing window. If they're
significantly different, check for vignetting as described in 2).
4) Chop calibrate doesn't work or not enough light
Verify that chopper HV (buzzzz) is on; sometimes gets turned off when people
are working in the lab.
Sometimes you can get a not-enough-light error if the last star tracked
the night before was too bright. In this case, set "Sum Low Lmt" to a more
reasonable value.
If the system had crashed during a ratio measurement, it's possible
the ratio shutter is still closed. Check it from the lights on its controller
box; to move, put in manual mode, select the open position, and put it
back in computer mode.
5) Chop calibrate doesn't complete.
This may be caused by running out of range on the chopper mirror. In this
case, you adjust the mirrors in front of the Questar telescope to center
the images manually. Reboot the star tracker so you're starting from the
default chop calibrate positions. Select "display this star tracker" on
the N star tracker window. Block the S beam entering the star-tracker telescope.
Adjust the WL source until you get a signal on the LCD meters mounted on
top of the secondary table. Temporarily block the N beam and verify that
you're seeing real signal. Adjust the N fold mirror in front of the telescope
until there is a signal on all four quadrants. As you get closer to the
correct position, you should need to turn down the WL source until it's
back to the typical value for chop calibrate. When done, deselect the N
star tracker, select the S tracker, unblock S, block N, and repeat the
procedure for S. Don't mess with this unless you're pretty sure it's the
problem.
SIDEROSTAT PROBLEMS