Nstars Documentation

Relevant Files:

SO90stage1.cl

SO90autoextall.cl

SO90final.cl

make_reduclist.cl

make_reduclist.pl

SO90_BADPIX_TRIM.dat

fluxall1_0 - located in /usr/bin

SO3flx - located in /home/flip/FluxCalibration/SO90/SO3flx/ (may have new version - check with ROG)

all commands enclosed { } are IRAF commands
all commands enclosed in ` ` are Unix commands

1. Copy fits files from the night's run into /usr/local/fits2/Chris_90in

2. Run SO90stage1.cl at cl> prompt in IRAF.

{ cl < SO90stage1.cl }

3. relieve fixpix problem (if necessary):

when SO90stage1.cl locks while doing fixpix, go to

unix prompt, kill the process "x_proto.e", type { cl> flpr }

a couple of times into IRAF, then cut an paste remaining

fixpix lines from Script directly onto the cl> prompt.

(explained in more detail below & in comments in script)

4. define task SO90autoextall

{ cl> task $SO90autoextall=SO90autoextall.cl }

5. run SO90autoextall

{ cl> SO90autoextall }

6. run make_reduclist.cl to create rough reduce.lst file

{ cl> cl < make_reduclist.cl }

7. Review and edit reduce.lst file from that night's logs to finalize reduce.lst

8. define as task SO90final

{ cl> task $SO90final = SO90final.cl }

9. run SO90final { cl> SO90final }

11. cd to output directory where .cal files were written to

This is set in SO90final.cl as /usr/local/SO90out

12. use XMK1x to rectify any early types and make .rec files

13. run the program fluxall1_0.pl within .cal directory to make flx, nfx or nor files. (or use SO3flx manually)

` fluxall1_0.pl `

(fluxall1_0 is not yet set up to flux SO90 data... this is a "TO DO")

(if it has not been done yet, simply edit fluxall1_0 to allow SO3flx to be called,

and set up all switches appropriately. I beleve SO4flx (newer version) is implemented)

Check with ROG for best fluxing program and see the fluxall1_0 code.

14. if satisfied with reduction, clean up IRAF directory with cleanup.cl

{ cl < cleanup.cl }

The SO90 reduction pipeline starts with the basic "stage1" reduction
which includes BIAS, & FLAT corrections, bad pixel correction.

SO90stage1:

the script SO90stage1.cl takes care of all this in one script:
first 12 lines cleanup old files - avoids overwrite errors

(I opted NOT to subtract dark frames - they are very few counts above bias)

biases are then collected, listed and combined.

flat files are collected & listed

objects are collected & listed (obj0001.imh -> )

comparisons are collected & listed. (hear0001.imh -> )

I wish to pause here and complain about Steward Observatory's FITS headers.
when it comes to comparison images.
- This has been fixed in images after M030311
Usually, a imhead or hselect test
will pick out "imagetype" to determine type of image (dark, bias, comp, etc).
Well, both flats and comparisons have imagetype = 'comp', which confuses
ccdlist. Flats should have imagetype = 'flat' or something.
This is the reason for the strange 'hselect' line instead of an
easy 'ccdlist'
Also, comparison files have "hear" written into the OBJECT field,
which should be (in my opinion) the object name that should use
the comparison file. This makes association between object files
and their comparison easier. Right now, comp files have IMAGETYPE='comp'
and OBJECT='hear', which is redundant. You will see this problem
re-appear in "make_reduclist.cl"


First pass through 'ccdproc' removes bias, & trims flats, objects and comparisons.

Master flat is created
imcombine()'s the trimmed, bias corrected flats
Boxcar is done to create FS - a "high pass filter" over the flat - this is the response curve
Original flat is then divided by FS, to take response curve out of flat.

Second Pass through 'ccdproc' flat corrects objects & comparisions with this Master flat file.

Fixpix is run using "SO90_BADPIX_TRIM.dat" as the bad pixel mask.
I spent way too much time trying to circumvent a parser problem
in iraf with fixpix. First, one call to fixpix would not complete -
only gets about 60 lines through the @list. So I decided to try to split
the @list into 50-line chunks. fixpix still locks after the first call
finishes, so I manually kill the IRAF process "x_proto.e", and continue
by cutting and pasting the fixpix calls directly into the cl> prompt
until all the @lists are processed, and all object files are run
through fixpix. See the comments in SO90stage1.cl for more info.

Cleanup is done, and Stage1 is complete

------

SO90autoextall.cl

This is an aperture extraction routine that pretty much automatically
finds your aperture, sets backgrounds, traces, and sums up to create
2-d extracted spectra. (.ms.imh files) It must be run as a task due to being an interactive
script. (you can always define tasks in you login.cl - if you wish)

dispersion axis is defined for Steward Observatory (should be done with setinstrument)
but never hurts to double-set

database is cleaned - ready for night's data.

object files listed

variables defined

While loop extracts every object file, one at a time.

preliminary apertures are found & adjusted

using the values determined by the first calls to apfind
and apresize (lines 39 & 40), we can define the background
regions automatically. this is done by reading the database
aperture file through some brute force.

the variable "bg_sample_string" is then created and the
parameter "apdefault.bsample" is set to this.

the database file is deleted, and a new apfind and apresize
is done to create the new database file with the new background
aperture.

This complex background aperture definition is done to compensate
for variable seeing - normally backgrounds are set and left alone
for the entire night. But if the seeing gets bad, then the
spectra can bleed into the background - this avoids that.

and interactive apall is called at the end to give the user
visual conformation that the correct aperture was found and background
apertures are set correctly. I usually get about 1 object per night
that has a cosmic ray on the column I define my aperture on - and apfind
will think this is the spectrum. This is the only reason why this
task is interactive.

-------------------

make_reduclist.cl

This script attempts to use fits header information to
create what we call a "reduce.lst" file, which describes
how separate spectra are to be combined and which comparison
to use to calibrate.

a reduce.lst file usually looks like so:
(example from Steward Obs. Apr 8, 2001 run reduce.lst)

2_Hya 2 hear0001
obj0001
obj0002
Zet_Leo 2 hear0002
obj0003
obj0004
.
.
.

This means 2_Hya has 2 images to combine (obj0001 & obj0002) and
should use hear0001 to calibrate.... etc.

the reduce.lst file lives in the same directory as
all the scripts and drives the next script, SO90final.cl

This would work great in theory, but there is no way within
fits headers (other than UT or RA&DEC) to associate comparisons
with objects automatically. So this script assumes sequential
comparisons, and a unique comparison image for every series
of objects. This, of course, isn't true (double stars, for example,
are 2 diff objects with the same comparison) so it's necessary
to open it up in your favorite editor and make sure it's correct.

It is nice, though, not to have to type the entire file out
for 100+ images.

Without going into too much depth on how it works, it creates
a file called "nameinfo.dat" in which a Perl script iterates
through (make_reduclist.pl) to create reduce.lst

-----------------

SO90final.cl

This is the script that calibrates each spectra file, combines them,
fills out the .cal file footer information from the fits headers, and
creates the output .cal files.

The first thing you will notice is the mess of variable declarations..
most of these are for the footer info. This is actually quite a small
script if the code that populates the footer was taken out.

The script is well commented, mostly because I inherited the script
from Dr. Richard Gray and had to understand it myself before altering
it into the mess it is today. I will try to resolve some issues
that may cause confusion.

On line 71, you will notice a call to apfind on the first comparison "hear0001".
I use this to set a wide aperture using apedit()'s graphical interface. I just
center an aperture, and open it wide (about 1/2 to 3/4 the width of the image).
This allows me to get a better signal on low-signal comparisons.

Then (line 76) I call 'identify' to create the reference solution (refSO90)
which is the "preliminary fit".

Next is a confusing if-ladder inside a for loop inside a while loop.

The while loop controls the entire reduce.lst file and (if reduce.lst is
correctly written) the call to fscan() in the while loop will only
read a line describing the object name, comparison file, and number of images.

The for loop iterates for each image of the object (from 1 to k, where k
is the number of images taken of the object) and calibrates each one.
It also collects a running total of airmasses to average later, and a total
exposure time.

the if-ladder decides which image you are currently processing,
and handles special cases:
if it's the first image in the series:
reidentify the comparison interactively
get the start date & time of series from fits header

if it's NOT the first of the series
reidentify the image non-interactively, using solution from first image

if it's the LAST of the series:
get the end date & time of the series


Edit the fits header with the comparison info (REFSPEC1), do a dispcor to calibrate.
(line 223)

Depending on the number of images in the series, we combine with an average or sum.

Most of the data reduction is done up to now, what's left is forming
the footer. Each if-clause checks to see if a value was read from the
fits header, and if not, prompts the user to enter a value.

at line 415, the script gets interesting again.
The footer is printed out to a .head file through a series of printf()'s
the .cal file is created through the call to listpix(),
and the .head file and .cal file are concatenated. to create the final .cal file.

(.head was chosen before we realized the info goes at the end of the file.)

From here, we go back to the while loop and process the next object.

----------------------

Use of XMK19 (or whatever version you have) is out of scope of this document,
but I'll mention that rectification of the .cal files is the next step, only
for completeness.

------------------

The final step is the flux calibration, which is done at the Unix prompt, outside
of IRAF. I use a Perl script I developed called "fluxall1_0.pl" to do this.

to flux calibrate a single .cal file, we use Dr. Richard Gray's flux calibration
program for Steward Observatory called "SO3flx" which calibrates given
an airmass (for .nor output) or optional v, b-y, and m1 photometry for .flx
and .nfx output.

`SO3flx [v] [b-y] [m1]`

I couldn't find the source (SO3flx.c) on my machine - but I can
get it from Dr. Richard Gray if needed.

airmass is easy to get:
`grep "#AIRMASS" `

I wrote a small script called fluxall1_0.pl which gathers all the information
for the command line and fluxes an entire directory by itself. I am including this
file although I know it won't get photometry through the MySQL database interface.

I can open the port up for remote queries (outside appstate.edu) if necessary, though.
and you will have to edit the subroutine "connect_to_database" to put an appropriate username
in.

--------------------

I believe that's all.
Be sure to read through the .cl scripts and enjoy the comments.

Relevant Files:

CTIOstage1.cl

CTIOautoextall.cl

CTIOfinal.cl

CTIOcleanup.cl

make_CTIO_reduclist.cl

make_CTIO_reduclist.pl

CTIO_BADPIX_TRIM.dat

fluxall1_0 - located in /usr/bin

ctiorun[1234] - located in /home/flip/FluxCalibration/CTIO

1. Copy fits files from the night's run into /usr/local/fits2/Chris_90in

2. Run CTIOstage1.cl at cl> prompt in IRAF.

{ cl < CTIOstage1.cl }

3. relieve fixpix problem (if necessary):

when CTIOstage1.cl locks while doing fixpix, go to

unix prompt, kill the process "x_proto.e", type { cl> flpr }

a couple of times into IRAF, then cut an paste remaining

fixpix lines from the script directly onto the cl> prompt.

(explained in more detail below & in comments in script)

4. define task CTIOautoextall

{ cl> task $CTIOautoextall=CTIOautoextall.cl }

5. run CTIOautoextall

{ cl> CTIOautoextall }

6. run make_CTIO_reduclist.cl AND make_CTIO_reduclist.pl to create rough reduce.lst file

{ cl> cl < make_CTIO_reduclist.cl }

` ./make_CTIO_reduclist.pl`

7. Review and edit reduce.lst file from that night's logs to finalize reduce.lst

8. define as task CTIOfinal

{ cl> task $CTIOfinal = CTIOfinal.cl }

9. run CTIOfinal { cl> CTIOfinal }

11. cd to output directory where .cal files were written to

This is set in CTIOfinal.cl as /usr/local/CTIOout

12. use XMK1x to rectify any early types and make .rec files

13. run the program fluxall1_0.pl within .cal directory to make flx, nfx or nor files. (or use ctiorun manually)

` fluxall1_0 `

14. if satisfied with reduction, clean up IRAF directory with cleanup.cl

{ cl < CTIOcleanup.cl }

The CTIO data reduction procedure varies only slightly from the SO90
procedure. This is because Steward Observatory was first to provide
us with images with FITS headers, and CTIO follows even more closely
to an NRAO 'standard' way of imaging spectra.
Therefore, the similarities in the datasets prompted a basic copy of the SO90
procedure with appropriate edits for CTIO data.

Copy Files:
Not much to be said here. Copy FITS files to reduce into /usr/local/fits2/CTIO
Be sure permissions are correct, and try renaming to .fit or .fits (lowercase)
if problems arrise reading the directory


CTIOstage1:

Stage1 reduction takes care of bias subtraction and flat correction of all the
frames, along with removing trim and biase sections

It also performs some bad pixel fixing, which is a big problem with CTIO. The
CCD entrance window (or some other part of the optical path) is quite dirty, and
there are quite a few very dark parts of the field. There has been work to correct
for these defects in the flat - by forcefully removing them from the responce flat
to better preserve them in the 'flattened flat'. Most correcton is still done in the
bad pixel mask, though. Concensus is that we would rather have falsified continum through
The dust covered area, than a flase spectral line. - Yes, these dust motes leave
a very convincing profile of a strong spectral line - one is famous for living
on or near the G-band.

Anyway, the bad pixel masks are CTIO_BADPIX*. trim means they are ment for images
that have been trimmed (so it matches the image dimentions and reference points)
and any dates on the filenames are specific for that set of data from CTIO.
If you have a new set of data from CTIO, do check the bad pixels and dust motes
manually and make a new pixel mask - but re-reducing old data with these
masks should be fine.

Since this script was inhereted from the SO90 script, it also inhereted
the 'fixpix' bug. See the SO90 reduction script above for instructions
on how to break out of a 'fixpix' lock and resume.

---------

CTIOautoextall:

(pretty much the same as SO90)

--------

make_CTIO_reduclist:

(same as SO90 except `./make_CTIO_reduclist.pl` is run afterwards manually to
create reduce.lst. Stop is in place so user can manually check intermediate file "nameinfo.dat"
Which the .pl script utilizes.
Recognize difference between .cl and .pl scripts (iraf and Perl).
Note commented out call to .pl script within .cl script

----------------

CTIOfinal.cl:

Same as SO90 - without comparision signal problem.

This script works quite well for CTIO.

Still should discuss resuming after screw-ups. Same as DSO procedure to resume

---------------

Rectification:
cd to /usr/local/CTIOout and work on the .cal files - rectify them with xmk1x
remember to rename to .ctio.rec if neccessary

-------------

Flux Calibration

Fluxall1_0 handles CTIO data. Each run should have it's own program to flux with
So far, there have been 4 runs, therefore there are 4 fluxing programs

Feb 2001 run -> /home/flip/FluxCalibration/CTIO/CTIOfeb01/ctiorun1
Aug 2001 run -> /home/flip/FluxCalibration/CTIO/CTIOaug01/ctiorun2
Jun 2002 run -> /home/flip/FluxCalibration/CTIO/CTIOjun02/ctiorun3
Dec 2002 run -> /home/flip/FluxCalibration/CTIO/CTIOdec02/ctiorun4

Use the appropriate one if re-reducing data from these runs. For new data
you really should observe a spectrphotometric standard and create a new
routine.


The SO90 reduction procedure is much more descriptive, and although it
is targeted to the SO90 data, much of the info in the documentation
is relevant to the CTIO reduction routine. Please have a look at it.

Introduction

1. /1200g - Dark Sky Observatory
   • /hb - DSO 'hb' files
   • /rec - DSO rectified 'hb' files
2. /600g - Dark Sky Observatory
   • /mb - DSO 'mb' files
   • /pfx - DSO '.rec' '.nfx' '.flx' '.nor' files
3. /SO90 - Steward Observatory 90" (VATT) "
   • /cal - raw '.cal' files
   • /proc - '.nor' '.nfx' '.flx' and '.rec' files
4. /ctio - CTIO 1.5m
   • /raw - raw '.cal' files
   • /flx - '.nor' '.nfx' '.flx' and '.rec' files
5. /toronto - David Dunlap Observatory
   • /4200 4200A center spectra
     • /raw - raw '.cal' files
     • /rec - '.rec' (and .nfx in the future) files.
   • /4700 - 4700A center spectra
     • /raw - raw '.cal' files
     • /rec - '.rec' (and .nfx in the future) files.

you will also notice extra directories, spectra files, and
Perl scripts laying about this directory. Most of the spectra
files you find outside this directory structure were pulled out
because they were not Nstars and most of these were put in "/others".
Most of the Perl scripts in these directories were "one time runs" and
the code should be well documented as to exactly what it does
inside the script itself.
These are the spectra which display on the web site in the "Nearby Star Spectra"
area. Having a spectrum come up for display on the web is not as simple
as copying it into it's appropriate directory - a reference to the file
must be inserted into the MySQL table 'SpectraFiles'. This can be done
manually or with the Perl script 'PushToServer' . See MySQL Basics and
MySQL Database Data - SpectraFiles for more information

Standards

MySQL Database Data

All "relational" data for the Nstars project is held on the MySQL
database server living on stellar.phys.appstate.edu, and is stored
in the database "Nstars".

This is why, as will be explained in depth later, one must specify

mysql> use Nstars;

as the first command on the MySQL command line, or using the '-D' switch when
starting the MySQL client to connect to stellar.phys.appstate.edu:

%mysql -h stellar -u flip -p -D Nstars

The tables within the Nstars Database:

NstarsList

The Primary Table.

Holds basic Nstars data from Hipparcos catalog.

In theory (but not practice) all hip# fields in all other tables are

Foreign keys into the hip field of this table.

Basically, what this means is that you are `not supposed to' insert

hip #'s into other tables without them first existing in this table.

In practice, though, there is no restrictions against this.

 
+----------+-------------+------+-----+---------+-------+
| Field    | Type        | Null | Key | Default | Extra |
+----------+-------------+------+-----+---------+-------+
| hip      | int(11)     |      | PRI | 0       |       |
| HD       | varchar(10) | YES  |     | NULL    |       |
| BD       | varchar(10) | YES  |     | NULL    |       |
| RA_2000  | varchar(11) |      |     |         |       |
| DEC_2000 | varchar(11) |      |     |         |       |
| PI       | varchar(8)  | YES  |     | NULL    |       |
| hip_V    | varchar(8)  | YES  |     | NULL    |       |
| hip_BmV  | varchar(8)  | YES  |     | NULL    |       |
| SpT      | varchar(12) | YES  |     |         |       |
| pi_err   | varchar(10) | YES  |     | NULL    |       |
+----------+-------------+------+-----+---------+-------+ 

ChromAct

Holds derived Chromospheric Activity indecies

+---------------+--------------+------+-----+---------+-------+
| Field         | Type         | Null | Key | Default | Extra |
+---------------+--------------+------+-----+---------+-------+
| hip           | int(11)      |      |     | 0       |       |
| name          | varchar(20)  | YES  |     | NULL    |       |
| DoubleFlag    | char(2)      | YES  |     | NULL    |       |
| Observatory   | varchar(20)  |      |     |         |       |
| S_mw          | varchar(6)   | YES  |     | NULL    |       |
| log_R_HK      | varchar(6)   | YES  |     | NULL    |       |
| ChromActClass | varchar(20)  | YES  |     | NULL    |       |
| Priority      | decimal(3,1) | YES  |     | NULL    |       |
+---------------+--------------+------+-----+---------+-------+  

ClassBatchInfo

Holds 'pipeline' information for each Nstar - basically, how far we've come with each star

The column "Batch" is quite important - it tells you that a hip # is included in

A classification batch, and what that batch is called.

+------------+-------------+------+-----+---------+-------+
| Field      | Type        | Null | Key | Default | Extra |
+------------+-------------+------+-----+---------+-------+
| hip        | int(11)     |      |     | 0       |       |
| name       | varchar(10) | YES  |     | NULL    |       |
| Classified | char(1)     |      |     | N       |       |
| Batch      | varchar(50) | YES  |     | NULL    |       |
| Published  | char(1)     |      |     | N       |       |
+------------+-------------+------+-----+---------+-------+ 

Classifications

Holds new Nstars Classification info.

+-------------+--------------+------+-----+---------+-------+
| Field       | Type         | Null | Key | Default | Extra |
+-------------+--------------+------+-----+---------+-------+
| hip         | int(11)      |      |     | 0       |       |
| name        | varchar(20)  |      |     |         |       |
| GCG_SpecTyp | varchar(50)  | YES  |     | NULL    |       |
| notes       | varchar(200) | YES  |     | NULL    |       |
| batch       | varchar(20)  |      |     |         |       |
+-------------+--------------+------+-----+---------+-------+  

Notes

Holds any notes that will appear on the web site.

+-------+---------+------+-----+---------+-------+
| Field | Type    | Null | Key | Default | Extra |
+-------+---------+------+-----+---------+-------+
| hip   | int(11) |      | PRI | 0       |       |
| note  | text    | YES  |     | NULL    |       |
+-------+---------+------+-----+---------+-------+

Observations

Holds information for every observation of Any Nstar

+----------------+--------------+------+-----+---------+-------+
| Field          | Type         | Null | Key | Default | Extra |
+----------------+--------------+------+-----+---------+-------+
| hip            | int(11)      |      |     | 0       |       |
| name           | varchar(20)  | YES  |     | NULL    |       |
| ObsvTime_Start | datetime     | YES  |     | NULL    |       |
| ObsvTime_End   | datetime     | YES  |     | NULL    |       |
| Airmass        | varchar(20)  | YES  |     | NULL    |       |
| Comments       | varchar(100) | YES  |     |         |       |
| Observatory    | varchar(20)  | YES  |     | NULL    |       |
| Project        | varchar(20)  | YES  |     | NULL    |       |
| reobserve      | int(11)      | YES  |     | 0       |       |
+----------------+--------------+------+-----+---------+-------+ 

Simplex

Holds all derived data for any one spectrum (from an observatory)

+-------------+--------------+------+-----+---------+-------+
| Field       | Type         | Null | Key | Default | Extra |
+-------------+--------------+------+-----+---------+-------+
| hip         | int(11)      |      |     | 0       |       |
| name        | varchar(20)  | YES  |     | NULL    |       |
| DoubleFlag  | char(2)      | YES  |     | NULL    |       |
| Observatory | varchar(20)  |      |     |         |       |
| T_eff       | varchar(8)   | YES  |     | NULL    |       |
| log_g       | varchar(5)   | YES  |     | NULL    |       |
| M_over_H    | varchar(5)   | YES  |     | NULL    |       |
| V_t         | varchar(5)   | YES  |     | NULL    |       |
| V_t_Fixed   | char(1)      | YES  |     | NULL    |       |
| Priority    | decimal(3,1) | YES  |     | NULL    |       |
| Fit_Qual    | varchar(15)  | YES  |     | NULL    |       |
+-------------+--------------+------+-----+---------+-------+ 

SpectraFiles

Holds filename locations for all spectra for any Nstar.

This table is queried when the web page is asked to display

a list of files associated with an Nstar. For any spectrum

to be displayed on the web page, it's filename must first

be in here, and associated with a hip #

+-------------+--------------+------+-----+---------+-------+
| Field       | Type         | Null | Key | Default | Extra |
+-------------+--------------+------+-----+---------+-------+
| hip         | int(11)      |      |     | 0       |       |
| name        | varchar(12)  | YES  |     | NULL    |       |
| Observatory | varchar(20)  |      |     |         |       |
| Filename    | varchar(50)  |      | PRI |         |       |
| priority    | decimal(6,4) | YES  |     | NULL    |       |
+-------------+--------------+------+-----+---------+-------+

Starnames

Holds names from other famous catalogs for every Nstar..

Good for finding the hip# for "VEGA" or "51 Peg"

+-----------+--------------+------+-----+---------+-------+
| Field     | Type         | Null | Key | Default | Extra |
+-----------+--------------+------+-----+---------+-------+
| hip       | int(11)      |      | PRI | -1      |       |
| hd        | varchar(12)  | YES  |     | NULL    |       |
| BD        | varchar(12)  | YES  |     | NULL    |       |
| CD        | varchar(12)  | YES  |     | NULL    |       |
| CPD       | varchar(12)  | YES  |     | NULL    |       |
| HR        | varchar(12)  | YES  |     | NULL    |       |
| GJ        | varchar(12)  | YES  |     | NULL    |       |
| Bayer     | varchar(30)  | YES  |     | NULL    |       |
| Flamstead | varchar(30)  | YES  |     | NULL    |       |
| NAME      | varchar(100) | YES  |     | NULL    |       |
| variable  | varchar(20)  | YES  |     | NULL    |       |
| LTT       | varchar(20)  | YES  |     | NULL    |       |
| SAO       | varchar(20)  | YES  |     | NULL    |       |
| GEN       | varchar(20)  | YES  |     | NULL    |       |
| uvby98    | varchar(20)  | YES  |     | NULL    |       |
| AKA1      | varchar(30)  | YES  |     | NULL    |       |
| AKA2      | varchar(30)  | YES  |     | NULL    |       |
| AKA3      | varchar(30)  | YES  |     | NULL    |       |
| AKA4      | varchar(30)  | YES  |     | NULL    |       |
| comments  | varchar(100) | YES  |     | NULL    |       |
+-----------+--------------+------+-----+---------+-------+ 

photometry

Holds All available photometry for all Nstars.

+--------------+----------+------+-----+---------+-------+
| Field        | Type     | Null | Key | Default | Extra |
+--------------+----------+------+-----+---------+-------+
| hip          | int(11)  |      | PRI | 0       |       |
| HD           | int(11)  | YES  |     | NULL    |       |
| BD           | char(10) | YES  |     | NULL    |       |
| hip_V_       | char(8)  | YES  |     | NULL    |       |
| hip_BmV_     | char(8)  | YES  |     | NULL    |       |
| strom_v      | char(8)  | YES  |     | NULL    |       |
| strom_bmy    | char(8)  | YES  |     | NULL    |       |
| strom_m1     | char(8)  | YES  |     | NULL    |       |
| strom_c1     | char(8)  | YES  |     | NULL    |       |
| H_beta       | char(8)  | YES  |     | NULL    |       |
| Jon_V        | char(8)  | YES  |     | NULL    |       |
| Jon_UmB      | char(8)  | YES  |     | NULL    |       |
| Jon_BmV      | char(8)  | YES  |     | NULL    |       |
| Jon_VmR      | char(8)  | YES  |     | NULL    |       |
| Jon_VmI      | char(8)  | YES  |     | NULL    |       |
| Jon_RmI      | char(8)  | YES  |     | NULL    |       |
| Jon_J        | char(8)  | YES  |     | NULL    |       |
| Jon_H        | char(8)  | YES  |     | NULL    |       |
| Jon_K        | char(8)  | YES  |     | NULL    |       |
| Cousin_VmRc  | char(8)  | YES  |     | NULL    |       |
| Cousin_VmIc  | char(8)  | YES  |     | NULL    |       |
| Cousin_RcmIc | char(8)  | YES  |     | NULL    |       |
| tmass_J      | char(8)  | YES  |     | NULL    |       |
| tmass_H      | char(8)  | YES  |     | NULL    |       |
| tmass_K      | char(8)  | YES  |     | NULL    |       |
+--------------+----------+------+-----+---------+-------+ 

spt2code

A special-use table - a translation table to turn MK types as strings into decimal numerical values

where Early types have low values and late types have high values

Used primarily to 'sort' or 'place' spectra by type.

+-------+--------------+------+-----+---------+-------+
| Field | Type         | Null | Key | Default | Extra |
+-------+--------------+------+-----+---------+-------+
| Spt   | varchar(8)   | YES  |     | NULL    |       |
| code  | decimal(6,4) | YES  |     | NULL    |       |
+-------+--------------+------+-----+---------+-------+ 

Starting / Stopping / Restarting the Server The program '/usr/sbin/apachectl' it a control interface for the http server.
By issuing either 'start', 'stop', or 'restart' argument to apachectl you can
perform the named action on the server
example: "/usr/bin/apachectl stop" will bring the server down
'man apachectl' for more information.

Website Layout

TableA:              TableB:
key_a | value_a        key_b | value_b
---------------        ---------------
  1   |   a             1    |   20
  2   |   b             3    |   40
  3   |   c		


Then, The query "Select * from TableA, TableB;" will produce:

key_a | value_a | key_b | value_b
---------------------------------
  1   |   a     |  1    |  20
  1   |   a     |  3    |  40
  2   |   b     |  1    |  20 
  2   |   b     |  3    |  40
  3   |   c     |  1    |  20
  3   |   c     |  3    |  40



As you can imagine, tables formed this way can get quite large.  Output size is always the product

of the size of the two tables crossed



Using a "join condition" makes this a bit more useful:

"SELECT * from TableA, TableB WHERE TableA.key_a=TableB.key_b;"



key_a | value_a | key_b | value_b
--------------------------------- 
  1   |   a     |  1    |  20   
  3   |   c     |  3    |  40




This is why most tables either have a primary key or a foreign key.

A  primary key  is a column that is Unique (can't contain the same value twice)

and cannot contain Null values.

In MySQL you can define a column as a primary key and it will enforce these rules for you when you
 
attempt to insert rows into the table.

A  foreign key  is a column in a table that REFERENCES a primary key in another

table.  They need not be unique, but they must always reference an existing primary key in another

table.  In MySQL, this is supported only under InnoDB tables, which is NOT what the Nstars Database

uses. (this would be nice to implement)
  See  http://www.mysql.com/doc/en/CREATE_TABLE.html 
for more info on column types and table types.




You can see this information about a table when you DESCRIBE it.
 



The two example tables above are valid tables for this design paradigm if key_a is a primary key and key_b is a foreign key

that references key_a.  I could insert another row into TableB with key_b = 1, but I could not add another 

row to TableA with key_a = 1 (violate uniqueness on primary key)



A LEFT JOIN is useful to join two tables where you know the second (right) table (usually the one with the foreign key)

doesn't have a row for every key in the first (left) table.



"SELECT * from TableA LEFT JOIN TableB on TableA.key_a=TableB.key_b;"


key_a | value_a | key_b | value_b
---------------------------------
  1   |   a     |  1    |  20
  2   |   b     | NULL  | NULL  
  3   |   c     |  3    |  40      



If there is no match in TableB for a key in TableA, then a row is placed with NULLS.

This becomes extremely useful to find out what DOESN'T exist in a table:




mysql> SELECT hip, HD, RA_2000, DEC_2000,  Observatory

FROM NstarsList LEFT JOIN Observations on NstarsList.hip=Observations.hip

WHERE Observations.Observatory IS NULL;




Would give you a list of ALL stars out of the NstarsList (the master list of nearby stars)

that are NOT present in the Observations Table, ie. Not Observed

If you removed the WHERE clause, you would get a complete list of ALL nearby stars, with rows

from NstarsList duplicated for multiple, non unique occurrences of hip #'s in Observations

(ie. Stars observed more than once)


To get a count of how many times a star has been observed, we can  Group By 
the hipparcos # and use the  Function  "count()":




mysql>SELECT hip, HD, count(hip) AS "Number Of Times Observed"

FROM NstarsList LEFT JOIN Observations on NstarsList.hip=Observations.hip

Group By NstarsList.hip;



The "Group By" collapses the table using duplicate values of column(s) specified in the

Group By clause.  The above query uses the complete column name (in the table.column format)

to avoid ambiguity with the 'hip' column in Observations.



count() is one of many functions you can use in Group By clauses.
  
See:

 http://www.mysql.com/doc/en/SELECT.html For a complete description of "Group By" and

 http://www.mysql.com/doc/en/Group_by_functions.html for a list of functions available with "Group By".



There are also a number of functions for string manipulation and other useful utilities:


mysql>SELECT LEFT(RA_2000, 2) AS HOUR, count(*)

FROM NstarsList

Group By LEFT(RA_2000, 2);




This extracts the first 2 characters from the RA to get the hour portion using the LEFT() function

Then groups by the same expression and counts the result rows.

The result is a count of the number of stars in each hour of RA.


 
You can find a complete list of these functions in  http://www.mysql.com/doc/en/Functions.html 

 INSERT/ UPDATE 

To add a new row to an existing table, use  INSERT 




mysql> INSERT into SpectraFiles VALUES (9999, "VEGA", "DSO600", "/600g/mb/VEGA.mb", "99");




will insert one row into the SpectraFiles table.

This would in turn allow the file "/600g/mb/VEGA.mb" to be visible from the web site - 

When someone queries for a star that matches hip 9999.  (a bogus entry) 



This would NOT work if Foreign Key Constraints were enforced - there is no 

hip 9999 in the NstarsList table, which is conceptually what this foreign key references

But since we are not running InnoDB tables and we haven't defined this constraint, the table

will allow us to insert the row



This is part of the manual process of putting a spectra file up on the web site.  Every file

than can be plotted on the "Query By Identifier" section has a row in this table, associating

a hip # with a filename.  The website will query this database to find what files it can plot.



You can also do INSERT...SELECT to select out rows from one table and insert them into another

or " INSERT set [column]=[value], [column=value] " to insert when you don't want to specify every

column in a VALUES() list.

see:  http://www.mysql.com/doc/en/INSERT.html 



 UPDATE  Does a similar action, but instead of inserting a NEW row into a table it changes

the value of a column or columns in an already existing row:




mysql>UPDATE SpectraFiles SET Filename="/600g/mb/hd172167.mb" where Filename = "/600g/mb/VEGA.mb";




Would change the row we just inserted (and all other rows that match the WHERE condition to a proper filename

 (hd number instead of proper name).  This would be the appropriate change to the table if you renamed a pixel file

in /var/SPECTRA 

Exercise caution with UPDATES.  If you forget a WHERE clause, your update will be done to EVERY row in the table

which is probably not what you wanted to do.  Also,  LIMIT  becomes useful here, as it can limit 

the possibility of damage to the table from an UPDATE gone crazy:



mysql>UPDATE SpectraFiles SET Filename="/600g/mb/hd172167.mb" where Filename = "/600g/mb/VEGA.mb" LIMIT 1;




Adding the "LIMIT" would prevent more than one update to the table, even if there was more than one row that met the

WHERE condition.
  This is good practice when doing updates.

see:

 http://www.mysql.com/doc/en/UPDATE.html 



 DELETE  is pretty straight forward:


mysql> DELETE from SpectraFiles where hip=9999 and name="VEGA" LIMIT 1;




This would clean up the bogus entry we placed in with INSERT and practiced with UPDATE.

Good practice in delete is also warranted - notice the "LIMIT 1" which is an extra

precaution in case the WHERE clause matches more than you want.  Also, the key 'hip' is

used in the delete along with the 'name' field - insuring that we are deleting EXACTLY what

we want to.  (The more fields you specify, the safer the delete).



It's also always good to try SELECTing what you are about to DELETE - whatever rows

come up after a SELECT with your WHERE clause will be the exact same set that gets deleted

with an identical WHERE clause.


 
 LOAD DATA INFILE  

This feature will do multiple inserts using a text file on the server's filesystem.
 
you can also do "LOAD DATA LOCAL INFILE" to use a text file on the local filesystem - you're computer

where the client is run from.
  

mysql> LOAD DATA INFILE "mydata.txt" into table "mytable";



will load a tab-delimited, newline ended file of single quoted ('') values.  You can change the

behavior of delimiters and field encapsulators with options to LOAD DATA INFILE.

The complement of LOAD DATA INFILE  is SELECT ... INTO OUTFILE 


A full description of this feature can be found here:
 http://www.mysql.com/doc/en/LOAD_DATA.html 



 Useful Shell Command Line Tools 
The client program, 'mysql' has many other features other than giving you a mysql> prompt

for example, you can specify a query right on the command line with the -e option:


shell% mysql -h stellar.phys.appstate.edu -u [username] -p -D Nstars -e "Select * from NstarsList;"



Will spit the results of the query "SELECT * from NstarsList" right to stdout where you can pipe it

into another process or redirect out to a file.



There are also a number of utilities in mysql's bin directory for other specific uses:


mysqlimport will do the same as LOAD DATA INFILE

mysqldump to dump out table structure and data to file (SELECT ... INTO OUTFILE)

mysqlhotcopy for copying tables 

mysqladmin for administering MySQL from the Unix command line 



Descriptions and more utilities can be found at  http://www.mysql.com/doc/en/Client-Side_Scripts.html 



 Administration

  /usr/bin/mysqladmin  

The program 'mysqladmin' is an extremely useful administration tool that is run from the command line.

the full documentation is at http://www.mysql.com/doc/en/mysqladmin.html

but one should at least be aware of it's existence in this document.  It makes simple administration tasks

quite easy



  User Accounts  


All Accounts for the MySQL database server are described in an SQL database inside MySQL called 'mysql'

viewing this database is useful as a start to see how administration is done:


mysql>use mysql;   #use the mysql database

mysql>SELECT * from user;




Will spit out quite a large table with each user's name, encrypted password, and permissions

they have to read/write or administer databases, tables, and even columns.



Administration can be done by INSERTing directly into this table or other tables in the MySQL

database, or through the use of  GRANT  statements.

see:

 http://www.mysql.com/doc/en/Adding_users.html 

 http://www.mysql.com/doc/en/GRANT.html 




  Starting & Stopping the server 


There are a couple of different ways to start and stop the server... all documented at

MySQL's web site.  The MySQL server should come up on boot, though, if all is OK. 


Manually, way to start it is to use the script 'safe_mysqld' in /usr/bin.  It should be as easy

as just running the script as root to bring up the server, but in extreme circumstances

you may have to pass it some options.  See  http://www.mysql.com/doc/en/safe_mysqld.html 


To bring the server down, see mysqladmin.
  

Although it may be tempting, and the documentation on mysql.com may hint at it, be advised

against sending a kill signal to a mysqld thread.  It has worked for me before, but I

have heard horror stories about corrupted tables and / or servers that won't start back up


Bringing the server up by running mysqld manually is possible though.  See the manual or read

what the script 'safe_mysqld' really does!



 
 Practical Examples Using the Database 

 Inserting new spectra files 

 Inserting Classifications 

 Inserting Derived Parameters 

 Adding Notes for Website spectra 

 Marking Stars for Re-Observation 

 Updating / Correcting data 












Perl - The Glue
Perl is a scripting language that is excellent for string manipulation.

The syntax is pretty readable as-is, although some of the regular expression

matching & substitutions may be unreadable.



The website's scripts are all written in Perl.  All of these scripts live in 

/usr/lib/cgi-bin.

 
 /usr/lib/cgi-bin (interface between Perl and web server)
 
Each one of the Perl scripts present in cgi-bin (and directories within) can be run

straight from the command line.  If you do, you will see that the script simply

prints out html straight to the console.  This is the same output that is sent to 

the client's browser, where it looks much nicer


The interactivity between the web user and the cgi scripts come from html forms

and the ability for Perl to get environmental variables set by apache.

Apache receives the data as name/value pairs via a "POST" (opposed to GET)

where the name/value pairs are actually present in the URL.
 
Apache puts the length of the string of data in an environmental variable called 'CONTENT _LENGTH'

and then puts the string onto stdin for reading

See the apache documentation for more details.

To make this exchange a little easier, there is a function called get_form_info()

in the Perl Function Library that does most of this work for you - returning an 

array of strings in the form "name=value". It is described in more detail below


From the point of view of the web server, all it's doing is reading HTML.  On a form

there will be an ACTION option to the FORM tag which points to a .pl script to pass

the form's data to.  The .pl script itself has within it the ability to logically decide

what to print to the user (in HTML form) based on the input provided from the user, via apache.

The variables passed are defined in the < FORM > < /FORM > block as < INPUT > tags.

Each one has a type, a name, a value, and a size to be defined. The name and value are the 

key elements, as that is what is passed to the cgi scripts beyond. Name must always be defined

but a value is set only if a default value is needed.  Without a value, the field appears

blank on the html form



 
 'use DBI;' (interface between Perl and MySQL) 
Any Perl script that requires access to the MySQL database internally - which is 

most of the scripts - will include the DBI module provided by the Perl user community

(see  http://www.cpan.org  )

DBI stands for DataBase Interface, and is a generic SQL interface that can connect

to many types of DB systems.



Perl scripts requiring this use need only specify 'use DBI;' near the beginning of the file.

This includes a set of new functions for Perl that easily communicate to the SQL server



Here is a quick example of a simple query done from Perl.  It simply prints out a list

of all stars classified in the project as A-type


#!/usr/bin/perl -w
use strict; 

use DBI; #use DBI module

#create connection handle - $dbh (DataBase Handle)
my $dbh = DBD->connect('dbi:mysql:Nstars','myusername','mypassword',{RaiseError => 1 } );

#create a SQL statement handle - $sth (StatemenT Handle) & prepare DB for query
# (let's get a list of all stars that have a spectral type that begins with "A")
my $sth = $dbh->prepare("SELECT hip, name, $GCG_SpecTyp 
                         FROM Classifications 
                         WHERE GCG_SpecTyp like \"A%\";");

#do the query
$sth->execute();

#the result is now available - the database server is holding on to it until we free it.

#first, create some variables to hold the return
#variable names can be anything, but I picked the same names as the column names for readability
my $hip, $name, $GCG_SpecTyp;

#tell the module to bind these variables to columns of the result.
# notice they are passed by-reference to bind_columns()
$sth->bind_columns(\$hip, \$name, \$GCG_SpecTyp);

#print the result - using a loop and fetch() function
while($sth->fetch()){
  print "$hip   $name    $GCG_SpecTyp\n";
}


#finish the query - free DB resources
$sth->finish();

#disconnect from database
$dbh->disconnect();




There are many other ways to do this same thing - as the motto of Perl is "There is more than
 one way to do it!"


I used  http://www.mysql.com/doc/en/Perl_DBI_Class.html  as my reference - but others exist. 
 Check out  http://dbi.perl.org/doc/index.html  also.





 
 Nstars Perl Function Library 

As Projects evolve, so do the commonly used pieces of code.  A rudimentary start has been 

made to create a library of frequently used Perl functions associated with the Nstars database.



Currently, this isn't the slickest form of a Perl library.  It's called Nstars_lib.pl and lives
 in /cgi-bin/lib/ . It is used in many scripts in cgi-bin, to use it, simply place



Require "/Nstars_lib.pl";



near the top of your script, and precede any calls to subroutines within with a '&'.  See

specdisplay.pl for an example.



This began with the rewrite of the specdisplay.pl script in /cgi-bin/spectra.  This is a 

beast of a script, and I functionalized many tasks within it.  Noticing that these functions

were likely to be needed on other scripts, I began the placing them in a file called  
lib_of_subs.pl  

which is currently in /home/flip/NstarsPerl_Subroutines/



Recently, it has evolved into a more dynamic library that can be included.  So a change

made to a subroutine the library will in turn affect every script on the website that uses

that subroutine.

Future users of this are encouraged to add new routines or improve existing ones.

Be aware though that changing return values or input parameters of existing functions may create conflicts and break already existing code.  Be sure you understand how an existing

subroutine is called, run, and every place that it is called from before changing these

properties.  Optimizing or rewriting internal code should not be a problem, as long

as the parameters and return are left unchanged



Overview of functions provided in lib_of_subs.pl:


 print_htm_header()
 simply prints the required header info to begin printing to a browser... Without it

Perl code will be printed to the browser instead of the printed html

 get_form_info()
 determines method of sending form information (POST or GET) and populates an array

of strings of "name=value" format.  The resultant array is returned;

 connect_readonly_to_database();
 function specific to connecting to MySQL Nstars Database as user 'webuser' - useful

for scripts that require only read access to the db and are called as cgi scripts


 connect_readwrite_to_database();
 function for connecting to the database as a privelaged user.  Right now, the script

has a hard-coded username of 'flip' - but it's a good example of how to prompt for a password
.  This function requires some more work to accept a username and turn off local terminal echo
 when providing a password


 get_hip()
 An extremely useful function to get the hipparcos # given a search string of a starname.

it intelegently determines what catalog the starname belongs to (hd123 = hd catalog, etc)

then searches the Starnames table through the already defined database handle.  If there are
 more than zero matches, it populates an array of hip #'s that are matches to the search string, but only returns the first match. This can be modified.
  
Returns -1 if no match found

For example, this piece of code:


.
.
# connect to database and such
.
.
my $hip = get_hip("Vega");
print $hip;
.
.



Would output:

91262


get_nstar_name()
 In a sence, does the reverse of get_hip.  Given a hip #, it finds the appropriate

friendly starname to be printed.  A "friendly" starname is defined as such:

If there is an HD number - use that

If not, use the BD designation

If BD not available, use the hip #

This also does a check to make sure the hip # provided is a nearby star, if it's not

it does not define the return.  use the 'defined()' function in perl to check for

an empty result
 (This needs to change to something a bit more standard)


get_notes()
Given a hip #, simply prints out any notes associated with the star.  Notes are held

in there own table in the Nstars database.  These are the same notes that are printed

below a plotted spectrum on the spectra page.


is_observed()
 Given a hip #, returns the number of observations of the star.

Usuially used for a boolean test to see if a star has been observed or not, but can

be used to get a count also


get_specfiles()
 Given a hip #, this returns a list of the filenames of all pixel files associated

with an observed and reduced spectrum.  It returns an empty list if there are no 

spectra files for the star

This is how specdisplay.pl determines what pixel files to show to the user when

a query is done from the spectra page.

The table, SpectraFiles in MySQL is where this info is stored.  There is also

a priority field in the table which describes the order in which the spectra files

are displayed on the page.  The file with the lowest priority is displayed first - 

Notice the "order by priority asc" in the query within the function itself. 


get_GCG_classification()
given a hip #, returns the GCG_spec_type stored in the Classificatons table - the NEW classification

get_hip_classification() 
given a hip #, returns the hipparcos classification from the NstarsList table - the OLD classification

get_parallax()
given a hip #, returns a list containing 2 elements - the parallax and the parallax error
 from the hipparcos catalog

get_photometry()
given a hip #, returns a list containing photometric data on the star in this order:

0 - hipparcos V

1 - hipparcos B-V

2 - v  (stromgren)

3 - b-y(stromgren) 

4 - m1 (stromgren)

5 - c1 (stromgren)

6 - H-beta 

7 - Johnson V

8 - Johnson U-B

9 - Johnson B-V

10 - Johnson V-R

11 - Johnson V-I

12 - Johnson R-I

13 - Johnson J

14 - Johnson H

15 - Johnson K

16 - Cousin V-R

17 - Cousin V-I

18 - Cousin R-I


 get_simplex_data()
given a hip #, returns a 2D list of simplex data currently in the list.

The array is 2D because for one hip #, there may be multuple stars of a double system

and each may have it's own data OR the star was observed at multuple observatories
 
and simplex values have been determined for each observation separately

Most of the time, it will only be one list wide,but propper use of this function should

check the entire result set

Remember, when passing arrays of arrays in Perl, it must be done by reference.

see specdisplay.pl for an example on how this is done and the result is traversed

The lists returned are in this format:

0 - Observatory

1 - T_eff   -effective tempature

2 - log_g   -log surface gravity

3 - M/H     -metallicity index

4 - V_t     -microturbulent velocity

5 - Fit_Qual -quality of simplex fit


get_ChromAct_data()
given a hip #, returns a 2D list of Derived Chromospheric Activity data.

much like get_simplex_data(), there may be more than one row for a given hip # 

and the function will return an array of arrays for this reason.

The lists returned are in this format:

0 - Observatory

1 - S_mw - S - value on Mount Wilson Chromospheric Activity scale

2 - log_R_HK - emission strength of K and H lines

3 - ChromActClass - Chromospheric Activity Class (active, inactive, etc..)


make_spectra_gif()
 This function uses ploticus to create a gif image of a spectrum plot

given a filename and a common name

The common name is used as a title on the image.  The filename is the partial-path

filename of the pixel file to plot - the same as what is returned from get_spectra_file()

This function creates the ploticus 'scriptfile' that drives the creation of the 

gif image and then does a call to ploticus to create the file.

It returns the filename of the new gif file created.

This function has only been implemented in specdisplay.pl, so it is still not quite

as general of a function as it should be. It currently is hard-coded to place created

gif spectra files in /htdocs/tmp and removes any .gif files that were there already

A little bit of work will make this into a nice, general function


make_spectra_jpeg()
 Something broke, I can't remember exactly what it was, but gif files no longer

worked after some upgrade I made or such.... No matter... ploticus can make jpeg's too!

And so this function was created.  Does the same as make_spectra_gif - but creates

a jpeg and returns the filename.


 get_date()
 Given a pixel list - passed as an array reference, returns the date in the 

footer of the pixel list in ISO 8061 format (yyyy-mm-dd) - no matter what form 

the date is within the actual footer of the pixel list.

The pixel lists are passed as arrays - by reference, not as a filename or a filehandle.

to prepare a pixel list for this function, simply OPEN the pixel list and assign 

an array to the handle:


  

open FILEHANDLE, "my_pixel_list.hb";

my @pixellist = < FILEHANDLE>

my $date = get_date(@pixellist);



 get_airmass()
 Just like get_date() the input is a reference to a pixel list array.

This function finds the airmass comment in the pixel list footer and returns it 

as a scalar


 get_observations()
 This function needs work - returns a list of observations for a hip #/

The return type is scalar - a single string - in the form

 "observatory|notes|airmass/observatory|notes|airmass

where '|' is a field delimeter and "/" is a line delimiter

This return type should be fixed to be a list of lists - each list being one observation.


 specfile_exsists()
 Returns a count of howmany spectra files are available for a hip #

Also useful as a boolean check to see if spectra files exsist.







 Data Archiving and Backup
 

Weekly Backup

On the webserver, there is a cron job that runs once a week to backup

all the Nstars data and webserver to a secure machine inside the network.

The script that runs is /etc/cron.weekly/Nstars_Backup.sh and it:

1) Copies, tar, and gzips database files.

2) tar and gzips spectra files.

3) tar and gzips website (/var/www/* /usr/lib/cgi-bin/* /var/log/apache*)

4) ftp all this data to barcud.phys.appstate.edu//usr/local/stellar_backup/



There are three types of files you will find on barcud after a backup:

databases.[date].tar.gz

spectra.[date].tar.gz

website.[date].tar.gz



where [date] is the date when the backup was created.



Restoring from Weekly Backup
In the worst case of a complete system failure, a new piece of hardware - preferably

running the same operating system as the failed one - could be built, and after

the install of the neccessary programs (apache, MySQL, ploticus, Perl, PHP, etc..)

all the tar files should be able to be extracted directly onto the new filesystem

and all data and funcitonality would be restored.  There would most definately

be loose ends and broken parts of the website, or missing symbolic links,

but all data pretaining to the project would be saved.



The more appropriate way to restore would be to get the appropriate file from the 

backup archives, unpack it, and overwrite any broken or corrupted data with the backup

data from the archive.

This is expecially useful for MySQL database data.  If the database happens to be corrupted

for any reason, the .MYI, .MYD and .frm files in /var/lib/mysql/Nstars can be overwritten

by the contents of the databases.[date].tar.gz file.  To do this, simply shutdown the MySQL

server, copy the files over you want to restore (each table has it's own backup), and restart
 the server.  All should be restored just fine.







 Written by Mike McFadden 

 May 6, 2003