	SDAST Weekly Chat #12: Friday 19th February 1999

Participants:

	Peter Kretschmar, Allan Hornstrup, Sami Maisala, Stefan Larsson,
	Carol Anne Oxborrow.

Subjects discussed:
	OSM: our wishes for parameters to be monitored
		- FRSS Monitoring
		- automatic vs. human monitoring
	Grey Filter Changes and Standard Products
	DAL3 - briefly
	
Action Items: None.

*** IRC session log started at Fri Feb 19 09:25:49 1999.
H allan@       #jemxadr 0 Allan Hornstrup                ~allan@apollo.dsri.dk
H oxborrow     #jemxadr 0 Carol Anne Oxborrow          ~oxborrow@tenma.dsri.dk
315 apollo.dsri.dk * End of /WHO list.      
#jemxadr> Hi Allan! END
<allan:#jemxadr> Hi -- you didn't join the breakfast club today?
<allan:#jemxadr> end
*** larsson <~larsson@tenma.dsri.dk> joined channel #jemxadr
#jemxadr> I always end up doing last minute things for this chat meeting, but I'm thi
+nking of becoming a breakfast club member too. END
#jemxadr> Hello Stefan. How's the weather in Stockholm? END
<larsson:#jemxadr> Morning there. END
<larsson:#jemxadr> Its cold. END
<larsson:#jemxadr> (On the other hand I slept on my office so I havnt been out today)
+ .END
#jemxadr> We're hovering around +-1--2 degrees. END
#jemxadr> You have to sleep in your office???? ARe you snowed in?? END
<larsson:#jemxadr> Not realy I gave a lecture late last evening.
<larsson:#jemxadr> I am awake now anyway. END
#jemxadr> Are you a member of a breakfast club Stefan? Sounds like you could really d
+o with it! END
<larsson:#jemxadr> I have had some coffee. As you remind me I think I am going to get
+ another one. END
*** Peter <~pkretsch@isdcul10.unige.ch> joined channel #jemxadr
<Peter:#jemxadr> Good morning everybody!
#jemxadr> Morning Peter! any snow in Versoix? END
<Peter:#jemxadr> Sorry for being slightly late - I'm just writing an email for you al
+l. END
<allan:#jemxadr> You could send the information through this channel, if that's easie
+r?
<allan:#jemxadr> END
<Peter:#jemxadr> Snow, yes still some. But unfortunbately the weather has turned rain
+y and warmer again ... END
<Peter:#jemxadr> Its almost finished and contains an attchment - its about DAL3. Give
+ me 30 seconds. Its noit long anyway.
<larsson:#jemxadr> I would'nt complain if it turned a little warmer (say 20-30 degree
+s) END
#jemxadr> I can't find NJW - I swear these men hide from me when they here me coming.
+ ENd
<Peter:#jemxadr> OK, the mail is out. 
<Peter:#jemxadr> Stephane has written a brief dal3jemx.h and wants from us feedback u
+ntil he's returning from his holidays. END
#jemxadr> Okay, the header looks good as a start. 
*** smaisala <~smaisala@taurus.astro.helsinki.fi> joined channel #jemxadr
#jemxadr> I've got some other suggestions, but I think we should save thse till the F
+inns are with us. END
#jemxadr> Talk of the devil......END
#jemxadr> Hello Sami, is Juhan joining us too? END
<smaisala:#jemxadr> Good morning. Sorry I'm late. Juhani is so busy that he is not wi
+th us today END
#jemxadr> So let's begin. First Allan, could you tell us briefly about OSM? END
<allan:#jemxadr> I have an action to provide a revised list of
<allan:#jemxadr> OSM parameters to monitor. We have discussed the issue
<allan:#jemxadr> and have a few request to add to the previous one
<allan:#jemxadr> presented at the SDAST#7 meeting.
<allan:#jemxadr> (And I should add, that the monitoring of the FRSS is still a reques
+t
<Peter:#jemxadr> Will you send the updated list to me via mail? END
<allan:#jemxadr> although I may have said differently sometime in thje past)
<allan:#jemxadr> The list just adds:
<allan:#jemxadr> - auxiliary: 
<allan:#jemxadr> add monitoring of the validity flag.
<allan:#jemxadr> -science data:
<allan:#jemxadr> add monitoring of header information of telemetry format changes (th
+is
<allan:#jemxadr> then dissappears from the HK data set, since it is actually not ther
+e anymore).
<allan:#jemxadr> add monitoring of greyfilter changes.
<allan:#jemxadr> I think that's it. END
#jemxadr> I Agree we should monitor the FRSS: I've been doing the fitting of these sp
+ectra and trying
<Peter:#jemxadr> Being intrinsically lazy, I'd still like to get a complete checklist
+ per mail - would you do that? END
<allan:#jemxadr> Well, I may :-) END
<allan:#jemxadr> Should we put the list on teh WWW, so that we have a simple referenc
+eEND
#jemxadr> to find ways to detect and signal dubious changes in the spectra, and the f
+itted parameters, and I think that actual visual inspection of the spectra is quite 
+important. END
<allan:#jemxadr> Right, note, though, that there is no
<allan:#jemxadr> fitting in the OSM, just monitoring of the mean rates, channel posit
+ions
<allan:#jemxadr> and widths of the 2 lines. END
#jemxadr> Yes, that's as much as we need: my routines do the fitting, and that provid
+es additional data. END
<Peter:#jemxadr> If I remember correctly, CA's software produces all the numbers and 
+RR's just checks those numbers against limits. END
#jemxadr> But I think OSM should have the option to LOOK at the spectra: check their 
+shape once in a while. END
<allan:#jemxadr> Isn't the OSM before CA's SW? END
#jemxadr> Yes, OSM should be way before our processing. But as a first warning sign, 
+odd behaviour in the calibration spectra is probably the best indication that someth
+ing's wrong. Afterall we know quite accurately what these signals should look lik
#jemxadr> e, and we have one spectrum for each part of the detector. END
<Peter:#jemxadr> Please remember that OSM is mostly an automatic process.
#jemxadr> Perhaps I'm thinking rather pessimistically, of major disasters here rather
+ than day to day running. END
<Peter:#jemxadr> As such to give alerts, it must mostly rely on numbers.
<allan:#jemxadr> Hey, lets back up:
<allan:#jemxadr> The OSM will ONLY check the FRSS with respect to 
<allan:#jemxadr> mean rates, channel positions and widths (FWHM, simply gotten from
<Peter:#jemxadr> There is no intrinsic problem in displaying the last calibration spe
+ctra all the time, but there is no guarantee that a human being will actually look a
+t them. END
<allan:#jemxadr> the data). These numbers are checked against predefined limits.
<allan:#jemxadr> No fancy SW goes before taht.  As far as I read the ADR for the OSM,
+ 
<allan:#jemxadr> there will be an option to do interactive monitoring, but
<allan:#jemxadr> that will only be needed if the numbers are close to the limits.
<allan:#jemxadr> We may discuss with RR of using a set of yellow numbers besides the
<allan:#jemxadr> set of red ones, to allow for non-emergency warnings.  I expoect
<allan:#jemxadr> further, that the instrument teams may be able to investigate e.g. t
+he
<allan:#jemxadr> long-term behaviour of a given HK parameter, say
<allan:#jemxadr> END
#jemxadr> So long as there's an option for a human operator to take a glance and comp
+are it with some printout of what a calibration spectrum should look like, I thinks 
+that's all we need. END
<Peter:#jemxadr> Let me please sort this out:
#jemxadr> Is that all on OSM? END
<Peter:#jemxadr> No, I'm just trying to clarify some matters
<Peter:#jemxadr> Allan is right in all he's saying. I'd just like to add that the num
+bers Reiner's software will use _are to be generated_ to a large part by the ISSW.
<Peter:#jemxadr> This can happen in DataPreparation and in Automatic Calibration Anal
+ysis, which both come in our processing _before_ OSM.
<Peter:#jemxadr> As far as I know, our architecture allows for this, assuming we have
+ defined all the data structures correctly.
<Peter:#jemxadr> It means we should take a good look at the list and at our design an
+d fill in any gaps. 
<allan:#jemxadr> I admit, that I haven't been sufficiently aware of that. I have to 
<allan:#jemxadr> think of the consequenses.
#jemxadr> In that case, monitoring of the fitted calibration parameters should be inc
+luded in the FRSS monitoring. END
<allan:#jemxadr> END
<allan:#jemxadr> ( Sorry to interrupt you peter, END)
#jemxadr> How is OSM different from JPerformance then ? END
<Peter:#jemxadr> In addition to the number crunching/ limit checking Reiner can and w
+ill display 'useful' displays, so in case of alerts the operators/scientists have so
+mething to look at immediately.
<Peter:#jemxadr> The content of those displays is rather free, and it might actually 
+be a good idea to continously display the FRSS spectra. We just should not rely on a
+ human reaction. END
<Peter:#jemxadr> OSM and JPerformance have a lot in common (I'm stating this since lo
+ng ago).
<allan:#jemxadr> - and we are not going to do anything in the performance.,.. that is
+ 
<allan:#jemxadr> being done in the OSM.
<Peter:#jemxadr> The main difference is that automatic OSM is really geared at automa
+tic, science window by science window processing.
<allan:#jemxadr> Let me have a further clarification:  If we say to RR, that we need 
+the
<allan:#jemxadr> 3 values from the science data (FRSS), and provide him with the 
<Peter:#jemxadr> The software Reiner has been developing for Interactive OSM (the  on
+e started in casse of alerts) would probably cover >90% of our JPerformance needs.
<allan:#jemxadr> positions of the sources will he then extract the information
<allan:#jemxadr> from the DAL and give the numbers we need, or do we 
<allan:#jemxadr> need to enter with some SW to extract the numbers for him?
<allan:#jemxadr> END
<Peter:#jemxadr> The latter, usually. 
#jemxadr> My routines will produce an alert if the calibration parameters seem odd in
+ any way. This could be a sign for OSM people to take a look at teh spectra themselv
+es. END
<allan:#jemxadr> There seem to be too much confusion here. We have to discuss this in
<allan:#jemxadr> real life. I.e. during the next meeting, or here internally and then
<allan:#jemxadr> provide a list of questions to the ISDC .END
<Peter:#jemxadr> You know, I;m slightly frustrated - I have been telling these things
+ several times already. END
#jemxadr> I think we should talk about this Allan. END
<Peter:#jemxadr> I'll give it another try and write an email later as basis for discu
+ssions - OK? END
<allan:#jemxadr> We may have a phone conversation. Please in that case have 
<allan:#jemxadr> our ADD at hand, so that you can explain to us where your previous
<allan:#jemxadr> statements may be reflected. END
<Peter:#jemxadr> Fine with me. END
<allan:#jemxadr> OK, on to the next?
<allan:#jemxadr> END
#jemxadr> Allan again: how to handle grey filter changes in a science window. END
<allan:#jemxadr> I dont have much information here, I just wanted us to discuss
<allan:#jemxadr> what we want to do in terms of standard products.  AS you recall
<allan:#jemxadr> we have agreed to provide a set of data products, where the 
<allan:#jemxadr> observer gets all data reduced to a format with the least 
<allan:#jemxadr> information of the telemetry formats used for that particular 
<allan:#jemxadr> observation.  But what do we do with possible greyfilter changes.
<allan:#jemxadr> I dont think we want to provide more than two sets (the total, or ra
+w)
<allan:#jemxadr> and the reduced (complete).  Thus, the greyfilter changes will also
<allan:#jemxadr> be present in the reduced - OK?
<allan:#jemxadr>  END
<Peter:#jemxadr> They have to, everywhere where we give fluxes, they should be taken 
+into account. END
#jemxadr> I though a little bit about this yesterday, and I agree with you that corre
+cting for grey filter for all the events in a window might be the wrong thing to do.
+ But where source fluxes and lightcurves are calculated, obviously there has to b
#jemxadr> ae corrected for grey fileter changes. Hence the standard products should b
+e defined in that way: event (not corrected for GF), list of source fluxes corrected
+ for GF and lightcurves corrected for GF. Then we let the astronomers themselves
#jemxadr> do what they want with the GF values: possibly with the help of a DAL3 rout
+ine to make corrections to event data in a science window (if that's possible). END
<allan:#jemxadr> (please remember to use CR when you write, it shows that the line is
+ busy) END
#jemxadr> Okay END
<Peter:#jemxadr> SP and me propose to generate a new datastructure called -GREY-STA a
+s part of DataPreparation, see the draft I sent around recently.
<Peter:#jemxadr> This data structure should contain _all_ grey filter changes found i
+n the TM. A similar table would be set up for the modes.
<Peter:#jemxadr> Incidentially these files could probably be used by OSM for some of 
+the tasks Allan mentioned. END
<allan:#jemxadr> We are rather looking at 
<allan:#jemxadr> what the observer gets.
<allan:#jemxadr> He is getting two sets of files and an explanation, as
<allan:#jemxadr> far as I recall the agreements.
<Peter:#jemxadr> In principle, he gets _everything_ the standard pipelines produce. E
+ND
<allan:#jemxadr> The one file contains the actual data, i.e. with all changes
<allan:#jemxadr> and everything. As a curtesy, we also provide a sat of data, 
<Peter:#jemxadr> Sorry that should have been, everything that's produced and archived
+. END
<allan:#jemxadr> where e.g. full event data are reduced to restricted imaging,
<allan:#jemxadr> if these were the two modes in question.  
<allan:#jemxadr> The problem is, do we further reduce the latter, so that
<allan:#jemxadr> the worst greyfiltering will be used throughout the (reduced) data s
+et.
<allan:#jemxadr> I agreee, that
<Peter:#jemxadr> D**m, I just realize that there's another executable for me to write
+ after the data structure changes ... END
<allan:#jemxadr> the observer of course do get all the other information - otherwise
<allan:#jemxadr> he cannot do the detailed analysis...END
<allan:#jemxadr> I think we should keep the greyfilter changes in the reduced dataEND
<Peter:#jemxadr> I'm slightly confused by the current discussion.
<larsson:#jemxadr> Gray filters should not be treated as mode changes to reduce
<Peter:#jemxadr> The only thing that seems clear to me is that we want to have the an
+alysis data products (images, fluxes, spectra, lightcurves)
<larsson:#jemxadr> data to "worst case".END
<Peter:#jemxadr> to be calculated taking grey filter changes into account.
<Peter:#jemxadr> (and if we didn't we'd produce some seriously wrong results occasion
+ally)
<Peter:#jemxadr> Also we seem to agree that in all previous steps, especially JCorrec
+tion
<Peter:#jemxadr> We do _not_ try to 'correct' for grey filter changes. Does everybody
+ agree with me so far? END
#jemxadr> I agree with Allan: it allows greater flexibility. And the researcher could
+ be given access to a routine to reduce everything to the heaviest filtering in a wi
+ndow if necessary. END
#jemxadr> Yes, I also agree with you Peter. END
<allan:#jemxadr> Peter, your list of data products is not complete. You are missing t
+he
<larsson:#jemxadr> Maybe with one exception.
<allan:#jemxadr> actual files, that the observer gets with the data imbedded. He is n
+ot
<allan:#jemxadr> supposed to use DAL - I know this has been debated a lot of times at
+ 
<allan:#jemxadr> the ISDC, but my impression is, that the observer CAN request a FITS
<larsson:#jemxadr> In case of final lightcurves we might give it normalized with erro
+r estimates?
<allan:#jemxadr> file with the data. END
<larsson:#jemxadr> END
<Peter:#jemxadr> Maybe I'm completely confused. My impression was that the observer g
+ets a directory tree of stuff, essentially a slice out of aur Archive filled with al
+l the myriad files prodouced. END
<allan:#jemxadr> Yes, and these are all FITS files, right?  In teh JEMX case includin
+g 
<allan:#jemxadr> the file with all the observed events, and a file where the full eve
+nts
<Peter:#jemxadr> Within this tree there is a place for the data products, which are s
+tored as FITS extensions. END
<allan:#jemxadr> have been reduced to restricted imaging formatsEND
<Peter:#jemxadr> We would probably have one file per JEM-X TM format, plus additional
+ files for the 'converted' data. END
<allan:#jemxadr> I have another meeting in two minutes.  I believe I have to
<allan:#jemxadr> look up my old documents to find an explanation of what the user act
+ually gets. 
<allan:#jemxadr> (If such exist). END
<larsson:#jemxadr> I also have to leave soon (2min). END
#jemxadr> I think we should call it a day now. END
<larsson:#jemxadr> I just have a comment to Peter regarding the
#jemxadr> Please would everyone circulate their wish lists for DAL3 and ISSW school b
+y email. END
<larsson:#jemxadr> data dictionary.
<larsson:#jemxadr> You say in theere: "JMXi-EVNT-WHT (obsolete)". My use of event wei
+ghts have not changed
<larsson:#jemxadr> so this data structure should remain. END.
<Peter:#jemxadr> Ooops, you're right there - sorry!
<Peter:#jemxadr> So this data structure is written and used by JSourceDataExtraction,
+ right? END
<larsson:#jemxadr> Yes.
<larsson:#jemxadr> END
<allan:#jemxadr> Peter, are you in all day?  I think that we after lunch could clarif
+y a few issues
<allan:#jemxadr> over the phone? END
<Peter:#jemxadr> Fine with me. shall we say 2pm?
<Peter:#jemxadr> END
<allan:#jemxadr> CA? END
<larsson:#jemxadr> Bye, bye everyone. END
*** Signoff: larsson ()
#jemxadr> 2pm it is. END
<allan:#jemxadr> OK END
*** Signoff: allan ()
<Peter:#jemxadr> OK. Talk to you later. END
#jemxadr> Have a good week end Sami. END
*** Error: Closing Link: oxborrow[~oxborrow@tenma.dsri.dk] () 
*** IRC session log ended at Fri Feb 19 10:35:30 1999.
