
	Data Structures Chat meeting #2: Wednesday 8th December 1999

Participants: Peter Kretschmar, Stefan Larsson, Niels Joergen Westergaard,
	Allan Hornstrup, Carol Anne Oxborrow, Sami Maisala, Eero Esko

Subjects covered: V2 component status: monthly reporting
		OGIP standardized column names for FITS extension headers
		PHA II FITS data format
		Corrections to energy binning
		COUNTS versus COUNTRATES
		FITS storage of background handling data structures
		GTI's and event evaluation: OSM
		Individual data structure names: ISDC approval
		Delivery of new data structure templates

Action Items:

AI991208_1	ALL 	Ongoing Send CAO complete component status summary
at the end of each month.
OPEN

AI991208_2	PK,NJW,SL,JH 10/01/2000 How do we want to store the 
background data structures in terms of fits extension(s)???
OPEN

AI991208_3	PK	10/01/2000	Make a presentation of Reiner's OSM
capabilities especially where GTI's are concerned, for the January meeting
OPEN

AI991208_4	PK, SL, NJW 10/01/2000 Determine what the later ISSW processing
needs by GTI and event evaluation information.
OPEN.

AI991208_5 	PK 	10/01/2000	Find an ISDC approved name for the
consolidated GTI data structure and other data structure names mentioned in
SL's summary of 8/12/99.
OPEN


**** Logging Started : Wed Dec 08 10:03:36 MET 1999
<allan> Could we start with V2 status? END
<Peter> Ooops that went in parallel :-) END
<Peter> Fine with me, who starts? END
> Hello, everyone! END
<smaisala> Hi there END
<larsson> Please wait a sec. My screen only shows one line at a time.  END
> Allan and I are using zircon, and it seems quite nice END
*** njw (~njw@heao1.dsri.dk) has joined channel #jemxadr
> I'll go and round up NJW END
*** Signoff: larsson ()
<njw> hi
<Peter> Not necessary anymore :-) END
> ......too late! END
*** larsson (~larsson@tenma.dsri.dk) has joined channel #jemxadr
<larsson> Hi again.. hope this works better. END
<larsson> It does. END
<allan> We can at least 'see' you END
> Perhaps Allan and I will send you all some diagrams later, if you're all goodEND
<larsson> Now I can even see my self. (!)END
<Peter> OK, let's get rolling - Allan wanted to have V2 status
> Okay, let's get started. V2 status: END
<Peter> For my part it's the same as last week:
<Peter> jspectimebinning executable written to ~80% others still outstanding. END
> My V2 Status: JCreateStatusTable will be delivered by Christmas
<njw> I must report same status as last handed over to CAO END
> JSciAdcCorr could also be delivered a week or two after that
<smaisala> For me it is still quite mess. I have to redesign my background handling and it take time END
> All other corrections could be delivered in FIRST form by end of January, but
*** eesko (~eesko@aquarius.astro.helsinki.fi) has joined channel #jemxadr
> with the proviso that quite a lot will change after talks with the IT. END
<larsson> My JSourceDataExtraction is about 55% as estimated last week. I looked
<eesko> Soory I'm late END
<larsson> through the datastructures again and just sent you a mail on that.
<larsson> I have invented one new data structure. END
> Could you send the rest of us copies of that email, Stefan? END
<larsson> I did just before we started. END
> Thanks. END
> As a general comment about V2 status: could you all please get in the habit
<allan> (It hasn't gone here, though)END
> of sending me a complete status list at the end of every month.
> You could keep just one list in a file and update that everytime you did
> some coding, then send the file as an attachment to me the last 
<larsson> RE MY MAIL. It did bounce for NJW and Allan! (on Apollo). I'll try again.END
> working day of each month. It gets a bit frustrating to have roland
<allan> (TO Stefan, never use anything else but @dsri.dk) END
> asking for this information and me never hearing anything.
> Could we make that an AI: everyone to send component status at the end of
> EVERY month? END
<Peter> OK with me. END
<allan> Y ;-) END
<smaisala> Y END
<njw> Stefan, try simply:  njw@dsri.dk  END
<njw> Y
<larsson> NJW and Allan: You should have it by now.
> I've got your email now Stefan. END
<allan> Stefan: THanks! END
<larsson> RE mail adresses, the same goes for me @idun.astro.su.se => @astro.su.se
<njw> Yes, I've got the mail END
<Peter> But I have nothing yet :-( END
> What about email addresses in Helsinki? I notice that the names of your servers
> seem to change pretty often. END
<larsson> Sorry for mail probl. Just sent one to Peter too. Any one else missing it?
<larsson> END
<smaisala> I think you get the best result using syntax name@astro.helsinki.fi END
<Peter> Now I've got ith - thanks! END
<larsson> You are welcome Peter, END
> Re your email Stefan: the grey/mode history table will be called JMXi-STAT-HIS
> and it will contain the following columns:
<larsson> CA: Thanks I will modify that. END
> OBT(4u, as per ISDC format for OBT), GREY_FILTER (1B) and MODE (1B) 
> I hope this fits in with ISDC's plans, and column-naming system, Peter.END
> I'll send you all a sample template now. END
<Peter> OBT should be OB_TIME
<Peter> This brings me to another point I forgot in may mail
<Peter> A few days several of us here at ISDC sat down and looked 
<Peter> at the long list of existing column and keyword names in our data structures
<Peter> We found that
<Peter> (a) Quite often names do not follow OGIP conventions
<Peter> (b) We have both cases of "same name for different things" 
<Peter>     (in data structures that might be accessed together, otherwise it's no problem)
<Peter>    and "different name for same thing" (for generic information)
<Peter> The conclusion was that we would like to make one attempt at
<Peter> sorting this out a little, because it's probably the last  chance
<Peter> we will have to do so.
<Peter> But this means that many templates will change and that any
<Peter> ISSW using those must adapt too.
> I would welcome a standard list of column names for things. Now is probably the
> best time to change these things, if we're going to do it at all.
<Peter> We hope we can make the change in January - so if you just withold your deliveries until February you could still adapt - I hope END
> Can we put and AI on you peter to supply a list of standardised column names
> for fits extensions. If so when would it be due? END
<Peter> Yes, but I'm not 100% sure if I will have it for our next meeting.
<Peter> The problem is people being completely tied up right now and in holidays afterwards. END
> Let's say mid Jan 2000 (it sounds so far away!) END
<Peter> OK. I'll try to collect as much as possible by then. END
> There are I'm sure some VERY basic quantities for which some standard
<Peter> Btw, looking at the status histroy template, I just realized that MODE
> name has slowly crystalised, for example OB_TIME rather than OBT.
<Peter> should probably be called DATAMODE or something like that.
<Peter> Someone else than me is looking into the OGIP standards. END
> Could you make a preliminary list of this sort in a couple of weeks? END
> I'll change MODE to DATAMODE END
<Peter> Yes, I will definitely come with some list for the January meeting. END
<smaisala> How about JMXi-MODE ??? END
<Peter> Let's keep MODE for the moment and wait for the recommendation from OGIP to be found by my colleagues END
> That's fine. I can easily change my code to whatever column names you want END
<Peter> Shall we move on? END
<Peter> To CAO: thanks! There are some people in this project who'd claim it cost them another month because of such changes ... END
> Then their programs must be very badly organised.....END
> Lets get on with the questions in Peter's email. do you all have that? END
<Peter> OK, let's move on. END
<njw> Y
<smaisala> Y END
> Could you describe this PHAII format, Peter? END
<Peter> This is a very generic format to store spectral data:
<Peter> The central features are that you have one table grouping multiple spectra
<Peter> where the actual channel-number and count(rate) information
<Peter> is contained in vector columns. 
<Peter> Each row of the table contains one spectrum, with no specific
<Peter> constraint on how these different spectra come about.
<Peter> So one could have one row per source for our case,
<larsson> http://heasarc.gsfc.nasa.gov/docs/heasarc/ofwg/docs/spectra/ogip_92_007/node9.html
<Peter> or for a single source many time- or pulse-phase-resolved
<larsson> Will give you the details. END
<Peter> spectra.
<Peter> Thanks Stefan, I was about to look up my bookmarks. END
> So this would be for data that has undergone a fair degree of processing:
<Peter> A description string identifies the individual spectra. Further columns
> being sorted by source and so on? END
<Peter> can hold specific information (e.g. PHASE).
<larsson> (It is yours becouse I got it from you Peter). END
<Peter> Yes, this is for the end products of our analysis S/W. END
<Peter> Side note: these end products are _NOT_ necessarily grouped anymore in SWGs or OGs. 
<Peter> The grouping we have makes a lot of sense, when we have lots
<Peter> of different data belonging logically together. 
<Peter> But at the end of an analysis (especially if its interactive)
<Peter> you may have the opposite problem: having multiple 
> In that case I'd like to leave this discussion to those of you further down the
<larsson> I agree with Peters suggestion of using PHA II. (Actually I like it a lot!) END.
> pipeline than I am. However I'd just like to point out that the way things
<Peter> instances of essentially the same data but obtained on different paths. END
> work in JCorrection JGainCalculation, the nominal telemetry energy bin numbers
<njw> The idea of having spectra from several sources in the same data structure seems to be a good one END
> (integers) get corrected for any ADC non-linearity by changing the bin
> boundaries, which thenceforth are floating point numbers. This means that if
> you're in nominal bin 3 you need to look up the bin boundaries in
> JMXi-SCIB-CAL for bin 3 to find you what actual extent this bin covers. END
<Peter> This brings me to a good question: 
<Peter> Does this mean that corrected spectra from different ScWs will have
<Peter> non-aligned channels? END
> No, it's expected that the bin-correction table will only be updated once or
> twice during the mission. Currently the test corrections are all zero. THis
> solution is only implemented if some damage occurs to the ADC that makes it
> very non-linear. END
<Peter> Phew! Shouldn't we label the structure -MOD then? 
> Truly corrected events are stored by PI channels that by definition overlap END
> Names are up to ISDC - I used CAL only to indicate that these were corrected 
<Peter> It was my understanding that MOD groups fixed and vert slowly evolving structures. END
> energy channel boundaries. I use MOD for the nominal ADC channel boundaries
> since the telemetry bins will be of non-uniform size. END
<Peter> yes, now I remember, sorry. END
> In theory -SCIB-CAL (for science energy bins) and -CALB-CAL (for calibration spectrum
> energy bins) can change every ScW, but they won't END
<Peter> OK, let's stay with -CAL END
> I say that with some confidence, but in fact it all depends on the behaviour
> of the on board ADC. END
> Sorry that I've made you digress so far on the PHAII format. Please continue
> that discussion. END
<Peter> It seems we all agree on using PHAII as output format. 
<Peter> Let's say we fix any not firmly defined column names at our
<Peter> January meeting - OK? END
> Fine with me END
<Peter> Up to that time we will also all have had time to think about the COUNTS vs RATE question END
<Peter> So what about lightcurves? END
<larsson> Yes, the same principle, I agree with that. END
> Me too, I think END
<Peter> Any objections from other astronomers? END
<njw> No, it's OK END
<smaisala> Ok END
> Next on Peter's email was storage of background information. How about it
> chaps? END
<larsson> I might have to think a little on that one.... END
<njw> The first guess is that the bkg info should be another vector column END
> Once you get into this vector bin thing (like for the calibration spectra) 
> it seems that you have a very compact way to store all similar information
> in just one extension, and my feeling is that this is a good thing in
> general. Whether it's a good thing in this specific instance is probably for
> the background Tzar and his minions to decide END
<smaisala> That is not simple question . It depends how we like to handle that bacground data. 
<Peter> Actually there may be an OGIP convention for the spectra 
<njw> Maybe one should think about bkg extraction in a similar way to spectrum extraction ? END
<Peter> so we better inform ourselves a bit better. END
<Peter> In general, I thought we had the following scheme for background handling:
<Peter> Helsinki's software provides us with all spatial, spectral
<Peter> and temporal information in the form of these 6(7/8?) data structures.
<Peter> And after that it's the task of the analysis software to extract
<Peter> from this data the relevant information for its own use.
<Peter> So this means we have a lot of freedom in interpreting what
<Peter> Tzar Juhani and his trusty henchmen give us.
<Peter> And I guess we should study a little bit on this would be
<Peter> best stored before coming to a conclusion.
<Peter> How about having an AI due for our next meeting on Stefan, Niels Joergen and me to study this? END
> Good idea. Due by January meeting? END
<Peter> Yes exactly. END
<larsson> Acepted. END
<smaisala> Thank you Peter :-) END
<larsson> Do we have a firm date for the jan meeting?
<larsson> END
> I've added JH to the AI too. Though it could be one of his minions END
<Peter> In my agenda it's January 10. END
> Monday 10th Jan END
<larsson> Fine END
<njw> OK END
<smaisala> OK END
<Peter> So on to the next question - has anybody thought deeply about GTIs? END
> So everyone can come on the 10th? END
<larsson> CA: Yes. END
> So let's move on to GTI's: shall we just use OSM? What are the other options
> Peter? END
<Peter> (A) we build and organize GTIs on our own 
<Peter> (B) we forget about GTIs and just access all HK and status data from within each analysis tool
<Peter> Please note that I do not endorse (B) and consider (A) currently superfluous. END
> I agree. What about the rest of you? END
<larsson> I am sorry if I am not up to date on this but what do you mean by "on our own" in A?
<larsson> END
<Peter> Having some more executables (not yet in ADD) that generate GTIs, merge them, manipulate them ... The SPI team wants this AFAIK. END
> Why don't SPI trust OSM to do this? END
<larsson> Is'nt something like that done in our present design? END
> JCorrection includes event evaluation, which will include GTI information from
<Peter> To CAO: Don't ask me. One reason may be that OSM usually only works in 8 sec timesteps. END
<Peter> To SL: No, just look at the ADD. END
> OSM, but also other considerations like how good the calibration data is
> for a given time interval etc. I.e. stuff that only our ISSW knows END
<larsson> OK Peter, I have to do my homework. END
<njw> The careful data analyst would like to make his/her own GTI (partly based on GTIs from OSM)
<njw> but maybe the job can be done by other means, such as FSELECT ? END
<Peter> In principle, Reiner's code can als be re-used interactively, 
<Peter> but maybe we need to take a look and see if there is missing
<Peter> functionality for JEM-X.
<Peter> END
> JEventEvaluation is currently the least well-defined component in the ISSW
> and I'm counting on a great deal of input from both SDAST and IT to determine
> how to evaluate the `goodness' of an event, so there's lots more to be done
> here. END
> However, I don't think we're lacking any actual functionality in this respect
> in the ISSW END
> What do the rest of you think? END
<Peter> The question in total may require some more time to study the problem.
<Peter> Maybe I should get an action to present Reiner's tools and their capabilities
> That sounds like a good idea Peter. Can you set that up for the Jan meeting?END
<Peter> and NJW, SL and me an Action to review their functionality.
<Peter> For the Jan meetin, yes. END
<Peter> Question: can we already decide if we will have a single
<Peter> merged GTI structure for the analysis (which an observer
<Peter> could recreate at whim for interactive work)? END
<larsson> Yes Peter, for interactive analysis one should be able
<larsson> to create such a merged GTI structure.
<larsson> END
> That's an AI due for the Januaryy meeting. I'll send the complete AI list at the
> end of this meeting, so that every one can see what's expected of them.END
> Now Stefan, you've sent us  along list of proposed data structures, would
> you like to tell us all alittle more about them? END
<larsson> It is mainly an overview, but there are some question
<larsson> marks and when you read it and have some comments
<larsson> I suggest you send me an email on those. END
<Peter> A very nice summary.
<Peter> I have 3 quick comments:
<Peter> 1. Since ISDC likes the idea, we should probably really endorse
<njw> Peter, now that this list has come up, did you check whether the JEMX-DATA
<njw> structure is acceptable ?
<njw> END
<Peter> NJW's group of detector descriptions. Niels Joergen: are you
<Peter> just talking about this? END
<njw> Yes END
<Peter> Can we have two groups: JMX1- and JMX2-DETE-GMD ?
<Peter> DETE for DETEctor description and GMD for Gropup of MOD data? END
<njw> Peter, do you mean two subgroups ? END
<larsson> Since it contains also the mask it should maybe not be caled DETE? END
<Peter> To NJW: Well we should have one per detector (quasi-instrument)
<Peter> simply because we keep the separate at all other places IMHO.
<Peter> To SL: good point - do you have a nicer name? END
<larsson> (Thats always more difficult.....END)
<larsson> Maybe just INST? END
<njw> Well, of course we can have two groups, but Stefan is right, what about JMXi-INST-GMD ? END
<Peter> Fine with me. END
<larsson> So lets go for that. END
<njw> Is 'GMD' and
<njw> sorry, is 'GMD' an ISDC approved term ? END
<Peter> Comment 2: JMXi-GRP1-GTI could be our general structure but we should have a nicer name. END
<Peter> to NJW: Stephane has loosened up under too much pressure :-) 
<Peter> More seriously: we are relatively free but are requested to keep to sane principles. END
<njw> OK, fine END
<Peter> In general, I think there will be still name changes together with all thos keyowrd and column changes we talked about at the beginning. END
<Peter> Any proposals on the GTI name or must I take an action for January? END
<njw> JMXi-GTI.-LST ?? END
> I'll put an AI on Peter for the GTI name. END
> Of course we're all free to send him suggestions END
> Any more questions about Stefan's summary list? END
<larsson> Peter, maybe you like to comment on the name of my new
<larsson> data structure too. JMXi-EXTR-SCP
<larsson> END
<Peter> Yes exactly. The ending -SCP might be unfortunate since 
<Peter> this ending is used by OSM to designate parameters to be
<Peter> displayed and checked against limits that are derived
<larsson> Do you have a suggestion Peter? END
<Peter> from scientific data. END
<Peter> Not right now. I'll think about it and email a suggestion
<larsson> Fine. END
<njw> what about -PRM ? END
<Peter> around later - OK? Or maybe we extend my AI to come up
<Peter> with a consolidated list of DS names at the meeting. END
<Peter> Naturally I can only do this if I have a rather good list
> Okay, consider it done END
<Peter> of what you actually want. END
<larsson> -PRM makes sence to if it is ok ISDC. END
<Peter> Keep -PRM for the moment. It has good changes to stay. END
<Peter> One reason for ISDC to be less fussy about names at this stage
<Peter> is that there simply is less commonality across instruments, 
<Peter> except for the final end products (images, spectra, lightcurves). END
> Okay, is that the last set of questions about Stefan's list? END
<Peter> Yes. END
<smaisala> Peter: What data structure is JMXi-SKYI-INT ???
<njw> Intermediate sky image END
<Peter> This was meant for INTermediated SKY Image. END
<Peter> Any more questions? Or are you all munching sandwiches already? END
> I have one very general question about data structures:
> if we don't deliver our templates directly with the software that uses
> them, how can we be sure that the templates are availble on the ISDC
> system at the time of delivery. This is essential because David cannot
<Peter> The short answer is: not at all :-) END
> possibly get newly delivered software to run if said S/W doenstn have
> access to the templates
<Peter> The long answer is: any new templates must be sent in to Stephane _in advance_
> So isn't David going to be spending a lot of time telling us that the SW
> doesn't work just because it can't find the required template? END
<Peter> so he can make the available to David. END
> How far in advance? END
<Peter> There is a certain danger of that and I'm pushing to get this whole business better organized. END
<Peter> Not really sure, a week at least I'd guess. 
> As I see things it should go a bit like this:
<Peter> Probably you need to mention in your comments that your
> 1) I make version 1.o of component, and send it in with version 1.0 of
> the templates
<Peter> delivery will fail, if David has not yet received updated templates. END
> 2) David installs tested SW, using my templates while Sgeafan looks over
> the templates.
> 3) Stefan gives me a list of all the changes he wants in my templates.
> 4) I make the changes in the templates and in SW for version 2.0
> 5) I deliver verison 2.0 to David, with  in-advance templates corrected
> after Stephane's whims (!!) END
> 6) Stephane makes new templates available to David before I deliver version 2.oEND
> Is this not reasonable? I don't want to waste time with `non-functioning'
> SW that's just missing a template or two to get working. END
<Peter> I see your point, but often there is no urgent time pressure onyour software
<Peter> to be actually delivered to David.
> That's not what Roland seems to think! Are there no deadlines? END
<Peter> And for various reasons ISDC does NOT want templates to go into Davids
<Peter> system on any other way then the one via Stephane. END
<Peter> Regarding deadlines: Yes, naturally. But if the dealine is March
<Peter> and your exceutable written and locally tested by January
> I'm not in principle in favour of sending templates to David as opposed to Stephane
> it's just that in the past, templates avalalble on  the net are always
<Peter> you should first get the templates off to Stephane and wait a little with the delivery to David. END
> rather old END
<Peter> The only relevant source for 'valid' templates, currently is what you can find under /isdc/dev/templates
> Okay, boss. I'll be a good girl and send the templates to Stephane! END
<Peter> (Maybe there is also something in the Reference platform)
<Peter> Now this directory does NOT contain many templates which we actually used and that should be changed. END
<Peter> The whole problem is that David's setup is very inflexible
<Peter> and introducing anything there that is not correct leads to
> (so I've noticed!)
<Peter> further trouble down the road. As I've said before:
<Peter> I want this to be improved and have raised it at the
<Peter> management level here at ISDC. Reactions still
<Peter> outstanding. END
<Peter> And to make a previous statement clearer:
> Well, you can count on me to try and play ball! END
<Peter> Regarding deadlines the most importnat thing is to have your software ready
<Peter> We will normally wasily accept a few days delay in delivery
<Peter> to the Configuration COntrol system for questions like this. END
> So AOB anyone? END
> I think we'll call it a day here. 
> Thank you to you all for a productive meetin.
> THere will be a chat meeting quickly on Friday morning just to go over the
> outstanding AIs. See you all then! END
<Peter> Without me, since I'll be with the OMC team. END
<njw> Bye bye END
*** Signoff: njw ()
<Peter> Bye. END
*** Signoff: Peter (Leaving)
<larsson> Bye. END
