From stdenis@fnal.gov Fri Mar 16 23:53:05 2001
Date: Fri, 01 Dec 2000 19:07:23 -0600 (CST)
From: Rick St. Denis <stdenis@fnal.gov>
Reply-To: r.stdenis@physics.gla.ac.uk
To: cdf-daq@fnal.gov, cdfonlineusers@fnal.gov, cdfdb-users@fnal.gov,
     cdf-comp-upg@fnal.gov
Subject: Minutes of Spontaneous Meeting on Valid/Used Sets Dec 1, 2000

hi all
  we had a discussion of valid and used sets that i thought would be of
general interest so i wrote minutes and am circulating them.  This is
the next attempt to understand how to associate a calibration with the
correct data run...


Minutes of the AdHoc Used and Valid  Set Database Meeting Dec 1, 2000

This is a spontaneous discussion that occurred between:
Rick Snider (phone)
Liz Sexton-Kennedy
Jack Cranshaw
Rick St. Denis

The following example was outlined:


Calibrations
============

CalibRunLists
-------------
Run  Status Version Table  (Mode)
101  Raw    1       SVXDH   C1
102  Raw    1       SIXCHP  C1
103  Raw    1       SVXDH   C2
104  Raw    1       SIXCHP  C2
106  Raw    1       CALI


Data Runs
=========
Run Mode
205  C1
206  C1
207  C2
208  C2


Valid Set
----------
JobSet  ProcessName  (runs) (Mode)
45      Reconst       101    C1
                      102
                      106
46      Reconst       103    C2
                      104
                      106

Used Set
--------
Run  Process Name   JobSet
205  Reconst        45
206  Reconst        45
207  Reconst        46
208  Reconst        46

In this example, the silicon calibrations were taken in two modes, C1
and C2. This could correspond to two bandwidth settings or dynamic
pedestal subtraction on and dynamic pedestal subtraction off  or any
of the many combinations of chip parameters.  For that matter there
could be different chips set to different parameters. The CalibRunList
table will not record a mode, but keeps the run numbers and with work
the modes can be deduced from the hardware database and the rundb.  The
information is not easy to find, but at least it is there.  There is
also a calorimeter calibration run that was taken.

The columns specified in the tables by (  ) are not really on the
database but we are using them here to keep track of what is going
on.

Now there are some data runs taken.  These are taken in the two silicon
modes.  Clearly, we want to associate runs 101 and 102 with runs 205
and 206 and then we want to associate runs 103 and 104 with runs 207 and
208. The calibration run is ok for all the data runs.

We first looked at how a valid set is made in the code in
<a

href="http://purdue-cdf.fnal.gov/CdfCode/source/DBUtils/SetsUtils/CalorDownloads.cc">
DBUtils/SetsUtils/CalorDownloads.cc</a>
This accumulates a list of tables, like SVXDH, SIXCHP and CALI,
specifies the run version and status, then writes the valids set.
To make the used set, the valid set is passed to a
<a
href="http://purdue-cdf.fnal.gov/CdfCode/source/DBUtils/SetsUtils/SetMakertest.cc#L127>
makeUsedSet</a> function with the run range specified for the runs for
which the valid set should be used.
There are some problems with this.  If one just makes some valid set,
then one cannot just make the used set as one would have to know how to
fetch the correct valid set.

For the example above, there is a Process name, Reconst, which Liz
and Rick Snider need to use for reconstruction, to indicate the valid
calibrations.  To do this, they need to find out from all the
subdetector people, in this case, Jean and Jack, which runs are ok for
205-208.  They then make a valid set with process name Reconst.  The
valid set is assigned a sequential number from the database, a JobSet.
They then make a second valid set with the same process name and get
another sequential number.

With this sequential number in hand, then they can make the used
set for each run, distinguishing the two modes of running with the JobSet.

This sequential number is different on every database instance
and is analogous to knowing the memory location of a variable and
having to keep track of it in order to work.

Furthermore, Rick and Liz have to get quite a number of people into one
room in order to get a run sorted out.

For the next two weeks, this is the way it will have to be.

Jack will modify the schema to allow for the calbration mode to be
specified.

The picture above becomes the following:

CalibRunLists  (no change)
-------------
Run  Status Version Table  (Mode)
101  Raw    1       SVXDH   C1
102  Raw    1       SIXCHP  C1
103  Raw    1       SVXDH   C2
104  Raw    1       SIXCHP  C2
106  Raw    1       CALI


Data Runs  (no change)
=========
Run Mode
205  C1
206  C1
207  C2
208  C2


Valid Set
----------
JobSet  ProcessName  (runs)  Calib_Mode  Valid_from
45      Reconst       101    C1          205
                      102
                      106
46      Reconst       103    C2          205
                      104
                      106

Used Set
--------
Run  Process Name   JobSet
205  Reconst        45
206  Reconst        45
207  Reconst        46
208  Reconst        46



A calib mode  and Valid from are added to the Valid Set to allow
for us to key on the inforamtion without using the JobSet Number.

In further discussion we also noted that there is no reason to trust
that the Used Set was REALLY used.  So we would like to have an
identically formatted table that is the "ToBeUsed Set" so that Used Set
is always what was used.   This implies that the Used Sets cannot have a
version: once it is a used set, always a Used Set. If one runs again and
changes the values, then the process name should change: ReconPass2.

Also, the fact that Rick and Liz have to convene a large meeting is a
big problem.  Therefore the solution that we need is for the individual
subdetectors to create their own UsedSets and then the UsedSet for
reconstruction would be made by combining the used sets for that run.
The combination algorithm would have to trace back through the valid
sets to the calibration run list and verify that there are no
contradictory requirements, signalling an error were that the case.



Conclusions
------------
o Struggle with current schema for commissioning run data
o Jean will immediately make used sets for the silicon to test
  out the ideas and run on some silicon data
o Once that is done, Rick and Liz will get the big groups together
  to do the reconstruction used set
o Jack will modify the database schema and software over the next couple
  of weeks.







*****************************
Dr. Richard St. Denis,Dept. of Phys.& Astr., Glasgow University
Glasgow G12 8QQ; United Kingdom; UK Phone:   [44] (141) 330 5887
UK Fax  :   [44] (141) 330 5881
======================================
FermiLab PO Box 500; MS 318 Batavia, Illinois 60510  USA
FNAL Phone: [00] 1-630-840-2943 FNAL Fax:   [00] 1-630-840-2968
Sidet:      [00] 1-630-840-8630 FCC:        [00] 1-630-840-3707