Workplan

------------------------------------------------------------------- LST2005: Work Plan for the EventBuilding DataCollection part ------------------------------------------------------------------- Aim: ---- Test scalability aspect of the EventBuilding part of the dataflow. General: -------- * Study of control aspects, robustness and stability on a full scale of the ATLAS EventBuilder * Take measurements for state transitions for varying sizes of the EventBuilder as a test for RunControl * Use of UDP and use of TCP * Don't aim to measure performance or scalability of the EventBuilder; i.e. very low data rates, and only a few dummy events built Details: -------- 1) In the first phases of the Large Scale Tests, we want to run with a partition having an EB segment with: one DFM and a few SFI's plus a ROS segment with a few ROS's (ROS-to-SFI ratio around 3:2; e.g., 15 ROS's and 10 SFI's). This is to prepare, test and understand the test procedures defined in step 2) and 3) below. 2) When we have around 300 machines in the LST05, we'd like to use a full scale version of the EB and ROS segments above: still have one DFM, but now have ~150 ROS's and ~100 SFI's, as foreseen for the final ATLAS DAQ system. Also a few intermediate steps will be measured with 1/4; 1/2; and 3/4 of the full scale system. 3) Integrate with ONLSW + LVL2 ROI Collection + Event Builder tests [i.e., step 2b) of LST workplan]: Go back to a smaller EB segment, always with one DFM and ROS-to-SFI machines in the ratio of 3:2 (e.g., 12 ROS's, 8 SFI's) and allocate the available PCs to the LVL2 subfarms for testing. 4) Integration: ONLSW + L2 ROI Collection + Event Builder + EF: [i.e., step 4) of LST workplan]: As step 3) above, but EF subfarms added. SFIs need to connect to EFDs. Need to add a few (max 5) SFOs in a SFO segment. 5) If at the end of the tests, as we approach the 19th of July, we have done all the tests we want with the Online, LVL2 and EF systems, then we could check an EB and ROS segment with twice the size of the full ATLAS configuration: 1 DFM, 300 ROS's and 200 SFI's just to see if we hit a wall there. This should not have the highest priority in the overall planning, nevertheless, if there is some time available this would be still a useful test.

for some of our new components: 1) Access Manager (Marius): Given a large partition (might be purely dummy or contain real DF applications) evaluate the performance loss at setup and transitions introduced by switching on the access management. Some time should be allowed to try to optimize AM (cache lifetime) and Corba parameters. For using the AM we need 1 machine configured as a MYSQL server, containing the AM database. These tests should be performed towards the end of the testing period, if and when the rest of the system is fairly stable.

At every change of size of the system, a partition should be run and timed with the AM deactivated and then with the AM active. Time differences for booting and transitions shall be recorded. If feasible, the AM should be then always kept active. - To use the AM we need a mysql server and a users database. Marius will follow up with Gokhan that we have a machine installed for this purpose on lxshare. - John-Erik and Marius will provide instructions on how to activate/deactivate authorization via the AM. (1 Variable in the database, at Partition level). - Marius will execute the timing tests for this component. 2) DVS (Alina): Scalability tests. Some time should be dedicated to see the performance of DVS, when very large number of tests are launched. In general, DVS shall be used throughout the testing period, for testing and logs browsing, etc... 3) Setup(Andrei): Setup shall be used for all tests. This LST period shall definitely assess its functioning. Dedicated time should be scheduled to see how well we manage to recover when infrastructure applications are killed.

- No specific test, but assistance from Andrei if there are problems. - Andrei will check that the text_gui works from within setup. 4) New PMG (Marc): on dedicated release Dedicated time to start/stop/monitor large number of processes and measure performance. These tests will be made with dedicated test programs, not with the DAQ. When these tests are ongoing the system shall be locked, in order not to mix up old and new pmg agents. We would like to get 2 days for this (one between 21-26 of June and one towards the end of the LST period). The second day might be dropped if everything works fine during the first tests.Tests will first be done on building 32 testbed. Then need 1/2 day exclusive mode.

- tests will be first carried out on bdg 32. - If successful, 1 day of parasitic running on (part of) the cluster will be asked (beginning of July) to install the software needed and do preliminary tests, followed by half day of measurements (exclusive usage) - If the first tests will have been succesfull another timeslot towards the end will be requested to run some automated measurements (can be done over night). 4) New RC (Giovanna): on dedicated release Dedicated tests on large (dummy) partitions to verify the functioning of the new run control and supervision. If everything works fine the new RC could be used for real partitions as well. These tests will require the installation of a second release containing the apps linked against the new RC. Tests could start after June 26th. Tests will first be done on building 32 testbed. Then need 1/2 day exclusive mode. If tests are successful, new controller could be used from then on. (boot and shutdown are expected to be faster because of distributed dsa_supervisor.

- Tests will first be carried out in bdg 32. - If successful, during July we will run a few equivalent partitions with old and new Controller and check if the behavior is similar, performance wise. One day will be enough for this. During the night we might then be able to make comparative automated timing tests. - Another slot of automated tests will be asked towards the end of the period, if the first tests were successful.

- at every change of size, run a partition with Controllers and ExampleApplications only to test completely the run control chain. - if there will be spare time, applications and controllers shall be killed on purpose, randomly, to check that they are restarted/ignored as specified in the DB and that the system behaves as expected.

Tests on large partitions on particular aspects of the scalability of the IGUI panels. Last year the IGUI has been used for configurations including up to 150 nodes. We will check the behaviour for larger configurations (for example 400 or 500 nodes), especially for PMG, Run Control, DataFlow and MRS panels.

- During setup time (at each relevant size change) the IGUI shall be tried out to check its scalability. - setup_daq shall always be started without GUI, then we will try out 2 different ways of running the IGUI: either started on a local machine outside the lxshare cluster (fast graphics, but potentially large latencies over the CERN network to send commands, etc...) or started on a dedicated machine within the lxshare cluster with a remote display (fast command distribution, but slow graphics response). - the general shifter in charge of setting up the system shall use the IGUI, assisted by Mihai, if/when needed.

Log Service: - it is requested that a partition stays up for sometime (30 minutes). - the log server will be started parasitically, but it might have an impact on the load in the mrs_server (keep an eye on CPU load!). - Log consumers will be requesting data and we will evaluate at which point log messages will start to be dropped. - this exercise should be repeated at each size change, if no evident problems have appeared. - Raul will contact Gokhan to tell him what is needed from an installation point of view (mysql and web servers). - These tests can be done with any partition. Raul will measure the behaviour of the LS.

The developers will be running the specific tests. that's why the first week is not suited for the pmg. For what concerns the AM, I will run the tests myself, since John-Erik will be leaving for good at the beginning of June. He has started tests on medium scale already (bdg 32). In general people in Controls will participate to the tests also when running in integrated mode. We'll have to sort out the coverage of the full period.

Alina: 09 June - 26 June Roul: 1.7 - 5.7 (possibly longer) and 18.7 - 29.7. Giovanna: 11.6 - 26.6, 16.7 - 24.7. Andrei: 2.07-25.07 Marc: 11.6 - 19.6, 11.7 - 17.7 (or 13.7 - 17.7) Mihai: 08.07 - 01.08.

•

–

Haimo coordinating the tests

here is the combined list of proposed tests from EF and LVL2.
For each of the tests, I added the 8 questions from the LST page
"HowTo participate in the test" with the answers from the HLT group.

   Q1. what is the purpose of the tests, which aspect of the system is 
       investigated,  what are the objectives (description of the test)?
   Q2. why can these tests only be done on a very large scale, 
       what are the challenges and/or expected results?
   Q3. who will prepare the tests, who will participate in the test preparation 
       (be the contact person, participate in meetings),  run the tests and 
       make the results available as part of the test report?
   Q4. will the tests have been tested and run on a medium scale on one of the 
       available farms before?
   Q5. can the tests be automated? Can they run autonomously, 
       i.e. over night or during parts of the week-end?
   Q6. do the tests require exclusive access to the farm or can they run in
      'shared' mode?
   Q7. how many nodes and how much testing time do they require 
       a) in shared mode b) in exclusive mode?
   Q8. are there any specific requirements on system parameters or any constraints?

I've grouped the tests by whether they need Athena, and combined those that are common
for LVL2 and EF. It's a bit long due to those 8 questions, but I think
they give us a nice view of what we want to do.

- Haimo


Proposed HLT test plan for 2005 Large Scale Tests
=================================================

Name abbreviations:

AdA - Andre dos Anjos
AB  - Andre Bogaerts
HG  - Hegoi Garitaonandia
SS  - Serge Sushkov
HZ  - Haimo Zobernig
AN  - Andrea Negri
SK  - Sander Klous

Tests which don't depend on HLT selection algorithms:
-----------------------------------------------------

Test 1) SW distribution with BitTorrent

Haimo coordinating the tests

  test distributing a "shrink-wrapped" sw distribution containing the 
  tdaq-01-01-00, offline 10.0.2 and hlt-02-00-00 releases, combined 
  into a single, pre-tested filesystem. This would be a compressed .iso 
  file of ca. 1.7 GB ( 5GB expanded ). 

  Q1: The method of distributing this to N nodes which we wish to test is 
  by using the peer-to-peer protocol BitTorrent. We have tested it with
  up to 50 nodes on Fast Ethernet. Maximum time observed was 30 mins to
  distribute 50*1.7 GB = 85 GB on 50 nodes. 
  (see https://uimon.cern.ch/twiki/bin/view/Atlas/HltSoftwareDownload ) 

  Q2: We would like to run this test in all the different sizes of the LST
  to get measured timing on its scaling behaviour. Expected scaling
  should be logarithmic with number of nodes, or better. But it will
  depend on other concurrent network activity.

  Q3: HZ AdA HG
  Q4: yes, already done on up to 50 machines
  Q5: yes, test can be run in automated way, eg by cron job
  Q6: for initial verification shared mode is ok, for scaling measurement
  exclusive access is preferred (largest configuration may need 1-2 hrs
  if log scaling applies, see Q2)

  Q7: the maximum available in each phase of LST (see Q2)
  Testing time initially a few hours if problems appear. BT itself is very
  well tested by the worldwide Internet community...

  Q8: requirements:
  a) a local disk area with at least 5 GB of disk space on every node
  b) ability to mount an .iso file as a read-only file system
  c) BitTorrent version 3.4.2 or higher. If tests are successfull we may
     want to use the new "trackerless" version 4.1.1 of BitTorrent, which 
     supports many thousands of simultaneous downloaders of the same file.
     This would be interesting at least for 500 or more nodes in the LST.

Test 2) verify this file system with known-to-work tdaq-only partition

  Q1, Q2: make sure larger partitions work, as tdaq-01-01-00 may behave 
  differently from tdaq-01-02-00
  Q3: HZ AdA AB  
  Q4: yes  Q5: yes  Q6: can run shared 
  Q7: some large partition should be tried, but not necessarily in all LST sizes
  Q8: no

Test 3) run w/ preloaded events, but dummy algorithms

  Q1, Q2: Learn behaviour of large partitions with preloaded events.
  Run a TDAQ partition w/ dummy algorithms, but with events preloaded
  in ROS (preferred, w/ ROSE as backup alternative)
  This test is also a prerequisite for test 6)
  Q3: HZ AdA AB 
  Q4: yes, done with small partitions in lab32
  Q5: yes, Q6: shared OK, fraction of cluster to be determined 
  Q7: same as Test 2)
  Q8: no

Tests with HLT sw (thus requiring tdaq-01-01-00, Offline 10.0.2, HLT-02-00-00):
-------------------------------------------------------------------------------

Test 4) HelloWorld and Level1 decoding test

  Q1: run algorithms 'HelloWorld' and 'HelloWorldLVL1'
  Both algorithms test much of the Athena infrastructure as used in HLT.
  Memory consumption, disk access patterns and shared library use will
  be studied. HelloWorld applies equally well for LVL2 and EF, while 
  HelloWorldLVL1 decodes the Level 1 result and is known to be working 
  - even in multi-threaded mode - and constitutes a LVL2-specific test

  Q2: verify use of Athena in LST for the first time.
  verify the LVL1 decoding part of the LVL2 trigger in multithreaded mode
  on large partitions. 
  Q3: HZ, AdA, AB, SS, HG, AN
  Q4: yes Q5: yes Q6: shared OK (part of cluster, not sharing same nodes)
  Q7: 50 to 100% of cluster (100% for a final test). Running time 5-30 minutes 
      per test 
  Q8: no

Tests which depend on availability of working algorithms:
---------------------------------------------------------
These tests must use tdaq-01-01-00 as Athena doesn't yet work with the new
tdaq release; all required sw will be distributed via the .iso file downloaded
in Test 1 - we assume this will work as tested in lab32.
The algorithms to be tested here may not be available during the first, smaller
phases of the LST. They will have a chance of getting distributed every time
Test 1 is run. 


Test 5) test HLT configure behaviour

  Q1: if any algorithm is available, that has been demonstrated to at least
  survive the 'configure' transition, we wish to test this transition
  in all the LST phases. This applies to LVL2 as well as EF.

  Q2: understand scaling problems with access to databases, loading shared
  libraries etc. Needs cooperation with DB management group due to large bursty
  load generated by the many DB clients.
  Q3: HZ, AdA, AB, SS, HG, AN
  Q4: yes  Q5: probably needs manual running to observe behaviour in real time
  Q6: shared mode OK 
  Q7: good fraction of cluster, maybe ~50%, probably not 100% (until everything
      is in good shape and under control)
  Q8: DB access may pose restrictions, but not yet known in detail
      Running time could be substantial if test is working at all, due to many
      parameters to be studied (many short runs). Details still need working out.

Test 6) test HLT run phase

  Q1: if any algorithm is available that is known to run on smaller testbeds
  in LVL2 and/or EF, we wish to test it also with all LST sizes,
  (using realistic, non-dummy events). Might be done separately for LVL2 and EF
  i.e. vertical slice probably only in the most optimistic of cases.

  Q2: study runtime behaviour of algorithms, look out for unexpected/forbidden
  database accesses during event processing. study resource consumption such as
  memory foot print (watch for leaks), network load, cpu usage etc. with many
  nodes..
  Q3: HZ, AdA, AB, SS, HG, AN
  Q4: yes  Q5: yes, but will also need some handholding...
  Q6: shared mode OK
  Q7: as Test 5), but running time may be substantial if all works in short runs
       (eg. long run on many nodes is a good way to find memory leaks and
        rare but serious problems)
  Q8: as Test 5)

Tests specific to EF requiring no Athena algorithms:
----------------------------------------------------

Andrea and Serge coordinating the tests
These tests should preferrably be done with the tdaq-01-02-00 release to 
benefit from recent improvements there. They can be scheduled independently
of the HLT tests requiring Athena.

note: the description of these tests has been moved to the section on sub-system tests.

Preparation tasks for the tests 
  
Software installation and administration

Gokhan and Matei responsible

Database generation tool

Gokhan  responsible

Tools

All to contribute

Preparation and documentation of testing tools is on-going. 
The documentation of those tools will be published on the LST web page, 
a number of documented items can already be found there. 
A twiki page has been created to give easy access for making  
contributions, corrections and discussion. Stable items will be moved 
into the main LST page.

Areas concerned are:

```
execution scripts
```
```
automatic execution scripts
```
```
analysis tools
```
```
farm management tools
```
```
debugging aid
```
```
How-to descriptions
```

Introduction