Experimental Design for Computer Science

  • We undertake experiments to identify if there is a difference or relationship
  • Data which is collected is either:
    • Quantitative (numeric)
    • Qualitative (non-numeric or opinions)
  • Computing students are in a difficult situation and may have to consider a range of experiment designs:
    • Experiment design to capture system data (below)
    • Experiment design to capture human data (e.g. interface testing)
  • Where possible I recommend that you capture your data during the computational process (benchmarking, hit counters, mouse and eye tracking maps etc…) as this reduces your workload
  • By using the computer to capture data during the computational process it can very quickly be turned into graphs, charts and tables within your testing section.
  • Most computer science student will use the Single-Case Experiment Design (A-B-A Design) or Prototyping

Key Information

Single-Case Experiment Design (A-B-A Design) or Prototyping

  • Used with individual computers (laptop, raspberry pi) or one group of computers (network)
  • A base-line assessment is completed to benchmark system (speed, range, bandwidth etc…)
  • Apply the changes (implement your artefact)
  • Complete the base-line assessment again, to test your changes
  • If there is a difference then the intervention worked (had an effect)
  • This is called ‘related samples,’ as the data comes from the same sample (set of data) but at different times

System Modelling or System Simulation

  • Used to simulate real-world
  • A base-line assessment is completed to benchmark the real world system (speed, range, bandwidth etc…)
  • Create a test sample (data set) to represent real-world input
  • Create artefact to process test sample and simulate real-world
  • Move to Non-Experimental Design or Single-Case Experiment Design (A-B-A Design)

Non-Experimental Design (Descriptive Study)

  • Non-experimental designs has no control (testing or randomness)
  • Examples of non-experimental designs could be monitoring the computer system to understand what is happening
  • Create an idea (hypothesis) base on the monitoring data
  • Move a Single-Case Experiment Design